Salesforce_http_call_out爬虫实例分享.pdf

Transcript

1. SFDC在日華人交流会 三回目
2. Salesforce http call out 爬虫实例分享 罗 海文
3. 概要 ▪ 场景 在项目中我们经常会用到通过http call out方式和其他系统交互 将外部的数据实时读入到salesforce系统里面 ▪ salesforce通过REST方式访问外界站点步骤如下： １、将Web Service的授权端点地址添加到Remote Site中： setup->Administer->Security Controls->Remote Site Settings ２、写代码进行访问 HttpRequest req = new HttpRequest(); req.setEndpoint('http://www.yahoo.com'); req.setMethod('GET'); String username = 'myname'; String password = 'mypwd'; Blob headerValue = Blob.valueOf(username + ':' + password); String authorizationHeader = 'BASIC ' + EncodingUtil.base64Encode(headerValue); req.setHeader('Authorization', authorizationHeader); Http http = new Http(); HTTPResponse res = http.send(req); System.debug(res.getBody());
4. 概要 ▪ 场景 在项目中我们经常会用到通过http call out方式和其他系统交互 将外部的数据实时读入到salesforce系统里面 ▪ salesforce通过REST方式访问外界站点步骤如下： １、将Web Service的授权端点地址添加到Remote Site中： setup->Administer->Security Controls->Remote Site Settings ２、写代码进行访问 HttpRequest req = new HttpRequest(); req.setEndpoint('http://www.yahoo.com'); req.setMethod('GET'); String username = 'myname'; String password = 'mypwd'; Blob headerValue = Blob.valueOf(username + ':' + password); String authorizationHeader = 'BASIC ' + EncodingUtil.base64Encode(headerValue); req.setHeader('Authorization', authorizationHeader); Http http = new Http(); HTTPResponse res = http.send(req); System.debug(res.getBody());
5. 小视频去水印爬虫实例讲解 ▪爬取对象 tiktok Id: 80348038 ▪爬取步骤： １、tiktok的授权端点地址添加到Remote Site中： https://api.tiktokv.com
6. 小视频去水印爬虫实例讲解 ▪爬取对象 tiktok Id: 80348038 ▪爬取步骤： １、tiktok的授权端点地址添加到Remote Site中： https://api.tiktokv.com ２、写代码对用户80348038的作品进行爬取 1)模拟tiktok的用户搜索操作获取uid https://api.tiktokv.com/aweme/v1/discover/search/?xxxx=yyyy&keyword=80348038 2)根据uid获取该用户的所有作品 https://api.tiktokv.com/aweme/v1/aweme/post/? xxxx=yyyy&user_id=' + uid + '&max_cursor=' + strMaxCusor + '&min_cursor=' + strMinCusor
7. 具体代码 ▪模拟tiktok的用户搜索操作获取uid Http h = new Http(); HttpRequest req = new HttpRequest(); HttpResponse res; req.setHeader('Accept-Encoding', 'gzip, deflate '); req.setHeader('Connection', 'close '); req.setHeader('Cookie', 'odin_tt=xxxxx'); req.setHeader('Host', 'api.tiktokv.com '); req.setHeader('User-Agent', 'User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'); req.setEndpoint(' https://api.tiktokv.com/aweme/v1/discover/search/?xxxx=yyyy&keyword=80348038'); req.setMethod('GET'); res = h.send(req); Map<String, Object> jsonMap1 = (Map<String, Object>)JSON.deserializeUntyped(res.getbody()); List<Object> userlist =(List<Object>) jsonMap1.get('user_list'); String strUid = ''; for (Object obj1 : userlist) { Map<String, Object> userMap = (Map<String, Object>)obj1; Map<String, Object> user_infoMap = (Map<String, Object> )userMap.get('user_info'); strUid = (String)user_infoMap.get('uid'); }
8. 具体代码 ▪模拟tiktok的用户搜索操作获取uid Http h = new Http(); HttpRequest req = new HttpRequest(); HttpResponse res; req.setHeader('Accept-Encoding', 'gzip, deflate '); req.setHeader('Connection', 'close '); req.setHeader('Cookie', 'odin_tt=xxxxx'); req.setHeader('Host', 'api.tiktokv.com '); req.setHeader('User-Agent', 'User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'); req.setEndpoint(' https://api.tiktokv.com/aweme/v1/discover/search/?xxxx=yyyy&keyword=80348038'); req.setMethod('GET'); res = h.send(req); Map<String, Object> jsonMap1 = (Map<String, Object>)JSON.deserializeUntyped(res.getbody()); List<Object> userlist =(List<Object>) jsonMap1.get('user_list'); String strUid = ''; for (Object obj1 : userlist) { Map<String, Object> userMap = (Map<String, Object>)obj1; Map<String, Object> user_infoMap = (Map<String, Object> )userMap.get('user_info'); strUid = (String)user_infoMap.get('uid'); }
9. 具体代码 ▪根据uid获取该用户的所有作品 Http h = new Http(); HttpRequest req = new HttpRequest(); HttpResponse res; req.setHeader('Accept-Encoding', 'gzip, deflate '); req.setHeader('Connection', 'close '); req.setHeader('Cookie', 'odin_tt=xxxxx'); req.setHeader('Host', 'api.tiktokv.com '); req.setHeader('User-Agent', 'User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'); req.setEndpoint('https://api.tiktokv.com/aweme/v1/aweme/post/? xxxx=yyyy&user_id=' + uid + '&max_cursor=' + strMaxCusor + '&min_cursor=' + strMinCusor '); req.setMethod('GET'); res = h.send(req); Map<String, Object> dataMap = (Map<String, Object>)JSON.deserializeUntyped(res.getbody()); List<Object> awemeListOjb = (List<Object>) dataMap.get('aweme_list'); for (Object obj2 : awemeListOjb) { Map<String, Object> awemeMap = (Map<String, Object>)obj2; Map<String, Object> videoMap = (Map<String, Object>) awemeMap.get('video’); Map<String, Object> addrMap = (Map<String, Object>) videoMap.get('play_addr’); List<Object> urlList = (List<Object>) addrMap.get('url_list') } ;
10. 具体代码 ▪根据uid获取该用户的所有作品 Http h = new Http(); HttpRequest req = new HttpRequest(); HttpResponse res; req.setHeader('Accept-Encoding', 'gzip, deflate '); req.setHeader('Connection', 'close '); req.setHeader('Cookie', 'odin_tt=xxxxx'); req.setHeader('Host', 'api.tiktokv.com '); req.setHeader('User-Agent', 'User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'); req.setEndpoint('https://api.tiktokv.com/aweme/v1/aweme/post/? xxxx=yyyy&user_id=' + uid + '&max_cursor=' + strMaxCusor + '&min_cursor=' + strMinCusor '); req.setMethod('GET'); res = h.send(req); Map<String, Object> dataMap = (Map<String, Object>)JSON.deserializeUntyped(res.getbody()); List<Object> awemeListOjb = (List<Object>) dataMap.get('aweme_list'); for (Object obj2 : awemeListOjb) { Map<String, Object> awemeMap = (Map<String, Object>)obj2; Map<String, Object> videoMap = (Map<String, Object>) awemeMap.get('video’); Map<String, Object> addrMap = (Map<String, Object>) videoMap.get('play_addr’); List<Object> urlList = (List<Object>) addrMap.get('url_list') } ;
11. 具体代码 ▪根据uid获取该用户的所有作品 Http h = new Http(); HttpRequest req = new HttpRequest(); HttpResponse res; req.setHeader('Accept-Encoding', 'gzip, deflate '); req.setHeader('Connection', 'close '); req.setHeader('Cookie', 'odin_tt=xxxxx'); req.setHeader('Host', 'api.tiktokv.com '); req.setHeader('User-Agent', 'User-Agent:Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)'); req.setEndpoint('https://api.tiktokv.com/aweme/v1/aweme/post/? xxxx=yyyy&user_id=' + uid + '&max_cursor=' + strMaxCusor + '&min_cursor=' + strMinCusor '); req.setMethod('GET'); res = h.send(req); Map<String, Object> dataMap = (Map<String, Object>)JSON.deserializeUntyped(res.getbody()); List<Object> awemeListOjb = (List<Object>) dataMap.get('aweme_list'); for (Object obj2 : awemeListOjb) { Map<String, Object> awemeMap = (Map<String, Object>)obj2; Map<String, Object> videoMap = (Map<String, Object>) awemeMap.get('video’); Map<String, Object> addrMap = (Map<String, Object>) videoMap.get('play_addr’); List<Object> urlList = (List<Object>) addrMap.get('url_list') } ; Pattern p = Pattern.compile('https://api.tiktokv.com/aweme/v1/playwm.*?line=0.*'); Matcher pm = p.matcher((String)str );
12. 具体代码 ▪添加一个页面来显示获取的作品 <apex:page controller="CL_GetDouinUserInfoConctroller" action="{!initDouyinVideoListById}"> <apex:repeat value="{!listDisObj}" var="roomInfo" > {!roomInfo.strurl}<br/> </apex:repeat> </apex:page> ▪添加domain 为外部用户提供访问作品一览的权限 setup->Administer->Domain Management-> My Domain
13. https://liveshow-developer- edition.ap5.force.com/VF_GetDouinVideoCSVPage?strIdGuoji=80348038&strKey=123
14. 参考资料 http://www.cnblogs.com/zero-zyq/p/5599632.html https://developer.salesforce.com/docs/atlas.en- us.apexcode.meta/apexcode/apex_callouts_timeouts.htm https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_gov_limits.htm https://developer.salesforce.com/docs/atlas.en- us.apexcode.meta/apexcode/apex_classes_restful_http_http.htm https://developer.salesforce.com/docs/atlas.en- us.apexcode.meta/apexcode/apex_classes_restful_http_httprequest.htm https://developer.salesforce.com/docs/atlas.en- us.apexcode.meta/apexcode/apex_classes_restful_http_httpresponse.htm https://developer.salesforce.com/docs/atlas.en- us.apexcode.meta/apexcode/apex_callouts_named_credentials.htm
15. ありがとうございました。