Basic Crawling

Geekcamp 2013

Transcript

1. Basic Crawling (Using Nokogiri and others) by ytbryan Wednesday, September 11, 13
2. I am running Digital Asia and we made these Briefcase Shoppink To myself Control your shop Browse shops easily Email yourself easily Wednesday, September 11, 13
3. •No complex solution. •No intelligent semantic algorithm. •Just simple crawling. Wednesday, September 11, 13
4. Disclaimer How I crawl the web, NOT how you should be crawling the web. Wednesday, September 11, 13
5. You should not be doing anything illegal, violate copyright or privacy or cause any damages. Wednesday, September 11, 13
6. nea.gov.sg weathersg.herokuapp. com Wednesday, September 11, 13
7. Wednesday, September 11, 13
8. DEMO Wednesday, September 11, 13
9. Basic Crawler HTML parser + Net/http Put in background HTML Parser + Net/http placed in 1 server. Put a scheduler like cron or any of your favorite scheduler. Distributed Crawling HTML Parser + Net/http x multiple servers Wednesday, September 11, 13
10. Basic Crawler HTML parser + Net/http Put in background (I’m doing this) HTML Parser + Net/http placed in 1 server. Put a scheduler like cron or any of your favorite scheduler. Distributed Crawling HTML Parser + Net/http x multiple servers Wednesday, September 11, 13
11. Basic Crawler HTML parser + Net/http Put in background (I’m doing this) HTML Parser + Net/http placed in 1 server. Put a scheduler like cron or any of your favorite scheduler. Distributed Crawling HTML Parser + Net/http x multiple servers Your need for structure data increase Wednesday, September 11, 13
12. How to Improve • Save to database. • Check database copy (cheaper) before Crawling (expensive) • Crawl in background to keep database fresh. • Distributed Crawling anyone? Wednesday, September 11, 13
13. •30 minutes, not 30 days. •One developer •One server •Go for solution that works. Wednesday, September 11, 13
14. Every shops in one place DEMO Wednesday, September 11, 13
15. DEMO Javascript Wednesday, September 11, 13
16. Get all images? • Document.images • Good for Images reuse • Good for aggregation of data or image data • Or use Nokogiri.doc.css(‘img’) Wednesday, September 11, 13
17. DEMO Javascript on iPhone Wednesday, September 11, 13
18. • Pass in the javascript command to //Convert  javascript  into  string NSString  *filePath  =  [[NSBundle  mainBundle]   pathForResource:@"javascript"  ofType:@"js"   inDirectory:@"www"]; NSData  *fileData  =  [NSData  dataWithContentsOfFile:filePath];                                                     NSString  *jsString  =  [[NSMutableString  alloc]   initWithData:fileData  encoding:NSUTF8StringEncoding];   //execute  your  javascript  on  webView result  =  [webView   stringByEvaluatingJavaScriptFromString:javascriptString];   • get back data in “result” Javascript on Webview Wednesday, September 11, 13
19. Image reuse on Facebook and Reddit Wednesday, September 11, 13
20. Web crawler of the century? Wednesday, September 11, 13
21. Wednesday, September 11, 13
22. Still possible. Let user overcome the captcha for you. Wednesday, September 11, 13
23. Let the user overcome captcha • Display captcha page. • Upon loading of next page, start crawling. If crawling fails, announce it. • How to tell if captcha is wrong. Wednesday, September 11, 13
24. Not worth doing. Wednesday, September 11, 13
25. A few hours to make. Service is running for over 2000 hours Wednesday, September 11, 13
26. 100x return on time invested. Wednesday, September 11, 13
27. Doesn’t always work. Wednesday, September 11, 13
28. Wednesday, September 11, 13
29. Wednesday, September 11, 13
30. STRUCTURE in unstructured data. Wednesday, September 11, 13
31. FIND them Wednesday, September 11, 13
32. Questions? Wednesday, September 11, 13