Scraping: 10 mistakes to avoid @ Breizhcamp 2016

From website, to storage, learn webscraping

#webscraping #tricks

Transcript

1. Fabien VAUCHELLES zelros.com / fabien.vauchelles.@zelros.com / @fabienv http://bit.ly/breizhscraping (24/03/2016)
2. FABIEN VAUCHELLES Developer for 16 years CTO of Expert in data extraction (scraping) Creator of Scrapoxy.io
3. What is Scraping
4. “Scraping is to transform human-readable webpage into machine-readable data.” Neo
5. Why do we do Scraping
6. EXAMPLES No API ! API with a requests limit Prices Emails Profiles Train machine learning models Addresses Face recognition
7. “I used Scraping to create my clients list !” Walter White
8. FORGET THE LAW 1.
9. THE LEGAL PATH Can we track the data ? Does the company tends to sue ? Data is private ? Is the company is in France ? Do the data provide added value ? no yes yes yes yes no no no no yes
10. THE LEGAL PATH Can we track the data ? Does the company tends to sue ? Data is private ? Is the company is in France ? Do the data provide added value ? no yes yes yes yes no no no no yes
11. THE LEGAL PATH Can we track the data ? Does the company tends to sue ? Data is private ? Is the company is in France ? Do the data provide added value ? no yes yes yes yes no no no no yes
12. THE LEGAL PATH Can we track the data ? Does the company tends to sue ? Data is private ? Is the company is in France ? Do the data provide added value ? no yes yes yes yes no no no no yes
13. THE LEGAL PATH Can we track the data ? Does the company tends to sue ? Data is private ? Is the company is in France ? Do the data provide added value ? no yes yes yes yes no no no no yes
14. RUBBER DUCK E-MARKET LET’S STUDY THE
15. BUILD YOUR OWN SCRIPT 2.
16. USE A FRAMEWORK Limit concurrents request by site Limit speed Change user agent Follow redirects Export results to CSV or JSON etc. Only 15 minutes to extract structured data !
17. USE THE ECOSYSTEM Frontera ScrapyRT PhantomJS Selenium PROXY EMULATION HELPER STORAGE
18. RUSH ON THE FIRST DATA SOURCE 3.
19. FIND THE EXPORT BUTTON
20. TAKE TIME TO FIND DATA
21. How to find a developer on Rennes
22. #1. GO TO BREIZHCAMP
23. #2. SCRAP GITHUB
24. #3. SCRAP GITHUB ARCHIVE
25. #4. USE GOOGLE BIG QUERY
29. KEEP THE DEFAULT USER-AGENT 4.
30. DEFAULT USER-AGENT SCRAPY Scrapy/1.0.3 (+http://scrapy.org) URLLIB2 (Python) Python-urllib/2.1
31. IDENTIFY AS A DESKTOP BROWSER CHROME Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3)↵ AppleWebKit/537.36 (KHTML, like Gecko)↵ Chrome/50.0.2661.37 Safari/537.36 200 503
32. SCRAP WITH YOUR DSL ACCESS 5.
33. BLACKLISTED
34. What is Blacklisting
35. TYPE OF BLACKLISTING Change HTTP status (200 -> 503) HTTP 200 but content change (login page) CAPTCHA Longer to respond And many others !
36. USE A PROXY SCRAPER PROXY TARGET 88.77.66.55 44.33.22.11 1.2.3.411
37. TYPE OF PROXIES PUBLIC PRIVATE
38. HIDE BEHIND SCRAPOXY SCRAPERS SCRAPOXY TARGET http://scrapoxy.io
39. TRIGGER ALERTS ON THE REMOTE SITE 6.
40. STAY OFF THE RADAR
41. ESTIMATE IP FLOW SCRAPER PROXY TARGET 10 requests / IP / minute ✔
42. ESTIMATE IP FLOW SCRAPER PROXY TARGET 10 requests / IP / minute ✔ 20 requests / IP / minute ✔
43. ESTIMATE IP FLOW SCRAPER PROXY TARGET 10 requests / IP / minute ✔ 20 requests / IP / minute ✔ 30 requests / IP / minute X
44. ESTIMATE IP FLOW The flow is 20 requests / IP / minute I want to refresh 200 items every minute I need 200 / 20 = 10 proxies !
45. MIX UP SCRAPER AND CRAWLER 7.
46. SCRAPERS ARE NOT CRAWLERS
47. FOCUS ON ESSENTIAL
48. What is the URL frontier
49. URL frontier is the list of URL to fetch.
50. TYPE OF URL FRONTIER FIX SEQUENTIAL TREE
51. STORE ONLY PARSED RESULTS 8.
52. SCRAPING IS AN ITERATIVE PROCESS EXTRACT AND CLEAN DATA SCRAP DATA USE DATA REFACTOR
53. SCRAP EVERYTHING... AGAIN ?
54. STORE FULL HTML PAGE
55. SCRAPING IS AN ITERATIVE PROCESS EXTRACT ALL CLEAN DATA SCRAP DATA USE DATA REFACTOR
56. STORE WEBPAGE ONE BY ONE 9.
57. STORAGE CAN’T MANAGE MILLIONS OF SMALL FILES !
58. BLOCK IS THE NEW STORAGE
59. BLOCK IS THE NEW STORAGE
60. BLOCK IS THE NEW STORAGE
61. BLOCK IS THE NEW STORAGE
62. BLOCK IS THE NEW STORAGE
63. BLOCK IS THE NEW STORAGE
64. BLOCK IS THE NEW STORAGE
65. BLOCK IS THE NEW STORAGE
66. BLOCK IS THE NEW STORAGE
67. STORE HTML IN 128 MO ZIPPED FILES
68. PARSING IS SIMPLE ! 10.
69. PARSERS There is a lot of parsers ! XPATH CSS REGEX TAGS TAG CLEANER
70. 2 METHODS TO EXTRACT DATA <div class=”parts> <div class=”part experience”> <div class=”year”>2014</div> <div class=”title”>Data Engineer</div> </div> </div> How to get the job title ?
71. #1. BY POSITION <div class=”parts> <div class=”part experience”> <div class=”year”>2014</div> <div class=”title”>Data Engineer</div> </div> </div> /div/div/div[2] (with XPath parser)
72. #1. BY POSITION <div class=”parts> <div class=”part experience”> <div class=”year”>2014</div> <div class=”location”>Paris</div> <div class=”title”>Data Engineer</div> </div> </div> /div/div/div[2] (with XPath parser)
73. #2. BY FEATURE <div class=”parts> <div class=”part experience”> <div class=”year”>2014</div> <div class=”title”>Data Engineer</div> </div> </div> .experience .title (with CSS parser)
74. LET’S RECAP !
75. STEP BY STEP FIND A SOURCE LIMIT THE URL FRONTIER SCRAP AND STORE PARSE BLOCS
76. STEP BY STEP FIND A SOURCE LIMIT THE URL FRONTIER SCRAP AND STORE PARSE BLOCS
77. STEP BY STEP FIND A SOURCE LIMIT THE URL FRONTIER SCRAP AND STORE PARSE BLOCS
78. STEP BY STEP FIND A SOURCE LIMIT THE URL FRONTIER SCRAP AND STORE PARSE BLOCS
79. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS
80. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS URL FRONTIER QUEUE
81. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS SCRAPERS SCRAPERS PROXIES URL FRONTIER QUEUE
82. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS SCRAPERS SCRAPERS PROXIES URL FRONTIER TARGET QUEUE
83. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS SCRAPERS SCRAPERS PROXIES URL FRONTIER STORAGE TARGET QUEUE
84. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS SCRAPERS SCRAPERS PROXIES URL FRONTIER SCRAPERS SCRAPERS PARSERS STORAGE TARGET QUEUE
85. ARCHITECTURE SCRAPERS SCRAPERS SCRAPERS SCRAPERS SCRAPERS PROXIES URL FRONTIER SCRAPERS SCRAPERS PARSERS STORAGE DATABASE TARGET QUEUE
86. Fabien VAUCHELLES zelros.com / fabien.vauchelles.@zelros.com / @fabienv http://bit.ly/breizhscraping The best opensource proxy for Scraping !