Web Scraping 101

Transcript

1. Web Scraping @cyrusstoller November 17, 2015
2. Repetitive tasks? No thank you.
5. Ruby gem install faraday nokogiri Python pip install scrapy Javascript / node.js npm install cheerio cURL / wget curl -o http://example.com ! wget -r --level=2 http://example.com/
8. Defining the data we want
9. You can look this up on your own
10. You can look this up on your own
11. What’s an HTTP request?
12. Making an HTTP request
13. Dealing with Authentication
16. Concurrency
17. Picking what you want
19. <code walkthrough>
20. Turn it up
21. Questions?
22. twitter: @cyrusstoller github: @cyrusstoller blog: cyrusstoller.com ! possible spring workshop series on automation and web scraping