Web Scraping with Python and Scrapy

Slides from my presentation on a tool using Python, Scrapy, and Selenium given on May 15th at DevCoMO in Columbia, MO.

Transcript

1. Web Scraping With Python and Scrapy Ceili Cornelison
2. Some background on me...
3. Some background on me... Developer at Delta Systems
4. Some background on me... Developer at Delta Systems NOT a Python developer
5. Some background on me... Developer at Delta Systems NOT a Python developer Student at the University of Missouri
6. Some background on me... Developer at Delta Systems NOT a Python developer Student at the University of Missouri @cornelisonc
7. Some background on scraping
8. Some background on scraping “a computer software technique of extracting information from websites”
9. Some background on scraping “a computer software technique of extracting information from websites” Transformation of unstructured web data into structured storable, analyzable data
10. Some background on scraping “a computer software technique of extracting information from websites” Transformation of unstructured web data into structured storable, analyzable data Online price comparison, weather data monitoring, and extracting data from really unfriendly web apps with no public API
11. With great power...
12. With great power... eBay, Inc. v. Bidder’s Edge, Inc.
13. With great power... eBay, Inc. v. Bidder’s Edge, Inc. ‘Trespass to Chattel’ - don’t cause people’s stuff problems
14. With great power... eBay, Inc. v. Bidder’s Edge, Inc. ‘Trespass to Chattel’ - don’t cause people’s stuff problems United States v. Andrew Auernheimer (weev)
15. With great power... eBay, Inc. v. Bidder’s Edge, Inc. ‘Trespass to Chattel’ - don’t cause people’s stuff problems United States v. Andrew Auernheimer (weev) Computer Fraud and Abuse Act
16. With great power... eBay, Inc. v. Bidder’s Edge, Inc. ‘Trespass to Chattel’ - don’t cause people’s stuff problems United States v. Andrew Auernheimer (weev) Computer Fraud and Abuse Act Can exist in a legal grey area
17. With great power... eBay, Inc. v. Bidder’s Edge, Inc. ‘Trespass to Chattel’ - don’t cause people’s stuff problems United States v. Andrew Auernheimer (weev) Computer Fraud and Abuse Act Can exist in a legal grey area But Google does it!
18. Signs you may be up to no good
19. Signs you may be up to no good
20. Signs you may be up to no good Evading captcha, bypassing firewalls, and other security features
21. Signs you may be up to no good Evading captcha, bypassing firewalls, and other security features Potential for server overload
22. Signs you may be up to no good Evading captcha, bypassing firewalls, and other security features Potential for server overload SHOULD you be accessing this data?
23. Signs you may be up to no good Evading captcha, bypassing firewalls, and other security features Potential for server overload SHOULD you be accessing this data? Read the Terms and Conditions! (No, really.)
24. Scrapy - open source Python web scraping
25. Scrapy - open source Python web scraping “Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing”
26. Scrapy - open source Python web scraping “Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing” Dependencies: Python 2.6 or 2.7, OpenSSL, and either pip or easy_install Python package managers
27. THE PROBLEM...
28. THE PROBLEM... https://www.courts.mo.gov/casenet/base/welcome.do
29. THE PROBLEM... https://www.courts.mo.gov/casenet/base/welcome.do Information is available, but not easy to access
30. THE PROBLEM... https://www.courts.mo.gov/casenet/base/welcome.do Information is available, but not easy to access Time consuming, repetitive, could be done by a monkey...
31. THE PROBLEM... https://www.courts.mo.gov/casenet/base/welcome.do Information is available, but not easy to access Time consuming, repetitive, could be done by a monkey... OR A COMPUTER!
32. THE SOLUTION???
33. THE SOLUTION??? Scrapy!
34. THE SOLUTION??? Scrapy! How to access the data?
35. THE SOLUTION??? Scrapy! How to access the data? XPath selectors.
36. THE SOLUTION??? Scrapy! How to access the data? XPath selectors. But there’s a form... and all the links are Javascript...
37. THE SOLUTION??? Scrapy! How to access the data? XPath selectors. But there’s a form... and all the links are Javascript... There’s a library for that.
38. Many Thanks...
39. Many Thanks... DevCoMO
40. Many Thanks... DevCoMO PyroCMS
41. Many Thanks... DevCoMO PyroCMS Dr. Dennis Crouch, University of Missouri