Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Transcript

1. Scrapy an overview
2. /skræpi/
3. Web Crawler vs. Web Scraper
6. Scrapy Framework Scraping / Crawling / Monitoring / Testing
7. Stable Active Large community
8. ~200 pages of docs
9. Commercial support
10. Framework?
14. Twisted event loop (reactor)
16. Your code goes here
17. The scraping logic
19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware
20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware
21. Media download Persistence Post-processing
22. Data flow control
23. Queuing
24. Talk is cheap, show me the code.
25. $ pip install Scrapy $ scrapy startproject home_news
26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...
27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Project root
28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Project config
29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Project module
30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Your items
31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Your pipelines
32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Your settings
33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ... Your spiders...
35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href
36. //*[@id="glb­materia"]/div[2]/h1/text()
37. //*[@id="materia­letra"]/div/div/p[1]/text()
39. $ pwd /home/caco/studies/scrapy_news/home_news
40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)
41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t json
42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t json
43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t json
44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t json (feed exporters: json,csv,xml)
48. Other nice features • scrapyd: run as a service • Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)
49. Obrigado! @cacovsky Thanks! @cacovsky
50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412