Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
PHPUnitの限界をPlaywrightで補完するテストアプローチ
Avatar for yuzneri yuzneri
0
370
[DevinMeetupTokyo2025] コード書かせないDevinの使い方
Avatar for yoshi takumiyoshikawa
2
250
DataformでPythonする / dataform-de-python
Avatar for snhryt snhryt
0
150
実践 Dev Containers × Claude Code
Avatar for Yuto Akiba touyu
1
120
Streamlitで実現できるようになったこと、実現してくれたこと
Avatar for Ayumu Yamaguchi ayumu_yamaguchi
2
270
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
Avatar for sushi-cat sushi0120
0
250
Dart 参戦！！静的型付き言語界の隠れた実力者
Avatar for Kuno Ayana kno3a87
0
160
ZeroETLで始めるDynamoDBとS3の連携
Avatar for afooooil afooooil
0
150
[SRE NEXT] 複雑なシステムにおけるUser Journey SLOの導入
Avatar for yakenji yakenji
1
910
Claude Code で Astro blog を Pages から Workers へ移行してみた
Avatar for Kei Kamikawa codehex
0
170
PHPカンファレンス関西2025　基調講演
Avatar for SUGIMOTO Kei sugimotokei
6
1.1k
JetBrainsのAI機能の紹介 #jjug
Avatar for yusuke yusuke
0
180

Featured

See All Featured
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
34
6k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
229
22k
Balancing Empowerment & Direction
Avatar for Lara Hogan lara
1
530
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
267
13k
Done Done
Avatar for chrislema chrislema
185
16k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
130
19k
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
53k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
51
8.7k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
337
57k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412