Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
生成AIを利用するだけでなく、投資できる組織へ
Avatar for pospome pospome
2
440
GISエンジニアから見たLINKSデータ
Avatar for nokonoko1203 nokonoko1203
0
190
組み合わせ爆発にのまれない - 責務分割 x テスト
Avatar for HAL halhorn
1
180
Pythonではじめるオープンデータ分析〜書籍の紹介と書籍で紹介しきれなかった事例の紹介〜
Avatar for Ryo YOSHI welliving
3
760
それ、本当に安全？ ファイルアップロードで見落としがちなセキュリティリスクと対策
Avatar for ikechi penpeen
6
1.8k
[AI Engineering Summit Tokyo 2025] LLMは計画業務のゲームチェンジャーか？ 最適化業務における活⽤の可能性と限界
Avatar for terry-u16 terryu16
2
240
CSC307 Lecture 01
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
650
Implementation Patterns
Avatar for Denys Poltorak denyspoltorak
0
140
ThorVG Viewer In VS Code
Avatar for Nor-s nors
0
540
これならできる！個人開発のすゝめ
Avatar for Tsubasa SEKIGUCHI tinykitten
PRO
0
140
PC-6001でPSG曲を鳴らすまでを全部NetBSD上の Makefile に押し込んでみた / osc2025hiroshima
Avatar for Izumi Tsutsui tsutsui
0
200
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
160

Featured

See All Featured
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.6k
Self-Hosted WebAssembly Runtime for Runtime-Neutral Checkpoint/Restore in Edge–Cloud Continuum
Avatar for Yuki Nakata chikuwait chikuwait
0
270
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon szymonslowik
1
180
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
57
6.7k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4.1k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
0
2.8k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
4.5k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
78
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
0
680

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412