Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Webエンジニアなのにブラウザの仕組みがわからないので、Pythonで自作してみた
Avatar for Tatsuki tatsuki12
4
1.7k
iOS開発×AI駆動開発 〜最近使って便利だったスキルの話〜
Avatar for nogu nogu66
0
140
in-process GraphQL のすすめ #ginzajs
Avatar for izumin5210 izumin5210
4
1.5k
The Good Stuff, Not the Slop: Engineering High-Quality Android Apps with Modern AI Tooling
Avatar for danybony danybony
1
180
Pythonの実行はどこまで賢くなったのか？ CPythonとPyPyから見る最適化のしくみ
Avatar for curekoshimizu curekoshimizu
4
2.8k
片田舎のおっさん、 Swift Buildのダイアモンド問題解決の不具合修正PRを出すが、解決方法がキャッシュをしないようにすることであり、ビルド時間が伸びると言われてマージされないので高速化もする/swiftbuild
Avatar for yimajo yimajo
0
360
Vibes Containers 〜AIで変わるコンテナ設計と運用〜
Avatar for Tetsuya Kikuchi tkikuc
1
480
FDEとは、何者なのか？
Avatar for Masapyon masapyon1212
0
140
MIZARU@SPAJAM2026 第二回予選
Avatar for drama 1901drama
0
110
Press start. Python's next generation.
Avatar for Carol Willing willingc
PRO
3
300
FastAPI の並行処理モデルを完全に理解する
Avatar for Yuki Ishikawa hoto17296
9
3.7k
Go を使い始めて 2 ヶ月の学び / My first two months with Go
Avatar for Daisuke Garaike contour_gara
0
420

Featured

See All Featured
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
What the history of the web can teach us about the future of AI
Avatar for Ines Montani inesmontani
PRO
1
680
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.4k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
11k
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
440
Redefining SEO in the New Era of Traffic Generation
Avatar for Szymon Słowik szymonslowik
1
400
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
1
890
WCS-LA-2024
Avatar for Leonardo Collado-Torres lcolladotor
0
820
Claude Code のすすめ
Avatar for schroneko schroneko
67
230k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
1k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
306
120k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412