Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
200

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
ポーリング処理廃止によるイベント駆動アーキテクチャへの移行
Avatar for Seitaro Fujigaki seitarof
3
1.3k
GoのDB アクセスにおける 「型安全」と「柔軟性」の両立 - Bob という選択肢
Avatar for tak tak848
0
290
Java 21/25 Virtual Threads 소개
Avatar for Sunghyouk Bae (Debop) debop
0
310
「速くなった気がする」をデータで疑う
Avatar for Leaf senleaf24
0
110
おれのAgentic Coding 2026/03
Avatar for TsukasaSekiguchi tsukasagr
1
120
Codex CLI でつくる、Issue から merge までの開発フロー
Avatar for amata1219 amata1219
0
250
CS教育のDX AIによる育成の効率化
Avatar for ニフティ株式会社 niftycorp
PRO
0
170
「接続」—パフォーマンスチューニングの最後の一手 〜点と点を結ぶ、その一瞬のために〜
Avatar for 武田 憲太郎 kentaroutakeda
4
2.2k
GC言語のWasm化とComponent Modelサポートの実践と課題 - Scalaの場合
Avatar for Rikito Taniguchi tanishiking
0
130
How to stabilize UI tests using XCTest
Avatar for Akio Itaya akkeylab
0
150
Tamach-sre-3_ANDPAD-shimaison93
Avatar for shimaison93 mane12yurks38
0
200
AI活用のコスパを最大化する方法
Avatar for Ochtum ochtum
0
360

Featured

See All Featured
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
61
43k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
230k
技術選定の審美眼（2025年版） / Understanding the Spiral of Technologies 2025 edition
Avatar for Takuto Wada twada
PRO
118
110k
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
1
11k
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
280
Deep Space Network (abreviated)
Avatar for Tony Rice tonyrice
0
97
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
A Soul's Torment
Avatar for Nat Ma seathinner
5
2.6k
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
82
6.2k
Testing 201, or: Great Expectations
Avatar for Joseph Mastey jmmastey
46
8.1k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412