Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Javaの型とAI時代に型が大事な理由 / java types and type in AI era
Avatar for Naoki Kishida kishida
2
140
Language Server 使ってる？ 〜VSCode と Zed の場合〜 / Are you using a Language Server? ~For VS Code and Zed~
Avatar for NAGATA Hiroaki handlename
0
790
Inside Stream API
Avatar for Yuichi.Sakuraba skrb
1
730
New "Type" system on PicoRuby
Avatar for pocke pocke
1
960
DynamoDBには集計系のクエリがないけどなんとかしたい
Avatar for 村上優稀 musan
1
180
Skillsは効率化、Agentsは"自分の拡張"——Builder時代のエージェント編成（CC Night 2026）
Avatar for Hiroki Uemura wemra
1
140
正しくソフトウェアを作る、前提を疑うための認知の視点 / doubt-premise
Avatar for MinoDriven minodriven
21
6.7k
Even G2とAWSで推しのエージェントを召喚しよう！
Avatar for Har1101 har1101
1
120
Vue × Nuxt × Oxc どこまで使える？実運用の現在地
Avatar for ANDPAD inc andpad
0
260
Snowflake Summitでの新機能 CoCo / CoWork / snowflake-summit-2026-overall-what-new-coco
Avatar for Tatsuhiro tatsuhiro
1
150
TypeScript+Orvalで実現する型安全かつ堅牢でスケーラブルなマルチチャネル通知基盤 / TSKaigi Night talks ~after conference~
Avatar for doriven d0riven
0
340
JJUG CCC 2026 Spring: JSpecify で実現する Kotlin フレンドリーな Java API 設計
Avatar for ternbusty ternbusty
1
180

Featured

See All Featured
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
410
Technical Leadership for Architectural Decision Making
Avatar for Kenny Baas-Schwegler baasie
3
410
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
16k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
27k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.9k
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
63k
From Legacy to Launchpad: Building Startup-Ready Communities
Avatar for Dug Song dugsong
0
230
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.4k
SERP Conf. Vienna - Web Accessibility: Optimizing for Inclusivity and SEO
Avatar for Sara Fernández Carmona sarafernandez
2
1.5k
Designing Powerful Visuals for Engaging Learning
Avatar for Mike Taylor tmiket
1
420
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
200
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
39
3.2k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412