Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
React Native New Architecture 移行実践報告
Avatar for taminif taminif
1
150
CSC509 Lecture 14
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
220
俺流レスポンシブコーディング 2025
Avatar for TAK tak_dcxi
14
8.4k
チームをチームにするEM
Avatar for hitode909 hitode909
0
290
バックエンドエンジニアによる Amebaブログ K8s 基盤への CronJobの導入・運用経験
Avatar for suna-big sunabig
0
140
手が足りない！兼業データエンジニアに必要だったアーキテクチャと立ち回り
Avatar for zinkosuke zinkosuke
0
580
LLMで複雑な検索条件アセットから脱却する！！ 生成的検索インタフェースの設計論
Avatar for po3rin po3rin
2
640
Rediscover the Console - SymfonyCon Amsterdam 2025
Avatar for Robin Chalas chalasr
2
160
認証・認可の基本を学ぼう前編
Avatar for kaza kouyuume
0
190
TUIライブラリつくってみた / i-just-make-TUI-library
Avatar for kazto kazto
1
360
AIエージェントを活かすPM術 AI駆動開発の現場から
Avatar for gyuta gyuta
0
350
Full-Cycle Reactivity in Angular: SignalStore mit Signal Forms und Resources
Avatar for Manfred Steyer manfredsteyer
PRO
0
120

Featured

See All Featured
It's Worth the Effort
Avatar for 3n 3n
187
29k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
990
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
15k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
76
5.2k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.8k
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
33
1.4k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.3k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
12
970
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.9k
BBQ
Avatar for Matthew Crist matthewcrist
89
9.9k
Scaling GitHub
Avatar for Zach Holman holman
464
140k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412