Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
Rediscover the Console - SymfonyCon Amsterdam 2025
Avatar for Robin Chalas chalasr
2
120
スタートアップを支える技術戦略と組織づくり
Avatar for pospome pospome
8
14k
Building AI with AI
Avatar for Ines Montani inesmontani
PRO
1
460
【Streamlit x Snowflake】データ基盤からアプリ開発・AI活用まで、すべてをSnowflake内で実現
Avatar for Ayumu Yamaguchi ayumu_yamaguchi
1
110
データファイルをAWSのDWHサービスに格納する / 20251115jawsug-tochigi
Avatar for kasacchiful kasacchiful
2
100
AIエージェントを活かすPM術 AI駆動開発の現場から
Avatar for gyuta gyuta
0
190
TypeScriptで設計する 堅牢さとUXを両立した非同期ワークフローの実現
Avatar for Matsumoto Moeka moeka__c
6
2.8k
AI時代もSEOを頑張っている話
Avatar for しらはま shirahama_x
0
210
dnx で実行できるコマンド、作ってみました
Avatar for Tomohisa Takaoka tomohisa
0
130
S3 VectorsとStrands Agentsを利用したAgentic RAGシステムの構築
Avatar for とすり tosuri13
4
250
Querying Design System デザインシステムの意思決定を支える構造検索
Avatar for ikuma-t ikumatadokoro
1
1.2k
TUIライブラリつくってみた / i-just-make-TUI-library
Avatar for kazto kazto
1
290

Featured

See All Featured
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
65
8k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.6k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
58
6.1k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
7k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
980
Writing Fast Ruby
Avatar for Erik Berlin sferik
630
62k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.3k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
690

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412