Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
リリース時」テストから「デイリー実行」へ！開発マネージャが取り組んだ、レガシー自動テストのモダン化戦略
Avatar for goataka (GOAMI Takaaki) goataka
0
130
tparseでgo testの出力を見やすくする
Avatar for utagawa kiki utgwkk
2
240
S3 VectorsとStrands Agentsを利用したAgentic RAGシステムの構築
Avatar for とすり tosuri13
6
320
Rediscover the Console - SymfonyCon Amsterdam 2025
Avatar for Robin Chalas chalasr
2
170
ローカルLLMを⽤いてコード補完を⾏う VSCode拡張機能を作ってみた
Avatar for NearMeの技術発表資料です nearme_tech
PRO
0
100
愛される翻訳の秘訣
Avatar for Kishikawa Katsumi kishikawakatsumi
3
330
Findy AI+の開発、運用におけるMCP活用事例
Avatar for starfish719 starfish719
0
1.2k
なあ兄弟、 余白の意味を考えてから UI実装してくれ！
Avatar for Ryomm ktcryomm
11
11k
AtCoder Conference 2025「LLM時代のAHC」
Avatar for Yuki Imajuku imjk
2
510
React Native New Architecture 移行実践報告
Avatar for taminif taminif
1
160
Navigation 3: 적응형 UI를 위한 앱 탐색
Avatar for Sungyong An fornewid
1
350
AIエンジニアリングのご紹介 / Introduction to AI Engineering
Avatar for r-kagaya rkaga
8
3k

Featured

See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
Avatar for Sam Stephenson sstephenson
162
16k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
186
22k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
55
3.1k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.6k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.7k
Visualization
Avatar for Eitan Lees eitanlees
150
16k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.4k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.1k
Navigating Team Friction
Avatar for Lara Hogan lara
191
16k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
2.9k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412