Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Hive Metastoreを通して学ぶIceberg REST Catalog ― 仕様から実装まで
Avatar for okumin okumin
0
310
JJUG CCC 2026 Spring: JSpecify で実現する Kotlin フレンドリーな Java API 設計
Avatar for ternbusty ternbusty
1
110
oxlintはeslint/typescript-eslintを置き換えられるのか
Avatar for shoma-fujita shomafujita
2
290
新規プロダクトを高速で生み出すハーネスエンジニアリング
Avatar for Ryohei Ikegami seanchas116
15
7.5k
CLIであることを活かしたGitHub Copilot CLI活用術 / GitHub Copilot CLI Pro Tips & Tricks
Avatar for Naoya.i nao_mk2
1
1.1k
不変条件と整合性境界—ビジネスが決める設計判断と実現パターン / Invariants and Consistency Boundaries
Avatar for nrs nrslib
11
3k
ReactとSvelteのその先、Ripple-TS / Beyond React and Svelte: Ripple-TS
Avatar for TOMIKAWA Sotaro ssssota
3
1.7k
Transactional Change Stream Processing With Debezium and Apache Flink
Avatar for Gunnar Morling gunnarmorling
1
140
脅威をエンジニアリングの糧にして――現場編 / Turning Threats into Engineering Fuel — Field Edition
Avatar for nrs nrslib
0
220
Agentic UI beyond Chats Architecture Patterns & Open Standards @ngMunich 05/2026
Avatar for Manfred Steyer manfredsteyer
PRO
0
180
密結合なバックエンドから TypeScript のコードを生成する
Avatar for kemuridama kemuridama
1
390
Modding RubyKaigi for Myself
Avatar for yui-knk yui_knk
0
810

Featured

See All Featured
Leo the Paperboy
Avatar for Maya Tellez mayatellez
7
1.8k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
930
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
5.8k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.7k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
128
55k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.9k
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
770
Stewardship and Sustainability of Urban and Community Forests
Avatar for Eric Wiseman pwiseman
0
220
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
2
310
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
15k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412