Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
エンジニアにデザインハーネスを 〜デザインプロセスを規定するためのハーネス〜 / Design harness from an engineer's perspective
Avatar for r-kagaya rkaga
2
1.8k
OpenSpecのproposalにbrainstormingを持たせてみた
Avatar for Toranosuke Minamikawa tigertora7571
1
150
琵琶湖の水は止められてもNet--HTTPのリトライは止められない / You might be able to stop the water flow of Lake Biwa but you can't stop Net::HTTP retries
Avatar for luccafort luccafort
PRO
0
480
ビデオ通話が繋がる0.2秒で何が起きているのか
Avatar for Miyamoto Naoyuki supurazako
2
160
【SRE NEXT 2026 Lunch Session】一人目専任SREの立ち上げを加速する ― AIと進めたオンボーディングで2分を0.04秒にした話
Avatar for PKSHA Technology（パークシャテクノロジー） pkshadeck
PRO
0
3.3k
AWS DevOps AgentのAzure接続機能を検証して見えた活用法／Use Cases Verified for the AWS DevOps Agent's Azure Connectivity Feature
Avatar for モブエンジニア（Masaki Okuda） masakiokuda
0
170
PostgreSQL 18で考えるUUID主キー
Avatar for Kazuhiro Seo kazuhiro1982
0
440
壊れたパーサから始める関数型設計と構成的なパーサ #fp_matsuri
Avatar for ライガー raiga0310
2
420
Augmenting AI with the Power of Jakarta EE
Avatar for ivargrimstad ivargrimstad
0
300
Terraform標準の組織で AWS CDKをどう使うか
Avatar for mu7889yoon / Yuta Nakamura mu7889yoon
1
430
コーディングルールの鮮度を保ちたい for SRE NEXT 2026 / keep-fresh-go-internal-conventions-sre-next-2026
Avatar for NAGATA Hiroaki handlename
0
160
Build-to-own AI: Agentic Development for Humans
Avatar for Ines Montani inesmontani
PRO
0
120

Featured

See All Featured
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
62k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
Avatar for Tech SEO Connect techseoconnect
PRO
0
540
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
1
820
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
1
270
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
1
2.3k
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
33
2k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
274
21k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
34
9.6k
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
2
13k
Neural Spatial Audio Processing for Sound Field Analysis and Control
Avatar for NII S. Koyama's Lab skoyamalab
0
390
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Between Models and Reality
Avatar for mayunak mayunak
4
380

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412