Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Scrapy Overview

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
Java 27新機能 / Java 27 new features
Avatar for Naoki Kishida kishida
2
190
Webの地図
Avatar for Yosuke Furukawa yosuke_furukawa
PRO
6
5k
UnityでSystem.Net.WebSocketsなWebSocketサーバが動かないのでUnity Monoのコードを覗いてみた / about implementing websocket server with unity mono
Avatar for にー兄さん drumath2237
1
520
iOSDC2026登壇資料.pdf
Avatar for RioFujimon riofujimon
0
200
なぜCTOを降りてFDEを選んだのか？〜なぜプロダクト企業がFDEで顧客の現場に踏み込むのか〜
Avatar for ryuta sakamoto gonta
1
120
Are APIs Still Relevant in the AI Era?
Avatar for Antoine Bluchet soyuka
0
400
技術的負債を組織課題として解く-増えすぎたマイクロサービスとの戦い-
Avatar for reimaru reimaru
1
2.6k
AHC070解法紹介
Avatar for eijirou eijirou
0
150
Vue Fes Japan 2026 タイムテーブル徹底解説
Avatar for 448jp | OKI Yoshiya 448jp
1
590
難しいけど、読めた。- OSSの入口に立った話。
Avatar for shota sts11142
0
130
iOS 27でニュースアプリはどう変わる！？ 〜日経電子版の新機能対応と、開発事例から〜
Avatar for Kazuki Nakashima lynnswap
7
13k
re:Inventに行く前に知っておきたい現地参加のノウハウ
Avatar for nokomoro3 nokomoro3
0
300

Featured

See All Featured
SEOcharity - Dark patterns in SEO and UX: How to avoid them and build a more ethical web
Avatar for Sara Fernández Carmona sarafernandez
0
290
Product Roadmaps are Hard
Avatar for C. Todd Lombardo iamctodd
55
13k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.9k
Gemini Prompt Engineering: Practical Techniques for Tangible AI Outcomes
Avatar for Mfonobong Umondia mfonobong
2
560
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
From π to Pie charts
Avatar for Rasagy Sharma rasagy
1
380
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
Avatar for Umar Saidu Auna auna
0
310
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
123
22k
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
10
1.1k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
508
140k
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
2
11k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412