Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

JusBrasil
April 12, 2013
Programming
2
190

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

JusBrasil

April 12, 2013
Tweet

Other Decks in Programming

See All in Programming
お前もAI鬼にならないか？👹Bolt & Cursor & Supabase & Vercelで人間をやめるぞ、ジョジョー！👺
taishiyade
7
4.1k
Rubyで始める関数型ドメインモデリング
shogo_tksk
0
130
Grafana Loki によるサーバログのコスト削減
mot_techtalk
1
140
5分で理解する SOLID 原則 #phpcon_nagoya
shogogg
1
280
Software Architecture
hschwentner
6
2.1k
Honoとフロントエンドの 型安全性について
yodaka
7
1.4k
CDK開発におけるコーディング規約の運用
yamanashi_ren01
2
230
Flutter × Firebase Genkit で加速する生成 AI アプリ開発
coborinai
0
170
Better Code Design in PHP
afilina
0
160
Unity Android XR入門
sakutama_11
0
170
Formの複雑さに立ち向かう
bmthd
1
910
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
920

Featured

See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
175
52k
Optimizing for Happiness
mojombo
376
70k
Designing for Performance
lara
604
68k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
Statistics for Hackers
jakevdp
797
220k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.6k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
12
980
Large-scale JavaScript Application Architecture
addyosmani
511
110k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.1k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412