Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scrapy Overview

Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
200
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
When benchmarks go bad - what I learned from measuring performance wrong
Avatar for Holly Cummins hollycummins
0
360
〜バイブコーディングを超えて〜 チームで実験し続けたAI駆動開発
Avatar for Toranosuke Minamikawa tigertora7571
0
190
ソフトウェア設計の結合バランス #phperkaigi
Avatar for Takuma Kajikawa kajitack
0
490
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
540
Agentic UI in the Frontend: Architectures with Open Standards @JAX 2026 in Mainz
Avatar for Manfred Steyer manfredsteyer
PRO
0
100
PHPでローカル環境用のSSL/TLS証明書を発行することはできるのか？ #phpconkagawa
Avatar for akase244 akase244
0
350
cloudnative conference 2026 flyle
Avatar for azihsoyn azihsoyn
0
130
書き換えて学ぶTemporal #fukts
Avatar for Hiroyuki ANAI pirosikick
2
360
なぜあなたのコードには「コシ」がないのか？〜AI時代に問う、最後まで美味しい設計と戦略〜 #phpconkagawa / phpconkagawa2026
Avatar for shogogg shogogg
0
150
UaaL×Androidアプリのメモリ計測 — Memory Profilerの先へ
Avatar for Ryosuke Shimizu (RIO) rio432
0
130
Lightning-Fast Method Calls with Ruby 4.1 ZJIT / RubyKaigi 2026
Avatar for Takashi Kokubun k0kubun
3
2.6k
Firefoxにコントリビューションして得られた学び
Avatar for ken7253 ken7253
2
160

Featured

See All Featured
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
3
350
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
120
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
Avatar for AKADEMIYA2063 akademiya2063
PRO
1
110
How to make the Groovebox
Avatar for あそなす asonas
2
2.2k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
3k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.9k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
10k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
140
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
470k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
0
1.3k

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412