Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Scrapy Overview

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for JusBrasil JusBrasil
April 12, 2013
Programming
210
2

Scrapy Overview

An overview of the Scrapy framework by @cacovsky

Avatar for JusBrasil

JusBrasil

April 12, 2013

Other Decks in Programming

See All in Programming
phperだってパンチカードでコーディングしたい
Avatar for hideki kinjyo o0h
PRO
0
480
エージェントを使い続けるためのメタワーク / Meta-work for keeping using the agent
Avatar for r-kagaya rkaga
6
1.9k
Groovy as a human and AI friendly programming language
Avatar for paulking paulk
0
120
自作Discord音声botをLambda MicroVMsに移行した話 / Migrating My Discord Voice Bot to Lambda MicroVMs
Avatar for アキキー | Akihisa Ikeda akihisaikeda
0
170
Web Framework Benchmark Battle 2026, Any%
Avatar for Akira Matsuda a_matsuda
3
3k
Performance Optimization With LLMs
Avatar for xLeitix xleitix
0
210
【加筆修正版】Laravel のアプリケーションをどこにデプロイするか #phpcon_ehime
Avatar for akase244 akase244
0
160
私なりのハーネスエンジニアリング ～独自の言語処理系の開発を通して～
Avatar for 瀬尾佳隆 seosoft
0
110
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
Avatar for Geunhyeok LEE leegeunhyeok
0
150
Spec-Driven Development - Why Vaadin Is the Perfect Fit
Avatar for Simon Martinelli simas
PRO
0
150
Jetpack Compose Mechanisms
Avatar for Jaewoong skydoves
3
880
クラウドエージェントは、 何がそんなに便利なのか / what makes cloud agents so useful
Avatar for r-kagaya rkaga
3
600

Featured

See All Featured
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
2
1.8k
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.9k
How to build an LLM SEO readiness audit: a practical framework
Avatar for Nick Samuel nmsamuel
2
960
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
1.2k
Mind Mapping
Avatar for Hélio Medeiros helmedeiros
1
410
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
7k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
9
2.8k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
143
7.8k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.6k
Test your architecture with Archunit
Avatar for Yoan thirion
2
2.5k
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
230
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
540

Transcript

  1. Scrapy an overview

  2. /skræpi/

  3. Web Crawler vs. Web Scraper

  4. None
  5. None

  6. Scrapy Framework Scraping / Crawling / Monitoring / Testing

  7. Stable Active Large community

  8. ~200 pages of docs

  9. Commercial support

  10. Framework?

  11. None
  12. None
  13. None

  14. Twisted event loop (reactor)

  15. None

  16. Your code goes here

  17. The scraping logic

  18. None

  19. ­ HttpErrorMiddleware ­ UrlLengthMiddleware ­ DepthMiddleware

  20. ­ HttpProxyMiddleware ­ HttpCacheMiddleware ­ RedirectMiddleware

  21. Media download Persistence Post-processing

  22. Data flow control

  23. Queuing

  24. Talk is cheap, show me the code.

  25. $ pip install Scrapy $ scrapy startproject home_news

  26. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

  27. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project root

  28. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project config

  29. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Project module

  30. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your items

  31. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your pipelines

  32. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your settings

  33. home_news/ scrapy.cfg home_news/ __init__.py items.py pipelines.py settings.py spiders/ __init__.py ...

    Your spiders...
  34. None

  35. //*[@id="glb­corpo"]/div/div[1]/div[1]/div[2]/div[1]/div[1]/div/div/a/@href

  36. //*[@id="glb­materia"]/div[2]/h1/text()

  37. //*[@id="materia­letra"]/div/div/p[1]/text()

  38. None

  39. $ pwd /home/caco/studies/scrapy_news/home_news

  40. $ pwd /home/caco/studies/scrapy_news/home_news (project root)

  41. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  42. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  43. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json

  44. $ pwd /home/caco/studies/scrapy_news/home_news $ scrapy crawl g1 -o scraped_data.json -t

    json (feed exporters: json,csv,xml)
  45. None
  46. None
  47. None

  48. Other nice features • scrapyd: run as a service •

    Webservice (issue commands via http requests) • Signals • Stats module • Contribs (CrawlSpider etc)

  49. Obrigado! @cacovsky Thanks! @cacovsky

  50. Images Spatula http://www.duebuoi.it/x/uk_usd/catalog/p/spatulas~805-16x10.html Spiderman http://tincan21.deviantart.com/art/muro-spidey-307810412