Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
0
36

Introduction to Scrapy

My talk about the framework python-based Scrapy.

Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019
Tweet

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
27
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
48
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
70
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
35

Other Decks in Programming

See All in Programming
Migration to Signals, Signal Forms, Resource API, and NgRx Signal Store @Angular Days 03/2026 Munich
Avatar for Manfred Steyer manfredsteyer
PRO
0
120
モックわからないマン卒業記 ~振る舞いを起点に見直した、フロントエンドテストにおけるモックの使いどころ~
Avatar for Tasuku Watanabe tasukuwatanabe
3
410
AHC061解説
Avatar for Shun_PI shun_pi
0
410
Vuetify 3 → 4 何が変わった？差分と移行ポイント10分まとめ
Avatar for kouki.miura koukimiura
0
170
ロボットのための工場に灯りは要らない
Avatar for watany watany
11
3k
DevinとClaude Code、SREの現場で使い倒してみた件
Avatar for karia/Y.Hisamatsu karia
1
1.1k
生成 AI 時代のスナップショットテストってやつを見せてあげますよ（α版）
Avatar for ojun ojun9
0
280
エンジニアの「手元の自動化」を加速するn8n 2026.02.27
Avatar for 西尾義英 symy2co
0
170
Redox OS でのネームスペース管理と chroot の実現
Avatar for isan isanethen
0
380
CS教育のDX AIによる育成の効率化
Avatar for ニフティ株式会社 niftycorp
PRO
0
150
[PHPerKaigi 2026]PHPerKaigi2025の企画CodeGolfが最高すぎて社内で内製して半年運営して得た内製と運営の知見
Avatar for Z.O.E. ikezoemakoto
0
240
Cyrius ーLinux非依存にコンテナをネイティブ実行する専用OSー
Avatar for n4mlz n4mlz
0
230

Featured

See All Featured
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
160
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.4k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
Avatar for Greg Gifford greggifford
PRO
0
110
The Impact of AI in SEO - AI Overviews June 2024 Edition
Avatar for Aleyda Solis aleyda
5
770
Measuring Dark Social's Impact On Conversion and Attribution
Avatar for Stephen Akadiri stephenakadiri
1
160
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
61k
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
150
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
630
How to build a perfect <img>
Avatar for Jono Alderson jonoalderson
1
5.3k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
2.9k

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!