Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
0
36

Introduction to Scrapy

My talk about the framework python-based Scrapy.

Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019
Tweet

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
27
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
48
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
70
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
35

Other Decks in Programming

See All in Programming
実は歴史的なアップデートだと思う AWS Interconnect - multicloud
Avatar for maroon1st maroon1st
0
370
AIエージェント、”どう作るか”で差は出るか？ / AI Agents: Does the "How" Make a Difference?
Avatar for r-kagaya rkaga
4
1.9k
責任感のあるCloudWatchアラームを設計しよう
Avatar for アキキー akihisaikeda
3
130
AI Agent の開発と運用を支える Durable Execution #AgentsInProd
Avatar for izumin5210 izumin5210
7
2.2k
余白を設計しフロントエンド開発を 加速させる
Avatar for karacoro / からころ tsukuha
7
2.1k
AIフル活用時代だからこそ学んでおきたい働き方の心得
Avatar for shinoyu shinoyu
0
120
AI によるインシデント初動調査の自動化を行う AI インシデントコマンダーを作った話
Avatar for azukiazusa azukiazusa1
1
610
AI Agent Tool のためのバックエンドアーキテクチャを考える #encraft
Avatar for izumin5210 izumin5210
6
1.7k
Fragmented Architectures
Avatar for Denys Poltorak denyspoltorak
0
140
AgentCoreとHuman in the Loop
Avatar for Har1101 har1101
5
200
.NET Conf 2025 の興味のあるセッ ションを復習した / dotnet conf 2025 quick recap for backend engineer
Avatar for Tomohisa Takaoka tomohisa
0
120
2年のAppleウォレットパス開発の振り返り
Avatar for muno92 muno92
PRO
0
190

Featured

See All Featured
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
33
1.5k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
66
36k
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
280
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
260
Exploring anti-patterns in Rails
Avatar for Elle Meredith aemeredith
2
230
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
269
14k
Docker and Python
Avatar for Tania Allard trallard
47
3.7k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
3.9k
Designing for Performance
Avatar for Lara Hogan lara
610
70k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
40k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
71k
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
302
51k

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!