Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
0
32

Introduction to Scrapy

My talk about the framework python-based Scrapy.
Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019
Tweet

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
24
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
45
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
66
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
33

Other Decks in Programming

See All in Programming
AIコーディング道場勉強会#2 君（エンジニア）たちはどう生きるか
Avatar for misaki.otb misakiotb
1
250
datadog dash 2025 LLM observability for reliability and stability
Avatar for 株式会社IVRy（社員登壇資料） ivry_presentationmaterials
0
110
GraphRAGの仕組みまるわかり
Avatar for とすり tosuri13
7
480
Is Xcode slowly dying out in 2025?
Avatar for 上ちょ / uetyo uetyo
1
190
Rubyでやりたい駆動開発 / Ruby driven development
Avatar for chobishiba chobishiba
1
370
イベントストーミング図からコードへの変換手順 / Procedure for Converting Event Storming Diagrams to Code
Avatar for nrs nrslib
1
340
#kanrk08 / 公開版 PicoRubyとマイコンでの自作トレーニング計測装置を用いたワークアウトの理想と現実
Avatar for bash0C7 bash0c7
1
360
DroidKnights 2025 - 다양한 스크롤 뷰에서의 영상 재생
Avatar for 이가은 gaeun5744
3
320
0626 Findy Product Manager LT Night_高田スライド_speaker deck用
Avatar for Mana Takada mana_takada
0
100
20250628_非エンジニアがバイブコーディングしてみた
Avatar for ponponmikan ponponmikankan
0
370
PHP 8.4の新機能「プロパティフック」から学ぶオブジェクト指向設計とリスコフの置換原則
Avatar for 武田 憲太郎 kentaroutakeda
2
490
C++20 射影変換
Avatar for Akira Takahashi faithandbrave
0
530

Featured

See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
Avatar for David Neal reverentgeek
48
5.4k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
130
19k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.6k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
30
5.8k
Facilitating Awesome Meetings
Avatar for Lara Hogan lara
54
6.4k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
188
11k
Reflections from 52 weeks, 52 projects
Avatar for Jefferson Lam jeffersonlam
351
20k
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
44
2.4k
Gamification - CAS2011
Avatar for David Bonilla davidbonilla
81
5.3k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
233
140k

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!