Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
0
34

Introduction to Scrapy

My talk about the framework python-based Scrapy.

Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019
Tweet

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
26
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
46
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
68
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
34

Other Decks in Programming

See All in Programming
CSC509 Lecture 11
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
310
アーキテクチャと考える迷子にならない開発者テスト
Avatar for irof irof
9
3.2k
Vueで学ぶデータ構造入門 リンクリストとキューでリアクティビティを捉える / Vue Data Structures: Linked Lists and Queues for Reactivity
Avatar for konkarin konkarin
1
320
Java_プロセスのメモリ監視の落とし穴_NMT_で見抜けない_glibc_キャッシュ問題_.pdf
Avatar for NTTドコモソリューションズ　Java担当 ntt_dsol_java
0
220
[SF Ruby Conf 2025] Rails X
Avatar for Vladimir Dementyev palkan
0
240
Patterns of Patterns (and why we need them)
Avatar for Denys Poltorak denyspoltorak
0
110
JEP 496 と JEP 497 から学ぶ耐量子計算機暗号入門 / Learning Post-Quantum Crypto Basics from JEP 496 & 497
Avatar for mackey0225 mackey0225
2
450
Honoを技術選定したAI要件定義プラットフォームAcsimでの意思決定
Avatar for Tadashi Shigeoka codenote
0
250
2026年向け会社紹介資料
Avatar for Yasutaka misu
0
250
目的で駆動する、AI時代のアーキテクチャ設計 / purpose-driven-architecture
Avatar for MinoDriven minodriven
9
3k
DartASTとその活用
Avatar for そた sotaatos
2
140
競馬で学ぶ機械学習の基本と実践 / Machine Learning with Horse Racing
Avatar for ShoheiMitani shoheimitani
14
13k

Featured

See All Featured
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
8
1.1k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
370
20k
Fantastic passwords and where to find them - at NoRuKo
Avatar for Phil Nash philnash
52
3.5k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Code Review Best Practice
Avatar for Trisha Gee trishagee
72
19k
Side Projects
Avatar for Sacha Greif sachag
455
43k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
9
410
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.2k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
61k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1371
200k

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!