Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
0
36

Introduction to Scrapy

My talk about the framework python-based Scrapy.

Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019
Tweet

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
27
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
48
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
70
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
35

Other Decks in Programming

See All in Programming
SourceGeneratorのススメ
Avatar for tkym htkym
0
200
20260127_試行錯誤の結晶を1冊に。著者が解説 先輩データサイエンティストからの指南書 / author's_commentary_ds_instructions_guide
Avatar for nash_efp nash_efp
1
970
AI & Enginnering
Avatar for codelynx codelynx
0
110
組織で育むオブザーバビリティ
Avatar for ryotaro kobayashi ryota_hnk
0
180
15年続くIoTサービスの SREエンジニアが挑む 分散トレーシング導入
Avatar for Melonps melonps
2
200
カスタマーサクセス業務を変革したヘルススコアの実現と学び
Avatar for Tomoyuki Hamaguchi _hummer0724
0
700
OSSとなったswift-buildで Xcodeのビルドを差し替えられるため 自分でXcodeを直せる時代になっている ダイアモンド問題編
Avatar for yimajo yimajo
3
620
コントリビューターによるDenoのすゝめ / Deno Recommendations by a Contributor
Avatar for petamoriken / 森建 petamoriken
0
200
Apache Iceberg V3 and migration to V3
Avatar for Tomohiro Tanaka tomtanaka
0
160
Package Management Learnings from Homebrew
Avatar for Mike McQuaid mikemcquaid
0
220
AIと一緒にレガシーに向き合ってみた
Avatar for nyafunta9858 nyafunta9858
0
240
CSC307 Lecture 03
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
490

Featured

See All Featured
Side Projects
Avatar for Sacha Greif sachag
455
43k
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
0
2k
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
1
1.6k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.1k
Six Lessons from altMBA
Avatar for Skipper Chong Warson skipperchong
29
4.1k
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
380
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
4.6k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
75
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
270
The Psychology of Web Performance [Beyond Tellerrand 2023]
Avatar for Tammy Everts tammyeverts
49
3.3k

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!