Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to Scrapy

Avatar for Lucas Hiago de Moura Vilela Lucas Hiago de Moura Vilela
November 30, 2019
Programming
43
0

Introduction to Scrapy

My talk about the framework python-based Scrapy.

Avatar for Lucas Hiago de Moura Vilela

Lucas Hiago de Moura Vilela

November 30, 2019

More Decks by Lucas Hiago de Moura Vilela

See All by Lucas Hiago de Moura Vilela
SQL com Arel no Rails
Avatar for Lucas Hiago de Moura Vilela luchiago
0
29
Brown Bag - Aplicação mobile de vídeo-chamadas
Avatar for Lucas Hiago de Moura Vilela luchiago
0
50
Gitpod
Avatar for Lucas Hiago de Moura Vilela luchiago
1
70
Design pattern Adapter
Avatar for Lucas Hiago de Moura Vilela luchiago
0
38

Other Decks in Programming

See All in Programming
ALB ログから Trace を気合で繋げる技術
Avatar for Hayato Kawai fohte
7
860
AIは賢い。でも実行環境は？ CLIおじさんがAI時代に伝えたいこと ~ CLIおじさんがAI時代に伝えたいこと ~
Avatar for curekoshimizu curekoshimizu
1
140
Building an Out-of-Order CPU
Avatar for Latte72 latte72
0
660
kubernetes コンポーネント開発入門 / 新卒N年目の勉強会&交流会！〜〇〇への誘い〜 #n_study
Avatar for mazrean mazrean
0
160
Can LLMs Replicate 4 Years of Compose Migration? Exploring the boundaries of automation with 279 XML files from a real product
Avatar for makun makun
0
130
ソフトウェアラスタライザ
Avatar for Fadis fadis
1
780
30年振りにコンパイラの定数整数除算を改善した
Avatar for herumi herumi
9
4.4k
GraphRAGのKnowledge Graphを 直接！見る/View-GraphRAG's-KnowledgeGraph-directly!
Avatar for tyumugiko tyumugi1113
0
210
変化を抱擁するドキュメントの作り方 - ビジネスルール駆動開発がもたらす、コードとの新しい関係
Avatar for ioki ioki
2
110
LoopHub - ローカルで動く GitHub で、AI と共同開発
Avatar for jugyo jugyo
0
400
異なる設計思想のフレームワークを経験して得た学び
Avatar for amekuhideki amekuhideki
2
1.5k
XP祭りでしか伝わらないフリップネタ #xpjug
Avatar for bayashi murabayashi
0
110

Featured

See All Featured
Claude Code どこまでも/ Claude Code Everywhere
Avatar for nwiizo nwiizo
67
58k
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
290
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
What Being in a Rock Band Can Teach Us About Real World SEO
Avatar for Ade Holder 427marketing
0
1.1k
Done Done
Avatar for chrislema chrislema
186
16k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
400
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
Avatar for Kevin Liebholz kevinliebholz
37
6.6k
Navigating Weather and Climate Data
Avatar for Ryan Abernathey rabernat
0
510
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.5k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
7.2k
SEO in 2025: How to Prepare for the Future of Search
Avatar for Michael King ipullrank
3
3.8k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
500

Transcript

  1. Introdução ao Scrapy Uma ferramenta para web scraping

  2. $ whoami > Estagiário na empresa CodeMiner42 > Back-end developer

    no projeto Colaboradados > Graduando em Ciência da Computação pela UFPI > Entusiasta da linguagem Python > Aventurando nas trilhas do Ruby on Rails /luchiago /luchiago

  3. A mercadoria mais valiosa do mundo após o tempo são

    os dados.

  4. Como obter esses dados? > Interface de Programação de Aplicativos

    > Requisições HTTP GET THEM ALL

  5. E quando o site não fornece uma API?

  6. Crawlers vs Scraping

  7. Colaborabot http://colaboradados.com.br/bot_colaboradados.html https://twitter.com/colabora_bot

  8. Web Scraping: problemas > Bloqueio de endereço IP > robots.txt

    > HTML mal estruturado

  9. Scrapy “Uma framework open source e colaborativa para extração dos

    dados que você precisa dos websites, em uma maneira rápida, simples e escalável” https://scrapy.org/

  10. Tecnologias semelhantes em Python Beautiful Soup https://www.crummy.com/software/BeautifulS oup/bs4/doc/ Selenium https://selenium-python.readthedocs.io/

    Requests https://2.python-requests.org//en/master/

  11. City Scrapers

  12. Obrigado!