Europython - Dive into Scrapy

Transcript

1. DIVE INTO SCRAPY Juan Riaza @juanriaza
2. @juanriaza Juan Riaza Who am I? • Developer Evangelist @ Scrapinghub • Pythonista, Djangonaut from 0.96 • I like to reverse engineer stuff
3. APIs Why Web Scraping Semantic web
4. What is Web Scraping The main goal in scraping is to extract structured data from unstructured sources, typically, web pages.
5. What for • Monitor prices • Leads generation • Aggregate information • Your imagination is the limit
6. Do you speak HTTP? Headers, Query String Status Codes Methods Persistence GET, POST, PUT, HEAD… 2XX, 3XX, 4XX, 418 , 5XX, 999 Accept-language, UA*… Cookies
7. Standard Library HTTP for humans Let’s perform a request urllib2 python-requests
8. import requests req = requests.get('http://scrapinghub.com/about/') Show me the code! What now?
9. HTML is not a regular language
10. lxml pythonic binding for the C libraries libxml2 and libxslt beautifulsoup html.parser, lxml, html5lib HTML Parsers
11. import requests import lxml.html req = requests.get('https://ep2015.europython.eu/schedule/list/') tree = lxml.html.fromstring(req.text) for tr in tree.xpath('//td[@data-talk]/div[@class="name"]/a'): name = tr.xpath('text()') url = tr.xpath('@href') Show me the code!
12. “Those who don't understand xpath are cursed to reinvent it, poorly.”
13. Scrapy-ify early on
14. “An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.”
15. $ scrapy shell <url> An interactive shell console Invaluable tool for developing and debugging your spiders
16. An interactive shell console >>> response.url 'http://example.com' >>> response.xpath('//h1/text()') [<Selector xpath='//h1/text()' data=u'Example Domain'>] >>> view(response) # open in browser >>> fetch('http://www.google.com') # fetch other URL
17. $ scrapy startproject <name> europython ├── europython │ ├── __init__.py │ ├── items.py │ ├── pipelines.py │ ├── settings.py │ └── spiders │ └── __init__.py └── scrapy.cfg Starting a project
18. What is a spider
19. import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ 'http://www.example.com/1.html', ] def parse(self, response): msg = 'A response from %s just arrived!' % response.url self.logger.info(msg) What is a Spider?
20. What is a Spider? import scrapy from europython.items import MyItem class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] def start_requests(self): yield scrapy.Request('http://www.example.com/1.html', self.parse) yield scrapy.Request('http://www.example.com/2.html', self.parse) yield scrapy.Request('http://www.example.com/3.html', self.parse) def parse(self, response): for h3 in response.xpath('//h3').extract(): yield MyItem(title=h3) for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse)
21. import scrapy class MySpider(scrapy.Spider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [ ‘http://www.example.com/1.html' ] def parse(self, response): for h3 in response.xpath(‘//h3/text()’).extract(): yield {‘title’: h3} for url in response.xpath('//a/@href').extract(): yield scrapy.Request(url, callback=self.parse) What is a Spider? 1.0
22. Items import scrapy class Product(scrapy.Item): name = scrapy.Field() price = scrapy.Field() stock = scrapy.Field() last_updated = scrapy.Field(serializer=str)
23. Item Loaders
24. Item exporters •Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends •Backends FTP, S3, local filesystem
25. from django.db import models class Person(models.Model): name = models.CharField(max_length=255) age = models.IntegerField() from scrapy.contrib.djangoitem import DjangoItem class PersonItem(DjangoItem): django_model = Person DjangoItem
26. Under the hood
27. Architecture
28. Item pipelines • Default field value • Validating scraped data • Checking for duplicates • Storing items • 3rd party integrations
29. Middlewares • Session handling • Retrying bad requests • Modifying requests • Randomize User Agent
30. Batteries included • Logging • Stats collection • Testing: contracts • Telnet console: inspect a Scrapy process
31. scrapinghub/pycon-speakers Project Example
32. Avoid getting banned • Rotate your User Agent • Disable cookies • Randomized download delays • Use a pool of rotating IPs • Crawlera
33. Broad Crawls: Frontera
34. A service daemon to run Scrapy spiders $ scrapyd-deploy Deployment 1.0 scrapyd
35. Scrapy Cloud $ shub deploy
36. TONS of Open Source Fully remote distributed team About us
37. Professional Services Products Mandatory Sales Slide • Scrapy Cloud • Crawlera
38. We’re hiring!
39. Gracias. Juan Riaza @juanriaza