From website to JSON data in 30 minutes with Scrapy

Scrapy is an open source framework to extract unstructured data from websites easily. Built for web scraping, Scrapy can be used for different scopes like crawling, monitoring and web applications tests.

After an introduction to all major features, a real use case will be shown to create a fully functional crawling system which is able to scrape and store all unstructured data gathered from web pages.

Transcript

1. WEB SITES TO JSON DATA IN 30 MINUTES WITH SCRAPY (Blame me if it takes more time!)
2. Emanuele Palazzetti (Python backend Developer) @ https://evonove.it git.palazzetti.me :: @palazzem :: github.com/palazzem
3. Before we begin, I want to tell you a story An unhappy story of me searching a "magic" framework: SCRAPY
4. (During an ordinary day) Everything goes right... ...until a customer knocks to your door... ...he wants to talk about a really important project...
5. Default approach: hear the customer (Not always the right thing to do)
6. Customer: "I've a brilliant idea!" Probably this is the "point of no return"
7. Customer: "Let's gather the Internet searching for companies information"
8. Google custom search search everything you need JSON/XML/... data extraction save all results in your (web) application Ok we have a deal!
9. Customer: IT'S TOO EXPENSIVE
10. Customer: WE MUST BYPASS GOOGLE
12. I found myself in a maelstorm Me: "Ok. What's your plan?" (I'll never say this again)
13. Customer: "Surf the Internet and you're done!" [...] "No?"
14. In this situation you should say what is good and what is bad Especially when we talk about ideas Especially when we talk about "surf"
15. Good idea / Bad idea
16. Customer requirements: gather information from customers website but we don't have customers website we have VAT, phone number, and other heterogeneous data use standard Google search page follow first N links grab (steal) emails!
17. ScraperWiki? SCRAPY! $ pip install scrapy $ scrapy startproject harvester
18. $ tree . ├── harvester │ ├── __init__.py │ ├── items.py # Item class │ ├── pipelines.py │ ├── settings.py │ └── spiders # Spiders class folder │ └── __init__.py └── scrapy.cfg
19. Item All web pages contain unstructured data Item class is a container used to collect scraped data Provides a dictionary-like API Class attributes are Field() objects
20. CompanyItem from scrapy.item import Item, Field class CompanyItem(Item): link = Field() emails = Field() Just a note: >>> company['name'] = 'Evonove' # setting unknown field Traceback (most recent call last): ... KeyError: 'CompanyItem does not support field: name'
21. Spider defines the custom behavior during scraping how Spider follows links how Spider finds unstructured data through Selectors how to populate Item() object
22. My first Spider class SpiderBot(Spider): name = 'bot' start_urls = [ "https://evonove.it/contacts/" ]
23. Parse the response! def parse(self, response): selector = Selector(response) # Use regex to find valid emails emails = selector.re(self.EMAIL_REGEX) if (emails): item = CompanyItem() item['link'] = response.url item['emails'] = emails return item return Easy thanks to Selector!
24. We don't have customers website def __init__(self, filename=None, *args, **kwargs): super(SpiderBot, self).__init__(*args, **kwargs) # Get start urls from file self.start_urls = search_urls_from_file(filename) if filename else [] search_urls_from_file https://www.google.it/search?q={}
25. What is missing? scrape Google search! find all results grab first 5 links follow them scrape resulting page! Did I tell you Scrapy is batteries included?
26. Hello CrawlSpider! class SpiderBot(CrawlSpider): name = 'bot' rules = ( Rule( SgmlLinkExtractor(), ), )
27. Ok maybe not *all* links class SpiderBot(CrawlSpider): name = 'bot' rules = ( Rule( SgmlLinkExtractor(restrict_xpaths=('//li//h3',)), ), )
28. And only first five class SpiderBot(CrawlSpider): name = 'bot' rules = ( Rule( SgmlLinkExtractor(restrict_xpaths=('//li//h3',)), process_links='filter_links', ), ) def filter_links(self, links): # Get only first 5 links if available return links[0:5]
29. Oh yes... I should parse all responses! class SpiderBot(CrawlSpider): name = 'bot' rules = ( Rule( SgmlLinkExtractor(restrict_xpaths=('//li//h3',)), process_links='filter_links', callback='parse_item', # parse() is now parse_item() ), ) def filter_links(self, links): # Get only first 5 links if available return links[0:5]
30. Customer's spider! class SpiderBot(CrawlSpider): name = 'bot' rules = ( Rule( SgmlLinkExtractor(restrict_xpaths=('//li//h3',)), process_links='filter_links', callback='parse_item', ), ) def filter_links(self, links): # Get only first 5 links if available return links[0:5] def __init__(self, filename=None, *args, **kwargs): super(SpiderBot, self).__init__(*args, **kwargs) # Get start urls from file self.start_urls = urls_from_file(filename) if filename else [] def parse_item(self, response): selector = Selector(response)
31. Launch our Spider! $ scrapy crawl bot -a filename=urls.txt
32. [...] TO JSON DATA [...] We need to collect scraped CompanyItem We need to serialize CompanyItem We need to write serialized data
33. Scrapy...my old scrapy... $ scrapy crawl bot -a filename=urls.txt -o emails.json -t json
34. (At the end of an ordinary day) The customer is happy... ...he has all data he wants...
35. ...and I need to get my invoice paid before Google blacklist his IPs! Thanks for your attention!
36. Any questions? git.palazzetti.me :: @palazzem :: github.com/palazzem github.com/palazzem/google-blacklist-me