Web Scraping for Fun and Profit using Scrapy

Transcript

1. Web Scraping for Fun and Profit using Scrapy Sigit Dewanto @seagatesoft
2. Tentang Saya • Software engineer di Scrapinghub (2014 - ?) ◦ Scrapinghub adalah Scrapy official maintainer • Sebelumnya Java dev di Wego.com, tim FPI ◦ Wego adalah travel meta-search engine
3. Web Scraping? Web Crawling?
4. Web Crawling • Bagaimana mendapatkan halaman web yang kita inginkan? • Yang kita butuhkan: HTTP Client ◦ Requests (Kenneth Reitz) ◦ Twisted web client
5. Web Scraping
6. Web Scraping • Kita sudah mendapatkan halaman web-nya. • Bagaimana mengekstrak data dari halaman web tersebut? • Yang kita butuhkan: ◦ HTML/XML parser: lxml, (Beautiful Soup) ◦ JSON parser: json, demjson ◦ Javascript parser: js2xml ◦ PDF: pdfquery
7. Scrapy • crawling + scraping framework
8. (Pakar Telematika)
9. 1. Buat proyek scrapy startproject <project_name>
10. 2. Definisikan skema data from scrapy import Item, Field class ProductItem(Item): sku = Field() name = Field() price = Field()
11. 3. Buat spider • Inspeksi sampel halaman web ◦ Firebug, Google Developer Tools ◦ scrapy shell • CSS Selector, XPath • Regular Expression (regex) ◦ When we need to extract part of string
12. 4. Run the spider scrapy crawl <spider_name> -o <output-file> --logfile=<log-file>
13. Scrapinghub Cloud • Dash: web dashboard • shub: command line tool untuk deployment • scrapinghub: wrapper untuk Scrapinghub API
14. Terima Kasih! • Source code ◦ https://github.com/seagatesoft/scrapydemo • Tertarik menggunakan Scrapinghub cloud? ◦ https://app.scrapinghub.com/account/signup/
15. Etika/Kesopanan • Crawl rate ◦ Jeda antara HTTP request ◦ Batasi concurrent requests • Patuhi robots.txt • Identifikasi crawler Anda dengan User-Agent header
16. Aspek Legal “It’s not scraping itself which is good or bad, or legal or illegal, but the circumstances in which you’re doing it.” Francis Irving - ScraperWiki (https://blog.scraperwiki.com/2012/04/is-scraping-legal/ )
17. Common issues • HTML page manipulated by Javascript • Antibot measurements ◦ Bot detection + ban ◦ Spider trap ◦ CAPTCHA • Huge number of links/pages • Irregular structure (no pattern)