Fun Learning Web Scraping

How to scrape, defend, and why it's ultimately futile.

Transcript

1. Fun Learning Web Scraping queensjs.s02e02.mp4
2. Danny Garcia @buzzedword
3. • Director of Engineering/Operations at ClassPass • Full stack engineer, DevOps day to day • First love is Javascript
4. Screen Scraping You should know about it
5. Simple Scraping
7. • Layout dependent • Difficult to traverse • Error prone • RegEx not made for XML • Doesn’t support AJAX
8. Advanced Scraping
9. • Layout dependent • Doesn’t support AJAX
10. Browser Emulation
11. • Layout dependent • Slow
13. Screen Scraping You can beat it
14. Don’t make it easy • Are you using IDs? • Are you HTML templating? • How identifiable are your components? • Randomly break your layout to stop scripts
15. All about AJAX • Defeats most RegEx scrapers • Defeats most XMLParsers • Browsers can render AJAX • Don’t use easy to access AJAX routes • https://queensjs.com/page/2 • use https://queensjs.com/_ajax/pagination&?q=1
16. Know your enemy! • CSRF tokens for fields • User authentication required • Use a paywall to discourage anonymity • Trace IP addresses • DMCA takedown request
18. Screen Scraping …it’ll still happen
19. Why? • Your data is amazing • You don’t have an API • Backdoor feature request • Hackers!
20. Scraping happens but who does it?
28. Is it worth it?
29. Costs • Development time • Anchoring bad features • Never stops • Alienating good engineers
30. What can you do? • No, really don’t use IDs. • Layout changes do break scraping • Integrate attack vectors as new product features • Don’t panic
31. • Scraping on an individual level is a query • Scraping in a cluster is an attack
32. Thank you! Danny Garcia @buzzedword ClassPass https://classpass.com/jobs