Primum non nocere

Principles for building and running web scrapers that are good internet citizens.

Transcript

1. Primum non nocere Lindsay Holmwood
2. Being good internet citizens
3. “Be nice”
5. Don’t disrupt the service for others
6. Your own private DoS max out bandwidth max out the app take out a database
7. Don’t incur costs
8. Bandwidth
9. Hosting
10. Don’t create (unnecessary) work
11. Alerts
12. Logs exhaust storage
13. Don’t break the law
14. Telecommunications Offences and Other Measures Act (No. 2) 2004 Section 474.6 (5) A person is guilty of an offence if: (a) the person uses or operates any apparatus or device (whether or not it is comprised in, connected to or used in connection with a telecommunications network); and (b) this conduct results in hindering the normal operation of a carriage service supplied by a carriage service provider. Penalty: Imprisonment for 2 years.
15. Be mindful of organisational factors
16. Charities
17. Government services
18. Strategies for your scrapers
19. Identification
20. Set a user agent
21. a = Mechanize.new a.user_agent_alias = 'My Scraper'
22. Capture and replay responses
23. scraper website scraper capture scraper capture
24. Testing tools do this already
25. require 'webmock' include WebMock::API WebMock.enable! WebMock.disable_net_connect!(allow: %r{unitedcinemas.com.au/session-times}) stub_request(:get, %r{http://www.unitedcinemas.com.au/session_data\.php\? date=.*&l=\w+&sort=title}). with(:body => "<h3>blah</h3>")
26. Test for failure, latency
27. require 'webmock' include WebMock::API WebMock.enable! WebMock.disable_net_connect!(allow: %r{unitedcinemas.com.au/session-times}) stub_request(:get, %r{http://www.unitedcinemas.com.au/session_data\.php\? date=.*&l=\w+&sort=title}). to_return { |request| sleep(rand(0.0..1.0)) { :body => "<h3>blah</h3>", :headers => { 'Content-Type' => 'text/html; charset=UTF-8' } } }
28. Use a cache
29. Intercept and serve a previously fetched document
30. Squid & Nginx
31. agent = Mechanize.new agent.set_proxy 'localhost', 8000
32. HTTPS?
33. Roll your own
34. page = get(url, cache: true)
35. def get(url, opts={}) options = { :cache => true }.merge!(opts) @agent ||= Mechanize.new case # Cache bypass when !options[:cache] page = @agent.get(url) # Cache hit when cached?(url) cache_fetch(url) # Cache miss else page = @agent.get(url) cache_store(url, page.body.to_s) end end
36. def cached?(url) cache_path(url).exist? end def cache_fetch(url) body = cache_path(url).read Nokogiri::HTML(body) end
37. def cache_store(url, content) cache_path(url).open('w') {|f| f << content} unless cached?(url) Nokogiri::HTML(content) end def cache_path(url) base = Pathname.new(__FILE__).parent.join('cache') hash = Digest::MD5.hexdigest(url) directory = base.join(hash[0]) directory.mkpath unless directory.directory? directory.join(hash[1..-1]) end
38. Rate limit requests
39. intentionally introduce a delay to web requests
40. serve errors
41. introduce delays
42. In the proxy Nginx https://lincolnloop.com/blog/rate-limiting-nginx/
43. In the proxy HAProxy http://blog.serverfault.com/2010/08/26/1016491873/
44. In the scraper Backoff: linear exponential middleware
45. Development vs Production
46. Limit impact during development
47. Doesn’t apply in production
48. Eventually you have to interact with the real thing
49. “Be nice”
50. Don’t disrupt the service for others
51. Don’t incur costs
52. Don’t create (unnecessary) work
53. Be mindful of organisational factors
54. I’m Lindsay
55. ❤ Thank you!