Keeping Throughput High on Green Saturday

This is a sponsored talk by Weedmaps.

Every April, we observe Earth Day and celebrate our planet’s beauty and resources, its oceans and trees. But only days earlier, another kind of tree is celebrated, and Weedmaps experiences its highest traffic of the year. Come see techniques we’ve used recently to lighten the latency on our most requested routes ahead of the elevated demand. Do you cache your API responses but want a lift in your hit ratio? Does that Elasticsearch best practice you know you’re ignoring cause nerves? We’ll pass our solutions to these problems—on the lefthand side.

Transcript

1. Keeping Throughput HIGH on Green Saturday Alexander Reiff
2. Discovery API
3. ! WM Core Discovery API
4. ! Discovery API
5. Green Saturday ⁉
6. Friday Saturday Sunday Monday 19 20 21 22 
 Green Saturday* Earth Day Passover (Night 1) Easter
7. 2016 2017 2018 2019 ❔ ❓
8. /location /brands/categories
9. /location 3 4 ! 5
10. microcache!
11. http {
 proxy_cache_path /tmp/cache keys_zone=micache: 150m max_size=200m; ...
 
 server { server_name railsconf2019.local
 
 location / { proxy_pass http: //localhost:3000;
 proxy_redirect off; ...
 
 proxy_cache micache;
 proxy_cache_valid 200 30s;
 1 2 3 4 5 6 7 8 9 10 11 12 13 14
12. 
 location / { proxy_pass http: //localhost:3000;
 proxy_redirect off; ...
 
 proxy_cache micache;
 proxy_cache_valid 200 30s;
 proxy_cache_key “$proxy_host$request_uri$http_authorization”; } }
 } 7 8 9 10 11 12 13 14 15 16 17 18
13. OpenResty
16. Decimal degree precision 1 2 3 4 5 6 Distance (meters) 0.01 0.1 1 10 100 1000 10000
18. class Types ::LatLon < Dry ::Struct ::Value attribute :latitude, Types ::Coercible ::Float attribute :longitude, Types ::Coercible ::Float def round(decimal_degrees) new(latitude: latitude.round(decimal_degrees), longitude: longitude.round(decimal_degrees)) end end 1 2 3 4 5 6 7 8 9
19. coord_params = params.permit(:latitude, :longitude).to_h rounded = Types ::LatLon[coord_params].round(4) Query ::Region.new(intersects_with: rounded).first 1 2 3
21. ❔
24. 3 4 ! 5 /location
25. 3 4 ! 5 /location
26. coord_params = params.permit(:latitude, :longitude).to_h rounded = Types ::LatLon[coord_params].round(4) cache_key = "region_ #{rounded}" Rails.cache.fetch(cache_key, expires_in: 10.minutes) do Query ::Region.new(intersects_with: rounded).first end 1 2 3 4 5 6 7
27. read-through cache
28. write-behind cache
29. class WriteBehindCache def initialize(ttl = 1.day) @ttl = ttl @store = Redis.current end def set(key, value) to_cache = { payload: value, expires_at: Time.current + @ttl } @ttl.set(key, to_cache) end def get(key) cached = @store.get(key) return unless cached refresh_cache(key) if cached[:expires_at] < Time.current cached[:payload] end private 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
30. cached[:payload] end private def refresh_cache(key) WriteBehindRefresher.perform_later(key, @ttl) end end class WriteBehindRefresher < ApplicationJob def perform(key, ttl) payload = fetch_resource(key) cache = WriteBehindCache.new(ttl) cache.set(key, payload) end def fetch_response(key) Faraday.get("https: //myapi.org/ #{key}").body end end 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
31. A Cautionary Tale
32. region_geometry_join bounding_box geometry … parent document child document id name aliases region_path region_geometry_join … regions index
33. region_geometry_join bounding_box geometry … parent document child document id name aliases region_path region_geometry_join … regions index region_geometry_join bounding_box geometry … id name aliases region_path region_geometry_join …
34. —Elasticsearch Query DSL docs
 has_child filter “If you care about query performance 
 you should not use this query.”
36. /location /brands/categories
40. class Query ::PromotedProducts def results - parser.parse(es_response) + @results ||= parser.parse(es_response) end end 1 2 3 4 5
41. 80 MS avg response time 100K RPM peak services throughput 9% microcache hit rate 53K RPM Discovery API 20% location requests
42. 100% uptime
44. Tune user input 
 to make it more useful
45. Limit external requests made while serving users
46. Reconfirm your schema is right for your queries
47. Benchmark your improvements!
48. For HIGH performance, Rails needs to be MELLOW.
50. ❓❓