LuaConf - Intelligent Load Balancing

How OpenResty and Lua are used to build 'intelligent load balancers', that can perform complex tasks such as: caching, throttling and application routing.

Transcript

1. BY @XLDENIS Intelligent Load Balancers 1
2. About Me 2 ! " # "
4. 7.7B+ GMV IN 2015 1200 EMPLOYEES 800k+ RPM 275k+ STORES Shopify 4
5. Intelligent Load Balancers 5
7. Load Balancers 7 terminate ssl redirect urls balance load proxy servers LBs can…
8. 8 what if LBs could…
9. 9 app specific routing across datacenters! (SORTING HAT)
10. 10 serve cached pages (EDGE CACHE)
11. 11 throttle sensitive areas (CHECKOUT THROTTLE)
12. 12 and more! HTTPDB CUSTOM BALANCERS KAFKALOGGER NOTARY IP THROTTLE
13. NGINX is rad 13
14. NGINX is rad 14 20,000 simultaneous connections per worker
15. NGINX is rad 15 C-API but no lua api
16. NGINX is rad 16 OSS and well written at that!
17. What if we could use Lua? 17
18. OpenResty 18
19. 19 OSS also well written!
20. OpenResty API 20 MATRIX / FLOW HERE
21. OpenResty API 21 MATRIX / FLOW HERE SO M ANY BOXES
22. 22 4 59 phases directives
23. 23 Full NGINX api in Lua
24. 24 Uses LuaJIT, so it’s fast
25. 25 MICROSECONDS PICTURES
27. nginx-routing-modules 27
28. nginx-routing-modules 28 30+ lua middlewares
30. 30 only 28 kloc small but critical
31. now for some examples… 31
32. Scaling is hard 32 2014 SHARDING 1 Shop = 1 Shard
33. Scaling is hard 33 2015 RESILIENCY
34. Scaling is hard 34 NOW PODS
35. Pods: wtf ? 35 • a pod is a fully isolated section of Shopify • can serve requests on its own • can fail on its own • can scale on its own
36. Pods: wtf ? 36
37. Pods: wtf ? 37
38. Pods: wtf ? 38
39. SortingHat 39
40. SortingHat 40
41. SortingHat 41 • requests belong to a pod and need to be assigned • extracts pod information • routes the request to correct upstreams
42. SortingHat 42 Serve from correct Pod Domain Lookup Analyze URL Found Not shop
43. • luckily, requests are for storefront urls • uses MySQL table mapping domains to stores • solves the first 90% s 43 Domain Lookup
44. s 44 What about the others? $ curl -v yeezysupply.com * Rebuilt URL to: yeezysupply.com/ * Trying 23.227.38.32... * Connected to yeezysupply.com (23.227.38.32) port 80 (#0) > GET / HTTP/1.1 > Host: yeezysupply.com > User-Agent: curl/7.43.0 > Accept: */* >
45. s 45 What about the others? $ curl -v yeezysupply.com * Rebuilt URL to: yeezysupply.com/ * Trying 23.227.38.32... * Connected to yeezysupply.com (23.227.38.32) port 80 (#0) > GET / HTTP/1.1 > Host: yeezysupply.com > User-Agent: curl/7.43.0 > Accept: */* > hostname
46. s 46 What about the others? $ curl -v app.shopify.com/amazon_mws/12345 * Trying 23.227.38.38... * Connected to app.shopify.com (23.227.38.38) port 80 (#0) > GET /amazon_mws/12345 HTTP/1.1 > Host: app.shopify.com > User-Agent: curl/7.43.0 > Accept: */* >
47. s 47 What about the others? $ curl -v app.shopify.com/amazon_mws/12345 * Trying 23.227.38.38... * Connected to app.shopify.com (23.227.38.38) port 80 (#0) > GET /amazon_mws/12345 HTTP/1.1 > Host: app.shopify.com > User-Agent: curl/7.43.0 > Accept: */* > shop id
48. url router 48
49. What about the others? 49 { pattern = "^/orders/(\\d+)", http = HTTP_METHODS, name = "checkout_legacy", method = shop_from_param }, { pattern = "^/favicon.ico$", http = {GET = true}, name = "favicon", method = get_favicon },
50. What about the others? 50 { pattern = "^/orders/(\\d+)", http = HTTP_METHODS, name = "checkout_legacy", method = shop_from_param }, { pattern = "^/favicon.ico$", http = {GET = true}, name = "favicon", method = get_favicon }, A table containing several key fields: pattern, http, name, method
51. What about the others? 51 1. match the pattern 2. match the http verb 3. apply the method 4. use results to route request 5. during an error log the name
52. What about the others? 52 • method can access full request information • POST data, UA, you name it • We currently have a dozen • pattern is a full regex, not a pattern (counterintuitively) • We have several hundred rules
53. 53 we have 397 routes… …and writing them manually sucks
54. 54 what if we could autogenerate them from applications!
55. Automatic Route Dumps 55 $ ./script/dump-routes export /services > routes Loading 2158 routes Selecting 661 routes Ouputting 397 rules
56. Automatic Route Dumps 56 { "pattern": "\\A\/services\/ping(?:\\.([^\/.?]+))?\\Z", "name": "services_ping", "http": [ "DELETE", "GET", "PATCH", "POST", "PUT", "HEAD" ], "method": "master_only" }
57. EdgeCache 57
59. 59 People love their lipstick
61. 61 a little too much in fact…
65. EdgeCache 65 • standard caches don't work • need to handle order of magnitude more traffic • Need to handle sensitive information
66. EdgeCache 66 • generates app aware cache keys • uses cookies to avoid mixing up privileged pages • uses a super short TTL (5s)
67. EdgeCache 67
68. Checkout Throttle 68
71. Checkout Throttle 71 • caching read-heavy pages is easy (sort of) • caching write-heavy areas is much harder • instead throttle access to write portions of the site
72. Checkout Throttle 72 required: stateless, fair
73. Checkout Throttle 73
74. Flow 74 1. You enter the queue 2. Browser long-polls 3. PID allows you in
75. Flow - PID 75 PROCESS PID CONTROLLER SENSOR Controller Output Setpoint Process Variable Measured Process Variable Disturbances
76. Flow - PID 76 • PID maximizes RPM, adjusts queue lag • On each request, look at cookie, compare to lag • If lag > cookie, request passes • If lag < cookie, and we have spare RPM, request passes • Otherwise, we bounce request
77. Custom Balancers 77
78. Custom Balancers 78 • often lb algorithms are not well suited to applications • bad algorithm causes lost performance • can increase errors
79. 79 heterogenous deploys you can’t always buy all new servers
80. 80 heterogenous load not every request is equal
81. 81 queuing behaviour rails vs node
82. 82 request priority not every request is equal
83. 83 balance_by_lua
84. 84 LBAFO Load Balancer Algorithm Face-Off
85. Death Ray 85 • use a single upstream until unhealthy • maximizes downtown
86. Mobile First 86 • mobile is the future • restrict non-mobile user capacity • create dedicated mobile pools • blackberry & windows get less
87. 87 some were serious…
88. Deterministic Hashing 88 • client maps to deterministic set • uses IP + shop_id • creates blast radius • Used by Google
89. EWMA 89 • Exponentially Weighted Moving Average • request latency per upstream • based on Finangle (Twitter) algorithms
90. HTTPDB 90 the true webscale db
91. HTTPDB 91 • not actually a db • shared dicts over HTTP • real-time configuration!
92. Thanks! Now, questions? 92 FOLLOW ME: @XLDENIS