Flash Sale Engineering

From stores with ads in the Super Bowl to selling Kanye’s latest album, Shopify has built a name for itself handling some of the world’s largest flash sales. These high profile events generate write-heavy traffic that can be four times our platform’s baseline throughput and don’t lend themselves to off-the-shelf solutions.

This talk is the story of how we engineered our platform to survive large bursts of traffic. Since it’s not financially sound for Shopify to have the required capacity always running, we built queueing and page caching layers into our Nginx load balancers with Lua. To guarantee these solutions worked, we tested them with a purpose-built load testing service.

Although flash sales are unique to commerce platforms, the lessons we learn from them are applicable to any services that experience bursts of traffic.

Online version: https://youtu.be/-I4tIudkArY

Transcript

1. Flash Sale Engineering by @EmilStolarsky
2. 2 Who’s this ugly baby?
4. 4 Flash Sales
5. 5 Black Friday
8. 8 Kanye West
9. 9 Super Bowl Ads
10. 10 #KylieLipKit
11. 11 tl;dr This is a big deal
12. 12 Write Heavy Traffic
19. 19 How Shopify works
20. 20 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
21. 21 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
22. 22 22 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
23. 23 23 THE INTERNET LB RAILS RESQUE WORKER MYSQL MEMCACHED REDIS
24. 24 24 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
25. 25 25 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
26. 26 26 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
27. 27 27 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS
28. 28 1. Right Tool 2. Heat Shield 3. Order out of Chaos
29. 29 Nginx
30. 30 Robust Piece of Software
31. 31 Lua
32. 32 1. Right Tool 2. Heat Shield 3. Order out of Chaos
33. 33 Back Pressure
35. 35 LB RAILS USER GET /checkout
36. 36 LB RAILS USER GET /checkout GET /throttle_page
37. 37 LB RAILS USER GET /checkout GET /throttle_page 200: (cached by LB) 200: throttle page
38. 38 LB RAILS USER GET /checkout GET /throttle_page 200: (cached by LB) 200: throttle page GET /checkout?poll=1 200: throttle page
39. 39 LB RAILS USER GET /checkout GET /throttle_page 200: (cached by LB) 200: throttle page GET /checkout?poll=1 200: throttle page GET /checkout?poll=1 GET /checkout?poll=1
40. 40 LB RAILS USER GET /checkout GET /throttle_page 200: (cached by LB) 200: throttle page GET /checkout?poll=1 200: throttle page 302: set checkout pass GET /checkout?poll=1 GET /checkout?poll=1 302: /checkout
41. 41 1. Right Tool 2. Heat Shield 3. Order out of Chaos
43. 43 Picture of shop queue
44. 44 Stateless* fair queueing
45. 45 * - Almost Stateless
46. 46 PROCESS PID CONTROLLER SENSOR Controller Output Setpoint Process Variable Measured Process Variable Disturbances Feedback System
47. 47 Stateless fair queueing t Timeline of timestamp in cookie
48. 48 Stateless fair queueing t Lag delimiter
49. 49 Stateless fair queueing t low priority high priority
50. 50 Stateless fair queueing 1 Bucket of high priority requests (max 16)
51. 51 Stateless fair queueing t 1 high priority
52. high priority 52 Stateless Fair queueing low priority 16 t
53. low priority 53 Stateless Fair queueing t 8 high priority
54. 54 Stateless Fair queueing t 12 low priority high priority
55. 55 Stateless Fair queueing t 10 low priority high priority
56. 56 Stateless fair queueing t 1 high priority
57. 57 But… Why?
59. 59 Reputation
60. 60 Flash Sales Payments
61. 61 Payments Gaming
62. 62 Gaming Tweets
63. 63 Nginx
64. 64 Throttling Queueing
65. 65 Blank Slide We’re hiring!
66. 66 Blank Slide Thanks! Fo llow m e o n Tw it te r @Em ilSt ol arsky