Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Flash Sale Engineering

Flash Sale Engineering

From stores with ads in the Super Bowl to selling Kanye’s latest album, Shopify has built a name for itself handling some of the world’s largest flash sales. These high profile events generate write-heavy traffic that can be four times our platform’s baseline throughput and don’t lend themselves to off-the-shelf solutions.

This talk is the story of how we engineered our platform to survive large bursts of traffic. Since it’s not financially sound for Shopify to have the required capacity always running, we built queueing and page caching layers into our Nginx load balancers with Lua. To guarantee these solutions worked, we tested them with a purpose-built load testing service.

Although flash sales are unique to commerce platforms, the lessons we learn from them are applicable to any services that experience bursts of traffic.

Online version: https://youtu.be/-I4tIudkArY

A6cafa6da915d3b3f513e63f1ad2c127?s=128

Emil Stolarsky

September 30, 2016
Tweet

Transcript

  1. Flash Sale Engineering by @EmilStolarsky

  2. 2 Who’s this ugly baby?

  3. 3

  4. 4 Flash Sales

  5. 5 Black Friday

  6. 6

  7. 7

  8. 8 Kanye West

  9. 9 Super Bowl Ads

  10. 10 #KylieLipKit

  11. 11 tl;dr This is a big deal

  12. 12 Write Heavy Traffic

  13. 13

  14. 14

  15. 15

  16. 16

  17. 17

  18. 18

  19. 19 How Shopify works

  20. 20 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS

  21. 21 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED REDIS

  22. 22 22 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED

    REDIS
  23. 23 23 THE INTERNET LB RAILS RESQUE WORKER MYSQL MEMCACHED

    REDIS
  24. 24 24 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED

    REDIS
  25. 25 25 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED

    REDIS
  26. 26 26 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED

    REDIS
  27. 27 27 RESQUE WORKER THE INTERNET LB RAILS MYSQL MEMCACHED

    REDIS
  28. 28 1. Right Tool 2. Heat Shield 3. Order out

    of Chaos
  29. 29 Nginx

  30. 30 Robust Piece of Software

  31. 31 Lua

  32. 32 1. Right Tool 2. Heat Shield 3. Order out

    of Chaos
  33. 33 Back Pressure

  34. 34

  35. 35 LB RAILS USER GET /checkout

  36. 36 LB RAILS USER GET /checkout GET /throttle_page

  37. 37 LB RAILS USER GET /checkout GET /throttle_page 200: (cached

    by LB) 200: throttle page
  38. 38 LB RAILS USER GET /checkout GET /throttle_page 200: (cached

    by LB) 200: throttle page GET /checkout?poll=1 200: throttle page
  39. 39 LB RAILS USER GET /checkout GET /throttle_page 200: (cached

    by LB) 200: throttle page GET /checkout?poll=1 200: throttle page GET /checkout?poll=1 GET /checkout?poll=1
  40. 40 LB RAILS USER GET /checkout GET /throttle_page 200: (cached

    by LB) 200: throttle page GET /checkout?poll=1 200: throttle page 302: set checkout pass GET /checkout?poll=1 GET /checkout?poll=1 302: /checkout
  41. 41 1. Right Tool 2. Heat Shield 3. Order out

    of Chaos
  42. 42

  43. 43 Picture of shop queue

  44. 44 Stateless* fair queueing

  45. 45 * - Almost Stateless

  46. 46 PROCESS PID CONTROLLER SENSOR Controller Output Setpoint Process Variable

    Measured Process Variable Disturbances Feedback System
  47. 47 Stateless fair queueing t Timeline of timestamp in cookie

  48. 48 Stateless fair queueing t Lag delimiter

  49. 49 Stateless fair queueing t low priority high priority

  50. 50 Stateless fair queueing 1 Bucket of high priority requests

    (max 16)
  51. 51 Stateless fair queueing t 1 high priority

  52. high priority 52 Stateless Fair queueing low priority 16 t

  53. low priority 53 Stateless Fair queueing t 8 high priority

  54. 54 Stateless Fair queueing t 12 low priority high priority

  55. 55 Stateless Fair queueing t 10 low priority high priority

  56. 56 Stateless fair queueing t 1 high priority

  57. 57 But… Why?

  58. 58

  59. 59 Reputation

  60. 60 Flash Sales Payments

  61. 61 Payments Gaming

  62. 62 Gaming Tweets

  63. 63 Nginx

  64. 64 Throttling Queueing

  65. 65 Blank Slide We’re hiring!

  66. 66 Blank Slide Thanks! Fo llow m e o n

    Tw it te r @Em ilSt ol arsky