Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Commerce-Centric Take on Queueing Fairly at High Throughput

Logan Martel
September 21, 2022

A Commerce-Centric Take on Queueing Fairly at High Throughput

When is a throttle more than a rate limiter? Historically, Shopify mitigated write-heavy traffic bursts up to 5x our baseline throughput via rate-limiting scripted in Nginx Lua modules at ingress on our load balancers. That solution served us well for years in scaling for some of the world's largest E-commerce flash sales. It also had drawbacks. Edge Tier overload protection divorced from Application Tier business logic meant inflexibility in testing, maintainability, & improving waiting room UX. High traffic on one shop could be throttled disproportionately from one load balancer to another. Users could wait 30 minutes, only to discover that their cart's inventory had gone out-of-stock 20 minutes prior. Lessons learned in moving from "off-the-shelf rate limiting" to "business-aware user queueing" broadly apply to any domain where traffic bursts could trigger a waiting room. This talk also covers our load testing & migration strategy in moving throttling away from the edge to our Rails monolith application tier.

Logan Martel

September 21, 2022
Tweet

Other Decks in Technology

Transcript

  1. 👈 Me • works on scaling Checkout @ Shopify •

    advocating stateful throttles today • shipped a scalable stateful throttle with: • Scott Francis • Bassam Mansoob • Jay Lim • Osama Sidat • Jonathan Dupuis 🛒 Docs as legal-lang
  2. The Plan (Roughly) 01 02 03 04 “Flash Sale” Thundering

    Herds Prior Work & Drawbacks “Stateful Throttle” Solutions Test in prod!
  3. 8 32M Requests per minute (peak) 11TB MySQL read I/O

    per second 24B Background jobs performed 42B API calls made to partner apps
  4. • blocking λ dequeues • “stateful” memory requirement • nevertheless,

    we’ll circle back to this idea Why not simply queue users in order (FIFO)?
  5. • “leaky bucket as queue” → stateful FIFO equivalent →

    buffered in-order requests • “leaky bucket as metre” → stateless throttle → requests either dropped (z > β) or forwarded (z ≤ β)
  6. • token buckets are equivalent mirror images to “leaky bucket

    as metre” • both statelessly throttle at rate ρ → support “bursty traffic” up to burst size z ≤ β
  7. Common Throttle Challenges • Capacity problem → limiting service rate

    to sustainable throughput • Starvation problem → ensuring prompt service for all buyers → (fast sellout) • Fairness problem → limiting deviations from FIFO service order (e.g. don’t incentivize a “race to poll”!)
  8. Variations on Windows • Sliding Window log→ track arrivals in-memory;

    pop outdated entries • Generic Cell Rate (GCRA) → metered leaky bucket with predicted arrivals • Concurrency & congestion controls → counting semaphores & TCP-style adaptive window sizes
  9. “Standard” Window Approaches Retro • Police Capacity → (limit concurrent

    buyers) ✅ • Don’t Starve Throughput → (fast sellout) ✅ • Promote Fairness → avoid a “race to poll” ❌
  10. Control Theory Idea: Adjust our “accepted traffic window” on-the-fly (à

    la TCP) Seeking stable fair throughput just as Thermostat “PID controllers”1 seek stable temperatures 1 Proportional-Integral-Derivative Controllers
  11. worked well at prioritizing “very lagged” user poll traffic difficult

    to stabilize → led to frequent window fluctuations never “quite” stateless → led to inconsistent behaviour across load balancers → complicated scaling across regional clusters Legacy Throttle “Adaptive Lag” Retro
  12. In Legacy throttle, users could also be queued for >30

    mins only to discover that their cart's inventory had already gone out-of-stock
  13. Intra-bin Queues y=10% into 1s . . . x-axis =

    arrival second x = 2s x = 1s . . . x = 3s y-axis = % into one-second bin y=25.2% into 1s y=33.33% into 1s (integer-valued) (decimal-valued)
  14. Tolerate unfairness within bins y=10% into 1s . . .

    x-axis = arrival second x = 2s x = 1s . . . x = 3s y-axis = % into one-second bin y=25.2% into 1s y=33.33% into 1s (integer-valued) (decimal-valued)
  15. Bin Scheduling • latest_bin - bin # currently assigned to

    arriving users • client_bin - bin # assigned to particular user (signed & encoded) • working_bin - max eligible bin to accept poll traffic from clients
  16. State as an enabler for scalability • Multi-layered caching -

    most requests don’t even reach Redis • Adaptive working_bin - increments can react to signals such as: ◦ compliance - do clients poll at advised poll times (not too early or late)? ◦ system health - do we have capacity to allow more traffic? • Sellout as backoff signal - traffic backoff after sellout → shorter queue times! • Horizontal Scaling - if needed, could shard bins over multiple Redis instances
  17. Some Takeaways 01 02 03 04 Race to poll drawback

    in rate limiters Benefits of queue state to fairness & UX Horizontal & adaptive scaling options Simulation-driven migrations! Thoughts? Chat with me sometime @martelogan !