a day in the life of a request

a day in the life of a request

A4b95be2145cc46f891707b6db9dd82d?s=128

Igor Wiedler

March 23, 2019
Tweet

Transcript

  1. 2.
  2. 3.
  3. 5.
  4. 6.
  5. 8.

    What is in the tail? 0 0.5 1 1.5 2

    2.5 3 3.5 4 4.5 5 0 20 40 60 80 100 0 20 40 60 80 100 Percentage of requests Latency (ms) ? Measuring and Optimizing Tail Latency, Kathryn McKinley
  6. 11.

    Example 2: Task Scheduling in Spark Driver W1 W2 W3

    5 SnailTrail, critical participation Window Conventional profiling Window % time SnailTrail, Hoffmann et al
  7. 14.
  8. 15.
  9. 18.
  10. 22.
  11. 23.

    func (rl *redisRateLimiter) RateLimit(...) { conn := rl.pool.Get() defer conn.Close()

    ctx, span := trace.StartSpan(ctx, "Redis.RateLimit") defer span.End() ... }
  12. 25.
  13. 31.

    EXPLAIN ANALYZE SELECT COUNT(*) FROM likes WHERE artist = 'CHVRCHES'

    /*request_id:123e4567-e89b-12d3- a456-426655440000*/
  14. 32.

    Aggregate Buffers: shared hit=74 read=41 -> Index Only Scan using

    likes_artist_idx on likes Index Cond: (artist = 'CHRVRCHES'::text) Heap Fetches: 10000 Buffers: shared hit=74 read=41 Planning Time: 0.344 ms Execution Time: 5.182 ms
  15. 35.
  16. 38.
  17. 40.
  18. 41.
  19. 48.
  20. 56.
  21. 57.
  22. 67.

    The Tail Longest 200 requests 15 0 20 40 60

    80 100 120 0 50 100 150 200 latency (ms) Top 200 requests Network and networking queueing time Idle time CPU time Dispatch queueing time latency Network & other Idle CPU work Queuing at worker not noise Network imperfections OS imperfections Long requests Overload }noise } Measuring and Optimizing Tail Latency, Kathryn McKinley
  23. 73.

    • Dapper, a Large-Scale Distributed Systems Tracing Infrastructure from Google,

    2010 • Scuba: Diving into Data at Facebook from Facebook, 2016 • Canopy: An End-to-End Performance Tracing And Analysis System from Facebook, 2017 • Performance Analysis of Cloud Applications from Google, 2018 • Systems Performance: Enterprise and the Cloud by Brendan Gregg, 2013 • The Tail at Scale by Jeff Dean and Luiz André Barroso, 2013 • Designs, Lessons and Advice from Building Large Distributed Systems by Jeff Dean, 2009 • Data Center Computers: Modern Challenges in CPU Design by Dick Sites, 2015 • Measuring and Optimizing Tail Latency by Kathryn McKinley, Strange Loop 2017 • Benchmarking "Hello, World!" by Dick Sites, 2018 • Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems by Mace et al, 2015 • RobinHood: Tail Latency Aware Caching by Berger et al, 2018 • SnailTrail: Generalizing Critical Paths for Online Analysis of Distributed Dataflows by Hoffmann et al, 2018