a day in the life of a request

Slide 1

Slide 1 text

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

hello!

Slide 4

Slide 4 text

why is it slow?

Slide 5

Slide 5 text

latency

Slide 6

Slide 6 text

t0 t1

Slide 7

Slide 7 text

Designs, Lessons and Advice from Building Large Distributed Systems, Jeff Dean

Slide 8

Slide 8 text

What is in the tail? 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0 20 40 60 80 100 0 20 40 60 80 100 Percentage of requests Latency (ms) ? Measuring and Optimizing Tail Latency, Kathryn McKinley

Slide 9

Slide 9 text

Benchmarking "Hello, World!", Dick Sites

Slide 10

Slide 10 text

Amdahl's law, Wikipedia

Slide 11

Slide 11 text

Example 2: Task Scheduling in Spark Driver W1 W2 W3 5 SnailTrail, critical participation Window Conventional profiling Window % time SnailTrail, Hoffmann et al

Slide 12

Slide 12 text

CPU Flame Graphs, Brendan Gregg

Slide 13

Slide 13 text

Systems Performance by Brendan Gregg

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Slide 16

Slide 16 text

The Gantt Chart: A Working Tool of Management, Henry Wallace Clark

Slide 17

Slide 17 text

Twitter Dot Com, Google Chrome

Slide 18

Slide 18 text

Symfony

Slide 19

Slide 19 text

Dapper, Google

Slide 20

Slide 20 text

func ProcessVideo(ctx, video) { ctx, span := trace.StartSpan(ctx, "ProcessVideo") defer span.End() video.Process() }

Slide 21

Slide 21 text

things this helps debug

Slide 22

Slide 22 text

Travis CI

Slide 23

Slide 23 text

func (rl *redisRateLimiter) RateLimit(...) { conn := rl.pool.Get() defer conn.Close() ctx, span := trace.StartSpan(ctx, "Redis.RateLimit") defer span.End() ... }

Slide 24

Slide 24 text

tx0 tx1 tx2 tx3 tx4 tx5 ... blocked

Slide 25

Slide 25 text

Slide 26

Slide 26 text

context propagation

Slide 27

Slide 27 text

Dapper, Google

Slide 28

Slide 28 text

X-Request-ID

Slide 29

Slide 29 text

SELECT COUNT(*) FROM likes WHERE artist = 'CHVRCHES'

Slide 30

Slide 30 text

SELECT COUNT(*) FROM likes WHERE artist = 'CHVRCHES' /*request_id:123e4567-e89b-12d3- a456-426655440000*/ Marginalia, Basecamp

Slide 31

Slide 31 text

EXPLAIN ANALYZE SELECT COUNT(*) FROM likes WHERE artist = 'CHVRCHES' /*request_id:123e4567-e89b-12d3- a456-426655440000*/

Slide 32

Slide 32 text

Aggregate Buffers: shared hit=74 read=41 -> Index Only Scan using likes_artist_idx on likes Index Cond: (artist = 'CHRVRCHES'::text) Heap Fetches: 10000 Buffers: shared hit=74 read=41 Planning Time: 0.344 ms Execution Time: 5.182 ms

Slide 33

Slide 33 text

req, err := http.NewRequest("GET", serviceURL, nil) req.Header.Add("X-Request-ID", requestID) resp, err := client.Do(req)

Slide 34

Slide 34 text

Canopy, Facebook

Slide 35

Slide 35 text

sampling

Slide 36

Slide 36 text

Dapper, Google

Slide 37

Slide 37 text

sampling decision

Slide 38

Slide 38 text

Travis CI

Slide 39

Slide 39 text

ﬁnding interesting traces

Slide 40

Slide 40 text

Honeycomb

Slide 41

Slide 41 text

LightStep

Slide 42

Slide 42 text

group by customer

Slide 43

Slide 43 text

happy path can also be interesting!

Slide 44

Slide 44 text

visualization

Slide 45

Slide 45 text

Jaeger, Uber

Slide 46

Slide 46 text

where do we go from here?

Slide 47

Slide 47 text

aggregation

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Canopy, Facebook

Slide 50

Slide 50 text

Canopy, Facebook

Slide 51

Slide 51 text

Pivot Tracing, Mace et al

Slide 52

Slide 52 text

Pivot Tracing, Mace et al

Slide 53

Slide 53 text

kernel tracing

Slide 54

Slide 54 text

Systems Performance by Brendan Gregg

Slide 55

Slide 55 text

Debugging Latency in Go 1.11, Jaana B. Dogan

Slide 56

Slide 56 text

eBPF

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Performance Analysis of Cloud Applications, Google

Slide 59

Slide 59 text

Performance Analysis of Cloud Applications, Google

Slide 60

Slide 60 text

Benchmarking "Hello, World!", Dick Sites

Slide 61

Slide 61 text

Benchmarking "Hello, World!", Dick Sites

Slide 62

Slide 62 text

Go Dynamic Tools, Dmitry Vyukov, GopherCon 2015

Slide 63

Slide 63 text

Visualization: Statemaps The Hurricane’s Butterﬂy, Bryan Cantrill

Slide 64

Slide 64 text

Stacked statemaps across machines Visualizing Systems with Statemaps, Bryan Cantrill

Slide 65

Slide 65 text

adaptively improving tail latency

Slide 66

Slide 66 text

"long requests reveal themselves" ~ Kathryn McKinley

Slide 67

Slide 67 text

The Tail Longest 200 requests 15 0 20 40 60 80 100 120 0 50 100 150 200 latency (ms) Top 200 requests Network and networking queueing time Idle time CPU time Dispatch queueing time latency Network & other Idle CPU work Queuing at worker not noise Network imperfections OS imperfections Long requests Overload }noise } Measuring and Optimizing Tail Latency, Kathryn McKinley

Slide 68

Slide 68 text

dealing with noise

Slide 69

Slide 69 text

speeding up work

Slide 70

Slide 70 text

recap • tail latency matters • tracing helps debug it

Slide 71

Slide 71 text

OpenCensus

Slide 72

Slide 72 text

the morning paper blog.acolyer.org

Slide 73

Slide 73 text

• Dapper, a Large-Scale Distributed Systems Tracing Infrastructure from Google, 2010 • Scuba: Diving into Data at Facebook from Facebook, 2016 • Canopy: An End-to-End Performance Tracing And Analysis System from Facebook, 2017 • Performance Analysis of Cloud Applications from Google, 2018 • Systems Performance: Enterprise and the Cloud by Brendan Gregg, 2013 • The Tail at Scale by Jeff Dean and Luiz André Barroso, 2013 • Designs, Lessons and Advice from Building Large Distributed Systems by Jeff Dean, 2009 • Data Center Computers: Modern Challenges in CPU Design by Dick Sites, 2015 • Measuring and Optimizing Tail Latency by Kathryn McKinley, Strange Loop 2017 • Benchmarking "Hello, World!" by Dick Sites, 2018 • Pivot Tracing: Dynamic Causal Monitoring for Distributed Systems by Mace et al, 2015 • RobinHood: Tail Latency Aware Caching by Berger et al, 2018 • SnailTrail: Generalizing Critical Paths for Online Analysis of Distributed Dataﬂows by Hoffmann et al, 2018

Slide 74

Slide 74 text

thanks! @igorwhilefalse