Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Tracing for Granularity
Search
JBD
June 02, 2018
Programming
2
1.8k
Tracing for Granularity
JBD
June 02, 2018
Tweet
Share
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2.1k
eBPF in Microservices Observability
rakyll
1
1.7k
OpenTelemetry at AWS
rakyll
1
1.9k
Debugging Code Generation in Go
rakyll
5
1.6k
Are you ready for production?
rakyll
8
2.8k
Servers are doomed to fail
rakyll
3
1.5k
Serverless Containers
rakyll
1
260
Critical Path Analysis
rakyll
0
630
Monitoring and Debugging Containers
rakyll
2
1.1k
Other Decks in Programming
See All in Programming
複雑化したリポジトリをなんとかした話 pipenvからuvによるモノレポ構成への移行
satoshi256kbyte
1
770
ポスターセッション: 「まっすぐ行って、右!」って言ってラズパイカーを動かしたい 〜生成AI × Raspberry Pi Pico × Gradioの試作メモ〜
komofr
0
950
ИИ-Агенты в каждый дом – Алексей Порядин, PythoNN
sobolevn
0
150
株式会社 Sun terras カンパニーデック
sunterras
0
220
2025年版 サーバーレス Web アプリケーションの作り方
hayatow
23
25k
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
4
1.3k
フロントエンド開発に役立つクライアントプログラム共通のノウハウ / Universal client-side programming best practices for frontend development
nrslib
7
3.9k
あなたの知らない「動画広告」の世界 - iOSDC Japan 2025
ukitaka
0
380
開発生産性を上げるための生成AI活用術
starfish719
1
170
ABEMAモバイルアプリが Kotlin Multiplatformと歩んだ5年 ─ 導入と運用、成功と課題 / iOSDC 2025
akkyie
0
320
CI_CD「健康診断」のススメ。現場でのボトルネック特定から、健康診断を通じた組織的な改善手法
teamlab
PRO
0
180
どの様にAIエージェントと 協業すべきだったのか?
takefumiyoshii
2
600
Featured
See All Featured
Being A Developer After 40
akosma
91
590k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
45
2.5k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
51k
RailsConf 2023
tenderlove
30
1.2k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
9
580
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.5k
Building Applications with DynamoDB
mza
96
6.6k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Faster Mobile Websites
deanohume
310
31k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
GitHub's CSS Performance
jonrohan
1032
460k
Transcript
tracing for granularity JBD, Google (@rakyll)
@rakyll
@rakyll tracing? What is tracing and why do we trace?
@rakyll
@rakyll clogged?
@rakyll leaking?
@rakyll path and direction?
@rakyll 100% availability (is a lie)
“ @rakyll A service is available if users cannot tell
there was an outage.
@rakyll Without an SLO, your team has no principled way
of saying what level of downtime is acceptable. • Error rate • Latency or throughput expectations Service Level Objectives (SLOs)
@rakyll 28 ms 100 ms 172 ms 56 ms 356
ms what user sees what else we can see sec.Check auth.AccessToken cache.Lookup spanner.Query GET /messages
@rakyll 182 ms 56 ms 245 ms what user sees
what else we can see sec.Check auth.AccessToken GET /messages 7 ms cache.Lookup
@rakyll latency...
@rakyll Go is the language to write servers. Many runtime
activities occur during the program execution: • scheduling • memory allocation • garbage collection Hard to associate a request with its impact on the runtime.
@rakyll clogged?
“ @rakyll There is no easy way to tell why
latency is high for certain requests. Is it due to GC, scheduler or syscalls? Can you review the code and tell us why? -SRE
@rakyll Execution tracer $ go tool trace • Reports fine-grained
runtime events in the lifetime of a goroutine. • Reports utilization of CPU cores. But cannot easily tell how handling a request impacts the runtime.
@rakyll 28 ms 100 ms 172 ms 56 ms 356
ms GET /messages auth.AccessToken cache.Lookup spanner.Query GET /messages
@rakyll 5 68µs 8 123µs networking serialization + deserialization garbage
collection blocking syscall what actually happens 172 ms auth.AccessToken
@rakyll 5 68µs 8 123µs epoll executing sys gc netwrite
@rakyll How? • Mark sections in code using runtime/trace. •
Enable execution tracer temporarily and record data. • Examine the recorded data.
@rakyll Go 1.11 introduces... • User regions, tasks and annotations.
• Association between user code and runtime. • Association with distributed traces.
@rakyll Go 1.11 runtime/trace import “runtime/trace” ctx, task := trace.NewTask(ctx,
“myHandler”) defer task.End() // Handler code here....
@rakyll region #1 task #1 Go 1.11 runtime/trace region #2
region #3 region #4 region #5 goroutine #1 goroutine #4 goroutine #5
@rakyll import _ "net/http/pprof" go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()
@rakyll $ curl http://server:6060/debug/pprof/trace?seconds=5 -o trace.out $ go tool trace
trace.out 2018/05/04 10:39:59 Parsing trace... 2018/05/04 10:39:59 Splitting trace... 2018/05/04 10:39:59 Opening browser. Trace viewer is listening on http://127.0.0.1:51803
Execution tracer tasks for RPCs (/usertasks)
Execution tracer tasks for RPCs (/usertasks)
RPCs overlapping with garbage collection
Execution tracer regions (/userregions)
Region summary for conn.ready
@rakyll Record in production $ curl http://server/debug/pprof/trace?seconds=5 -o trace.out $
go tool trace trace.out
@rakyll Try It! Install the Go 1.11 beta1! golang.org/dl
@rakyll $ go get go.opencensus.io/trace import rt “runtime/trace” ctx, span
:= trace.StartSpan(ctx, “/messages”) defer span.End() rt.WithRegion(ctx, “foo”, func(ctx) { // Do something... })
@rakyll Limitations • Execution tracer cannot do accounting for cross-goroutine
operations automatically. • Exposition format is hard to parse if `go trace tool` is not used.
thank you! JBD, Google
[email protected]
@rakyll