Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
You, latency and profiling @ GopherCon India 2017
Search
Filippo Valsorda
February 25, 2017
Programming
13
3.8k
You, latency and profiling @ GopherCon India 2017
Filippo Valsorda
February 25, 2017
Tweet
Share
More Decks by Filippo Valsorda
See All by Filippo Valsorda
Asynchronous networking @ GopherCon 2018
filosottile
3
1.9k
Le note cifrate di Antonio Marzi
filosottile
1
340
Why cgo is slow @ CapitalGo 2018
filosottile
2
4.2k
Squeezing a key through a carry bit @ 34c3
filosottile
0
1.5k
Calling Rust from Go, without cgo @ GothamGo 2017
filosottile
1
2.5k
You, latency and profiling @ GolangUK 2017
filosottile
0
1.1k
Encrypting the Internet with Go @ GopherCon 2017
filosottile
9
2.6k
TLS 1.3 @ 33c3
filosottile
4
6.6k
Stealing Bitcoin With Math - HOPE XI
filosottile
13
29k
Other Decks in Programming
See All in Programming
ECMAScript、Web標準の型はどう管理されているか / How ECMAScript and Web standards types are maintained
petamoriken
3
390
The Shape of a Service Object
inem
0
520
Perl 5 OOP機構30年史 - Perl 5's OOP Mechanism over the past 30 years
moznion
0
250
rbs-inlineを導入してYARDからRBSに移行する
euglena1215
1
290
GraphQL あるいは React における自律的なデータ取得について
quramy
11
3k
Amazon Neptuneで始める初めてのグラフDB ー グラフDBを使う意味を考える ー
satoshi256kbyte
2
260
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
470
令和トラベルにおけるLLM活用事例:社内ツール開発から得た学びと実践
ippo012
0
210
仮想ファイルシステムを導入して開発環境のストレージ課題を解消する
segadevtech
2
550
大公開!iOS開発の悩みトップ5 〜iOSDC Japan 2024〜
ryunakayama
0
190
RAGの回答精度評価用のQAデータセットを生成AIに作らせた話
kurahara
0
250
サーバーレスで負荷試験!Step Functions + Lambdaを使ったk6の分散実行
shuntakahashi
6
1.6k
Featured
See All Featured
We Have a Design System, Now What?
morganepeng
48
7.1k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.4k
Building Applications with DynamoDB
mza
90
6k
The Straight Up "How To Draw Better" Workshop
denniskardys
230
130k
Creatively Recalculating Your Daily Design Routine
revolveconf
215
12k
Infographics Made Easy
chrislema
239
18k
Web development in the modern age
philhawksworth
205
10k
Being A Developer After 40
akosma
84
590k
How GitHub (no longer) Works
holman
310
140k
[RailsConf 2023] Rails as a piece of cake
palkan
48
4.6k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
32k
How To Stay Up To Date on Web Technology
chriscoyier
786
250k
Transcript
You, latency and profiling Filippo Valsorda — @FiloSottile
What is fast?
A fast hash… … can hash many MB per second
A fast regex… … can process many MB/s
A fast database… … can store many GB per second
A fast database… … can store many GB per second
… or replies to queries in a few milliseconds
A fast website… … can handle many requests at the
same time … or loads in a few milliseconds
A fast API… … can handle many clients at the
same time … or answers in a few milliseconds
Fast is… Throughput and Latency
CPU profiling -cpuprofile /debug/pprof/profile
CPU profiling SIGPROF
runtime/proc.go
CPU profiling http.Handler 1 Running on CPU http.Handler 2 I/O
I/O http.Handler 3 I/O http.Handler 4 I/O
CPU profiling Running on CPU I/O I/O I/O I/O http.Handler
1 http.Handler 2 http.Handler 3 http.Handler 4
CPU profiling I/O Observed latency http.Handler 1
func Write(data []byte) { for i := 0; i <
50; i++ { tmpfile, _ := ioutil.TempFile("", "ex") defer os.Remove(tmpfile.Name()) _, err = tmpfile.Write(data) tmpfile.Close() } } func Hash(data []byte) { for i := 0; i < 50; i++ { sha256.Sum256(data) } }
$ time curl http://127.0.0.1:12345/hash-and-write 9.831 total go tool pprof -web
slowserver cpu.pprof
None
$ time curl http://127.0.0.1:12345/hash-and-write 9.831 total $ time
curl http://127.0.0.1:12345/write-no-hash 7.692 total
Latency is not all about CPU
CPU profiling is for throughput The tracer is for latency
$ time curl http://127.0.0.1:12345/write-no-hash 7.692 total go tool trace -pprof=syscall
trace.out > syscall.pprof go tool pprof -web syscall.pprof
None
The tracer Detailed nanosecond-level log of execution events: • goroutine
scheduling, blocking • syscalls, network, I/O • garbage collection
The tracer • log files are BIG • 25% overhead
(after Go 1.7) • symbols are embedded (Go 1.7)
Trace vs. profile • Full event context • Goroutine metadata
and history • Discrete events, not samples
The tracer import "runtime/trace" import "net/http/pprof" /debug/pprof/trace?seconds=5
None
go tool trace -pprof=TYPE trace.out Supported profile types are: -
net: network blocking profile - sync: synchronization blocking profile - syscall: syscall blocking profile - sched: scheduler latency profile
go tool trace -pprof=syscall trace.out > syscall.pprof go tool pprof
-web syscall.pprof
go tool trace -pprof=syscall trace.out > syscall.pprof go-torch -b syscall.pprof
&& open torch.svg
None
None
-pprof=syscall func Write(data []byte) { for i := 0; i
< 50; i++ { tmpfile, _ := ioutil.TempFile("", "ex") defer os.Remove(tmpfile.Name()) _, err = tmpfile.Write(data) tmpfile.Close() } }
-pprof=sync // Block waits on a channel func Block() {
<-time.After(1 * time.Second) }
-pprof=sync
-pprof=sync
-pprof=net func Download() []byte { r, err := http.Get("https://golang.org/doc/go1.8") body,
err := ioutil.ReadAll(r.Body) r.Body.Close() return body }
-pprof=net
-pprof=net
None
Writing our own profile prof := make(map[uint64]pprof.Record) for _, ev
:= range events { if ev.Type != trace.EvGoBlockNet { continue } rec := prof[ev.StkID] rec.Stk = ev.Stk rec.N++ rec.Time += ev.Link.Ts - ev.Ts prof[ev.StkID] = rec } pprof.BuildProfile(prof).Write(os.Stdout)
var childG = make(map[uint64]struct{}) var lastGLen int for { for
_, ev := range events { if ev.Type != trace.EvGoCreate { continue } if _, ok := childG[ev.G]; !ok && !filterStack(ev.Stk, re) { continue } childG[ev.Args[0]] = struct{}{} } if len(childG) == lastGLen { break } lastGLen = len(childG) }
if _, ok := childG[ev.G]; !ok && !filterStack(ev.Stk, re) {
continue }
tracefocus tracefocus -filter=ServeHTTP trace.out > net.pprof github.com/FiloSottile/tracetools
None
None
Build more tools! • Focus on goroutine number • Aggregate
all blocking types • Visualizations • …
tracehist tracehist -filter=Close trace.out github.com/FiloSottile/tracetools
None
Build more tools! Profile latency!
Questions? Filippo Valsorda @FiloSottile filippo@cloudflare.com