Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
You, latency and profiling @ GopherCon India 2017
Search
Filippo Valsorda
February 25, 2017
Programming
13
3.8k
You, latency and profiling @ GopherCon India 2017
Filippo Valsorda
February 25, 2017
Tweet
Share
More Decks by Filippo Valsorda
See All by Filippo Valsorda
Asynchronous networking @ GopherCon 2018
filosottile
3
1.9k
Le note cifrate di Antonio Marzi
filosottile
1
330
Why cgo is slow @ CapitalGo 2018
filosottile
2
4.1k
Squeezing a key through a carry bit @ 34c3
filosottile
0
1.5k
Calling Rust from Go, without cgo @ GothamGo 2017
filosottile
1
2.4k
You, latency and profiling @ GolangUK 2017
filosottile
0
1k
Encrypting the Internet with Go @ GopherCon 2017
filosottile
9
2.5k
TLS 1.3 @ 33c3
filosottile
4
6.5k
Stealing Bitcoin With Math - HOPE XI
filosottile
13
29k
Other Decks in Programming
See All in Programming
Activities at Cairo Library
cairolibrary720
0
1.2k
Android開発者のための Kotlin Multiplatform入門
ntaro
0
190
Google's Recipe for Scaling (Web) Security – LocoMocoSec 2024
lweichselbaum
0
170
12年前の『型システム入門』翻訳の思い出話
mame
11
1.2k
初心者がおさえておきたいAWS CDKのベストプラクティス 2024
konokenj
15
7.3k
Ruby メモリ管理 プログラミング
megmogmog1965
0
130
ドメイン駆動設計の実践
masuda220
PRO
19
5.2k
OpenAI/Gemini APIを使って EPUBを翻訳するCLIツールをつくってみた
tomiyan
0
790
最古の関数型言語「Lisp」ことはじめ / lisp_in_kamiyama
uhooi
1
190
Javaの現状2024夏 / Java current status 2024 summer
kishida
4
1.4k
スクラムマスターって孤独じゃないですか?
yoshitaroyoyo
1
140
feature環境をGitHub ActionsとCloudFormationでいい感じに管理する
nealle
2
310
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
Automating Front-end Workflow
addyosmani
1362
200k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
23
1.9k
The Language of Interfaces
destraynor
151
23k
Practical Orchestrator
shlominoach
185
10k
What the flash - Photography Introduction
edds
65
11k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.3k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
44
4.7k
Music & Morning Musume
bryan
43
5.9k
Faster Mobile Websites
deanohume
303
30k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
121
18k
Transcript
You, latency and profiling Filippo Valsorda — @FiloSottile
What is fast?
A fast hash… … can hash many MB per second
A fast regex… … can process many MB/s
A fast database… … can store many GB per second
A fast database… … can store many GB per second
… or replies to queries in a few milliseconds
A fast website… … can handle many requests at the
same time … or loads in a few milliseconds
A fast API… … can handle many clients at the
same time … or answers in a few milliseconds
Fast is… Throughput and Latency
CPU profiling -cpuprofile /debug/pprof/profile
CPU profiling SIGPROF
runtime/proc.go
CPU profiling http.Handler 1 Running on CPU http.Handler 2 I/O
I/O http.Handler 3 I/O http.Handler 4 I/O
CPU profiling Running on CPU I/O I/O I/O I/O http.Handler
1 http.Handler 2 http.Handler 3 http.Handler 4
CPU profiling I/O Observed latency http.Handler 1
func Write(data []byte) { for i := 0; i <
50; i++ { tmpfile, _ := ioutil.TempFile("", "ex") defer os.Remove(tmpfile.Name()) _, err = tmpfile.Write(data) tmpfile.Close() } } func Hash(data []byte) { for i := 0; i < 50; i++ { sha256.Sum256(data) } }
$ time curl http://127.0.0.1:12345/hash-and-write 9.831 total go tool pprof -web
slowserver cpu.pprof
None
$ time curl http://127.0.0.1:12345/hash-and-write 9.831 total $ time
curl http://127.0.0.1:12345/write-no-hash 7.692 total
Latency is not all about CPU
CPU profiling is for throughput The tracer is for latency
$ time curl http://127.0.0.1:12345/write-no-hash 7.692 total go tool trace -pprof=syscall
trace.out > syscall.pprof go tool pprof -web syscall.pprof
None
The tracer Detailed nanosecond-level log of execution events: • goroutine
scheduling, blocking • syscalls, network, I/O • garbage collection
The tracer • log files are BIG • 25% overhead
(after Go 1.7) • symbols are embedded (Go 1.7)
Trace vs. profile • Full event context • Goroutine metadata
and history • Discrete events, not samples
The tracer import "runtime/trace" import "net/http/pprof" /debug/pprof/trace?seconds=5
None
go tool trace -pprof=TYPE trace.out Supported profile types are: -
net: network blocking profile - sync: synchronization blocking profile - syscall: syscall blocking profile - sched: scheduler latency profile
go tool trace -pprof=syscall trace.out > syscall.pprof go tool pprof
-web syscall.pprof
go tool trace -pprof=syscall trace.out > syscall.pprof go-torch -b syscall.pprof
&& open torch.svg
None
None
-pprof=syscall func Write(data []byte) { for i := 0; i
< 50; i++ { tmpfile, _ := ioutil.TempFile("", "ex") defer os.Remove(tmpfile.Name()) _, err = tmpfile.Write(data) tmpfile.Close() } }
-pprof=sync // Block waits on a channel func Block() {
<-time.After(1 * time.Second) }
-pprof=sync
-pprof=sync
-pprof=net func Download() []byte { r, err := http.Get("https://golang.org/doc/go1.8") body,
err := ioutil.ReadAll(r.Body) r.Body.Close() return body }
-pprof=net
-pprof=net
None
Writing our own profile prof := make(map[uint64]pprof.Record) for _, ev
:= range events { if ev.Type != trace.EvGoBlockNet { continue } rec := prof[ev.StkID] rec.Stk = ev.Stk rec.N++ rec.Time += ev.Link.Ts - ev.Ts prof[ev.StkID] = rec } pprof.BuildProfile(prof).Write(os.Stdout)
var childG = make(map[uint64]struct{}) var lastGLen int for { for
_, ev := range events { if ev.Type != trace.EvGoCreate { continue } if _, ok := childG[ev.G]; !ok && !filterStack(ev.Stk, re) { continue } childG[ev.Args[0]] = struct{}{} } if len(childG) == lastGLen { break } lastGLen = len(childG) }
if _, ok := childG[ev.G]; !ok && !filterStack(ev.Stk, re) {
continue }
tracefocus tracefocus -filter=ServeHTTP trace.out > net.pprof github.com/FiloSottile/tracetools
None
None
Build more tools! • Focus on goroutine number • Aggregate
all blocking types • Visualizations • …
tracehist tracehist -filter=Close trace.out github.com/FiloSottile/tracetools
None
Build more tools! Profile latency!
Questions? Filippo Valsorda @FiloSottile filippo@cloudflare.com