Exploring Go Runtime Metrics

Transcript

1. 1 Exploring Go Runtime Metrics Chin-Ming Huang & Mohit Pokharna
2. 2 Software engineer at Mercari since 2020, primarily focusing on ML-assisted Listing and Search backend. Chin-Ming Huang Software engineer at Mercari since 2019, primarily focusing on ML price and Search backend.
 Mohit Pokharna
3. 3 What’s Runtime? Agenda The runtime/metrics Module 02 01
4. 4 What’s Runtime?
5. 5 Runtime • Go Runtime is the library to execute Go programs. • Runtime contains goroutine scheduler, memory allocator, garbage collector, etc.
6. 6 Runtime Your program Go runtime OS syscall T T T T executable
7. 7 A Glimpse - Get nCPU before Scheduling src/runtime/asm_arm64.s
8. 8 src/runtime/os_darwin.go
9. 9 src/runtime/os_darwin.go
10. 10 /Library/Developer/CommandLineTools/SDKs/MacOSX<VERSION>.sdk/usr/include/sys/sysctl.h /Library/Developer/CommandLineTools/SDKs/MacOSX<VERSION>.sdk/usr/include/sys/syscall.h
11. 11 The runtime/metrics Module
12. 12 Proposal: API for unstable runtime metrics (#37112, design doc) Before runtime/metrics, runtime metrics are exposed in two ways: 1. by struct-based sampling APIs, e.g. runtime.ReadMemStats and runtime/debug.GCStats. 2. via GODEBUG flags which emit strings containing metrics to standard error (e.g. gctrace, gcpacertrace, scavtrace).
13. 13 MemStats is hard to evolve because it must obey the Go 1 compatibility rules. The existing metrics are confusing, but we can‘t change them. Some of the metrics are now meaningless (like EnableGC and DebugGC), and several have aged poorly (like hard-coding the number of size classes at 61, or only having a single pause duration per GC cycle). Hence, we tend to shy away from adding anything to this because we’ll have to maintain it for the rest of time. by struct-based sampling APIs
14. 14 The gctrace format is unspecified, which means we can evolve it (and have completely changed it several times). But it‘s a pain to collect programmatically because it only comes out on stderr and, even if you can capture that, you have to parse a text format that changes. Hence, automated metric collection systems ignore gctrace. via GODEBUG flags
15. 15 • Sampling-based API is taken in opposition to a stream-based or event-based API. • Consider “sets of metrics” as the unit of API instead of individual metrics. • Metric name be built from two components: a forward-slash-separated path to a metric where each component is lowercase words separated by hyphens, and its unit. E.g. /memory/classes/heap/free:bytes New Design
16. 16 Example Take one metric of heap for example:
18. 18 Take a Rest..
19. 19 Trace Read Method src/runtime /metrics.go src/runtime/metrics/sample.go
20. 20 src/runtime /metrics.go
21. 21 statAggregate
22. 22 metricData src/runtime /metrics.go
23. 23 agg.ensure(...) src/runtime /metrics.go
24. 24 • runtime/metrics provides an unified interface to access different metrics, including heap, sys, cpu and gc stats. • Underneath, each metric is corrected from different part of runtime, so 2 steps are performed for metrics reading: ensure & compute. Remarks
25. 25 Supported Metrics Metric Prefix Count Metrics /cgo 1 go-to-c-calls /cpu 11 gc, idle, scavenge, total, user /gc 22 cycles, scan, heap, gomemlimit, stack /godebug 23 GODEBUG /memory 14 heap, mcache, mspan, os-stack /sched 7 gomaxprocs, goroutines, latencies, pauses, mutex /sync 1 mutex wait.
26. 26 Metrics: /cpu Metric Use Case /cpu/classes/total:cpu-seconds /cpu/classes/user:cpu-seconds /cpu/classes/idle:cpu-seconds Measure CPU utilization for Go code and/or Go runtime. /cpu/classes/gc/… Monitor CPU for garbage collection. /cpu/classes/scavenge/… Monitor CPU for returning unused memory to the underlying platform ** IMPORTANT: These metrics are overestimate, and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics.
27. 27 Metrics: /gc Metric Use Case /gc/cycles/… Counts for GC cycles. /gc/heap/… Heap allocations /gc/scan/… The amount of space that GC is scannable. /gc/limiter/last-enabled:gc-cycle GC cycle the last time the GC CPU limiter was enabled. (OOM) /gc/gogc:percent Heap size target percentage. Set by: GOGC, runtime/debug.SetGCPercent /gc/gomemlimit:bytes Go runtime memory limit. Set by: GOMEMLIMIT, runtime/debug.SetMemoryLimit /gc/stack/starting-size:bytes The stack size of new goroutines. /gc/pauses:seconds Deprecated.
28. 28 Metrics: /memory Metric Use Case /memory/classes/total:bytes All memory mapped by the Go runtime into the current process. /memory/classes/heap/… Details of memory managements, including objects, free, released, unused, stack, etc. (example) /memory/classes/metadata/… Memory that is used for runtime mcache, mspan, etc. /memory/classes/os-stacks:bytes Stack memory allocated by OS. (unstable) /memory/classes/profiling/buckets:bytes Memory that is used by the stack trace for profiling. /memory/classes/other:bytes Memory used for trace, debugging, profiling, etc.
29. 29 Metrics: /sched Metric Use Case /sched/latencies:seconds Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running. (example) /sched/goroutines:goroutines Count of live goroutines. /sched/gomaxprocs:thread The current runtime.GOMAXPROCS setting, or the number of operating system threads. /sched/pauses/… GC-related or non-GC-related stop-the-world stopping latencies.
30. 30 • /sync/mutex/wait/total:seconds Approximate cumulative time goroutines have spent blocked on a sync.Mutex, sync.RWMutex, or runtime-internal lock. • /godebug/... count non-default behaviors due to GODEBUG setting. • /cgo/go-to-c-calls:calls Count of calls made from Go to C by the current process. Metrics: Others
31. 31 Q&A