Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Exploring Go Runtime Metrics

Exploring Go Runtime Metrics

Mohit Pokharna

March 26, 2024

More Decks by Mohit Pokharna

Other Decks in Technology


  1. 2 Software engineer at Mercari since 2020, primarily focusing on

    ML-assisted Listing and Search backend. Chin-Ming Huang Software engineer at Mercari since 2019, primarily focusing on ML price and Search backend.
 Mohit Pokharna
  2. 5 Runtime • Go Runtime is the library to execute

    Go programs. • Runtime contains goroutine scheduler, memory allocator, garbage collector, etc.
  3. 12 Proposal: API for unstable runtime metrics (#37112, design doc)

    Before runtime/metrics, runtime metrics are exposed in two ways: 1. by struct-based sampling APIs, e.g. runtime.ReadMemStats and runtime/debug.GCStats. 2. via GODEBUG flags which emit strings containing metrics to standard error (e.g. gctrace, gcpacertrace, scavtrace).
  4. 13 MemStats is hard to evolve because it must obey

    the Go 1 compatibility rules. The existing metrics are confusing, but we can‘t change them. Some of the metrics are now meaningless (like EnableGC and DebugGC), and several have aged poorly (like hard-coding the number of size classes at 61, or only having a single pause duration per GC cycle). Hence, we tend to shy away from adding anything to this because we’ll have to maintain it for the rest of time. by struct-based sampling APIs
  5. 14 The gctrace format is unspecified, which means we can

    evolve it (and have completely changed it several times). But it‘s a pain to collect programmatically because it only comes out on stderr and, even if you can capture that, you have to parse a text format that changes. Hence, automated metric collection systems ignore gctrace. via GODEBUG flags
  6. 15 • Sampling-based API is taken in opposition to a

    stream-based or event-based API. • Consider “sets of metrics” as the unit of API instead of individual metrics. • Metric name be built from two components: a forward-slash-separated path to a metric where each component is lowercase words separated by hyphens, and its unit. E.g. /memory/classes/heap/free:bytes New Design
  7. 17

  8. 24 • runtime/metrics provides an unified interface to access different

    metrics, including heap, sys, cpu and gc stats. • Underneath, each metric is corrected from different part of runtime, so 2 steps are performed for metrics reading: ensure & compute. Remarks
  9. 25 Supported Metrics Metric Prefix Count Metrics /cgo 1 go-to-c-calls

    /cpu 11 gc, idle, scavenge, total, user /gc 22 cycles, scan, heap, gomemlimit, stack /godebug 23 GODEBUG /memory 14 heap, mcache, mspan, os-stack /sched 7 gomaxprocs, goroutines, latencies, pauses, mutex /sync 1 mutex wait.
  10. 26 Metrics: /cpu Metric Use Case /cpu/classes/total:cpu-seconds /cpu/classes/user:cpu-seconds /cpu/classes/idle:cpu-seconds Measure

    CPU utilization for Go code and/or Go runtime. /cpu/classes/gc/… Monitor CPU for garbage collection. /cpu/classes/scavenge/… Monitor CPU for returning unused memory to the underlying platform ** IMPORTANT: These metrics are overestimate, and not directly comparable to system CPU time measurements. Compare only with other /cpu/classes metrics.
  11. 27 Metrics: /gc Metric Use Case /gc/cycles/… Counts for GC

    cycles. /gc/heap/… Heap allocations /gc/scan/… The amount of space that GC is scannable. /gc/limiter/last-enabled:gc-cycle GC cycle the last time the GC CPU limiter was enabled. (OOM) /gc/gogc:percent Heap size target percentage. Set by: GOGC, runtime/debug.SetGCPercent /gc/gomemlimit:bytes Go runtime memory limit. Set by: GOMEMLIMIT, runtime/debug.SetMemoryLimit /gc/stack/starting-size:bytes The stack size of new goroutines. /gc/pauses:seconds Deprecated.
  12. 28 Metrics: /memory Metric Use Case /memory/classes/total:bytes All memory mapped

    by the Go runtime into the current process. /memory/classes/heap/… Details of memory managements, including objects, free, released, unused, stack, etc. (example) /memory/classes/metadata/… Memory that is used for runtime mcache, mspan, etc. /memory/classes/os-stacks:bytes Stack memory allocated by OS. (unstable) /memory/classes/profiling/buckets:bytes Memory that is used by the stack trace for profiling. /memory/classes/other:bytes Memory used for trace, debugging, profiling, etc.
  13. 29 Metrics: /sched Metric Use Case /sched/latencies:seconds Distribution of the

    time goroutines have spent in the scheduler in a runnable state before actually running. (example) /sched/goroutines:goroutines Count of live goroutines. /sched/gomaxprocs:thread The current runtime.GOMAXPROCS setting, or the number of operating system threads. /sched/pauses/… GC-related or non-GC-related stop-the-world stopping latencies.
  14. 30 • /sync/mutex/wait/total:seconds Approximate cumulative time goroutines have spent blocked

    on a sync.Mutex, sync.RWMutex, or runtime-internal lock. • /godebug/... count non-default behaviors due to GODEBUG setting. • /cgo/go-to-c-calls:calls Count of calls made from Go to C by the current process. Metrics: Others