How to instrument Go code in 2020 - Björn Rabenstein - Grafana Labs

If you are an astute follower of Peter Bourgon's [Go best practices](https://peter.bourgon.org/go-best-practices-2016/#logging-and-instrumentation), you have known it since 2016: “You should be instrumenting every significant component of your codebase.” But what does that even mean? Back then, Peter was mostly talking about metrics (and flatteringly recommended [Prometheus](https://prometheus.io) for it). However, logging can also be seen as instrumentation. (Peter considered it related but separate in his post from 2016.) And then there is instrumentation for distributed tracing. All three of them, metrics, logs, and traces, are nowadays often referred to as the [three pillars ob observability](https://www.oreilly.com/library/view/distributed-systems-observability/9781492033431/ch04.html). Wouldn't it be nice if you could do all three of them within the same framework? [OpenTelemetry](https://opentelemetry.io) is a project promising exactly that. There is a wide field of techniques between simple standard Go packages like `expvar` and `log` and the newest and most ambitious projects like OpenTelemetry. This talk doesn't intend to give you hands-on instructions for using any of them. Instead, it will provide a (necessarily incomplete) overview of established and new approaches and some guidelines how to navigate the many choices.

Transcript

1. Björn “Beorn” Rabenstein GoDays, Berlin – 2020-01-23 How to instrument Go code in 2020
2. | 2
3. https://peter.bourgon.org/go-best-practices-2016/ | 3
4. https://peter.bourgon.org/go-best-practices-2016/ | 4
5. https://peter.bourgon.org/go-best-practices-2016/ | 5
6. | 6
7. | 7
8. | 8
9. | 9
10. | 10
11. | 11 What Go “built in” “Modern” needs Example Debugging Profiling/Tracing runtime/pprof runtime/trace Logging log Metrics expvar
12. | 12 What Go “built in” “Modern” needs Example Debugging Almost useless… You’ll find one if you really need it. Profiling/Tracing runtime/pprof runtime/trace Distributed tracing opentracing/ opentracing-go Logging log Structured logging go-kit/log Metrics expvar Labeled metrics prometheus/ client_golang
13. | 13 What Go “built in” “Modern” needs Example Debugging Almost useless… You’ll find one if you really need it. Profiling/Tracing runtime/pprof runtime/trace Distributed tracing opentracing/ opentracing-go Logging log Structured logging go-kit/log Metrics expvar Labeled metrics prometheus/ client_golang
14. | 14 https://speakerdeck.com/chimeracoder/building-resilient-services-in-go
15. | 15
16. | 16 Unified instrumentation for metrics, logging, and distributed tracing
17. | 17 https://xkcd.com/927/ One way of doing it
18. | 18 Go!!! Python Java Javascript C# PHP C/C++ … For ALL the languages
19. A rich and universal data model | 19 Metrics are labeled. Logs are structured. OpenTracing.
20. For ALL data collection models | 20 Push vs pull. On prem vs. cloud. Distributed vs. replicated. Location of sampling, aggregation, buffering. Exposition formats. Transport protocols.
21. | 21 https://opentelemetry.io/
22. | 22 https://opentelemetry.io/about/
23. | 23 https://github.com/open-telemetry/opentelemetry-collector
24. | 24
25. | 25 How will I instrument my Go code in 2020? Now what?
26. | 26 Bet on OpenTelemetry as the winner for tracing. Step 1:
27. | 27 Relax! Logs and metrics are the easy part anyway. And not everything is a trace. Step 2:
28. | 28 As always: Take your context into account. (Service mesh? Microservice kit?) Step 3:
29. https://github.com/beorn7/talks beorn@grafana.com