Observability in Go Application
Jun 23 2021
Sakti Dwi Cahyono
Built in tools
Logging, tracing, metrics
“Observability refers to understanding a system through
observation, and classiﬁes the tools that accomplish this.
These tools includes tracing tools, sampling tools, and tools
based on ﬁxed counters. It does not include benchmark tools,
which modify the state of the system by performing a
Excerpt From: Brendan Gregg. “BPF Performance Tools: Linux
System and Application Observability.”
● Microservices/services create complex interactions.
● Failures don't exactly repeat.
● Debugging multi-tenancy is painful.
● Monitoring no longer can help us.
What is observability?
● We need to answer questions about our systems.
What characteristics did the queries that timed out at 500ms
share in common? Service versions?
● Instrumentation produces data.
● Querying data answers our questions.
Telemetry aids observability
● Telemetry data isn't observability itself.
● Instrumentation code is how we get telemetry.
● Telemetry data can include traces, logs, and/or metrics.
All different views into the same underlying truth.
● Proﬁling, package runtime/pprof,
○ cpu,heap, threadcreate, goroutine,
● Tracing, package x/net/trace
● Debugging, GODEBUG environment variable
● Runtime statistics and events, package
Metrics, logs, and traces, oh my!
○ Aggregated summary statistics.
○ Detailed debugging information emitted by processes.
● Distributed Tracing
○ Provides insights into the full lifecycles, aka traces of
requests to a system, allowing you to pinpoint failures and
Structured data can be transmuted into any of these!
Metrics concepts in a nutshell
○ Instantaneous point-in-time value (e.g. CPU utilization)
● Cumulative counters
○ Cumulative sums of data since process start (e.g. request counts)
● Cumulative histogram
○ Grouped counters for a range of buckets (e.g. 0-10ms, 11-20ms)
○ The derivative of a counter, typically. (e.g. requests per second)
● Aggregation by tags
○ Data can be joined along shared tags (e.g. hostname, cluster
Tracing concepts in a nutshell
○ Represents a single unit of work in a system.
○ Typically encapsulates: operation name, a start and ﬁnish timestamp,
the parent span identiﬁer, the span identiﬁer, and context items.
○ Deﬁned implicitly by its spans. A trace can be thought of as a
directed acyclic graph of spans where the edges between spans are
deﬁned as parent/child relationships.
○ Contains the tracing identiﬁers, tags, and options that are propagated
from parent to child spans
Add more context to traces with Span Events
● Span Events are context-aware logging.
● An event contains timestamped information added to a span. You can
think of this as a structured log, or a way to annotate your spans with
speciﬁc details about what happened along the way.
■ the name of the event
■ one or more attributes
■ a timestamp
OpenCensus + OpenTracing = OpenTelemetry
○ Provides APIs and instrumentation for distributed tracing
○ Provides APIs and instrumentation that allow you to collect
application metrics and distributed tracing.
○ An effort to combine distributed tracing, metrics and logging into a
single set of system components and language-speciﬁc libraries.
● Speciﬁcation (API, SDK, Data)
● Instrumentation Libraries
We are hiring
Backend Engineer Wanted!
Sampingan is currently seeking a Backend Engineer
What you will do:
● Translating business requirements into scalable technical solutions;
● Producing high-quality maintainable code, testing and collaboratively review it to ensure eﬃciency;
● Pairing with team members on functional and nonfunctional requirements and spread design philosophy, goals and improve the
code quality across the team;
● Participating in preparing systems requirements, speciﬁcations and design;
● Ensuring maintainability of core app assets and artifacts;
● Researching new tools, learn and experiment with new languages and technologies and growing continuously with us;
● Continuously refactoring applications and architectures to maintain high quality levels and experience in troubleshooting server
performance - memory issues, GC tuning and resource leaks
If you are interested, send your CV to [email protected]. Don't forget to include the role name and your name on the subject line.
Sampingan @sampingan.business & @sampingan.id
Follow us on