Observability in Go Application using OpenTelemetry

Transcript

1. Observability in Go Application
   using OpenTelemetry
   Jun 23 2021
   Sakti Dwi Cahyono
   

   View Slide

2. Intro 01
   Observability
   Built in tools
   Logging, tracing, metrics
   Tracing implementation
   Q&A
   02
   03
   04
   05
   06
   

   View Slide

3. Observability
   

   View Slide

4. “Observability refers to understanding a system through
   observation, and classifies the tools that accomplish this.
   These tools includes tracing tools, sampling tools, and tools
   based on fixed counters. It does not include benchmark tools,
   which modify the state of the system by performing a
   workload experiment”
   Excerpt From: Brendan Gregg. “BPF Performance Tools: Linux
   System and Application Observability.”
   Definition
   

   View Slide

5. Source: http://www.brendangregg.com/blog/2014-08-23/linux-perf-tools-linuxcon-na-2014.html
   

   View Slide

6. Why observability?
   ● Microservices/services create complex interactions.
   ● Failures don't exactly repeat.
   ● Debugging multi-tenancy is painful.
   ● Monitoring no longer can help us.
   

   View Slide

7. What is observability?
   ● We need to answer questions about our systems.
   What characteristics did the queries that timed out at 500ms
   share in common? Service versions?
   ● Instrumentation produces data.
   ● Querying data answers our questions.
   

   View Slide

8. Telemetry aids observability
   ● Telemetry data isn't observability itself.
   ● Instrumentation code is how we get telemetry.
   ● Telemetry data can include traces, logs, and/or metrics.
   All different views into the same underlying truth.
   

   View Slide

9. Built-in tools
   

   View Slide

10. Go diagnostics
    Source: https://golang.org/doc/diagnostics
    ● Profiling, package runtime/pprof,
    net/http/pprof
    ○ cpu,heap, threadcreate, goroutine,
    block, mutex
    ● Tracing, package x/net/trace
    ● Debugging, GODEBUG environment variable
    ● Runtime statistics and events, package
    runtime, runtime/debug
    

    View Slide

11. Logging,
    tracing, metrics
    

    View Slide

12. Metrics, logs, and traces, oh my!
    ● Metrics
    ○ Aggregated summary statistics.
    ● Logs
    ○ Detailed debugging information emitted by processes.
    ● Distributed Tracing
    ○ Provides insights into the full lifecycles, aka traces of
    requests to a system, allowing you to pinpoint failures and
    performance issues.
    Structured data can be transmuted into any of these!
    

    View Slide

13. Metrics concepts in a nutshell
    ● Gauges
    ○ Instantaneous point-in-time value (e.g. CPU utilization)
    ● Cumulative counters
    ○ Cumulative sums of data since process start (e.g. request counts)
    ● Cumulative histogram
    ○ Grouped counters for a range of buckets (e.g. 0-10ms, 11-20ms)
    ● Rates
    ○ The derivative of a counter, typically. (e.g. requests per second)
    ● Aggregation by tags
    ○ Data can be joined along shared tags (e.g. hostname, cluster
    name).
    

    View Slide

14. Tracing concepts in a nutshell
    ● Span
    ○ Represents a single unit of work in a system.
    ○ Typically encapsulates: operation name, a start and finish timestamp,
    the parent span identifier, the span identifier, and context items.
    ● Trace
    ○ Defined implicitly by its spans. A trace can be thought of as a
    directed acyclic graph of spans where the edges between spans are
    defined as parent/child relationships.
    ● DistributedContext
    ○ Contains the tracing identifiers, tags, and options that are propagated
    from parent to child spans
    

    View Slide

15. Source:
    https://sgryphon.wordpress.com/2020/11/16/a-guide-to-w3c-trace-context/
    

    View Slide

16. Add more context to traces with Span Events
    ● Span Events are context-aware logging.
    ● An event contains timestamped information added to a span. You can
    think of this as a structured log, or a way to annotate your spans with
    specific details about what happened along the way.
    ○ Contains:
    ■ the name of the event
    ■ one or more attributes
    ■ a timestamp
    

    View Slide

17. Implementation
    

    View Slide

18. OpenCensus + OpenTracing = OpenTelemetry
    ● OpenTracing:
    ○ Provides APIs and instrumentation for distributed tracing
    ● OpenCensus:
    ○ Provides APIs and instrumentation that allow you to collect
    application metrics and distributed tracing.
    ● OpenTelemetry:
    ○ An effort to combine distributed tracing, metrics and logging into a
    single set of system components and language-specific libraries.
    

    View Slide

19. Source: https://opentelemetry.io/docs/
    

    View Slide

20. OpenTelemetry Components
    ● Proto
    ● Specification (API, SDK, Data)
    ● Collector
    ● Instrumentation Libraries
    

    View Slide

21. Demo Repository
    https://github.com/sakti/o11ygo
    

    View Slide

22. We are hiring
    https://sampingan.company/career
    

    View Slide

23. Backend Engineer Wanted!
    Sampingan is currently seeking a Backend Engineer
    What you will do:
    ● Translating business requirements into scalable technical solutions;
    ● Producing high-quality maintainable code, testing and collaboratively review it to ensure efficiency;
    ● Pairing with team members on functional and nonfunctional requirements and spread design philosophy, goals and improve the
    code quality across the team;
    ● Participating in preparing systems requirements, specifications and design;
    ● Ensuring maintainability of core app assets and artifacts;
    ● Researching new tools, learn and experiment with new languages and technologies and growing continuously with us;
    ● Continuously refactoring applications and architectures to maintain high quality levels and experience in troubleshooting server
    performance - memory issues, GC tuning and resource leaks
    If you are interested, send your CV to careers@sampingan.co.id. Don't forget to include the role name and your name on the subject line.
    Sampingan @sampingan.business & @sampingan.id
    Follow us on
    www.sampingan.co.id
    

    View Slide

24. Q&A
    

    View Slide