Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Propagating context and tracing across your distributed process boundaries using OpenTelemetry

Propagating context and tracing across your distributed process boundaries using OpenTelemetry

Presented at API Days Australia LIVE - 16 Sep 2021

Everyone is building distributed systems these days. Some better than others. One thing the teams building and running distributed systems well have in common is they have very good observability of the components and services. Conversely, the teams that don't have good observability struggle when things go wrong in a distributed system because it's often terribly time consuming to put the pieces together to analyse the crime scene. The logs might sit in disparate log aggregation systems and even when in one place, leave you with having to do the hard work to correlate and visualize the system workflows yourself.

OpenTelemetry is an observability framework for cloud-native software which aims to solve some of these issues by having a common set of definitions of concepts around observability and exposing them to the tool of your choice.

In this talk, I examine how to propagate your tracing context across process boundaries and visualize the flow of requests through your distributed services (Microservices/Serverless/Other) easily using tools like Zipkin and Jaeger. We will see how to use already instrumented libraries and also how to propagate the trace information yourself. At the end of this talk you will know how to easily trace and observe distributed components of the systems you build.

B4c7e56ec325c0c55d5e29c7ab42e49f?s=128

Dasith Wijesiriwardena

September 15, 2021
Tweet

Transcript

  1. Tracing across your distributed process boundaries using 15-16 Sep 2021

    Dasith Wijesiriwardena @dasiths OpenTelemetry
  2. I am here because I have a love hate relationship

    with distributed systems. HELLO! I AM Dasith @dasiths dasith.me
  3. Distributed tracing using OpenTelemetry Introduction to distributed tracing Agenda Demo

    using OpenTelemetry .NET SDK Introduction to OpenTelemetry @dasiths
  4. “ Peter Drucker @dasiths You can’t manage what you can’t

    measure..
  5. @dasiths

  6. Introduction to Distributed Tracing @dasiths

  7. Distributed Tracing @dasiths Distributed tracing, also called distributed request tracing,

    is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance. https://opentracing.io/docs/overview/what-is-tracing/
  8. @dasiths Example Distributed Tracing API Gateway Middleware Backend Database APP

  9. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP
  10. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP
  11. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP
  12. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP
  13. @dasiths Example Request Distributed Tracing Middleware Backend Database Correlation_ID =

    ABC123 API Gateway APP
  14. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP fields @timestamp, @message | filter @message like /some-unique-thing-for-the-request/ | sort @timestamp desc | limit 20
  15. @dasiths Example Distributed Tracing Middleware Backend Database Correlation_ID = ABC123

    Request API Gateway APP fields @timestamp, @message | filter @message like /some-unique-thing-for-the-request/ | sort @timestamp desc | limit 20
  16. @dasiths Example Request Distributed Tracing Middleware Backend Database https://link.springer.com/chapter/10.1007/978-3-030-61140-8_9 Correlation_ID

    = ABC123 API Gateway APP
  17. @dasiths Example Request Distributed Tracing Middleware Backend Database https://link.springer.com/chapter/10.1007/978-3-030-61140-8_9 Correlation_ID

    = ABC123 API Gateway APP
  18. @dasiths Example Request Distributed Tracing Backend Database Middleware API Gateway

    APP
  19. @dasiths Example Request Distributed Tracing Backend Database Middleware API Gateway

    APP
  20. @dasiths Example Request Distributed Tracing Backend Database Middleware API Gateway

    APP
  21. Looking at the Modern Landscape @dasiths

  22. • Runs natively on the cloud • Distributed (i.e. Microservices)

    • Managed by many teams •Broad set of languages and frameworks • Built on top of open source technologies Modern Software.. @dasiths
  23. Enter OpenTelemetry @dasiths

  24. @dasiths An observability framework for cloud-native software. OpenTelemetry is a

    collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior. https://opentelemetry.io/
  25. @dasiths An observability framework for cloud-native software. OpenTelemetry is a

    collection of tools, APIs, and SDKs. You can use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior. https://opentelemetry.io/
  26. All About The SPAN @dasiths

  27. OpenTelemetry Architecture @dasiths Signals Tracing Metrics Baggage Context Propagation

  28. Defined By All About The Span @dasiths Tracing Span https://medium.com/nikeengineering/hit-the-ground-running-with-distributed-tracing-core-concepts-ff5ad47c7058

  29. Defined By All About The Span @dasiths Tracing Span https://medium.com/nikeengineering/hit-the-ground-running-with-distributed-tracing-core-concepts-ff5ad47c7058

    Temporal Relationship Casual Relationship
  30. Context Propagation @dasiths Span (A) Time APP TraceId: 1 ParentSpan:

    Null
  31. @dasiths Context Propagation Span (A) Span (B) Time Context Propagation

    TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A APP
  32. @dasiths Context Propagation Context Propagation Span (C) Span (D) Context

    Propagation Span (A) Span (B) Time @ Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  33. @dasiths Context Propagation Context Propagation Span (C) Span (D) •

    TraceId • SpanId • TraceFlags • TraceState Context Propagation Span (A) Span (B) Span Context Time @ Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  34. @dasiths Context Propagation Context Propagation Span (C) Span (D) •

    TraceId • SpanId • TraceFlags • TraceState Context Propagation Span (A) Span (B) Span Context Time Process 2 Process 1 @ Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  35. Pr @dasiths Context Propagation Context Propagation Span (D) • TraceId

    • SpanId • TraceFlags • TraceState Context Propagation Span Context Span (C) Span (A) Span (B) Over-the-wire vs in-process Time Process 1 Process 2 @ Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  36. Pr @dasiths Context Propagation Context Propagation Span (D) • TraceId

    • SpanId • TraceFlags • TraceState Context Propagation Span Context Span (C) Span (A) Span (B) Inject Context To Outgoing Request Over-the-wire vs in-process Time Process 2 Process 1 @ Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B Extract Context From Incoming Request APP
  37. Pr @dasiths Context Propagation Context Propagation Span (D) Context Propagation

    Baggage Baggage Baggage Span (C) Span (A) Span (B) Baggage Time Process 1 Process 2 @ Context Propagation Inject Context To Outgoing Request Extract Context From Incoming Request TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  38. Pr @dasiths Context Propagation Context Propagation Span (D) Context Propagation

    Baggage Baggage Baggage Span (C) Span (A) Span (B) Time Process 1 Process 2 @ Inject Context To Outgoing Request Extract Context From Incoming Request KeyValue Pairs Baggage Data Baggage Context Propagation TraceId: 1 ParentSpan: Null TraceId: 1 ParentSpan: A TraceId: 1 ParentSpan: B TraceId: 1 ParentSpan: B APP
  39. DEMO @dasiths https://github.com/dasiths/OpenTelemetryDistributedTracingSample

  40. • Span (Activity) + Baggage ▪ Trace Context, Parent->Child, Tags,

    Events • Context Propagation ▪ Injecting & extracting, built-in vs custom instrumentation ▪ HTTP W3C trace context vs custom serialisation • Exporting ▪ Forwarding to & observing the traces in Zipkin Demo Highlights @dasiths
  41. Pr @dasiths Propagation

  42. Exporters (Traces + Metrics) @dasiths And lots more…

  43. Start using OpenTelemetry today! Distributed tracing is essential for performance

    and reliability of modern cloud native software. It’s no longer a “nice to have”. @dasiths
  44. Any questions? THANKS! @dasiths dasith.me https://www.nationalgeographic.com/travel/destinations/asia/sri-lanka/ Great content curated by

    our ANZ Dev team: https://aka.ms/DevNewsletterJoin 1. Sign up for our monthly developer newsletter We have a great Azure APIM Architecture Learning Path you can complete at: http://aka.ms/LearnAzureAPIM 3. Head to Microsoft Learn Email a screenshot of the newsletter subscription confirmation to ANZAzureDevs@microsoft.com. First 20 requests only – 1 Pass per person. 2. Request your $50 USD Azure Pass Get hands-on with Azure API Management for free!
  45. Presentation template designed by powerpointify.com Special thanks to all people

    who made and shared these awesome resources for free: CREDITS Photographs by unsplash.com Free Fonts used: https://www.fontsquirrel.com/fonts/oswald @dasiths