Standards in Distributed Tracing

An introduction to the standards used in Distributed Tracing, specifically OpenTracing, OpenCensus, and W3C TraceContext.

1. Thomas Watson March 2019 Standards in Distributed Tracing
2. @wa7son Agenda • Distributed Tracing overview • TraceContext • OpenTracing • OpenCensus
3. @wa7son React Tracing Microservices X
4. @wa7son React Tracing Microservices X
5. @wa7son React Tracing Microservices X
6. @wa7son React Tracing Microservices X
7. @wa7son Distributed Tracing Source: Giphy
8. @wa7son The APM library with many names Agents, Clients, and Tracers
9. @wa7son REST API Payment Service Order Service MySQL Payment Provider
10. @wa7son REST API Payment Service Order Service Payment Provider MySQL MySQL Spans Timeline
11. @wa7son React Tracing Microservices APM
12. @wa7son Tracing Microservices APM
13. @wa7son Tracing Microservices APM
14. @wa7son Challenges Source: Giphy
15. @wa7son Interservice Communication HTTP Binary Apache Kafka Message Queues …
16. @wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com <magic header>
17. @wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com <magic header>
18. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent
19. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 Trace Context Working Group DRAFT
20. @wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent
21. @wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID 1 byte 8 bits 128 bit random number 64 bit random number Parent Span ID Trace Flags traceparent
22. @wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID Parent Span ID Trace Flags 1 byte 8 bits 128 bit random number 64 bit random number Trace Flags Recorded — 0b00000001 traceparent
23. @wa7son Congo Rojo Multi-Vendor Environments Congo Service B Service C Service A
24. @wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0
25. @wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 : <vendorname1=opaqueValue1>,<vendorname2=opaqueValue2>,… tracestate DRAFT
26. @wa7son tracestate <vendorname1=opaqueValue1>,<vendorname2=opaqueValue2>,… Parent (potentially vendor specific) Up to 32 unique vendors
27. @wa7son Programmatic API Integrations (frameworks, db’s, etc) Context Propagation (in process) Context Propagation (cross service) Data format Backend transport protocol Backend / UI
28. @wa7son TraceContext Programmatic API Integrations (frameworks, db’s, etc) Context Propagation (in process) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI
29. @wa7son
30. @wa7son npm install opentracing
31. @wa7son
32. @wa7son
33. @wa7son const opentracing = require('opentracing') const tracer = new opentracing.Tracer() const span = tracer.startSpan('some work') // Do some work span.finish()
34. @wa7son span.setTag('user.id', user.getId()) span.log({ 'event': ‘error', 'error.object': err })
35. @wa7son // Start a new (parentless) root Span: const parent = Tracer.startSpan('DoWork') // Start a new (child) Span: const child = Tracer.startSpan('load-from-db', { childOf: parent.context() }) // Start a new async (FollowsFrom) Span: const child = Tracer.startSpan('async-cache-write', { references: [ opentracing.followsFrom(parent.context()) ] })
36. @wa7son const headersCarrier = {} // "Serialize" Span Context into headersCarrier tracer.inject( clientSpan.context(), Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) // Add tracing headers to headers of outgoing request Object.assign(outboundHTTPReq.headers, headersCarrier)
37. @wa7son // "De-serialize" Span Context from inbound HTTP request headers const headersCarrier = inboundHTTPReq.headers const wireCtx = tracer.extract( Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) const serverSpan = tracer.startSpan('...', { childOf: wireCtx })
38. @wa7son Tracer.FORMAT_TEXT_MAP Tracer.FORMAT_HTTP_HEADERS Tracer.FORMAT_BINARY
39. @wa7son const { Tracer } = require('my-custom-tracer') const tracer = new Tracer() const opentracing = require('opentracing') opentracing.initGlobalTracer(tracer) // then later... const tracer = opentracing.globalTracer()
40. @wa7son const { Tracer } = require('my-custom-tracer') const tracer = new Tracer() const express = require('express') const expressOpenTracing = require('express-opentracing') const app = express() app.use(expressOpenTracing({ tracer })) // ...
41. @wa7son https://docs.google.com/document/d/ 1C_thV0lGRe0fVqUq7-qn7eZfbH2oyuJQqFgRnGRTSBY/edit WIP: ScopeManager
42. @wa7son ScopeManager: Possible Solutions • async-listener • async_hooks • async-hook-jl (AsyncWrap binding wrapper) • zone.js • domain
43. @wa7son
44. @wa7son Contributions per language at github.com/opentracing-contrib • Java: 50 • Python: 11 • Go: 9 • C#: 3 • Scala: 3 • JavaScript: 2 • Ruby: 2 • C++: 1 • Erlang: 1 • nginx: 1
45. @wa7son TraceContext Programmatic API Integrations (frameworks, db’s, etc) Context Propagation (in process) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI
46. @wa7son TraceContext OpenTracing Programmatic API ✅ Integrations (frameworks, db’s, etc) (✅) Context Propagation (in process) ((✅)) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI
47. @wa7son OpenCensus
48. @wa7son npm install @opencensus/nodejs
49. @wa7son
50. @wa7son App OpenCensus Library Monitoring Service Backend VM OpenCensus Agent OpenCensus Collector
51. @wa7son Implementation Details • continuation-local-storage + async_hooks • require-in-the-middle • shimmer • @opencensus/instrumentation-all • @opencensus/instrumentation-grpc • @opencensus/instrumentation-http • @opencensus/instrumentation-https • @opencensus/instrumentation-http2 • @opencensus/instrumentation-mongodb • (@opencensus/instrumentation-redis)
52. @wa7son TraceContext OpenTracing Programmatic API ✅ Integrations (frameworks, db’s, etc) (✅) Context Propagation (in process) ((✅)) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI
53. @wa7son TraceContext OpenTracing OpenCensus Programmatic API ✅ ✅ Integrations (frameworks, db’s, etc) (✅) (✅) Context Propagation (in process) ((✅)) ✅ Context Propagation (cross service) ✅ ✅ Data format (✅) Backend transport protocol (✅) Backend / UI
54. @wa7son But I heard a rumor that OpenTracing and OpenCensus are planning to merge?!
55. @wa7son github.com/watson Thank you