Standards in Distributed Tracing

Slide 1

Slide 1 text

Thomas Watson March 2019 Standards in Distributed Tracing

Slide 2

Slide 2 text

@wa7son Agenda • Distributed Tracing overview • TraceContext • OpenTracing • OpenCensus

Slide 3

Slide 3 text

@wa7son React Tracing Microservices X

Slide 4

Slide 4 text

@wa7son React Tracing Microservices X

Slide 5

Slide 5 text

@wa7son React Tracing Microservices X

Slide 6

Slide 6 text

@wa7son React Tracing Microservices X

Slide 7

Slide 7 text

@wa7son Distributed Tracing Source: Giphy

Slide 8

Slide 8 text

@wa7son The APM library with many names Agents, Clients, and Tracers

Slide 9

Slide 9 text

@wa7son REST API Payment Service Order Service MySQL Payment Provider

Slide 10

Slide 10 text

@wa7son REST API Payment Service Order Service Payment Provider MySQL MySQL Spans Timeline

Slide 11

Slide 11 text

@wa7son React Tracing Microservices APM

Slide 12

Slide 12 text

@wa7son Tracing Microservices APM

Slide 13

Slide 13 text

@wa7son Tracing Microservices APM

Slide 14

Slide 14 text

@wa7son Challenges Source: Giphy

Slide 15

Slide 15 text

@wa7son Interservice Communication HTTP Binary Apache Kafka Message Queues …

Slide 16

Slide 16 text

@wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com

Slide 17

Slide 17 text

@wa7son Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 GET /products HTTP/1.1 Host: www.example.com

Slide 18

Slide 18 text

Slide 19

Slide 19 text

@wa7son GET /products HTTP/1.1 Host: www.example.com : - - - Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 traceparent GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 Trace Context Working Group DRAFT

Slide 20

Slide 20 text

Slide 21

Slide 21 text

@wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID 1 byte 8 bits 128 bit random number 64 bit random number Parent Span ID Trace Flags traceparent

Slide 22

Slide 22 text

@wa7son 00 82c5500f40667e5500e9ae8e9711553c 992631f881f78c3b 01 Version Trace ID Parent Span ID Trace Flags 1 byte 8 bits 128 bit random number 64 bit random number Trace Flags Recorded — 0b00000001 traceparent

Slide 23

Slide 23 text

@wa7son Congo Rojo Multi-Vendor Environments Congo Service B Service C Service A

Slide 24

Slide 24 text

@wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0

Slide 25

Slide 25 text

@wa7son GET /products HTTP/1.1 Host: www.example.com traceparent: 00-82c5500f40667e5500e9ae8e9711553c-992631f881f78c3b-01 Date: Mon, 29 Oct 2018 16:11:05 GMT Connection: keep-alive Content-Length: 0 : ,,… tracestate DRAFT

Slide 26

Slide 26 text

@wa7son tracestate ,,… Parent (potentially vendor specific) Up to 32 unique vendors

Slide 27

Slide 27 text

@wa7son Programmatic API Integrations (frameworks, db’s, etc) Context Propagation (in process) Context Propagation (cross service) Data format Backend transport protocol Backend / UI

Slide 28

Slide 28 text

@wa7son TraceContext Programmatic API Integrations (frameworks, db’s, etc) Context Propagation (in process) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI

Slide 29

Slide 29 text

@wa7son

Slide 30

Slide 30 text

@wa7son npm install opentracing

Slide 31

Slide 31 text

@wa7son

Slide 32

Slide 32 text

@wa7son

Slide 33

Slide 33 text

@wa7son const opentracing = require('opentracing') const tracer = new opentracing.Tracer() const span = tracer.startSpan('some work') // Do some work span.finish()

Slide 34

Slide 34 text

@wa7son span.setTag('user.id', user.getId()) span.log({ 'event': ‘error', 'error.object': err })

Slide 35

Slide 35 text

@wa7son // Start a new (parentless) root Span: const parent = Tracer.startSpan('DoWork') // Start a new (child) Span: const child = Tracer.startSpan('load-from-db', { childOf: parent.context() }) // Start a new async (FollowsFrom) Span: const child = Tracer.startSpan('async-cache-write', { references: [ opentracing.followsFrom(parent.context()) ] })

Slide 36

Slide 36 text

@wa7son const headersCarrier = {} // "Serialize" Span Context into headersCarrier tracer.inject( clientSpan.context(), Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) // Add tracing headers to headers of outgoing request Object.assign(outboundHTTPReq.headers, headersCarrier)

Slide 37

Slide 37 text

@wa7son // "De-serialize" Span Context from inbound HTTP request headers const headersCarrier = inboundHTTPReq.headers const wireCtx = tracer.extract( Tracer.FORMAT_HTTP_HEADERS, headersCarrier ) const serverSpan = tracer.startSpan('...', { childOf: wireCtx })

Slide 38

Slide 38 text

@wa7son Tracer.FORMAT_TEXT_MAP Tracer.FORMAT_HTTP_HEADERS Tracer.FORMAT_BINARY

Slide 39

Slide 39 text

@wa7son const { Tracer } = require('my-custom-tracer') const tracer = new Tracer() const opentracing = require('opentracing') opentracing.initGlobalTracer(tracer) // then later... const tracer = opentracing.globalTracer()

Slide 40

Slide 40 text

@wa7son const { Tracer } = require('my-custom-tracer') const tracer = new Tracer() const express = require('express') const expressOpenTracing = require('express-opentracing') const app = express() app.use(expressOpenTracing({ tracer })) // ...

Slide 41

Slide 41 text

@wa7son https://docs.google.com/document/d/ 1C_thV0lGRe0fVqUq7-qn7eZfbH2oyuJQqFgRnGRTSBY/edit WIP: ScopeManager

Slide 42

Slide 42 text

@wa7son ScopeManager: Possible Solutions • async-listener • async_hooks • async-hook-jl (AsyncWrap binding wrapper) • zone.js • domain

Slide 43

Slide 43 text

@wa7son

Slide 44

Slide 44 text

@wa7son Contributions per language at github.com/opentracing-contrib • Java: 50 • Python: 11 • Go: 9 • C#: 3 • Scala: 3 • JavaScript: 2 • Ruby: 2 • C++: 1 • Erlang: 1 • nginx: 1

Slide 45

Slide 45 text

Slide 46

Slide 46 text

@wa7son TraceContext OpenTracing Programmatic API ✅ Integrations (frameworks, db’s, etc) (✅) Context Propagation (in process) ((✅)) Context Propagation (cross service) ✅ Data format Backend transport protocol Backend / UI

Slide 47

Slide 47 text

@wa7son OpenCensus

Slide 48

Slide 48 text

@wa7son npm install @opencensus/nodejs

Slide 49

Slide 49 text

@wa7son

Slide 50

Slide 50 text

@wa7son App OpenCensus Library Monitoring Service Backend VM OpenCensus Agent OpenCensus Collector

Slide 51

Slide 51 text

@wa7son Implementation Details • continuation-local-storage + async_hooks • require-in-the-middle • shimmer • @opencensus/instrumentation-all • @opencensus/instrumentation-grpc • @opencensus/instrumentation-http • @opencensus/instrumentation-https • @opencensus/instrumentation-http2 • @opencensus/instrumentation-mongodb • (@opencensus/instrumentation-redis)

Slide 52

Slide 52 text

Slide 53

Slide 53 text

@wa7son TraceContext OpenTracing OpenCensus Programmatic API ✅ ✅ Integrations (frameworks, db’s, etc) (✅) (✅) Context Propagation (in process) ((✅)) ✅ Context Propagation (cross service) ✅ ✅ Data format (✅) Backend transport protocol (✅) Backend / UI

Slide 54

Slide 54 text

@wa7son But I heard a rumor that OpenTracing and OpenCensus are planning to merge?!

Slide 55

Slide 55 text

@wa7son github.com/watson Thank you