Distributed Tracing: From Chaos to Clarity

Distributed Tracing From Chaos to Clarity

@PurpleBooth About me ▪ Quite tall ▪ Help people use
best practices ▪ Capybaras are my favourite animal

@PurpleBooth Tracing

@PurpleBooth https:/ /www.flickr.com/photos/bnilsen/5206109741

@PurpleBooth Tracing

@PurpleBooth

@PurpleBooth Quick Easy Wins

@PurpleBooth Story

@PurpleBooth Half True

@PurpleBooth This is me Billie

@PurpleBooth This is Rob (we’re pairing) Rob Billie

@PurpleBooth Tim (one of our bosses), introduces us to Emma
Tim Emma

@PurpleBooth Photo by NeONBRAND on Unsplash

@PurpleBooth Photo by Gabriel Alenius on Unsplash

@PurpleBooth Emma has a promotion coming up

@PurpleBooth The site feels slow, and she’s worried it won’t
keep up with demand

@PurpleBooth She wants us to load test and fix the
slowness

@PurpleBooth (Spoiler: It can’t)

@PurpleBooth And we have no idea why

@PurpleBooth If you remember back to the start

@PurpleBooth This is the exact same problem Emma had

@PurpleBooth Emma’s team is smart

@PurpleBooth “Stop this from being slow now, but also in
the future.”

@PurpleBooth We need to increase the observability of the application

@PurpleBooth Personal Favourite

@PurpleBooth Contents ▪ Current & Future ▪ Primitives of OpenTracing
▪ Implementation

@PurpleBooth

@PurpleBooth Simon Wardley - Evolution Curve

@PurpleBooth • Rare • Poorly Understood • Talked about in
terms of wonder

@PurpleBooth

@PurpleBooth • Slightly more common • Rapid creating new functionality
• Publications talk about how to build it yourself

@PurpleBooth

@PurpleBooth • Common • Actually works! • Publications talk about
how to install it, not build it

@PurpleBooth

@PurpleBooth • It’s unusual if you don’t do it •
You know it, and you know how to use it • Publications talk about how to use it

@PurpleBooth

@PurpleBooth Open Tracing ▪ Istio, Traefik, Linkerd ▪ Open and
Paid ▪ Coming merge with OpenCensus

@PurpleBooth How does it work?

@PurpleBooth

@PurpleBooth Not Servers

@PurpleBooth Microservices

@PurpleBooth Nanoservices

@PurpleBooth Functions

@PurpleBooth Also Servers

@PurpleBooth traefik Balance Customer Cashier Account Transaction

@PurpleBooth traefik Balance Customer Cashier Account Transaction Customer Details, Name
DOB

@PurpleBooth traefik Balance Customer Cashier Account Transaction Span Span

@PurpleBooth traefik Balance Customer Cashier Account Transaction Trace

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Special Headers

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Probability

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Latency

@PurpleBooth traefik Balance Customer Cashier Account Transaction Accounts to Customer
Mapping

@PurpleBooth traefik Balance Customer Cashier Account Transaction Deposit or Withdraw
Cash

@PurpleBooth traefik Balance Customer Cashier Account Transaction Log all Transactions

@PurpleBooth traefik Balance Customer Cashier Account Transaction Update the current
balance

@PurpleBooth

@PurpleBooth Bits ▪ Client Interface ▪ Semantic Specification ▪ Semantic
Conventions

@PurpleBooth Semantic Specification ▪ Spans ▪ References ▪ SpanContexts ▪
Tracer

@PurpleBooth traefik Balance Customer Cashier Account Transaction Span Span

Tracer

@PurpleBooth traefik Balance Customer Cashier Account Transaction Hi Mom!

Tracer

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar

Tracer

@PurpleBooth Semantic Conventions ▪ Tags ▪ Logs ▪ Baggage

@PurpleBooth Tags ▪ http.method ▪ db.instance

@PurpleBooth Logs ▪ stack

@PurpleBooth Baggage ▪ SpanID ▪ TraceID

@PurpleBooth Where to start?

@PurpleBooth Where to start Edge of the network Simplest Tracing
Custom Tracing

@PurpleBooth Registries And Meshes! ▪ Traefik ▪ Amazon API Gateway
▪ Google Cloud Endpoints

@PurpleBooth

# traefik.toml [tracing] backend = "jaeger" [tracing.jaeger] samplingServerURL = "http:/
/jaeger-agent:5778/sampling" localAgentHostPort = "jaeger-agent:6831" @PurpleBooth

helm install stable/traefik --name my-release \ --set tracing.enabled=true \ --set
tracing.backend=jaeger \ --set tracing.jaeger.localAgentHostPort=jaeger-agent:6831 \ --set tracing.jaeger.samplingServerURL=http:/ /jaeger-agent:5778/sampling \ --namespace kube-system @PurpleBooth

@PurpleBooth

@PurpleBooth Simple Tracing

@PurpleBooth

import ( // TRIMMED "github.com/opentracing/opentracing-go" "github.com/uber/jaeger-client-go/config" // TRIMMED ) @PurpleBooth

func getTracer() (opentracing.Tracer, io.Closer, error) { jaegerHostPort := ":6831" cfg
:= config.Configuration{ ServiceName: "account-service", Sampler: &config.SamplerConfig{ Type: "const", Param: 1, }, Reporter: &config.ReporterConfig{ LogSpans: false, BufferFlushInterval: 1 * time.Second, LocalAgentHostPort: jaegerHostPort, }, } return cfg.NewTracer() } @PurpleBooth

func getTracer() (opentracing.Tracer, io.Closer, error) { jaegerHostPort := ":6831" cfg
:= config.Configuration{ // Snipped } return cfg.NewTracer() } @PurpleBooth

cfg := config.Configuration{ // Snipped ServiceName: "account-service", Sampler: &config.SamplerConfig{ Type:
"const", Param: 1, }, } @PurpleBooth

import ( // TRIMMED "github.com/gorilla/mux" "github.com/opentracing-contrib/go-gorilla/gorilla" "github.com/opentracing/opentracing-go" // TRIMMED )
@PurpleBooth

@PurpleBooth tracer, _, err := getTracer() opentracing.InitGlobalTracer(tracer) if err !=
nil { log.Fatal("cannot initialize Tracer", err) } _ = r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error { route.Handler( gorilla.Middleware(tracer, route.GetHandler())) return nil })

import ( // TRIMMED otredis "github.com/smacker/opentracing-go-redis" // TRIMMED ) @PurpleBooth

@PurpleBooth func HomeHandler(writer http.ResponseWriter, request *http.Request) { client := redis.NewClient(&redis.Options{
Addr: "localhost:6379", Password: "", DB: 0, }) ctx := request.Context() // Pass me to other opentracing libraries client = otredis.WrapRedisClient(ctx, client) // Do other stuff

@PurpleBooth

@PurpleBooth Use the AMQP Lib

@PurpleBooth Custom tracing

@PurpleBooth type RabbitBroker interface { Produce(*Event, map[string]string) error } type
TracedRabbitBroker struct { RabbitBroker RabbitBroker context context.Context hostname string exchange string } func (t TracedRabbitBroker) Produce(event *Event, headers map[string]string) error { span, _ := opentracing.StartSpanFromContext(t.context, "emit") defer span.Finish() ext.MessageBusDestination.Set(span, t.exchange) ext.PeerHostname.Set(span, t.hostname) ext.SpanKindConsumer.Set(span) tracer := opentracing.GlobalTracer() if err := tracer.Inject(span.Context(), opentracing.TextMap, opentracing.TextMapCarrier(headers)); err != nil { ext.Error.Set(span, true) return err } err := t.RabbitBroker.Produce(event, headers) ext.Error.Set(span, err != nil) return err }

@PurpleBooth type RabbitBroker interface { Produce(*Event, map[string]string) error } func
(t TracedRabbitBroker) Produce(event *Event, headers map[string]string) error { }

@PurpleBooth span, _ := opentracing.StartSpanFromContext(t.context, "emit") defer span.Finish() ext.MessageBusDestination.Set(span, t.exchange)
ext.PeerHostname.Set(span, t.hostname) ext.SpanKindProducer.Set(span)

@PurpleBooth tracer := opentracing.GlobalTracer() if err := tracer.Inject(span.Context(), opentracing.TextMap, opentracing.TextMapCarrier(headers));
err != nil { ext.Error.Set(span, true) return err } err := t.RabbitBroker.Produce(event, headers)

@PurpleBooth func TraceConsume(hostname string, exchange string, callback func(msg string, headers
map[string]string) error) func(msg string, headers map[string]string) error { return func(msg string, headers map[string]string) error { tracer := opentracing.GlobalTracer() spanCtx, err := tracer.Extract(opentracing.TextMap, opentracing.TextMapCarrier(headers)) if err == nil { return err } span := opentracing.StartSpan("consume", opentracing.ChildOf(spanCtx)) defer span.Finish() ext.MessageBusDestination.Set(span, exchange) ext.PeerHostname.Set(span, hostname) ext.SpanKindConsumer.Set(span) return callback(msg, headers) } }

@PurpleBooth tracer := opentracing.GlobalTracer() spanCtx, err := tracer.Extract(opentracing.TextMap, opentracing.TextMapCarrier(headers)) if
err == nil { return err }

@PurpleBooth span := opentracing.StartSpan("consume", opentracing.ChildOf(spanCtx)) defer span.Finish() ext.MessageBusDestination.Set(span, exchange) ext.PeerHostname.Set(span,
hostname) ext.SpanKindConsumer.Set(span)

@PurpleBooth return callback(msg, headers)

@PurpleBooth

@PurpleBooth Wrapping up

“1. Know your options

“1. Know your options 2. Know how it works

“1. Know your options 2. Know how it works 3.
Implementation strategy

@PurpleBooth Thank you!

Distributed Tracing: From Chaos to Clarity

Distributed Tracing: From Chaos to Clarity

More Decks by Armakuni

Other Decks in Programming

Featured

Transcript