Distributed Tracing: From Chaos to Clarity

Slide 1

Slide 1 text

Distributed Tracing From Chaos to Clarity

Slide 2

Slide 2 text

@PurpleBooth About me ▪ Quite tall ▪ Help people use best practices ▪ Capybaras are my favourite animal

Slide 3

Slide 3 text

@PurpleBooth Tracing

Slide 4

Slide 4 text

@PurpleBooth https:/ /www.flickr.com/photos/bnilsen/5206109741

Slide 5

Slide 5 text

@PurpleBooth Tracing

Slide 6

Slide 6 text

@PurpleBooth

Slide 7

Slide 7 text

@PurpleBooth Quick Easy Wins

Slide 8

Slide 8 text

@PurpleBooth Story

Slide 9

Slide 9 text

@PurpleBooth Half True

Slide 10

Slide 10 text

@PurpleBooth This is me Billie

Slide 11

Slide 11 text

@PurpleBooth This is Rob (we’re pairing) Rob Billie

Slide 12

Slide 12 text

@PurpleBooth Tim (one of our bosses), introduces us to Emma Tim Emma

Slide 13

Slide 13 text

@PurpleBooth Photo by NeONBRAND on Unsplash

Slide 14

Slide 14 text

@PurpleBooth Photo by Gabriel Alenius on Unsplash

Slide 15

Slide 15 text

@PurpleBooth Emma has a promotion coming up

Slide 16

Slide 16 text

@PurpleBooth The site feels slow, and she’s worried it won’t keep up with demand

Slide 17

Slide 17 text

@PurpleBooth She wants us to load test and fix the slowness

Slide 18

Slide 18 text

@PurpleBooth (Spoiler: It can’t)

Slide 19

Slide 19 text

@PurpleBooth And we have no idea why

Slide 20

Slide 20 text

@PurpleBooth If you remember back to the start

Slide 21

Slide 21 text

@PurpleBooth This is the exact same problem Emma had

Slide 22

Slide 22 text

@PurpleBooth Emma’s team is smart

Slide 23

Slide 23 text

@PurpleBooth “Stop this from being slow now, but also in the future.”

Slide 24

Slide 24 text

@PurpleBooth We need to increase the observability of the application

Slide 25

Slide 25 text

@PurpleBooth Personal Favourite

Slide 26

Slide 26 text

@PurpleBooth Contents ▪ Current & Future ▪ Primitives of OpenTracing ▪ Implementation

Slide 27

Slide 27 text

@PurpleBooth

Slide 28

Slide 28 text

@PurpleBooth

Slide 29

Slide 29 text

@PurpleBooth

Slide 30

Slide 30 text

@PurpleBooth

Slide 31

Slide 31 text

@PurpleBooth

Slide 32

Slide 32 text

@PurpleBooth Simon Wardley - Evolution Curve

Slide 33

Slide 33 text

@PurpleBooth ● Rare ● Poorly Understood ● Talked about in terms of wonder

Slide 34

Slide 34 text

@PurpleBooth

Slide 35

Slide 35 text

@PurpleBooth ● Slightly more common ● Rapid creating new functionality ● Publications talk about how to build it yourself

Slide 36

Slide 36 text

@PurpleBooth

Slide 37

Slide 37 text

@PurpleBooth ● Common ● Actually works! ● Publications talk about how to install it, not build it

Slide 38

Slide 38 text

@PurpleBooth

Slide 39

Slide 39 text

@PurpleBooth ● It’s unusual if you don’t do it ● You know it, and you know how to use it ● Publications talk about how to use it

Slide 40

Slide 40 text

@PurpleBooth

Slide 41

Slide 41 text

@PurpleBooth

Slide 42

Slide 42 text

@PurpleBooth Open Tracing ▪ Istio, Traefik, Linkerd ▪ Open and Paid ▪ Coming merge with OpenCensus

Slide 43

Slide 43 text

@PurpleBooth How does it work?

Slide 44

Slide 44 text

@PurpleBooth

Slide 45

Slide 45 text

@PurpleBooth

Slide 46

Slide 46 text

@PurpleBooth

Slide 47

Slide 47 text

@PurpleBooth Not Servers

Slide 48

Slide 48 text

@PurpleBooth Microservices

Slide 49

Slide 49 text

@PurpleBooth Nanoservices

Slide 50

Slide 50 text

@PurpleBooth Functions

Slide 51

Slide 51 text

@PurpleBooth Also Servers

Slide 52

Slide 52 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 53

Slide 53 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 54

Slide 54 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 55

Slide 55 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 56

Slide 56 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Customer Details, Name DOB

Slide 57

Slide 57 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Span Span

Slide 58

Slide 58 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Trace

Slide 59

Slide 59 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar

Slide 60

Slide 60 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Special Headers

Slide 61

Slide 61 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Probability

Slide 62

Slide 62 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar Latency

Slide 63

Slide 63 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 64

Slide 64 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 65

Slide 65 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 66

Slide 66 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Accounts to Customer Mapping

Slide 67

Slide 67 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 68

Slide 68 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 69

Slide 69 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 70

Slide 70 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Deposit or Withdraw Cash

Slide 71

Slide 71 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 72

Slide 72 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Log all Transactions

Slide 73

Slide 73 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 74

Slide 74 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Update the current balance

Slide 75

Slide 75 text

@PurpleBooth

Slide 76

Slide 76 text

@PurpleBooth

Slide 77

Slide 77 text

@PurpleBooth

Slide 78

Slide 78 text

@PurpleBooth Bits ▪ Client Interface ▪ Semantic Specification ▪ Semantic Conventions

Slide 79

Slide 79 text

@PurpleBooth Semantic Specification ▪ Spans ▪ References ▪ SpanContexts ▪ Tracer

Slide 80

Slide 80 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Span Span

Slide 81

Slide 81 text

@PurpleBooth Semantic Specification ▪ Spans ▪ References ▪ SpanContexts ▪ Tracer

Slide 82

Slide 82 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction Hi Mom!

Slide 83

Slide 83 text

@PurpleBooth Semantic Specification ▪ Spans ▪ References ▪ SpanContexts ▪ Tracer

Slide 84

Slide 84 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction jaegar

Slide 85

Slide 85 text

@PurpleBooth Semantic Specification ▪ Spans ▪ References ▪ SpanContexts ▪ Tracer

Slide 86

Slide 86 text

@PurpleBooth Semantic Conventions ▪ Tags ▪ Logs ▪ Baggage

Slide 87

Slide 87 text

@PurpleBooth Tags ▪ http.method ▪ db.instance

Slide 88

Slide 88 text

@PurpleBooth Logs ▪ stack

Slide 89

Slide 89 text

@PurpleBooth Baggage ▪ SpanID ▪ TraceID

Slide 90

Slide 90 text

@PurpleBooth Where to start?

Slide 91

Slide 91 text

@PurpleBooth Where to start Edge of the network Simplest Tracing Custom Tracing

Slide 92

Slide 92 text

@PurpleBooth Registries And Meshes! ▪ Traefik ▪ Amazon API Gateway ▪ Google Cloud Endpoints

Slide 93

Slide 93 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 94

Slide 94 text

@PurpleBooth

Slide 95

Slide 95 text

# traefik.toml [tracing] backend = "jaeger" [tracing.jaeger] samplingServerURL = "http:/ /jaeger-agent:5778/sampling" localAgentHostPort = "jaeger-agent:6831" @PurpleBooth

Slide 96

Slide 96 text

helm install stable/traefik --name my-release \ --set tracing.enabled=true \ --set tracing.backend=jaeger \ --set tracing.jaeger.localAgentHostPort=jaeger-agent:6831 \ --set tracing.jaeger.samplingServerURL=http:/ /jaeger-agent:5778/sampling \ --namespace kube-system @PurpleBooth

Slide 97

Slide 97 text

@PurpleBooth

Slide 98

Slide 98 text

@PurpleBooth Simple Tracing

Slide 99

Slide 99 text

@PurpleBooth

Slide 100

Slide 100 text

@PurpleBooth

Slide 101

Slide 101 text

import ( // TRIMMED "github.com/opentracing/opentracing-go" "github.com/uber/jaeger-client-go/config" // TRIMMED ) @PurpleBooth

Slide 102

Slide 102 text

func getTracer() (opentracing.Tracer, io.Closer, error) { jaegerHostPort := ":6831" cfg := config.Configuration{ ServiceName: "account-service", Sampler: &config.SamplerConfig{ Type: "const", Param: 1, }, Reporter: &config.ReporterConfig{ LogSpans: false, BufferFlushInterval: 1 * time.Second, LocalAgentHostPort: jaegerHostPort, }, } return cfg.NewTracer() } @PurpleBooth

Slide 103

Slide 103 text

func getTracer() (opentracing.Tracer, io.Closer, error) { jaegerHostPort := ":6831" cfg := config.Configuration{ // Snipped } return cfg.NewTracer() } @PurpleBooth

Slide 104

Slide 104 text

cfg := config.Configuration{ // Snipped ServiceName: "account-service", Sampler: &config.SamplerConfig{ Type: "const", Param: 1, }, } @PurpleBooth

Slide 105

Slide 105 text

import ( // TRIMMED "github.com/gorilla/mux" "github.com/opentracing-contrib/go-gorilla/gorilla" "github.com/opentracing/opentracing-go" // TRIMMED ) @PurpleBooth

Slide 106

Slide 106 text

@PurpleBooth tracer, _, err := getTracer() opentracing.InitGlobalTracer(tracer) if err != nil { log.Fatal("cannot initialize Tracer", err) } _ = r.Walk(func(route *mux.Route, router *mux.Router, ancestors []*mux.Route) error { route.Handler( gorilla.Middleware(tracer, route.GetHandler())) return nil })

Slide 107

Slide 107 text

import ( // TRIMMED otredis "github.com/smacker/opentracing-go-redis" // TRIMMED ) @PurpleBooth

Slide 108

Slide 108 text

@PurpleBooth func HomeHandler(writer http.ResponseWriter, request *http.Request) { client := redis.NewClient(&redis.Options{ Addr: "localhost:6379", Password: "", DB: 0, }) ctx := request.Context() // Pass me to other opentracing libraries client = otredis.WrapRedisClient(ctx, client) // Do other stuff

Slide 109

Slide 109 text

@PurpleBooth

Slide 110

Slide 110 text

@PurpleBooth

Slide 111

Slide 111 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 112

Slide 112 text

@PurpleBooth Use the AMQP Lib

Slide 113

Slide 113 text

No content

Slide 114

Slide 114 text

@PurpleBooth Custom tracing

Slide 115

Slide 115 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 116

Slide 116 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 117

Slide 117 text

@PurpleBooth type RabbitBroker interface { Produce(*Event, map[string]string) error } type TracedRabbitBroker struct { RabbitBroker RabbitBroker context context.Context hostname string exchange string } func (t TracedRabbitBroker) Produce(event *Event, headers map[string]string) error { span, _ := opentracing.StartSpanFromContext(t.context, "emit") defer span.Finish() ext.MessageBusDestination.Set(span, t.exchange) ext.PeerHostname.Set(span, t.hostname) ext.SpanKindConsumer.Set(span) tracer := opentracing.GlobalTracer() if err := tracer.Inject(span.Context(), opentracing.TextMap, opentracing.TextMapCarrier(headers)); err != nil { ext.Error.Set(span, true) return err } err := t.RabbitBroker.Produce(event, headers) ext.Error.Set(span, err != nil) return err }

Slide 118

Slide 118 text

@PurpleBooth type RabbitBroker interface { Produce(*Event, map[string]string) error } func (t TracedRabbitBroker) Produce(event *Event, headers map[string]string) error { }

Slide 119

Slide 119 text

@PurpleBooth span, _ := opentracing.StartSpanFromContext(t.context, "emit") defer span.Finish() ext.MessageBusDestination.Set(span, t.exchange) ext.PeerHostname.Set(span, t.hostname) ext.SpanKindProducer.Set(span)

Slide 120

Slide 120 text

@PurpleBooth tracer := opentracing.GlobalTracer() if err := tracer.Inject(span.Context(), opentracing.TextMap, opentracing.TextMapCarrier(headers)); err != nil { ext.Error.Set(span, true) return err } err := t.RabbitBroker.Produce(event, headers)

Slide 121

Slide 121 text

@PurpleBooth traefik Balance Customer Cashier Account Transaction

Slide 122

Slide 122 text

@PurpleBooth func TraceConsume(hostname string, exchange string, callback func(msg string, headers map[string]string) error) func(msg string, headers map[string]string) error { return func(msg string, headers map[string]string) error { tracer := opentracing.GlobalTracer() spanCtx, err := tracer.Extract(opentracing.TextMap, opentracing.TextMapCarrier(headers)) if err == nil { return err } span := opentracing.StartSpan("consume", opentracing.ChildOf(spanCtx)) defer span.Finish() ext.MessageBusDestination.Set(span, exchange) ext.PeerHostname.Set(span, hostname) ext.SpanKindConsumer.Set(span) return callback(msg, headers) } }

Slide 123

Slide 123 text

@PurpleBooth tracer := opentracing.GlobalTracer() spanCtx, err := tracer.Extract(opentracing.TextMap, opentracing.TextMapCarrier(headers)) if err == nil { return err }

Slide 124

Slide 124 text

@PurpleBooth span := opentracing.StartSpan("consume", opentracing.ChildOf(spanCtx)) defer span.Finish() ext.MessageBusDestination.Set(span, exchange) ext.PeerHostname.Set(span, hostname) ext.SpanKindConsumer.Set(span)