Tracing your money

At Mondo we’re building a new kind of bank in Go on a microservice platform, which has a huge number of benefits, at the cost of increased complexity. We use Phosphor, an open source system tracing tool built in Go, to track and diagnose problems in our platform. We'll cover how we developed and integrated this using the Context package, and how this enables system behaviour to be observed and monitored in high volume, low latency, distributed Go applications.

Transcript

1. TRACING YOUR MONEY @mattheath London Go User Group Sep 2015
2. @mattheath
6. monoliths traditional dev
17. DATABASE APPLICATION
18. DATABASE APPLICATION
19. DATABASE DATABASES APPLICATION
20. DATABASE DATABASES APPLICATION SEARCH
21. DATABASE DATABASES APPLICATION CACHE SEARCH
22. DATABASE DATABASES APPLICATION CACHE SEARCH CAT GIFS
23. DATABASE DATABASES MONOLITH CACHE SEARCH CAT GIFS
24. MONOLITH
27. LOAD BALANCER
28. LOAD BALANCER HTTP API & ROUTING LAYER
29. SOME
 API LOAD BALANCER HTTP API & ROUTING LAYER
30. SOME
 API SERVICE ONE SERVICE TWO LOAD BALANCER HTTP API & ROUTING LAYER
31. SOME
 API SERVICE ONE SERVICE TWO LOAD BALANCER HTTP API & ROUTING LAYER DATABASE
32. SOME
 API SERVICE ONE SERVICE TWO LOAD BALANCER HTTP API & ROUTING LAYER DATABASE DATABASE
33. SOME
 API SERVICE ONE SERVICE TWO LOAD BALANCER HTTP API & ROUTING LAYER DATABASE DATABASE EXTERNAL PROVIDER
34. ? ? ? LOAD BALANCER HTTP API & ROUTING LAYER ? ? ?? ? ? ? ? ? ? ?
36. REQUEST
37. ERROR?
41. DISTRIBUTED
 TRACING
43. api api api.customer api.customer service.customer service.customer
44. HTrace Zipkin X-Trace AppDynamics Salp Jaeger CrossStitch
45. HTrace Zipkin X-Trace AppDynamics Salp Jaeger CrossStitch
46. HTrace Zipkin X-Trace AppDynamics Salp Jaeger CrossStitch
47. github.com/mondough/phosphor
48. api api api.customer api.customer service.customer service.customer
49. api api api.customer api.customer service.customer service.customer SEND RECV SEND RECV RECV SEND RECV SEND
50. api api.customer service.customer SEND RECV SEND RECV RECV SEND RECV SEND phosphor
51. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
52. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
53. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
54. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
56. api api api.customer api.customer service.customer service.customer
57. api api api.customer api.customer service.customer service.customer
58. 8096820c-3b7b-47ec-bce6-1c239252ab40
59. api api api.customer api.customer service.customer service.customer
60. api api api.customer api.customer service.customer service.customer
61. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
62. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
63. package context type Context interface { Deadline() (deadline time.Time, ok bool) Done() <-chan struct{} Err() error Value(key interface{}) interface{} }
64. ctx := context.Background()
65. ctx := context.Background() ctx = context.WithValue(ctx, "TraceID", "boop")
66. ctx := context.Background() ctx = context.WithValue(ctx, "TraceID", "boop") ctx.Value("TraceID") => "boop"
67. mondough/typhon mondough/mercury
68. client.Req( ctx context.Context, service string, endpoint string, req proto.Message, res proto.Message )
69. type Client interface { Add(Call) Client // various other funcs Middleware() []ClientMiddleware SetMiddleware([]ClientMiddleware) AddMiddleware(ClientMiddleware) }
70. if traceId, ok := ctx.Value("TraceID").(string); ok && parentId != "" { req.SetHeader("TraceID", traceId) }
71. api api api.customer api.customer service.customer service.customer
72. api api api.customer api.customer service.customer service.customer
73. type Server interface { // various funcs Middleware() []ServerMiddleware SetMiddleware([]ServerMiddleware) AddMiddleware(ServerMiddleware) }
74. ctx := context.Background() if v := req.Headers()["TraceID"]; v != "" { ctx = context.WithValue(ctx, "TraceID", v) }
75. api api api.customer api.customer service.customer service.customer
76. api api api.customer api.customer service.customer service.customer
77. &Annotation{ Type: AnnotationType_SERVER_SEND, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{}, Payload: []byte{}, // extra fields }
78. &Annotation{ Type: AnnotationType_SERVER_SEND, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{}, Payload: []byte{}, // extra fields }
79. &Annotation{ Type: AnnotationType_SERVER_SEND, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{}, Payload: []byte{}, // extra fields }
80. &Annotation{ Type: AnnotationType_SERVER_SEND, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{}, Payload: []byte{}, // extra fields }
81. &Annotation{ Type: AnnotationType_ANNOTATION, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{}, Payload: []byte{}, // extra fields }
82. &Annotation{ Type: AnnotationType_ANNOTATION, TraceId: "ffeae21f-c5c1-4...", ParentId: "31f4ebd0-a34b-4...", SpanId: "2c97d609-46ef-4...", Timestamp: microTime(time.Now()), Hostname: util.Hostname(), KeyValue: map[string]string{ "query": "SELECT xxx FROM...", }, Payload: results, // extra fields }
83. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
84. listener, err := net.ListenUDP("udp", address) check(err) for { message := make([]byte, packetSize) n, _, err := listener.ReadFrom(message) buf := bytes.NewBuffer(message[0:n]) writeToTransport(buf) }
85. ch := make(chan []byte, 10000) for { message := make([]byte, packetSize) n, _, err := listener.ReadFrom(message) buf := bytes.NewBuffer(message[0:n]) select { case ch <- buf.Bytes(): // log.Infof("Wrote message to channel") default: // abort! // log.Infof("Dropped message") } }
86. func (w *worker) loop() { var b []byte timeout := time.NewTicker(bufferWindow) defer timeout.Stop() // Bonus points // Spin and forward on traces every time our // buffer fills, or when our time window elapses for { select { case b = <-w.ch: w.buf = append(w.buf, b) if len(w.buf) >= bufferSize { w.send() } case <-timeout.C: w.send() } } }
87. PhosphorD Service A Trace Library goroutine chan UDP Service B Trace Library goroutine chan UDP PHOSPHOR HOST INSTANCES PUBLISH DASHBOARDS AGGREGATION PERSISTANCE NSQ
89. e433eacb-c894-4efe-8df2-b56425179510
90. 611bb3f2-746d-46fe-baf8-495009acd5bc
93. NEXT?
94. LOGGING
 METRICS INTEGRATIONS
95. log.Errorf(ctx, "Uh oh! %s", err.Error()) log.Warnf(ctx, "Uh oh! %s", err.Error()) log.Infof(ctx, "Uh oh! %s", err.Error()) log.Debugf(ctx, "Uh oh! %s", err.Error())
96. 44ac4b44-a874-4e60-8fe5-8d05229dbb49 127.0.0.1 - - [30/Sep/2015 05:38:15] "POST /oauth2/token HTTP/1.1" 200 720 "" "Go 1.1 package http" 0. "{"access_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaSI6Im9hdXRoY2xpZW50XzAwMDA5MGJ2RkdZRFJ5T2Z0UE9KSWYiLCJleHAiOjE0NDM2NTYyOTUsImlhdC AwMDA5MGJ2Rkg2Yk83dTFjM1RsdGgiLCJ1aSI6InVzZXJfMDAwMDkwYnZGRkRjUDRLSmxFa0xZWCIsInYiOiIxIn0.gGS_VD6tY7whmkVTZunsE- Fo8Si8fkUDOulCA8cbZxI","client_id":"oauthclient_000090bvFGYDRyOftPOJIf","expires_in": 21599,"refresh_token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhaSI6InRva18wMDAwOTBidkZINmJPN3UxYzNUbHRoIiwiY2kiOiJvYXV0aGNsaWVudF8wMDAwOTBid DQzNjM0Njk1LCJ1aSI6InVzZXJfMDAwMDkwYnZGRkRjUDRLSmxFa0xZWCIsInYiOiIxIn0.rqWHmWWORQXoitn3HeJU743rMwB9pW7g3YJ96BGYSoE","token_type":"Bearer","us LYX"}" 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695745954288 [Debug] Bearer token recovered: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjaSI6Im9hdXRoY2xpZW50XzAwMDA5MGJ2RkdZRFJ5T2Z0UE9KSWYiLCJleHAiOjE0NDM2NTYyOTUsImlhdCI6MTQ0MzYzNDY5NSwi dTFjM1RsdGgiLCJ1aSI6InVzZXJfMDAwMDkwYnZGRkRjUDRLSmxFa0xZWCIsInYiOiIxIn0.gGS_VD6tY7whmkVTZunsE-Fo8Si8fkUDOulCA8cbZxI 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695746092845 [Debug] [Mercury:Client] Sending request to service.platform.authentication/read_a 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695761353836 [Debug] [Mercury:Client] Sending request to service.platform.permission/read… 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695762559395 [Debug] [Mercury:Client] Sending request to service.platform.permission/read… 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695762610999 [Debug] [Mercury:Client] Sending request to service.oauth2-client/read_client… 44ac4b44-a874-4e60-8fe5-8d05229dbb49 1443634695781366551 [Debug] [Mercury:Client] Sending request to service.api.card-activation/handlehttp
97. THANKS! @mattheath @getmondo Join our alpha at getmondo.co.uk!