Understand Your System Behavior By Implement Distributed Tracing Using Jaeger

Engineering Sharing in Qiscus. Also published in my Medium: https://medium.com/@yusufs/understand-your-system-behavior-by-implement-distributed-tracing-using-jaeger-9041df1104ec

Transcript

1. Understand Your System Behavior By Implement Distributed Tracing Using Jaeger Yusuf Syaifudin - Yogyakarta, December 12th, 2018
2. Why We Need Tracing?
3. Why We Need Tracing? • Even in upfront the API/endpoint just looks like request-response model, but it may contains some complicated process behind. Let say about: validation, check data in cache, query in database, or even calling 3rd party service or another internal service. • We may can see log, but reading log generally is hard to see what system’s do in behind.
4. Log may just look like this: { "level": "info", "client_ip": "182.0.201.240", "status": 200, "method": "POST", "endpoint": "/api/v2/sdk/login_or_register", "app_code": "helo-mhbiymsjdqhzwymd", "user_id": 0, "time": "2018-12-12T08:05:46Z", "latency": 191.69, "user_agent": "Chrome/63.0.3239.111", "sdk_version": "WEB_2.6.5-beta3", "sdk_platform": "WEB", "sdk_device_brand": "", "sdk_device_model": "", "sdk_device_os_version": "", "message": "API REQUEST" } Log like this is not define anything, but just a plain log.
5. “Distributed tracing helps pinpoint where failures occur and what causes suboptimal performance.” - From https://searchitoperations.techtarget.com/definition/distributed-tracing
6. How tracing works?
7. Meet Dora the Explorer If you don’t know who is she, watch here: https://www.youtube.com/watc h?v=N33ldfKwdLs
8. Plot • The series centers around Dora with a love of embarking on quests related to an activity that she wants to partake of or a place that she wants to go to. • Each episode is based around a series of cyclical events that occur along the way during Dora's travels, along with obstacles that she and Boots are forced to overcome or puzzles that they have to solve relating to riddles, the Spanish language, or counting. To see which route she should take, she always ask to Map:
9. What is the relation to tracing in Software Development? Suppose that: • Dora’s mission is the one API request in server. • Map is the route from user’s request to response. • Obstacle is like a function. So, to finish one mission/travel, Dora may face several obstacles. Dora the Explorer map template. Source: https://commixus.com/dora-the-explorer-map-template-58475/ (accessed at 2 Dec 2018.)
10. Do the tracing on Dora’s mission • Suppose that we want Dora to report to us what causing her to take the long time and every detail that she may find it good to report. • So, Dora can records the start time and end time of each obstacle, note some additional information when you find it useful to report, etc. Dora will leave a footprint in each of obstacle. Here's the visual:
11. Do the tracing on Dora’s mission
12. Let’s code it! • Define each obstacle as a function. • Then each function must be delayed in n millisecond so we can visualize it better.
13. We can call all of defined function like this: serverSpan := eCtx.Get("serverSpan").(opentracing.Span) ctx := opentracing.ContextWithSpan( context.Background(), serverSpan, ) defer ctx.Done() … out, _ = startTheJourneyFromTreeHouse(ctx, in) out, _ = passTheForest(ctx, out) out, _ = crossTheLake(ctx, out, isRainyDay) out, _ = enterThePyramid(ctx, out)
14. When we access Jaeger, it will look something like this: We can know easily that the most consumed task is in function crossTheLake when the parameter is_rainy_day is true.
15. Thanks! https://medium.com/@yusufs/u nderstand-your-system-behavior- by-implement-distributed-tracing- using-jaeger-9041df1104ec