Distributed Tracing: Understand how your systems work together

Understanding failures or latencies in monoliths or small systems usually starts with looking at a single component in isolation. Microservices architecture invalidates this assumption because end user requests now traverse dozen of components and a single component simply does not give you enough information: each part is just one side of a bigger story.

In this talk we’ll look at distributed tracing which summarizes all sides of the story into a shared timeline and also distributed tracing tools like Apache Zipkin, which highlights the relationship between components, from the very top of the stack to the deepest aspects of the system.

Transcript

1. Distributed Tracing Understand how your components work together
2. About me José Carlos Chávez • Software Engineer at Typeform focused on the aggregate of responses services. • Zipkin core team and open source contributor for Observability projects.
3. Distributed Systems
4. Distributed systems A collection of independent components appears to its users as a single coherent system. Characteristics: • Concurrency • No global clock • Independent failures
5. Water heater Gas supplier Cold water storage tank Shutoff valve First floor branch Tank valve 爆$❄#☭ Distributed systems
6. Auth service Images service Videos service DB2 DB3 DB4 TCP error (2003) 500 Internal Error 500 Internal Error GET /media/e5k2 API Proxy Distributed systems: Understanding failures DB1 Media API
7. Water heater Gas supplier Cold water storage tank Shutoff valve First floor branch Tank valve 爆$❄#☭ I AM HERE! First floor distributor is clogged! Distributed systems: Understanding failures
8. We do have that, it is called logs!
9. API Proxy Auth service Media API Images service Videos service DB2 DB3 DB4 TCP error (2003) 500 Internal Error 500 Internal Error GET /media/e5k2 Logs & Concurrency DB1
10. [24/Oct/2017 13:50:07 +0000] “GET /media HTTP/1.1” 200 … **0/13548” [24/Oct/2017 13:50:07 +0000] “GET /media HTTP/1.1” 200 … **0/23948” [24/Oct/2017 13:50:08 +0000] “GET /media HTTP/1.1” 200 … **0/12396” [24/Oct/2017 13:50:07 +0000] “GET /videos HTTP/1.1” 200 … **0/23748” [24/Oct/2017 13:50:07 +0000] “GET /images HTTP/1.1” 200 … **0/23248” [24/Oct/2017 13:50:07 +0000] “GET /auth HTTP/1.1” 200 … **0/26548” [24/Oct/2017 13:50:07 +0000] “POST /media HTTP/1.1” 200 … **0/13148” [24/Oct/2017 13:50:07 +0000] “GET /media HTTP/1.1” 200 … **0/2588” [24/Oct/2017 13:50:07 +0000] “GET /auth HTTP/1.1” 500 … **0/3248” [24/Oct/2017 13:50:07 +0000] “POST /media HTTP/1.1” 200 … **0/23548” [24/Oct/2017 13:50:07 +0000] “GET /images HTTP/1.1” 200 … **0/22598” [24/Oct/2017 13:50:07 +0000] “GET /videos HTTP/1.1” 200 … **0/13948” ... ? ? Logs & Concurrency
11. Water heater Gas supplier Cold water storage tank Shutoff valve First floor branch Tank valve 爆$❄#☭ I AM HERE! First floor distributor is clogged! Distributed systems: Understanding failures
12. Distributed Tracing to unclog your pipes
13. API Proxy Media API Auth Videos Images Time 500 error Distributed tracing [1508410442] no cache for resource, retrieving from DBc TraceID d52d38b69b0fb15efa
14. Distributed Tracing: What answers I get? • What services did a request pass through? • What occurred in each service for a given request? • Where did the error happen? • Where are the bottlenecks? • What is the critical path for a request? • Who should I page?
15. Distributed Tracing & friends Credits: Peter Bourgon
16. Benefits of Distributed Tracing • (almost) Immediate feedback • System insight, clarifies non trivial interactions • Visibility to critical paths and dependencies • Understand latencies • Request scoped, not request’s lifecycle scoped.
17. Trace’s Anatomy • A trace shows an execution path through a distributed system • A span in the trace represents a logical unit of work (with a start and end) • A context includes information that should be propagated across services • Tags and logs (optional) add complementary information to spans. /things auth.Auth Time GET /videos mysql.Get T R A C E
18. Elements of distributed tracing Credits: Nic Munroe Leg 1: inbound propagation Leg 2: outbound propagation Leg 3: in-process propagation Distributed Tracing
19. Leg 1: Inbound propagation When your service process a request or consume a message. API Proxy Media API GET /media TraceID: fAf3oXL6DS SpanID: dZ0xHIBa1A ...
20. Leg 2: Outbound propagation When your service makes an outbound call to another service Media API Video service GET /videos TraceID: fAf3oXL6DS ParentID: dZ0xHIBa1A SpanID: y74fr5udj http/get
21. mysql.Query redis.Get Leg 3: In process propagation When performing an operation inside the service Media API Cache service Images service GET /images
22. API Proxy Media API Auth Videos Images Time 500 error Distributed tracing [1508410442] no cache for resource, retrieving from DBc TraceID d52d38b69b0fb15efa
23. Any overhead? For users: • Observability tools are meant to be unintrusive • Sampling reduces overhead • (Don’t) trace every single operation For developers: • Not all libraries are ready to plug instruments • Instrumentation can be delegated to common frameworks
24. Introducing Apache Zipkin
25. Apache Zipkin Based on BigBrotherBird (B3) and inspired on Google Dapper (2010). It was open sourced by Twitter (2012) and joined Apache Incubator on September 2018. • Mature tracing model emerged from users’ needs. • Used by large companies like Netflix, SoundCloud and Yelp but also not too big ones. • Strong community: ◦ @zipkinproject ◦ gitter.im/openzipkin
26. Service (instrumented) Transport Collect spans Collector API UI Storage DB Visualize Retrieve data Store spans http/kafka/grpc Receive spans Deserialize and schedule for storage Cassandra/MySQL/ElasticSearch Zipkin: architecture
27. Zipkin: traces
28. Zipkin: traces
29. Zipkin: traces
30. Zipkin: trace overview
31. Zipkin: tags and logs
32. Zipkin: traces with errors
33. Zipkin: traces for async operations
34. Zipkin: dependency graph
35. Zipkin: dependency graph
36. Q&As twitter.com/jcchavezs Find more: http://bit.ly/dist-trac