Why Tracing is important for Distributed Systems.pdf

Transcript

1. Why Tracing is important for Distributed Systems
2. Introduction Raja Sundaram Ganesan Email: grajasundaram@gmail.com Twitter: @zeagord Github: @zeagord
3. TakeAways • Why we need distributed tracing • How we can instrument using Zipkin’s Brave and then Visualize • Who is using Zipkin and How?
4. Observability What is observability? • Past, Present, Future of a system What is Monitoring? • Overall report of the system
5. Pillars of Observability • Logs - Recording the events • Metrics - Aggregated measure of events • Traces - Recording events with Ordering • Monitoring, Alerting and Visualization Credit: Coda Hale
6. Credit: Peter Bourgon
7. Hypothetical bill payment request API Gateway Auth Service Billier Service Payment Service User Service Biller Gateways Payment Gateway Payment History DB DB What services and unit of work does this request touches What services are my customers Who is blocking me most of the time How does a request gets affected if I fail
8. Why Distributed Tracing? • Narrow down issues quickly • Find the latency • Understand Complex applications • See your architecture • Visualize traces and Share with the teams
9. Tracing Jargon • Span - Basic unit of work. Timed. Scoped. Can be stopped and started • Trace - Collection of spans that represents a single request • Tracer - Records timing and metadata about the operations that takes place in your application • Instrumentation - Happens inside your application using a tracer
10. Tracing Jargon ...continued • Collector - Collects the spans using different transport and saves the spans • Tags/Annotations - Metadata about the operation
11. How does it look? Service Names span Trace
12. What is Zipkin Zipkin is a collection of Distributed Tracing libraries originated from Twitter and now maintained by the community named OpenZipkin. Zipkin has tracers, collectors and a UI to visualize traces
13. Zipkin under the hood • Zipkin Server: Armeria (Netty), Spring boot, Java, Kotlin • Instrumentation for most of the popular languages ◦ Java, Javascript, Python, Ruby, GoLang, C++, PHP
14. How to instrument code? • Spring Cloud Sleuth ◦ Auto Instruments HTTP Requests ◦ Messaging ◦ Have annotations for custom unit of work
15. How to instrument code? @NewSpan(name = "convertToDatabaseColumn") @Override public String convertToDatabaseColumn (String creditCardNumber ) { VaultOperations vaultOperations = BeanUtil.getBean(VaultOperations .class); Plaintext plaintext = Plaintext.of(creditCardNumber); return vaultOperations .opsForTransit() .encrypt("customer", plaintext) .getCiphertext(); }
16. How to instrument code? ScopedSpan span = tracer.startScopedSpan( "encode"); try { // The span is in "scope" meaning downstream code such as loggers can see trace IDs return encoder.encode(); } catch (RuntimeException | Error e) { span.error(e); // Unless you handle exceptions, you might not know the operation failed! throw e; } finally { span.finish(); // always finish the span }
17. Transport for Spans • HTTP • RabbitMQ • Kafka • AWS Kinesis/SQS • Google Pub/Sub
18. Processing and Storing Traces • Elastic Search • Cassandra • MySQL (Not for production) • In-Memory (Not for production) • 3rd party (ex VoltDB, Google StackDriver)
19. How to set up Zipkin for production • Where to begin ? • How to choose storage ? • Transport
20. Case Studies • Netflix • Twitter • InfoStellar (Stellar Station) • Samsung • Ascend • TypeForm
21. Twitter Scala Build
22. Netflix
23. Ascend • Use Zipkin to understand the latencies caused and bottlenecks and the response time of various request during system failure • Measure latency improvements before and after refactoring the services • Understand the system architecture using Zipkin Dependencies and also identify non-conformant service communications that deviates from the design
24. Ascend
25. LINE corp • Uses it for Latency Investigation • Metrics (~ 50 millions metrics) (in-house build using opentsdb / mysql) • Logging (in-house build using elasticsearch) • Tracing (Using openzipkin)
26. InfoStellar (StellarStation) • Offers communication infrastructure to satellite operators - low latency end-to-end is a must • Tracing provides detailed insight into component latencies for troubleshooting • Need to ensure as much of the flow as possible is traced, can't skip Pub/Sub, etc.
27. SoundCloud • 100 million tracks on the platform and more than 12 hours of audio gets uploaded every minute. • 100 engineers and hundreds of microservices. • Enrich with Kubernetes pod data and other metadata
28. A typical Service graph
29. What to watch out for in Observability in the next few years? • Adaptive Alerting • Anomaly Detection • Graph Visualization • Navigation instead of querying in the UI • Uniform Format
30. What to watch out for in Observability in the next few years? “Graph analysis is probably the answer. tools like adaptive alerting pushing you insight based on anomalies and deltas in the traces going across the system. Similar to deltas in a metric value, but based on topology instead. Challenge to this would be the unstable nature of call graphs. For example, people are still increasing how many nodes are traced. This will mean that there will be shape differences due to a higher degree of instrumentation. The system will have to figure out the difference between things like this and things like orphaned services or problems. Another challenge is trace data is built to be lossy, the system will have to be able to tolerate missing parts of traces. It is likely commercial products already do some of this on the backend, but if brought to open source a lot of things could progress. For example deployment tools could know if they are causing vs fixing a problem. Tracing UIs could self-navigate or push suggestions instead of a query based approach. I think this would be very interesting.” - Adrian Cole
31. Demo Time
32. Questions
33. References Site docs - https://github.com/openzipkin/openzipkin.github.io/wiki/Sites Talk to Zipkin folks at - https://gitter.im/openzipkin/zipkin Visit https://zipkin.io/ for more info Twitter Scala Days - https://portal.klewel.com/watch/webcast/scala-days-2019/talk/13/ Visualization - http://simianviz.surge.sh/netflix