Velocity 2017 SJ: Application Tracing

Workshops slides

Transcript

1. Application Tracing Velocity San Jose 2017
2. Your instructor •Bryan Liles •bryanliles@gmail.com •@bryanl
3. WIFI: OReilly Conf
4. Agenda • Workstation Setup • Goals • Tracing Introduction • Exercise 1 • Break • Exercise 2 • Exercise 3 • Beyond the basics • Discussion • Closing
5. Workshop Setup • https://github.com/bryanl/velocity-sj-2017 • http://bit.ly/vsj2017-apptracing
6. • Jaeger • Language Specific Exercises • Jaeger Demo App • Postgres
7. Today’s applications are complex
8. Shop
9. Shop Inventory Management Shipping
10. Shop Inventory Management Shipping Redis Postgres
11. How do we know what’s going on?
12. Shop
13. Shop Metrics For every resource, check utilization, saturation, and errors. • Average busy time • Amount of capacity available • Error count
14. Shop Logging Capture events to help identify incidents and application specific data • Auth success/failure • Validation failures • So many more
15. Shop Tracing Capture timing metrics from one or multiple resources participating in a transaction. • Find latency issues and errors across resources
16. Shop Tracing
17. Tracing Request scoped Metrics Aggregates Logging Events Request scoped events Aggregate events Request scoped metrics
18. Tracing Request scoped Metrics Aggregates Logging Events Prometheus ELK OpenTracing
19. Tracing
21. Terminology
22. Span: A period of time A span contains the following: * Operation name * Start/Finish timestamps * Tags * Logs * References to other spans
23. Trace: a directed acyclic graph (DAG) of spans Span A Span B Span C Span D Span E Span F Span G Span H
24. Trace: a DAG of spans Time ➡ Span A Span B Span D Span C Span E Span F Span G Span H
25. OpenTracing
26. Google Dapper "We built dapper to provide Google’s developers with more information about the behavior of complex distributed systems"
27. Google Dapper Design Goals • Low Overhead • Application-level transparency • Scalability
28. Zipkin • Created by Twitter • Zipkin and Dapper are based on the same principals
29. OpenTracing isn’t a framework
30. OpenTracing describes an abstraction for tracing distributed systems
31. Jaeger: OpenTracing Compatible Implementation
32. Jaeger Driver OpenTracing API App Jaeger
33. Standardization Why OpenTracing?
34. • Standardize span management • Standardize inter-process propagation • Standardize active span management • Standardize in-band context encoding • Standardize out-of-band trace data encoding
35. Why Standardize? • Tracing should be easy and unobtrusive • Competing standards dilute progress • The ecosystem is complex and diverse. It’s easier to drive to a single standard rather than having multiple.
36. Open Tracing Demo
37. pip install -U docker-compose
38. Workshop Setup
39. Activity: Tracing Concepts
40. Talking to Jaeger
41. Reporter Sampler Jaeger
42. Activity: More Tracing Concepts
43. Discussion / Lessons Learned
44. What should I log or tag? https://github.com/opentracing/specification/blob/master/semantic_conventions.md
45. Instrumenting
46. Searching for tags or events
47. Baggage Sometimes, you want to make data available for child spans
48. Closing