Distributed System Tracing

Once you’ve broken up your monolithic application into a micro services architecture like everyone’s been telling you to you’ll come upon a horrible realization. The profiling and analysis tools you’ve been using to monitor the performance of your application are no longer quite as useful as they once were. Most traditional profilers have no way to cross process and service boundaries and give meaningful insights into inter-process performance issues. This session will demonstrate the a tracing infrastructure you can put in place to monitor your distributed service oriented architecture. You’ll discover just how easy it is to instrument a system for tracing and visualize tracing data for highlighting issues and insights.

Transcript

1. DISTRIBUTED SYSTEM TRACING @dustyburwell Monitoring Performance in Distributed Systems
3. Dusty Burwell @dustyburwell
10. Traditional Options
11. CPU Tracing Statistical or Detailed
15. Memory Dumping
16. Structured Log Aggregation
17. Structured Log Aggregation a.k.a. read the f’n error
18. Microservices Tracing
19. Dapper http://bit.ly/cm-dapper
20. Vocabulary
21. Trace
23. Trace ID
24. Span
25. Trace ID
26. Trace ID
27. Trace ID Span ID
28. Trace ID Span ID
29. Trace ID Span ID Parent Span ID
30. Trace ID Span ID Parent Span ID Client Start - timestamp Server Start - timestamp Server End - timestamp Client End - timestamp
31. Annotation
32. Trace ID Span ID Parent Span ID Client Start - timestamp Server Start - timestamp Server End - timestamp Client End - timestamp
33. Trace ID Span ID Parent Span ID Client Start - timestamp Server Start - timestamp Server End - timestamp Client End - timestamp
34. Trace ID Span ID Parent Span ID Client Start - timestamp Server Start - timestamp Server End - timestamp Client End - timestamp Service Name Endpoint Is Error? Error code Etc.
36. Parts
37. Instrumentation
38. Transport libraries (HTTP, etc) Middleware or similar Headers passed from instrumented libraries and received by middleware pass along Trace ID and Parent Span ID
39. Sampling
40. Tryfer https://github.com/tryfer
41. Data Pipeline
42. Aggregation, Visualization, and Reporting
43. Specialized Tooling
45. Generalized Tooling
46. Generalized Tooling e.g. MapReduce
47. Challenges
48. Achieving Buy In
49. Achieving Buy In a.k.a. Chicken and Egg
50. Polyglot Probs
52. Callback Oriented Frameworks
53. Exercise for the Reader
54. Dapper http://bit.ly/cm-dapper
55. Zipkin http://zipkin.io/
56. Working Groups Distributed Tracing and SOA
57. Log Aggregation Splunk, ELK, etc.
58. NewRelic http://newrelic.com/
59. Stackify http://stackify.com/