Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Velocity 2017 SJ: Application Tracing

Velocity 2017 SJ: Application Tracing

Workshops slides


Bryan Liles

June 21, 2017

More Decks by Bryan Liles

Other Decks in Programming


  1. Application Tracing Velocity San Jose 2017

  2. Your instructor •Bryan Liles •bryanliles@gmail.com •@bryanl

  3. WIFI: OReilly Conf

  4. Agenda • Workstation Setup • Goals • Tracing Introduction •

    Exercise 1 • Break • Exercise 2 • Exercise 3 • Beyond the basics • Discussion • Closing
  5. Workshop Setup • https://github.com/bryanl/velocity-sj-2017 • http://bit.ly/vsj2017-apptracing

  6. • Jaeger • Language Specific Exercises • Jaeger Demo App

    • Postgres
  7. Today’s applications are complex

  8. Shop

  9. Shop Inventory Management Shipping

  10. Shop Inventory Management Shipping Redis Postgres

  11. How do we know what’s going on?

  12. Shop

  13. Shop Metrics For every resource, check utilization, saturation, and errors.

    • Average busy time • Amount of capacity available • Error count
  14. Shop Logging Capture events to help identify incidents and application

    specific data • Auth success/failure • Validation failures • So many more
  15. Shop Tracing Capture timing metrics from one or multiple resources

    participating in a transaction. • Find latency issues and errors across resources
  16. Shop Tracing

  17. Tracing Request scoped Metrics Aggregates Logging Events Request scoped events

    Aggregate events Request scoped metrics
  18. Tracing Request scoped Metrics Aggregates Logging Events Prometheus ELK OpenTracing

  19. Tracing

  20. None
  21. Terminology

  22. Span: A period of time A span contains the following:

    * Operation name * Start/Finish timestamps * Tags * Logs * References to other spans
  23. Trace: a directed acyclic graph (DAG) of spans Span A

    Span B Span C Span D Span E Span F Span G Span H
  24. Trace: a DAG of spans Time ➡ Span A Span

    B Span D Span C Span E Span F Span G Span H
  25. OpenTracing

  26. Google Dapper "We built dapper to provide Google’s developers with

    more information about the behavior of complex distributed systems"
  27. Google Dapper Design Goals • Low Overhead • Application-level transparency

    • Scalability
  28. Zipkin • Created by Twitter • Zipkin and Dapper are

    based on the same principals
  29. OpenTracing isn’t a framework

  30. OpenTracing describes an abstraction for tracing distributed systems

  31. Jaeger: OpenTracing Compatible Implementation

  32. Jaeger Driver OpenTracing API App Jaeger

  33. Standardization Why OpenTracing?

  34. • Standardize span management • Standardize inter-process propagation • Standardize

    active span management • Standardize in-band context encoding • Standardize out-of-band trace data encoding
  35. Why Standardize? • Tracing should be easy and unobtrusive •

    Competing standards dilute progress • The ecosystem is complex and diverse. It’s easier to drive to a single standard rather than having multiple.
  36. Open Tracing Demo

  37. pip install -U docker-compose

  38. Workshop Setup

  39. Activity: Tracing Concepts

  40. Talking to Jaeger

  41. Reporter Sampler Jaeger

  42. Activity: More Tracing Concepts

  43. Discussion / Lessons Learned

  44. What should I log or tag? https://github.com/opentracing/specification/blob/master/semantic_conventions.md

  45. Instrumenting

  46. Searching for tags or events

  47. Baggage Sometimes, you want to make data available for child

  48. Closing