Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Debugging applications with Distributed Tracing

Debugging applications with Distributed Tracing

Presented at the March Squamish Technology Meetup, this presentation gave an overview of distributed tracing describing its core concepts.

Alex Boten

March 07, 2019
Tweet

More Decks by Alex Boten

Other Decks in Technology

Transcript

  1. W I T H D I S T R I

    B U T E D T R A C I N G D E B U G G I N G A P P L I C AT I O N S
  2. D E B U G G I N G A

    P P L I C AT I O N S
  3. FAV O U R I T E D E B

    U G G I N G T O O L S ?
  4. FAV O U R I T E D E B

    U G G I N G T O O L S ? printf(“#### HERE”)
  5. FAV O U R I T E D E B

    U G G I N G T O O L S ?
  6. T R A C I N G • application: debuggers

    (gdb, pdb…) • system: strace/dtrace/ptrace/dtruss… • network: tcpdump
  7. W H AT A B O U T D I

    S T R I B U T E D A P P L I C AT I O N S ?
  8. D I S T R I B U T E

    D T R A C I N G
  9. D I S T R I B U T E

    D T R A C I N G set of tools and practices to monitor the health and reliability of distributed applications - trace context w3c working draft
  10. C O N C E P T S • Context

    • Propagation
  11. C O N C E P T S • Context

    • Propagation
  12. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 localhost:6831 localhost:16686
  13. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… localhost:6831 localhost:16686 curl localhost:8080/v1/reservation?city=Whistler
  14. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: hotels-api … localhost:6831 localhost:16686 curl localhost:8080/v1/reservation?city=Whistler X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234…
  15. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: hotels-api … X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: weather-api … localhost:6831 localhost:16686 curl localhost:8080/v1/reservation?city=Whistler X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234…
  16. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: hotels-api … X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: weather-api … X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: trace-demo … localhost:6831 localhost:16686 curl localhost:8080/v1/reservation?city=Whistler X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234…
  17. W E AT H E R - A P I

    L O C A L H O S T: 8 0 8 2 H O T E L S - A P I L O C A L H O S T: 8 0 8 1 T R A C E - D E M O L O C A L H O S T: 8 0 8 0 X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: hotels-api … X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: weather-api … X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234… Service: trace-demo … localhost:6831 localhost:16686 curl localhost:8080/v1/reservation?city=Whistler X-B3-TraceId: 1234… X-B3-ParentSpanId: 1234… X-B3-SpanId: 1234… X-B3-Sampled: 1234…
  18. V I S I B I L I T Y

    • context for events/transactions across entire systems • together client & server applications
  19. V I S I B I L I T Y

    • system behaviour under production loads • hotspots
  20. V I S I B I L I T Y

    • annotations & attributes • deadcode
  21. U S E C A S E S • debugging

    an application • gaining visibility into existing system
  22. G O T C H A S • more work

    upfront • sampling
  23. A LT E R N AT I V E S

    • event logging • metrics
  24. F U R T H E R L E A

    R N I N G • Dapper, a large scale distributed systems tracing infrastructure: https:// static.googleusercontent.com/media/research.google.com/en//archive/ papers/dapper-2010-1.pdf • https://opentracing.io & https://opencensus.io • Trace context - W3C working draft: https://www.w3.org/TR/trace-context/ • You know you want to know more about printf: https://en.wikipedia.org/ wiki/Printf_format_string
  25. Q U E S T I O N S ?

    A L E X B O T E N @ C O D E B O T E N