Making sense of event-driven dataflows - Kafka Summit NYC 2019

Tracing Apache Kafka-based applications with Zipkin

Transcript

1. @jeqo89 at #kafkasummit Making sense of event-driven dataflows Distributed tracing for Apache Ka a®-based applications with Zipkin
2. @jeqo89 at #kafkasummit
3. @jeqo89 at #kafkasummit
4. @jeqo89 at #kafkasummit
5. @jeqo89 at #kafkasummit
6. @jeqo89 at #kafkasummit
7. @jeqo89 at #kafkasummit
8. @jeqo89 at #kafkasummit
9. @jeqo89 at #kafkasummit
10. @jeqo89 at #kafkasummit
11. @jeqo89 at #kafkasummit Complexity happens…
12. @jeqo89 at #kafkasummit “Complexity is anything [...] that makes a system hard to understand and modify” John Ousterhout, “A Philosophy of Software Design”
13. @jeqo89 at #kafkasummit “Complexity is caused by two things: dependencies and obscurity” John Ousterhout, “A Philosophy of Software Design”
14. @jeqo89 at #kafkasummit twitter.com/rakyll/status/971231712049971200
15. @jeqo89 at #kafkasummit Jorge Esteban Quilcate Otoya twitter: @jeqo89 | github: jeqo Peruvian in Oslo, Norway Integration team at SYSCO AS Part of Apache Kafka and Zipkin communities
16. @jeqo89 at #kafkasummit Talk: “Making sense of your event-driven dataflows” 40 min Q&A Why? What distributed tracing? How to instrument Kafka apps? Demo What’s next? Demo time
17. @jeqo89 at #kafkasummit Trace Span
18. @jeqo89 at #kafkasummit “Demystifying” Kafka client configurations Kafka producers Kafka Streams Kafka Consumer github.com/jeqo/tracing-kafka-apps
19. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Trace ID Trace metrics Trace timeline Spans
20. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps
21. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps
22. @jeqo89 at #kafkasummit Is Kafka producer `send` sync or async? github.com/jeqo/tracing-kafka-apps Blocking call
23. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Non-blocking call Is Kafka producer `send` sync or async?
24. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Batched record
25. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps auto.commit=true
26. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps commit per record
27. @jeqo89 at #kafkasummit services Report
28. @jeqo89 at #kafkasummit “The more accurately you try to measure the position of a particle, the less accurately you can measure its speed” Heisenberg's uncertainty principle
29. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc tracer tracer Traces reporting: Annotation-based approach TRACES
30. @jeqo89 at #kafkasummit PRODUCER CONSUMER tracer tracer TraceContext=abc BROKER TraceContext=abc TRACES
31. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
32. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
33. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
34. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer<K, V> producer = new KafkaProducer<>(settings); Producer<K, V> tracedProducer = kafkaTracing.producer(producer); producer.send( new ProducerRecord<>( "my-topic", key, value ));
35. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer<K, V> producer = new KafkaProducer<>(settings); Producer<K, V> tracedProducer = kafkaTracing.producer(producer); // wrap tracedProducer.send( new ProducerRecord<>( "my-topic", key, value ));
36. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer<K, V> consumer = new KafkaConsumer<>(settings); Consumer<K, V> tracedConsumer = kafkaTracing.consumer(consumer); while (running) { var records = consumer.poll(1000); records.forEach(this::process); }
37. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer<K, V> consumer = new KafkaConsumer<>(settings); Consumer<K, V> tracedConsumer = kafkaTracing.consumer(consumer); // wrap while (running) { var records = tracedConsumer.poll(1000); records.forEach(this::process); }
38. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ void process(ConsumerRecord<K, V> record){ // extract span from record headers Span span = kafkaTracing.nextSpan(record) .name("process") .start(); try (var ws = tracer.withSpanInScope(span)) { doProcess(record); } catch (RuntimeException | Error e) { span.error(e); throw e; } finally { span.finish(); } }
39. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = new KafkaStreams(b.build(), config); kafkaStreams.start();
40. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = // wrap ksTracing.kafkaStreams(b.build(), config); kafkaStreams.start();
41. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .transform(ksTracing.map(“parse”, this::parseRecord)) .join(table, this::tableJoiner) .transformValues(ksTracing.transformValues( “transform”, this::transform))) .to("output-topic");
42. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: Black-box approach agent agent TRACES
43. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: mixed approach agent agent TRACES tracer tracer
44. @jeqo89 at #kafkasummit Report Transport services
45. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = URLConnectionSender.create( "http://localhost:9411/api/v2/spans" ); var reporter = AsyncReporter.create(sender);
46. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = KafkaSender.newBuilder() .bootstrapServers( "localhost:9092") .build(); var reporter = AsyncReporter.create(sender);
47. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = BringYourOwnSender.newBuilder() .build(); var reporter = AsyncReporter.create(sender);
48. @jeqo89 at #kafkasummit Report Transport Storage BringYourOwnDB services
49. @jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) services Data-at-rest
50. @jeqo89 at #kafkasummit Streaming Messaging Kafka Clients REST Proxy KSQL Kafka Source Connector Kafka Streams Kafka Sink Connector
51. @jeqo89 at #kafkasummit REST Proxy KSQL Kafka Source Connector Kafka Sink Connector Kafka Interceptors
52. @jeqo89 at #kafkasummit Producer Interceptor API
53. @jeqo89 at #kafkasummit Consumer Interceptor API
54. @jeqo89 at #kafkasummit Demo: Tracing Kafka-based applications github.com/jeqo/talk-kafka-zipkin
55. @jeqo89 at #kafkasummit What’s next?
56. @jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) Distributed Tracing IS A Stream Processing Problem Data-at-rest services
57. @jeqo89 at #kafkasummit Distributed Tracing IS A Stream Processing Problem Span Consumer Trace Aggregation Span Store spans-collected traces-completed Dependencies Store github.com/jeqo/zipkin-storage-kafka Custom processors
58. @jeqo89 at #kafkasummit
59. @jeqo89 at #kafkasummit
60. @jeqo89 at #kafkasummit Canopy: An End-to-End Performance Tracing And Analysis System”
61. @jeqo89 at #kafkasummit
62. @jeqo89 at #kafkasummit
63. @jeqo89 at #kafkasummit
64. @jeqo89 at #kafkasummit
65. @jeqo89 at #kafkasummit Peter Alvaro et al., “Automating Failure Testing Research at Internet Scale”
66. @jeqo89 at #kafkasummit is there anyone applying this?
67. @jeqo89 at #kafkasummit Haystack: tracing and analysis platform Tracing Trends and Metrics Anomaly Detection Remediation Alerting services
68. @jeqo89 at #kafkasummit Demo: Extending Zipkin with Kafka and Haystack github.com/jeqo/talk-kafka-zipkin
69. @jeqo89 at #kafkasummit * Demo 1, source code: github.com/jeqo/tracing-kafka-apps * Demo 2, source code: github.com/jeqo/talk-kafka-zipkin * Blog post: confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications * Zipkin: github.com/openzipkin (moving to Apache foundation), gitter.im/openzipkin/zipkin * Sites using Zipkin: cwiki.apache.org/confluence/display/ZIPKIN/Sites * Haystack: github.com/ExpediaDotCom/haystack, gitter.im/expedia-haystack * Zipkin Kafka Backend: github.com/jeqo/zipkin-storage-kafka * Kafka Interceptor for Zipkin: github.com/sysco-middleware/kafka-interceptor-zipkin * Martin Kleppmann et al. 2019. Online Event Processing. https://dl.acm.org/citation.cfm?id=3321612 * John Ousterhout. A Philosophy of Software Design. www.amazon.com/Philosophy-Software-Design-John-Ousterhout/dp/1732102201 * Jonathan Kaldor et al.2017. Canopy: An End-to-End Performance Tracing And Analysis System.SOSP’17(2017). doi.org/10.1145/3132747.3132749 * Peter Alvaro et al.2016. Automating Failure Testing Research at Internet Scale.SoCC ’16. dx.doi.org/10.1145/2987550.2987555 Resources
70. @jeqo89 at #kafkasummit fin github.com/jeqo