@jeqo89 at #kafkasummit Making sense of event-driven dataflows Distributed tracing for Apache Ka a®-based applications with Zipkin

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit Complexity happens…

@jeqo89 at #kafkasummit “Complexity is anything [...] that makes a system hard to understand and modify” John Ousterhout, “A Philosophy of Software Design”

@jeqo89 at #kafkasummit “Complexity is caused by two things: dependencies and obscurity” John Ousterhout, “A Philosophy of Software Design”

@jeqo89 at #kafkasummit twitter.com/rakyll/status/971231712049971200

@jeqo89 at #kafkasummit Jorge Esteban Quilcate Otoya twitter: @jeqo89 | github: jeqo Peruvian in Oslo, Norway Integration team at SYSCO AS Part of Apache Kafka and Zipkin communities

@jeqo89 at #kafkasummit Talk: “Making sense of your event-driven dataflows” 40 min Q&A Why? What distributed tracing? How to instrument Kafka apps? Demo What’s next? Demo time

@jeqo89 at #kafkasummit Trace Span

@jeqo89 at #kafkasummit “Demystifying” Kafka client configurations Kafka producers Kafka Streams Kafka Consumer github.com/jeqo/tracing-kafka-apps

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Trace ID Trace metrics Trace timeline Spans

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps

@jeqo89 at #kafkasummit Is Kafka producer `send` sync or async? github.com/jeqo/tracing-kafka-apps Blocking call

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Non-blocking call Is Kafka producer `send` sync or async?

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Batched record

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps auto.commit=true

@jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps commit per record

@jeqo89 at #kafkasummit services Report

@jeqo89 at #kafkasummit “The more accurately you try to measure the position of a particle, the less accurately you can measure its speed” Heisenberg's uncertainty principle

@jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc tracer tracer Traces reporting: Annotation-based approach TRACES

@jeqo89 at #kafkasummit PRODUCER CONSUMER tracer tracer TraceContext=abc BROKER TraceContext=abc TRACES

@jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer producer = new KafkaProducer<>(settings); Producer tracedProducer = kafkaTracing.producer(producer); producer.send( new ProducerRecord<>( "my-topic", key, value ));

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer producer = new KafkaProducer<>(settings); Producer tracedProducer = kafkaTracing.producer(producer); // wrap tracedProducer.send( new ProducerRecord<>( "my-topic", key, value ));

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer consumer = new KafkaConsumer<>(settings); Consumer tracedConsumer = kafkaTracing.consumer(consumer); while (running) { var records = consumer.poll(1000); records.forEach(this::process); }

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer consumer = new KafkaConsumer<>(settings); Consumer tracedConsumer = kafkaTracing.consumer(consumer); // wrap while (running) { var records = tracedConsumer.poll(1000); records.forEach(this::process); }

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ void process(ConsumerRecord record){ // extract span from record headers Span span = kafkaTracing.nextSpan(record) .name("process") .start(); try (var ws = tracer.withSpanInScope(span)) { doProcess(record); } catch (RuntimeException | Error e) { span.error(e); throw e; } finally { span.finish(); } }

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = new KafkaStreams(b.build(), config); kafkaStreams.start();

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = // wrap ksTracing.kafkaStreams(b.build(), config); kafkaStreams.start();

@jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .transform(ksTracing.map(“parse”, this::parseRecord)) .join(table, this::tableJoiner) .transformValues(ksTracing.transformValues( “transform”, this::transform))) .to("output-topic");

@jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: Black-box approach agent agent TRACES

@jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: mixed approach agent agent TRACES tracer tracer

@jeqo89 at #kafkasummit Report Transport services

@jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = URLConnectionSender.create( "http://localhost:9411/api/v2/spans" ); var reporter = AsyncReporter.create(sender);

@jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = KafkaSender.newBuilder() .bootstrapServers( "localhost:9092") .build(); var reporter = AsyncReporter.create(sender);

@jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = BringYourOwnSender.newBuilder() .build(); var reporter = AsyncReporter.create(sender);

@jeqo89 at #kafkasummit Report Transport Storage BringYourOwnDB services

@jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) services Data-at-rest

@jeqo89 at #kafkasummit Streaming Messaging Kafka Clients REST Proxy KSQL Kafka Source Connector Kafka Streams Kafka Sink Connector

@jeqo89 at #kafkasummit REST Proxy KSQL Kafka Source Connector Kafka Sink Connector Kafka Interceptors

@jeqo89 at #kafkasummit Producer Interceptor API

@jeqo89 at #kafkasummit Consumer Interceptor API

@jeqo89 at #kafkasummit Demo: Tracing Kafka-based applications github.com/jeqo/talk-kafka-zipkin

@jeqo89 at #kafkasummit What’s next?

@jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) Distributed Tracing IS A Stream Processing Problem Data-at-rest services

@jeqo89 at #kafkasummit Distributed Tracing IS A Stream Processing Problem Span Consumer Trace Aggregation Span Store spans-collected traces-completed Dependencies Store github.com/jeqo/zipkin-storage-kafka Custom processors

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit Canopy: An End-to-End Performance Tracing And Analysis System”

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit

@jeqo89 at #kafkasummit Peter Alvaro et al., “Automating Failure Testing Research at Internet Scale”

@jeqo89 at #kafkasummit is there anyone applying this?

@jeqo89 at #kafkasummit Haystack: tracing and analysis platform Tracing Trends and Metrics Anomaly Detection Remediation Alerting services

@jeqo89 at #kafkasummit Demo: Extending Zipkin with Kafka and Haystack github.com/jeqo/talk-kafka-zipkin

@jeqo89 at #kafkasummit * Demo 1, source code: github.com/jeqo/tracing-kafka-apps * Demo 2, source code: github.com/jeqo/talk-kafka-zipkin * Blog post: confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications * Zipkin: github.com/openzipkin (moving to Apache foundation), gitter.im/openzipkin/zipkin * Sites using Zipkin: cwiki.apache.org/confluence/display/ZIPKIN/Sites * Haystack: github.com/ExpediaDotCom/haystack, gitter.im/expedia-haystack * Zipkin Kafka Backend: github.com/jeqo/zipkin-storage-kafka * Kafka Interceptor for Zipkin: github.com/sysco-middleware/kafka-interceptor-zipkin * Martin Kleppmann et al. 2019. Online Event Processing. https://dl.acm.org/citation.cfm?id=3321612 * John Ousterhout. A Philosophy of Software Design. www.amazon.com/Philosophy-Software-Design-John-Ousterhout/dp/1732102201 * Jonathan Kaldor et al.2017. Canopy: An End-to-End Performance Tracing And Analysis System.SOSP’17(2017). doi.org/10.1145/3132747.3132749 * Peter Alvaro et al.2016. Automating Failure Testing Research at Internet Scale.SoCC ’16. dx.doi.org/10.1145/2987550.2987555 Resources

@jeqo89 at #kafkasummit fin github.com/jeqo