Making sense of event-driven systems Distributed tracing for Apache Ka a®-based applications with Zipkin @jeqo89 | #codeone

@jeqo89 | #codeone “Complexity symptoms: - Change amplification - Cognitive load - Unknown unknowns” John Ousterhout, “A Philosophy of Software Design”

@jeqo89 | #codeone “Complexity is caused by two things: dependencies and obscurity” John Ousterhout, “A Philosophy of Software Design”

@jeqo89 | #codeone SERVER CLIENT SERVER

@jeqo89 | #codeone CLIENT SERVER CLIENT SERVER

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone

@jeqo89 | #codeone Complexity happens…

@jeqo89 | #codeone twitter.com/rakyll/status/971231712049971200

@jeqo89 | #codeone Jorge Esteban Quilcate Otoya twitter: @jeqo89 | github: jeqo Peruvian in Oslo, Norway Integration team at SYSCO AS Unclogging Data pipelines by day Contributing to Apache Kafka and OpenZipkin communities by night

@jeqo89 | #codeone Talk: “Making sense of your event-driven systems” 37 min Q&A Why? What’s distributed tracing? How to instrument Kafka apps? Demo What’s next? Demo time depth sync/async causal relation

@jeqo89 | #codeone

@jeqo89 | #codeone Traces in Zipkin

@jeqo89 | #codeone “Demystifying” Kafka client configurations Kafka producers Kafka Streams Kafka Consumer github.com/jeqo/tracing-kafka-apps

@jeqo89 | #codeone github.com/jeqo/tracing-kafka-apps Blocking call kafkaProducer.send(record, (metadata, exception) -> { //... }).get(); // Synchronous send

@jeqo89 | #codeone github.com/jeqo/tracing-kafka-apps Non-blocking call kafkaProducer.send(record, (metadata, exception) -> { //... }); // Async send

@jeqo89 | #codeone github.com/jeqo/tracing-kafka-apps Batched send var producerConfig = new Properties(); // ... producerConfig.put(ProducerConfig.LINGER_MS_CONFIG, 1_000); producerConfig.put(ProducerConfig.BATCH_SIZE_CONFIG, 100_000);

@jeqo89 | #codeone github.com/jeqo/tracing-kafka-apps auto.commit=true config = new Properties(); //... config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);

@jeqo89 | #codeone github.com/jeqo/tracing-kafka-apps commit per record var records = consumer.poll(TIMEOUT); records.forEach(record -> { doSomething(record); consumer.commitSync(Map.of(new TopicPartition(...), new OffsetAndMetadata(...))); });

@jeqo89 | #codeone Record

@jeqo89 | #codeone CLIENT SERVER TraceContext=abc tracer tracer Annotation-based TRACES

@jeqo89 | #codeone PRODUCER CONSUMER tracer tracer TraceContext=abc BROKER TraceContext=abc TRACES

@jeqo89 | #codeone /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 | #codeone /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 | #codeone /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }

@jeqo89 | #codeone github.com/openzipkin/b3-propagation B3 Propagation

@jeqo89 | #codeone github.com/openzipkin/b3-propagation B3 Propagation

@jeqo89 | #codeone CLIENT SERVER TraceContext=abc Black-box agent agent TRACES linkerd.io/2019/08/09/service-mesh-distributed-tracing-myths/

@jeqo89 | #codeone linkerd.io/2019/08/09/service-mesh-distributed-tracing-myths/

@jeqo89 | #codeone linkerd.io/2019/08/09/service-mesh-distributed-tracing-myths/

@jeqo89 | #codeone CLIENT SERVER TraceContext=abc mixed agent agent TRACES tracer tracer linkerd.io/2019/08/09/service-mesh-distributed-tracing-myths/

@jeqo89 | #codeone “The more accurately you try to measure the position of a particle, the less accurately you can measure its speed” Heisenberg's uncertainty principle

@jeqo89 | #codeone Record services Collect

@jeqo89 | #codeone Record Collect Store BringYourOwnDB services

@jeqo89 | #codeone Record Collect Store Dependencies (batch) services

@jeqo89 | #codeone Streaming Messaging Kafka Clients REST Proxy KSQL Kafka Source Connector Kafka Streams Kafka Sink Connector

@jeqo89 | #codeone /** Instrumentation for Kafka Clients **/ Producer producer = new KafkaProducer<>(settings); Producer tracedProducer = kafkaTracing.producer(producer); producer.send( new ProducerRecord<>( "my-topic", key, value ));

@jeqo89 | #codeone /** Instrumentation for Kafka Clients **/ Producer producer = new KafkaProducer<>(settings); Producer tracedProducer = kafkaTracing.producer(producer); // wrap tracedProducer.send( new ProducerRecord<>( "my-topic", key, value ));

@jeqo89 | #codeone /** Instrumentation for Kafka Clients **/ Consumer consumer = new KafkaConsumer<>(settings); Consumer tracedConsumer = kafkaTracing.consumer(consumer); while (running) { var records = consumer.poll(1000); records.forEach(this::process); }

@jeqo89 | #codeone /** Instrumentation for Kafka Clients **/ Consumer consumer = new KafkaConsumer<>(settings); Consumer tracedConsumer = kafkaTracing.consumer(consumer); // wrap while (running) { var records = tracedConsumer.poll(1000); records.forEach(this::process); }

@jeqo89 | #codeone /** Instrumentation for Kafka Clients **/ void process(ConsumerRecord record){ // extract span from record headers Span span = kafkaTracing.nextSpan(record) .name("process") .start(); try (var ws = tracer.withSpanInScope(span)) { doProcess(record); } catch (RuntimeException | Error e) { span.error(e); throw e; } finally { span.finish(); } }

@jeqo89 | #codeone /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::prepare)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); var topology = b.build(); KafkaStreams kafkaStreams = new KafkaStreams(topology, config); kafkaStreams.start();

@jeqo89 | #codeone /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::prepare)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); var topology = b.build(); KafkaStreams kafkaStreams = // wrap ksTracing.kafkaStreams(topology, config); kafkaStreams.start();

@jeqo89 | #codeone /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::prepare)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); var topology = b.build();

@jeqo89 | #codeone /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .transform(ksTracing.map(“preparing”, () -> this::prepare)) .join(table, this::tableJoiner) .transformValues(ksTracing.transformValues( “transforming”, () -> this::transform))) .to("output-topic");

@jeqo89 | #codeone REST Proxy KSQL Kafka Source Connector Kafka Sink Connector Kafka Interceptors

@jeqo89 at #kafkasummit Demo: Tracing Kafka-based applications github.com/jeqo/talk-kafka-zipkin

@jeqo89 | #codeone https://www.confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications

@jeqo89 | #codeone Record Collect Store Dependencies (batch) Distributed Tracing is a Event Streaming problem Data-at-rest services

@jeqo89 | #codeone Netflix pipeline

@jeqo89 | #codeone Netflix pipeline

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem Span Collected partitioned-spans Traces Store github.com/openzipkin-contrib/zipkin-storage-kafka

@jeqo89 | #codeone Scatter-gather/scalable back-end kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html#adding-an-rpc-layer-to-your-application

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem Span Collected Trace Aggregation partitioned-spans traces-completed Traces Store github.com/openzipkin-contrib/zipkin-storage-kafka

@jeqo89 | #codeone Session windows and suppression kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#session-windows

@jeqo89 | #codeone Session windows and suppression kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#session-windows

@jeqo89 | #codeone var b = new StreamsBuilder(); b.stream(spansTopic, ...).groupByKey() // how long to wait for another span .windowedBy(SessionWindows.with(timeout)...) .aggregate(ArrayList::new, aggregateSpans(), joinAggregates(), ...) // hold until a new record tells that a window is closed and we can process it further .suppress(untilWindowCloses(unbounded())) .toStream()

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem Span Collected Trace Aggregation partitioned-spans traces-completed dependencies Traces Store github.com/openzipkin-contrib/zipkin-storage-kafka

@jeqo89 | #codeone var b = new StreamsBuilder(); b.stream(spansTopic, ...).groupByKey() ... // session windows and suppression .toStream() // traceStream .flatMapValues(spansToDependencyLinks()) .selectKey((key, value) -> linkKey(value)) .to(dependencyTopic, ...));

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem Span Collected Trace Aggregation Traces Store partitioned-spans traces-completed Dependencies Store dependencies github.com/openzipkin-contrib/zipkin-storage-kafka

@jeqo89 | #codeone Tumbling window aggregates kafka.apache.org/documentation/streams/developer-guide/dsl-api.html#session-windows

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem Span Collected Trace Aggregation Traces Store partitioned-spans traces-completed Dependencies Store Custom processors dependencies github.com/openzipkin-contrib/zipkin-storage-kafka

@jeqo89 | #codeone Distributed Tracing is a Stream Processing Problem partitioned-spans traces-completed error-traces dependencies github.com/jeqo/zipkin-storage-kafka-experiments github.com/openzipkin/openzipkin.github.io/wiki/2018-07-18-Aggregation-and-Analysis-at-Netflix Long-term Error Trace Store path-aggregate

@jeqo89 | #codeone Haystack: tracing and analysis platform Tracing Trends and Metrics Anomaly Detection Remediation Alerting services expediadotcom.github.io/haystack/

@jeqo89 | #codeone expediadotcom.github.io/haystack/ Haystack: tracing and analysis platform

@jeqo89 | #codeone expediadotcom.github.io/haystack/ Haystack

@jeqo89 | #codeone Wrapping up: Consider Distributed Tracing as better data source to deal with system collaboration complexity Distributed Tracing pipeline is an Event Streaming pipeline Focus on aggregation, model extraction and signals (!)

@jeqo89 | #codeone * Demos, source code: github.com/jeqo/tracing-kafka-apps github.com/jeqo/talk-kafka-zipkin * Blog post: confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications * Zipkin Kafka Backend: github.com/openzipkin-contrib/zipkin-storage-kafka * Kafka Interceptor for Zipkin: github.com/sysco-middleware/kafka-interceptor-zipkin * Sites using Zipkin: github.com/openzipkin/openzipkin.github.io/wiki/Sites * Haystack: github.com/ExpediaDotCom/haystack, gitter.im/expedia-haystack * Martin Kleppmann et al. 2019. Online Event Processing. https://dl.acm.org/citation.cfm?id=3321612 * John Ousterhout. A Philosophy of Software Design. amazon.com/Philosophy-Software-Design-John-Ousterhout/dp/1732102201 * Jonathan Kaldor et al.2017. Canopy: An End-to-End Performance Tracing And Analysis System.SOSP’17(2017). doi.org/10.1145/3132747.3132749 * Peter Alvaro et al.2016. Automating Failure Testing Research at Internet Scale.SoCC ’16. dx.doi.org/10.1145/2987550.2987555 * Mark Burgess 2019. From Observability to Significance in Distributed Information Systems. https://arxiv.org/abs/1907.05636 Resources

@jeqo89 | #codeone github.com/openzipkin/openzipkin.github.io/wiki/Sites

@jeqo89 | #codeone fin github.com/jeqo/talk-kafka-zipkin gitter.im/openzipkin/zipkin github.com/openzipkin