Spring Kafka beyond the basics Lessons learned on our Kafka journey at ING Bank Tim van Baarsen JFALL – NOVEMBER 3RD 2022

Who am I? Tim van Baarsen Software Engineer @ ING Netherlands The Netherlands Team Dora Amsterdam

ING is active in more than 40 counties 3

Kafka @ ING 4 Frontrunners in Kafka Running in production: • 7 years • 5000+ topics • Serving 1000+ Development teams • Self service topic management • Many different patterns

Kafka @ ING 5 0 100.000 200.000 300.000 400.000 500.000 600.000 700.000 800.000 900.000 1.000.000 2015 2016 2017 2018 2019 2020 2021 2022 Messages produced per second (average) Messages produced per second (average) Traffic is growing with 10%+ monthly

Agenda 6 • Kafka in a nutshell • Spring Kafka essentials • Scenario: Poison pill / Deserialization exceptions • Scenario: Lack of exception handling • Testing • Monitoring • Wrap-up • Questions

Kafka in a nutshell 7 Consumer Consumer Producer Kafka client Kafka broker Consumer Kafka client 0100101001101 0100101001101 poll send Responsibilities: - subscribe - deserialization • key • Value - heartbeat Not responsible for: • Type checking • Schema validation • Other constraints Responsibilities: - send - serialization • key • value 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 Data in a Kafka topic are just stored as bytes! new Responsible for: • Append only log • Distribute • Replicate

Kafka in a nutshell Consumer Consumer Producer Kafka client Kafka broker Consumer Kafka client 0100101001101 0100101001101 poll send 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 new Confluent Schema Registry REST API Load schema Responsible for: • Schema validation • Avro • Protobuf • Json schema KafkaAvroSerializer KafkaAvroDeserializer

Spring for Apache Kafka - essentials 9 Kafka Kafka Clients Listener Container Error Handling Deserializer spring-kafka- test User Code Error Handler ConsumerRecord Recoverer KafkaTemplate @KafkaListener Consumer Producer Kafka Spring Kafka Your code @EmbeddedKafka https://spring.io/projects/spring-kafka Kafka Streams @EnableKafkaStreams

Spring Kafka & Spring Boot - Producer 10 @Component @Slf4j public class StockQuoteProducer { @Autowired private KafkaTemplate kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } }

Spring Kafka & Spring Boot - Consumer 11 @Component @Slf4j public class StockQuoteProducer { @Autowired private KafkaTemplate kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } } @Component @Slf4j public class StockQuoteConsumer { @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partition) { log.info("Consumed from partition: {} value: {}", partition, stockQuote); } }

Spring Kafka & Spring Boot – Kafka streams 12 @Configuration @EnableKafkaStreams public class KafkaStreamsConfig { @Bean public KStream kStream(StreamsBuilder streamsBuilder) { KStream branchedStream = new KafkaStreamBrancher() .branch((key, value) -> value.getExchange().equalsIgnoreCase("NYSE"), kStream -> kStream.to("stock-quotes-nyse")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("NASDAQ"), kStream -> kStream.to("stock-quotes-nasdaq")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("AMS"), kStream -> kStream.to("stock-quotes-ams")) .defaultBranch(kStream -> kStream.to("stock-quotes-exchange-other")) .onTopOf(streamsBuilder.stream("stock-quotes")); return branchedStream; } }

Spring Kafka & Spring Boot – Configuration (application.yml) 13 spring: application: name: producer-application kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer client-id: ${spring.application.name} properties: schema.registry.url: http://localhost:8081

After the honeymoon phase is over 14 🧨 😱 🚀 😎 Traps Lessons learned The hard way! Apply in your own project(s) Share with your fellow developers Tips & Tricks 👩💻

🚀 Tip: Local development setup 15 • Docker • Kafka Cluster • Zookeeper • Confluent Schema Registry • CLI Tools • Kafka CLI, Confluent CLI, Kafka cat • UI Tools • Confluent Control Center • Conduktor • Klaw • Kafka UI • Etc

🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 16 💊 💀 Corrupted record What is a Poison Pill? Deserialization failure A record that always fails when consumed, no matter how many times it is attempted. Different forms: ❓

🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 17 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker Consumer Kafka client KafkaAvroDeserializer KafkaAvroSerializer StringSerializer 0100101001101 0100101001101 poll send Producer Kafka client send >_ 💊 Confluent Schema Registry REST API Load schema Register schema 010001101

🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 18 Scenario: • Consumer of topics • Someone produced a ‘poison’ message • Consumer fails to deserialize • Consequence • Blocks consumption of the topic/partition • application can’t ‘swallow the pill’ • Try again and again and again (very fast) • Log line for every failure AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’ Consumer Another Consumer (different application)

🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 19 • Result: log file will grow very fast, flood your disc • Impact: High • How to survive this scenario? • Wait until the retention period of the topic has passed • Change consumer group • Manually / Programmatically update the offset • Configure ErrorHandlingDeserializer (Provided by Spring Kafka) 👎 👍 👎 👎 AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’

🧨 Scenario: ‘Poison Pill’ - Demo 20

Consumer Consumer Real life Poison Pill scenario: E2E encryption 21 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker Consumer Kafka client CustomKafkaAvroSerializer CustomKafkaAvroDeserializer $ecret $ecret yolo $ecret 0100101001101 0100101001101 poll send Confluent Schema Registry REST API

🧨 Scenario: Lack of proper exception handling 22 Scenario: • Consumer • Exception is thrown in the method handling the message Result (by default) • Records that fail are: • Retried • Logged • We move on to the next one Consequence • You lose that message! • Not acceptable in many use-cases! @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote) { if ("KABOOM".equalsIgnoreCase(stockQuote.getSymbol())) { throw new RuntimeException("Whoops something went wrong..."); } }

🧨 Scenario: Lack of proper exception handling 23 Impact • Dependents on your use-case How to survive this scenario? • Replace / configure DefaultErrorHandler by: • CommonLoggingErrorHandler • ContainerStoppingErrorHandler • Use backoff strategy for recoverable exceptions • Configure ConsumerRecordRecoverer • Dead letter topic • Implement your own recoverer

🧨 Scenario: Lack of proper exception handling - Demo 24

Testing 25 • Support for Integration testing @EmbeddedKafka @EmbeddedKafka @SpringJUnitConfig public class EmbeddedKafkaIntegrationTest { @Autowired private EmbeddedKafkaBroker embeddedKafkaBroker; @Test public void yourTestHere() throws Exception { // TODO implement ;) } }

Testing – Alternatives to EmbeddedKafka 26 • Test containers • Focus on unit test first • Kafka streams • Topology test driver

Testing – Test topology test driver 27 @Test void stockQuoteFromAmsterdamStockExchangeEndUpOnTopicQuotesAmsTopic() { StockQuote stockQuote = new StockQuote("INGA", "AMS", "10.99", "EUR", "Description", Instant.now()); stockQuoteInputTopic.pipeInput(stockQuote.getSymbol(), stockQuote); assertThat(stockQuoteAmsOutputTopic.isEmpty()).isFalse(); assertThat(stockQuoteAmsOutputTopic.getQueueSize()).isEqualTo(1L); assertThat(stockQuoteAmsOutputTopic.readValue()).isEqualTo(stockQuote); assertThat(stockQuoteNyseOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteNasdaqOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteOtherOutputTopic.isEmpty()).isTrue(); }

Monitoring - Three Pillars observability 28 Tracing Logging Metrics (Aggregatable) (Events) (Request scoped) Micrometer + Prometheus Spring Cloud + Zipkin Sleuth

Monitoring – Metrics 29 • Out of the box Kafka metrics • consumers • producer • streams • Micrometer • Spring Boot Actuator Metrics

Monitoring – Metrics 30

Monitoring – Distributed tracing 31

Monitoring – Distributed tracing 32

🎓 Lessons learned 33 • Invest time in your local development environment (fast feedback loop) • Use Spring Kafka but also understand the core Kafka APIs • Consumer: • Expect the unexpected! • Handle Deserialization exceptions a.k.a. ‘poison pills’ • Proper exception handling • Validate incoming data • Producer: • Don’t change your serializers • Leverage Apache Avro + Confluent Schema registry • Don’t break compatibility for your consumers!

🎓 Lessons learned 34 • Security • Who can produce data to your topics? • Who can consume? • Monitor your topics, consumer groups & applications in production • Micrometer • Spring Cloud Sleuth • Don’t overdo integration test!

Questions 35 🤔 ❔

Thanks for joining my talk! @TimvanBaarsen https://www.medium.com/@TimvanBaarsen https://github.com/j-tim/jfall-2022-spring-kafka-beyond-the-basics