JFall 2022 - Spring Kafka beyond the basics - Lessons learned on our Kafka journey at ING Bank

Slides for my talk at JFall 2022 NLJUG

https://jfall.nl/speakers-2022/

Source code demo: https://github.com/j-tim/jfall-2022-spring-kafka-beyond-the-basics

Transcript

1. Spring Kafka beyond the basics Lessons learned on our Kafka journey at ING Bank Tim van Baarsen JFALL – NOVEMBER 3RD 2022
2. Who am I? Tim van Baarsen Software Engineer @ ING Netherlands The Netherlands Team Dora Amsterdam
3. ING is active in more than 40 counties 3
4. Kafka @ ING 4 Frontrunners in Kafka Running in production: • 7 years • 5000+ topics • Serving 1000+ Development teams • Self service topic management • Many different patterns
5. Kafka @ ING 5 0 100.000 200.000 300.000 400.000 500.000 600.000 700.000 800.000 900.000 1.000.000 2015 2016 2017 2018 2019 2020 2021 2022 Messages produced per second (average) Messages produced per second (average) Traffic is growing with 10%+ monthly
6. Agenda 6 • Kafka in a nutshell • Spring Kafka essentials • Scenario: Poison pill / Deserialization exceptions • Scenario: Lack of exception handling • Testing • Monitoring • Wrap-up • Questions
7. Kafka in a nutshell 7 Consumer Consumer Producer Kafka client Kafka broker Consumer Kafka client 0100101001101 0100101001101 poll send Responsibilities: - subscribe - deserialization • key • Value - heartbeat Not responsible for: • Type checking • Schema validation • Other constraints Responsibilities: - send - serialization • key • value 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 Data in a Kafka topic are just stored as bytes! new Responsible for: • Append only log • Distribute • Replicate
8. Kafka in a nutshell Consumer Consumer Producer Kafka client Kafka broker Consumer Kafka client 0100101001101 0100101001101 poll send 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 new Confluent Schema Registry REST API Load schema Responsible for: • Schema validation • Avro • Protobuf • Json schema KafkaAvroSerializer KafkaAvroDeserializer
9. Spring for Apache Kafka - essentials 9 Kafka Kafka Clients Listener Container Error Handling Deserializer spring-kafka- test User Code Error Handler ConsumerRecord Recoverer KafkaTemplate @KafkaListener Consumer Producer Kafka Spring Kafka Your code @EmbeddedKafka https://spring.io/projects/spring-kafka Kafka Streams @EnableKafkaStreams
10. Spring Kafka & Spring Boot - Producer 10 @Component @Slf4j public class StockQuoteProducer { @Autowired private KafkaTemplate<String, StockQuote> kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } }
11. Spring Kafka & Spring Boot - Consumer 11 @Component @Slf4j public class StockQuoteProducer { @Autowired private KafkaTemplate<String, StockQuote> kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } } @Component @Slf4j public class StockQuoteConsumer { @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partition) { log.info("Consumed from partition: {} value: {}", partition, stockQuote); } }
12. Spring Kafka & Spring Boot – Kafka streams 12 @Configuration @EnableKafkaStreams public class KafkaStreamsConfig { @Bean public KStream<String, StockQuote> kStream(StreamsBuilder streamsBuilder) { KStream<String, StockQuote> branchedStream = new KafkaStreamBrancher<String, StockQuote>() .branch((key, value) -> value.getExchange().equalsIgnoreCase("NYSE"), kStream -> kStream.to("stock-quotes-nyse")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("NASDAQ"), kStream -> kStream.to("stock-quotes-nasdaq")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("AMS"), kStream -> kStream.to("stock-quotes-ams")) .defaultBranch(kStream -> kStream.to("stock-quotes-exchange-other")) .onTopOf(streamsBuilder.stream("stock-quotes")); return branchedStream; } }
13. Spring Kafka & Spring Boot – Configuration (application.yml) 13 spring: application: name: producer-application kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer client-id: ${spring.application.name} properties: schema.registry.url: http://localhost:8081
14. After the honeymoon phase is over 14 🧨 😱 🚀 😎 Traps Lessons learned The hard way! Apply in your own project(s) Share with your fellow developers Tips & Tricks 👩💻
15. 🚀 Tip: Local development setup 15 • Docker • Kafka Cluster • Zookeeper • Confluent Schema Registry • CLI Tools • Kafka CLI, Confluent CLI, Kafka cat • UI Tools • Confluent Control Center • Conduktor • Klaw • Kafka UI • Etc
16. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 16 💊 💀 Corrupted record What is a Poison Pill? Deserialization failure A record that always fails when consumed, no matter how many times it is attempted. Different forms: ❓
17. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 17 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker Consumer Kafka client KafkaAvroDeserializer KafkaAvroSerializer StringSerializer 0100101001101 0100101001101 poll send Producer Kafka client send >_ 💊 Confluent Schema Registry REST API Load schema Register schema 010001101
18. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 18 Scenario: • Consumer of topics • Someone produced a ‘poison’ message • Consumer fails to deserialize • Consequence • Blocks consumption of the topic/partition • application can’t ‘swallow the pill’ • Try again and again and again (very fast) • Log line for every failure AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’ Consumer Another Consumer (different application)
19. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 19 • Result: log file will grow very fast, flood your disc • Impact: High • How to survive this scenario? • Wait until the retention period of the topic has passed • Change consumer group • Manually / Programmatically update the offset • Configure ErrorHandlingDeserializer (Provided by Spring Kafka) 👎 👍 👎 👎 AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’
20. 🧨 Scenario: ‘Poison Pill’ - Demo 20
21. Consumer Consumer Real life Poison Pill scenario: E2E encryption 21 0 1 2 3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker Consumer Kafka client CustomKafkaAvroSerializer CustomKafkaAvroDeserializer $ecret $ecret yolo $ecret 0100101001101 0100101001101 poll send Confluent Schema Registry REST API
22. 🧨 Scenario: Lack of proper exception handling 22 Scenario: • Consumer • Exception is thrown in the method handling the message Result (by default) • Records that fail are: • Retried • Logged • We move on to the next one Consequence • You lose that message! • Not acceptable in many use-cases! @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote) { if ("KABOOM".equalsIgnoreCase(stockQuote.getSymbol())) { throw new RuntimeException("Whoops something went wrong..."); } }
23. 🧨 Scenario: Lack of proper exception handling 23 Impact • Dependents on your use-case How to survive this scenario? • Replace / configure DefaultErrorHandler by: • CommonLoggingErrorHandler • ContainerStoppingErrorHandler • Use backoff strategy for recoverable exceptions • Configure ConsumerRecordRecoverer • Dead letter topic • Implement your own recoverer
24. 🧨 Scenario: Lack of proper exception handling - Demo 24
25. Testing 25 • Support for Integration testing @EmbeddedKafka @EmbeddedKafka @SpringJUnitConfig public class EmbeddedKafkaIntegrationTest { @Autowired private EmbeddedKafkaBroker embeddedKafkaBroker; @Test public void yourTestHere() throws Exception { // TODO implement ;) } }
26. Testing – Alternatives to EmbeddedKafka 26 • Test containers • Focus on unit test first • Kafka streams • Topology test driver
27. Testing – Test topology test driver 27 @Test void stockQuoteFromAmsterdamStockExchangeEndUpOnTopicQuotesAmsTopic() { StockQuote stockQuote = new StockQuote("INGA", "AMS", "10.99", "EUR", "Description", Instant.now()); stockQuoteInputTopic.pipeInput(stockQuote.getSymbol(), stockQuote); assertThat(stockQuoteAmsOutputTopic.isEmpty()).isFalse(); assertThat(stockQuoteAmsOutputTopic.getQueueSize()).isEqualTo(1L); assertThat(stockQuoteAmsOutputTopic.readValue()).isEqualTo(stockQuote); assertThat(stockQuoteNyseOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteNasdaqOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteOtherOutputTopic.isEmpty()).isTrue(); }
28. Monitoring - Three Pillars observability 28 Tracing Logging Metrics (Aggregatable) (Events) (Request scoped) Micrometer + Prometheus Spring Cloud + Zipkin Sleuth
29. Monitoring – Metrics 29 • Out of the box Kafka metrics • consumers • producer • streams • Micrometer • Spring Boot Actuator Metrics
30. Monitoring – Metrics 30
31. Monitoring – Distributed tracing 31
32. Monitoring – Distributed tracing 32
33. 🎓 Lessons learned 33 • Invest time in your local development environment (fast feedback loop) • Use Spring Kafka but also understand the core Kafka APIs • Consumer: • Expect the unexpected! • Handle Deserialization exceptions a.k.a. ‘poison pills’ • Proper exception handling • Validate incoming data • Producer: • Don’t change your serializers • Leverage Apache Avro + Confluent Schema registry • Don’t break compatibility for your consumers!
34. 🎓 Lessons learned 34 • Security • Who can produce data to your topics? • Who can consume? • Monitor your topics, consumer groups & applications in production • Micrometer • Spring Cloud Sleuth • Don’t overdo integration test!
35. Questions 35 🤔 ❔
36. Thanks for joining my talk! @TimvanBaarsen https://www.medium.com/@TimvanBaarsen https://github.com/j-tim/jfall-2022-spring-kafka-beyond-the-basics