Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Spring Kafka beyond the basics - Lessons learned on our Kafka journey at ING Bank

Spring Kafka beyond the basics - Lessons learned on our Kafka journey at ING Bank

83ea5a35fafca28db79ff9752533de75?s=128

Tim van Baarsen

May 27, 2022
Tweet

Other Decks in Programming

Transcript

  1. Spring Kafka beyond the basics Lessons learned on our Kafka

    journey at ING Bank Tim van Baarsen SPRING I/O BARCELONA – MAY 27TH 2022
  2. Who am I? Tim van Baarsen Software Engineer @ ING

    Netherlands The Netherlands Team Dora Amsterdam
  3. ING is active in more than 40 counties 3

  4. Kafka @ ING 4 Frontrunners in Kafka Running in production:

    • 7 years • 5000+ topics • Serving 850+ Development teams • Self service topic management
  5. Kafka @ ING 5 0 100.000 200.000 300.000 400.000 500.000

    600.000 2015 2016 2017 2018 2019 2020 2021 2022 Messages produced per second (average) Messages produced per second (average) Traffic is growing with 10%+ monthly
  6. Agenda 6 • Kafka in a nutshell • Spring Kafka

    essentials • Scenario: Poison pill / Deserialization exceptions • Scenario: Lack of exception handling • Testing • Monitoring • Wrap-up • Questions
  7. Kafka in a nutshell 7 Consumer Consumer Producer Kafka client

    Kafka broker Consumer Kafka client 0100101001101 0100101001101 poll send Responsibilities: - subscribe - deserialization • key • Value - heartbeat Not responsible for: • Type checking • Schema validation • Other constraints Responsibilities: - send - serialization • key • value 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 Data in a Kafka topic are just stored as bytes! new Responsible for: • Append only log • Distribute • Replicate
  8. Kafka in a nutshell Consumer Consumer Producer Kafka client Kafka

    broker Consumer Kafka client 0100101001101 0100101001101 poll send 0 1 2 3 4 5 old 0 1 2 3 4 5 6 0 1 2 3 topic: ‘stock- quotes’ Partition 0 Partition 1 Partition 2 new Confluent Schema Registry REST API Load schema Responsible for: • Schema validation • Avro • Protobuf • Json schema KafkaAvroSerializer KafkaAvroDeserializer
  9. Spring for Apache Kafka - essentials 9 Kafka Kafka Clients

    Listener Container Error Handling Deserializer spring-kafka- test User Code Error Handler ConsumerRecord Recoverer KafkaTemplate @KafkaListener Consumer Producer Kafka Spring Kafka Your code @EmbeddedKafka https://spring.io/projects/spring-kafka Kafka Streams @EnableKafkaStreams
  10. Spring Kafka & Spring Boot - Producer 10 @Component @Slf4j

    public class StockQuoteProducer { @Autowired private KafkaTemplate<String, StockQuote> kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } }
  11. Spring Kafka & Spring Boot - Consumer 11 @Component @Slf4j

    public class StockQuoteProducer { @Autowired private KafkaTemplate<String, StockQuote> kafkaTemplate; public void produce(StockQuote stockQuote) { kafkaTemplate.send("stock-quotes", stockQuote.getSymbol(), stockQuote); log.info("Produced stock quote: {}", stockQuote); } } @Component @Slf4j public class StockQuoteConsumer { @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partition) { log.info("Consumed from partition: {} value: {}", partition, stockQuote); } }
  12. Spring Kafka & Spring Boot – Kafka streams 12 @Configuration

    @EnableKafkaStreams public class KafkaStreamsConfig { @Bean public KStream<String, StockQuote> kStream(StreamsBuilder streamsBuilder) { KStream<String, StockQuote> branchedStream = new KafkaStreamBrancher<String, StockQuote>() .branch((key, value) -> value.getExchange().equalsIgnoreCase("NYSE"), kStream -> kStream.to("stock-quotes-nyse")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("NASDAQ"), kStream -> kStream.to("stock-quotes-nasdaq")) .branch((key, value) -> value.getExchange().equalsIgnoreCase("AMS"), kStream -> kStream.to("stock-quotes-ams")) .defaultBranch(kStream -> kStream.to("stock-quotes-exchange-other")) .onTopOf(streamsBuilder.stream("stock-quotes")); return branchedStream; } }
  13. Spring Kafka & Spring Boot – Configuration (application.yml) 13 spring:

    application: name: producer-application kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer client-id: ${spring.application.name} properties: schema.registry.url: http://localhost:8081
  14. After the honeymoon phase is over 14 🧨 😱 🚀

    😎 Traps Lessons learned The hard way! Apply in your own project(s) Share with your fellow developers Tips & Tricks 👩💻‍‍
  15. 🚀 Tip: Local development setup 15 • Docker • Kafka

    Cluster • Zookeeper • Confluent Schema Registry • CLI Tools • Kafka CLI, Confluent CLI, Kafka cat • UI Tools • Confluent Control Center • Kafka UI • Conduktor • etc
  16. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 16 💊 💀 Corrupted

    record What is a Poison Pill? Deserialization failure A record that always fails when consumed, no matter how many times it is attempted. Different forms: ❓
  17. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 17 0 1 2

    3 4 5 6 7 8 old new Kafka topic: ‘stock-quotes’ Producer Kafka client Kafka broker Consumer Kafka client KafkaAvroDeserializer KafkaAvroSerializer StringSerializer 0100101001101 0100101001101 poll send Producer Kafka client send >_ 💊 Confluent Schema Registry REST API Load schema Register schema 010001101
  18. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 18 Scenario: • Consumer

    of topics • Someone produced a ‘poison’ message • Consumer fails to deserialize • Consequence • Blocks consumption of the topic/partition • application can’t ‘swallow the pill’ • Try again and again and again (very fast) • Log line for every failure AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’ Consumer Another Consumer (different application)
  19. 🧨 Scenario: ‘Poison Pill’ (Deserialization exception) 19 • Result: log

    file will grow very fast, flood your disc • Impact: High • How to survive this scenario? • Wait until the retention period of the topic has passed • Change consumer group • Manually / Programmatically update the offset • Configure ErrorHandlingDeserializer (Provided by Spring Kafka) 👎 👍 👎 👎 AAPL NASDAQ $243.65 INGA AEX €10.34 NFLX NASDAQ $280.30 INGA AEX €10.66 💊 PILL ? Consumer Kafka client Consumes from: Kafka topic: ‘stock-quotes’
  20. 🧨 Scenario: ‘Poison Pill’ - Demo 20

  21. 🧨 Scenario: Lack of proper exception handling 21 Scenario: •

    Consumer • Exception is thrown in the method handling the message Result (by default) • Records that fail are: • Retried • Logged • We move on to the next one Consequence • You lose that message! • Not acceptable in many use-cases! @KafkaListener(topics = "stock-quotes") public void on(StockQuote stockQuote) { if ("KABOOM".equalsIgnoreCase(stockQuote.getSymbol())) { throw new RuntimeException("Whoops something went wrong..."); } }
  22. 🧨 Scenario: Lack of proper exception handling 22 Impact •

    Dependents on your use-case How to survive this scenario? • Replace / configure DefaultErrorHandler by: • CommonLoggingErrorHandler • ContainerStoppingErrorHandler • Use backoff strategy for recoverable exceptions • Configure ConsumerRecordRecoverer • Dead letter topic • Implement your own recoverer
  23. 🧨 Scenario: Lack of proper exception handling - Demo 23

  24. Testing 24 • Support for Integration testing @EmbeddedKafka @EmbeddedKafka @SpringJUnitConfig

    public class EmbeddedKafkaIntegrationTest { @Autowired private EmbeddedKafkaBroker embeddedKafkaBroker; @Test public void yourTestHere() throws Exception { // TODO implement ;) } }
  25. Testing – Alternatives to EmbeddedKafka 25 • Test containers •

    Focus on unit test first • Kafka streams • Topology test driver
  26. Testing – Test topology test driver 26 @Test void stockQuoteFromAmsterdamStockExchangeEndUpOnTopicQuotesAmsTopic()

    { StockQuote stockQuote = new StockQuote("INGA", "AMS", "10.99", "EUR", "Description", Instant.now()); stockQuoteInputTopic.pipeInput(stockQuote.getSymbol(), stockQuote); assertThat(stockQuoteAmsOutputTopic.isEmpty()).isFalse(); assertThat(stockQuoteAmsOutputTopic.getQueueSize()).isEqualTo(1L); assertThat(stockQuoteAmsOutputTopic.readValue()).isEqualTo(stockQuote); assertThat(stockQuoteNyseOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteNasdaqOutputTopic.isEmpty()).isTrue(); assertThat(stockQuoteOtherOutputTopic.isEmpty()).isTrue(); }
  27. Monitoring - Three Pillars observability 27 Tracing Logging Metrics (Aggregatable)

    (Events) (Request scoped) Micrometer + Prometheus Spring Cloud + Zipkin Sleuth
  28. Monitoring – Metrics 28 • Out of the box Kafka

    metrics • consumers • producer • streams • Micrometer • Spring Boot Actuator Metrics
  29. Monitoring – Metrics 29

  30. Monitoring – Distributed tracing 30

  31. Monitoring – Distributed tracing 31

  32. 🎓 Lessons learned 32 • Invest time in your local

    development environment (fast feedback loop) • Use Spring Kafka but also understand the core Kafka APIs • Consumer: • Expect the unexpected! • Handle Deserialization exceptions a.k.a. ‘poison pills’ • Proper exception handling • Validate incoming data • Producer: • Don’t change your serializers • Leverage Apache Avro + Confluent Schema registry • Don’t break compatibility for your consumers!
  33. 🎓 Lessons learned 33 • Security • Who can produce

    data to your topics? • Who can consume? • Monitor your topics, consumer groups & applications in production • Micrometer • Spring Cloud Sleuth • Don’t overdo integration test!
  34. Questions 34 🤔 ❔

  35. Thanks for joining my talk! https://github.com/j-tim/spring-io-barcelona-2022-spring-kafka-beyond-the-basics @TimvanBaarsen https://www.medium.com/@TimvanBaarsen