Event Driven in the Wild with kafka - BeJUG

Transcript

1. 1 Jonas Geiregat Event Driven in the Wild with Kafka
2. How to avoid event horizon? ``In astrophysics, an event horizon is a boundary beyond which events cannot affect an outside observer`` -- wikipedia
3. Semantics the branch of linguistics and logic concerned with meaning
4. Event Driven Architecture (EDA)
5. Event Driven Architecture (EDA) Something that happened in the past Named as such - past tense Immutable Ubiquitous language
6. Given an unregistered customer When they complete their registration Then the CRM system is notified Name the event UserCreated CustomerDTO CustomerCreated UserCreated CustomerDTO CustomerCreated CustomerRegistered ✅
7. coarse-grained or fine-grained events ? Event Granularity
8. record UserUpdated( String id, UserStatus status, ... Too Coarse-Grained One service is only interested when the status goes to deactivated record UserDeactivated(String id) {}
9. record UserStreetNameChanged( String id, String streetName, ... Too Fine-Grained One service is only interested when a user his location changes record UserRelocated( String id, String streetName, …
10. Event Driven Architecture (EDA) Reactive Async Loose coupling Scalable
11. EDA enables looser coupling a minimum of coupling is required for two system to work together
12. Types of coupling • Operational - a system depends on the presence or operation of another system • Development - change in one system require changes in another • Semantic - systems share identical concepts • Functional - overlapping responsibilities • Temporal - dependency based on time or sequence • Stamp - share more information than necessary • Latent or Indirect - shared knowledge of a third system
13. 📦 order µservice 📦 shipping µservice HTTP Broker ShipOrder OrderSubmitted 📦 order µservice 📦 shipping µservice
14. Publishing events without caring or needing to know about the consumers is the theory, but unfortunately rarely the reality
15. Pubsub vs Messaging
16. Messaging Pubsub Point-to-Point One-to-Many 📦 µservice 📦 µservice 📦 µservice 📦 µservice 📦 µservice Queue Topic
17. Anatomy of a Kafka Topic 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 0 1 2 Topic with 3 partitions offset message (bytes) consists out of a key, payload and headers
18. Consumer Groups 📦 µservice 📦 µservice 📦 µservice topic 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 0 1 2 GroupA GroupB thread 1 : p0 thread 2 : p1 thread 3 : p2 thread 1 : p0 thread 2 : p1 thread 1: p2
20. How to organize for an Event Driven Architecture ?
21. How do you manage topics?
22. There are 2 hard problems in computer science: cache invalidation naming things -- Leon Bambrick especially topics -- Jonas
23. Establish a topic naming convention early on
24. DNS as a topic naming analogy broad scope narrow scope docs.google.com
25. DNS as a topic naming analogy narrow scope broad scope dev.orders.online.checkout.events
26. env.bounded-context.domain.aggregate-root.[type] dev.finance.fraud-detection.alert.events ^(dev|qa|prd)\.([a-z0-9-]+)\.([a-z0-9-]+)\.([a-z0-9-]+)\.(event|state|message)$
27. env.bounded-context.app-abbreviation.aggregate-root dev.boost.fa.report
28. env.bounded-context.app-abbreviation.aggregate-root dev.boost.fa.report
29. env.bounded-context.domain.aggregate-root.[type] dev.finance.fraud-detection.alert.events
30. env.bounded-context.domain.aggregate-root.[type] dev.finance.fraud-detection.alert.events dev.finance.fraud-detection.alert.messages dev.finance.fraud-detection.alert.state
31. env.bounded-context.domain.aggregate-root.[type] singular or plural ? dev.finance.fraud-detection.alerts.events
32. How do you create, update and delete topics?
33. Centralized vs Distributed approach
34. 🛠 IAC 📦 µservice 📦 µservice 📦 µservice Centralized
35. Centralized • central repository ◦ - one repository coupled to many others ◦ - blocking unaware gatekeeper risk ◦ ~ who is responsible ◦ - release coordination (more convoluted and risky) ◦ - does not scale well with larger number of teams and or services • + less complex with a minimum of teams and or services • + easier to enforce company wide rules
36. Distributed 🛠 IAC 🛠 IAC 🛠 IAC 📦 µservice 📦 µservice 📦 µservice 📦 µservice 📦 µservice 📦 µservice
37. Distributed • + logically coupled to the code using the topic • + high degree of autonomy • + scales well with larger number of teams or services • - distributed code/scripts/pipeline configs • ~ less easy to enforce company wide rules
38. @Bean public NewTopic orderTopic() { return TopicBuilder.name("procurement.order") .partitions(10) .replicas(3) .compact() .build(); } Distributed: from the application itself spring-kafka example
39. How long shall we retain our data?
40. Do you really want to limit yourself to the events published years ago?
41. CQRS and Event Sourcing 📦 command service 📦 query service 💾 DB Command Event Query Materialized 💾 DB
42. CQRS and Event Sourcing are NOT top-level architectures Only use Event Sourcing where you can make a competitive advantage
43. To use or not to use schemas?
44. The same question, centralized vs distributed applies to schemas
45. auto.register.schemas: false
46. AVSC { "type": "record", "name": "OrderCreated", "namespace": "com.example.events", "fields": [ { "name": "orderId", "type": "string" }, { "name": "customerName", "type": "string" }, { "name": "orderAmount", "type": "double" }, { "name": "price", "type": { "type": "record", "name": "Money", "fields": [ { "name": "amount", "type": "double" }, { "name": "currency", "type": "string" @namespace("com.example.events") protocol OrderEventProtocol { record Money { double amount; string currency; } record OrderCreated { string orderId; string? customerName; double orderAmount; Money price; } } AVDL Only needed within an RPC context
47. Distributed schema approach separate releasable maven module kafka-schema-registry-maven-plugin allows for validating and registering schemas avro-maven-plugin allows for generating java code from avdl mvn schema-registry:register -pl events -P dev
48. Distributed pipeline
49. spring-kafka or kafka-streams Which framework
50. Which framework kafka + IO trx synchronization retry support kafka to kafka exactly once kafka-streams spring-kafka dlt support stateful processing
51. Message Delivery Guarantees At most once At least once Exactly once 0 or 1 1+ 1 might be lost duplicates Risk Guarantee
52. Message Producing Guarantees At most once At least once Exactly once 0 or 1 1+ 1 never published duplicates Risk Guarantee
53. Kafka Poll Loop Poll X records Process X records Commit if auto.commit.interval.ms has elapsed and enable.auto.commit=true This provides an at-least-once delivery guarantee.
54. Kafka Transactions
55. Determine the transaction coordinator • _transaction_state internal topic for managing transactions • coordinator is determined by ◦ hash(transactional.id) / mod(len(__transaction_state partitions)) ◦ the resulting partition number, its leader will become the coordinator
56. Kafka Transaction Flow
57. isolation.level: read_committed default setting isolation.level: read_uncommitted
58. transactional.id must be unique for each producer instance kafka client producer property
59. spring.kafka.transaction-id-prefix: my-app- Enable transactions using spring-kafka with spring-boot autoconfigures a TransactionManager Which results in the following transactional.ids my-app-0 my-app-1 my-app-2 Ideal for one instance apps
60. Fencing: occurs when a new transaction with a higher epoch starts using the same transactional ID as an active transaction. ERROR 4395 --- [producer-my-app-0] o.a.k.clients.producer.internals.Sender : [Producer clientId=producer-my-app-0, transactionalId=my-app-0] Aborting producer batches due to fatal error org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.
61. spring.kafka.transaction-id-prefix: my-app-${random.uuid} Making sure multiple instance have an unique transaction.id Which results in the following transaction.ids my-app-4d567be0-3bb2-4b87-9b5d-a9440abd2dc9-0 my-app-4d567be0-3bb2-4b87-9b5d-a9440abd2dc9-1 my-app-4d567be0-3bb2-4b87-9b5d-a9440abd2dc9-2
62. Kafka has a lot similar properties of Database Infinite retention Transactions but that does not make it a database replacement
63. Client buys something on the website order service takes in the HTTP request and needs to notify the stock service
64. @RestController void handleOrderProductCreated(OrderProductRequest request) { orderService.orderProduct(request); } class OrderService { void orderProduct(OrderProductRequest request) { repository.save(toOrder(request)) kafkaTemplate.send(toOrderCreatedEvent(request)) } } } Not an atomic operation
65. @RestController void handleOrderProductCreated(OrderProductRequest request) { orderService.orderProduct(request); } class OrderService { @Transactional void orderProduct(OrderProductRequest request) { save(request) kafkaTemplate.send(toOrderCreatedEvent(request)) } } Annotations magically make things work .. or not? Request processing failed: org.springframework.beans.factory.NoUniqueBeanDefinitionExcepti on: No qualifying bean of type 'org.springframework.transaction.TransactionManager' available: expected single matching bean but found 2: transactionManager,kafkaTransactionManager
66. @RestController void handleOrderProductCreated(OrderProductRequest request) { orderService.orderProduct(request); } class OrderService { @Transactional(“transactionManager”) void orderProduct(OrderProductRequest request) { save(request) kafkaTemplate.send(toOrderCreatedEvent(request)) } } And the kafka transaction will magically also work ? Yes, because • KafkaTransactionManager synchronizes itself with the JpaTransactionManager • The JpaTransactionManager commits first • Upon success the KafkaTransactionManager commits
67. class OrderService { orderProduct(OrderProductRequest request) { eventPublisher.publish(request); orderRepo.save(request); } class EventPublisher { @Transactional("kafkaTransactionManager") publish(OrderProductRequest request) { this.kafkaTemplate.send(toOrderCreatedEvent(request)); } class OrderRepository { @Transactional("transactionManager") void save(OrderProductRequest request) { } Use nested methods to commit kafka before database
68. Transactional outbox pattern: db first @Post(“/orders”) { save(order) saveOrderCreatedEvent(order) } 📦 µservice 💾 DB
69. Transactional outbox pattern @Thread { event = readUnpublishedEvent() publish(event) markAsPublished(event) } 📦 µservice 💾 DB 1. Read Event 2. Publish Event
70. Listen to yourself 📦 µservice 💾 DB 1. HTTP 2. Produce 3. Consume 4. Store Eventually consistent
71. spring-kafka architecture kafka client library ListenerContainer consumer producer Your Own Code @KafkaListener enable.auto.commit=false ErrorHandling
72. Upon Created Orders send out shipment @Consumer(“orders-topic”) { publish(orderShipped) }
73. @KafkaListener void consumeOrderCreatedEvent(OrderCreatedEvent event) { orderService.shipOrder(event); } class OrderService { @Transactional("kafkaTransactionManager") publish(OrderProductRequest request) { this.kafkaTemplate.send(toOrderShipped(request)); } } With Spring Boot, it is only necessary to set the spring.kafka.producer.transaction-id-prefix property - Spring Boot will automatically configure a KafkaTransactionManager bean and wire it into the listener container. Unnecessary
74. Prepare for failures
75. Retries or not? @Bean DefaultErrorHandler handler() { ExponentialBackOffWithMaxRetries bo = new ExponentialBackOffWithMaxRetries(6); bo.setInitialInterval(1_000L); bo.setMultiplier(2.0); bo.setMaxInterval(10_000L); return new DefaultErrorHandler(myRecoverer, bo); } This will retry after 1, 2, 4, 8, 10, 10 seconds, before calling the recoverer.
76. Prepare or accept poison pills Dead Letter Topic (DLT)
77. DLT per consumer or topic • - Will double the number of topics and increase the number of partitions • + easier to manager when things go wrong DLT per service • - uses less topics and partitions • + harder to manage when things go wrong ◦ due to mixed formats ◦ due to mixed sources
78. DLTs are powerful, but without a clear plan to address issues, they can become a liability
80. 80 Jonas Geiregat Thanks for listening