Key Insights from Using Kafka in Large-Scale Projects

Transcript

2. 2 WE ARE Follow us on LinkedIn:
3. 3 INGREDIENTS OF A WEBSHOP • Connecting Multiple Domains • Complex Integration Problem • 13 development teams • 28 Microservices and growing • > 50 Kafka Topics / Cluster
4. 4 ORCHESTRATION VS CHOREOGRAPHY Choreography is decoupled but can make debugging and control flow difficult to follow Orchestration is more observable, debuggable and centralized, but results in a single point of failure. Source: https://www.milanjovanovic.tech/blog/orchestration-vs-choreography
5. 5 ARCHITECTURE • Connecting Multiple Domains • Complex Integration Problem • 13 development teams • 28 Microservices and growing • > 50 Kafka Topics / Cluster
6. 6 8 TIPS HOW TO MAKE YOUR JOURNEY WITH KAFKA AS UNPLEASANT AS POSSIBLE Tested in the field
7. KAFKA IS LIKE ANY OTHER MESSAGING SYSTEM – HOW HARD CAN IT BE 1
8. 8 IS KAFKA LIKE ANY OTHER MESSAGING SYSTEM? • Kafka is very powerful but a complex beast to handle compared to AMQP, PubSub, JMS and co. • Developers have to deal with more low-level details like consumer groups, managing offsets, partitions, and error handling. • Migration from other messaging system takes time and careful planning • Design mistakes are expensive
9. 9 WHAT CAN WE DO ABOUT IT SETUP GOLDEN PATH (STANDARDS) • Provide application blueprints • Local playgrounds • Manage schema centrally • Define sensitive defauls LEARNING • Workshops for developers • Share pitfalls and your learnings with the team • Competence center
10. BUILD A DISTRIBUTED MONOLITH 2 Building an event based system but still maintaining strong dependencies between services
11. 11 HOW TO BUILD A DISTRIBUTED MONOLITH • Messages are not self-contained • Consumers have strong dependencies on the producers and other services • Tricky error handling (especially with stateless services) • Race conditions are very likely or strict versioning of entities is required • Internal DOS Attacks updateEvent
12. 12 WHAT CAN WE DO • Self Contained Messages • To not affect all consumers • Add additional Topics • Introduce additional event types on the same topic • Request-Reply Pattern SVC 1 SVC2 SVC4 SVC4
13. EVERYTHING IN KAFKA IS AN EVENT 3 Current IT trends and literature guides us into thinking that using Kafka is equal to building an event-based system always.
14. 14 ARE THERE ONLY EVENTS IN KAFKA? A pure “event-based” system comes with preconditions and design constraints. Example: Rendering Receipt Documents (Invoice) The side effect is - If I just create invoices on event states, I have to recreate the event state to retrigger invoices - This may have side effects on other things and is not clean (because it was an artificial state change) - And it will usually always be created unless I take extra measures - But it is very likely I want to recreate invoices at any time not just for specific events
15. 15 WHAT WE CAN DO ABOUT IT From EIP there are four message types to consider • Event Message (1:n) • Document Message (1:1) or (1:n) • Command Message (1:1) • Request-Reply Messages (1:1) And a combination thereof
16. 16 HOW WE SOLVED IT • Make deliberate choice what kind of message pattern you want to use (e.g. events, documents, commands) • Make the choice transparent, e.g. in the topic and record name. • Follow a consistent pattern for constructing the schema • Use consistent schema upgrade policy • FORWARD(_TRANSITIVE) is a good fit for ”event” and “document” topics, the producer owns the schema • BACKWARD(_TRANSITIVE) is a good fit for ”command” topics, the consumer owns the schema Make the message intent clear
17. THINK ABOUT KAFKA ERROR HANDLING ONLY IF (WHEN) YOU HAVE ERRORS IN PRODUCTION 4
18. 18 • We tend to optimize for the happy path • We assume, if something goes wrong we can fix it manually, like in the good old days with DBMS. • When error handling comes in place, things get more complicated IGNORE ERROR HANDLING
19. 19 WHAT WE CAN DO ABOUT IT • Decide on end-to-end delivery semantics early on (at-least-once, exactly-once, etc.) • Distinguish transient errors from permanent errors • Choose an error handling pattern • Is time critical? • Is it acceptable to lose messages? • Is ordering important? Make the message intent clear https://www.confluent.io/blog/error-handling-patterns-in-kafka/
20. 20 HOW WE SOLVED IT • Pause container listener and resume if underlying cause has been resolved A transient error, e.g. upstream dependency broken • Dead Letter Queue A permanent error, e.g. message validation fails • Transactional Inbox / Outbox for at-least-once delivery with idempotency (Idempotent receiver) keys and dead letter queue If order has to be preserved and messages must not be lost. • Keep it as simple as possible • Guidelines, not strict rules • Observability is key
21. ONE SCHEMA TO RULE THEM ALL 5 • You don’t need a schema • Just assume a schema will not change ever • Do not define a schema upgrade policy • Do not assign clear responsibilities for topics and schemes
22. 22 IS THERE ONLY ONE SCHEMA FOR A TOPIC ALWAYS? • Requirements are likely to change • Inventing schemes later is tricky • Schema registry is an add-on convention, so is it really needed?
23. 23 WHAT WE CAN DO ABOUT IT • Clear ownership of Topics / Schemes • All affected parties are visible • Define a strategy for compatible updates • Define a strategy for breaking changes • Set up a playbook • Have multiple stages to test it • Document Failures
24. 24 HOW WE SOLVED IT • Any (new) topic is assigned to a product team • This team can create the topic and propose a schema (upgrade) • All affected parties are invited to contribute and have to approve the MR for schema changes • Schema is rolled out to the schema registry and respective Java client libraries are deployed • Run the playbook, depending on the update strategy
25. 25 HOW WE SOLVED IT
26. ROLL A DICE TO DECIDE ON THE NUMBER OF PARTITIONS 6 • Better be on the safe side, more partitions are always better • Ignore the limit of 4000 partitions we can have per broker • Always create topics years prior to usage to reserve capacity
27. 27 DOES THE NUMBER OF PARTITIONS MATTER? • More partitions mean more overhead in compute (and memory) on the brokers • Can significantly impact costs • Will not automatically increase availability • Your Kafka colleagues will love you
28. 28 WHAT CAN WE DO ABOUT IT OPERATIONS • Define and enforce policies (e.g. Policy as Code) • Provide guidelines • Provide upscaling of partitions DEVELOPMENT TEAMS • Max consumers available in consumergroups • Consider data volume • Time criticality • Type of workload • Number of brokers available Uphold friendship with your Kafka Operations Team
29. 29 HOW WE SOLVED IT • Used low partition count on test environments • Apply standard setting of three partitions for a productive setup and 3 in-sync replicas (eq number of brokers) Unless • High data volume or dynamic load expected • Compute intense workloads
30. 30 HOW WE SOLVED IT To do leader election in a distributed system. The one consumer assigned to the partition is the elected leader. Deliberately choose a single partition
31. NEVER TOUCH DEFAULT SETTINGS 7
32. 32 • Leaving everything with default settings, we usually get an idempotent Producer -> exactly once semantics • Periodic Auto Commit Offset on Consumer End (e.g. every 5 seconds) independently of the actual Unit of Work -> something like maybe once or more. DEFAULT SETTINGS
33. 33 WHAT WE CAN DO ABOUT IT • Define a set of reasonable defaults for your use • Consider different default settings using different languages and Frameworks Some Hints Consumer: • enable.auto.commit • max.poll.records • max.poll.interval.ms • session.timeout.ms Schema Registry • auto.register.schemas • use.latest.version
34. I DESIGN MY KAFKA MESSAGES SO THAT I CAN LOG IT NICELY 6
35. 35 LOGGING STRATEGY • We log all produced events • We log all consumed events • We make sure the payload fits in the log message to not get truncated SVC 1 SVC2 Request: HTTP/POST /do/some/thing Response /do/some/thing SVC3 SVC4
36. 36 8) BUT, • It is expensive • It is just replicating the same thing • We litter the logs and blurr the important info
37. 37 WHAT WE CAN DO ABOUT IT • Use one id to correlate all logs • Use a tracing system to induce sub spans • Consistently for the scope of one business transaction SVC 1 SVC2 Request: HTTP/2 200 H: x3423452d /do/some/thing Response /do/some/thing SVC3 SVC4 x3423452 d x3423452 d Kafka Header x3423452d x3423452 d
38. 38 HOW WE SOLVED IT • Built Observability on • Monitoring • Logging • Traces • Request-id returns all logs accross all systems for a user-session • Trace-id for a request • Span-id enables timings between specific use cases SVC 1 SVC 1 Trace API Establish Context Request-id: 123 TraceId: 567/891 Call Trace API (sampled) Request-id: 123 TraceId: 567/891 Call Trace API (sampled) Request-id: 123 TraceId: 567/999 Call Trace API (sampled) Request-id: 123 TraceId: 567/555 Kafka Header: Request-id: 123 TraceId: 567/999 Monitoring Request-id: 123 TraceId: 567/555 Collect Kafka Metrics
39. 39 8) LOGS Logging-System Kafka-UI (AKHQ)
40. 40 8) TRACES
41. 41 MONITORING Monitoring Consumer Group Offsets for each Topic (including alerts)
42. 43 WHAT WE HAD TO LEAVE OUT A LOT MORE THINGS WE HAVE NOT COVERED • SubjectNamingStrategies • Implement Idempotency • Subject References • GDPR Concerns • DeletionPolicy • Certificate Handling • Flink Integration