4 Patterns to Jumpstart your Event-Driven Architecture Journey @ Current 2023, San Jose California

Abstract:

The shift from monolithic applications to microservices is anything but easy. Since services usually don't operate in isolation, it's vital to implement proper communication models among them. A crucial aspect in this regard is to avoid tight coupling and numerous point-to-point connections between any two services. One effective approach is to build upon messaging infrastructure as a decoupling element and employ an event-driven application architecture.

During this session, we explore selected event-driven architecture patterns commonly found in the field: the claim-check pattern, the content enricher pattern, the message translator pattern, and the outbox pattern. For each of the four patterns, we look into a live demo scenario based on Apache Kafka and discuss some variations and trade-offs regarding the chosen implementation.

You will walk away with a solid understanding of how the discussed event-driven architecture patterns help you with building robust and decoupled service-to-service communication and how to apply them in your next Apache Kafka-based project.

Demo Repository: https://github.com/hpgrahsl/eda-pattern-examples

Recording: https://www.confluent.io/events/current/2023/4-patterns-to-jumpstart-your-event-driven-architecture-journey/

Transcript

1. 4 Patterns to Jumpstart your Event-Driven Architecture Journey Hans-Peter Grahsl Developer 🥑 Advocate 1
2. Who am I ? 2 Developer 🥑 Advocate @ Red Hat based in Graz, Austria 󰎈 open-source enthusiast 5x Confluent Community Catalyst MongoDB Champion since 2020 speaker at dev & tech conferences @hpgrahsl@mastodon.social Hans-Peter Grahsl “I’m passionate about event-driven architectures, distributed stream processing systems and data engineering.”
3. Agenda 3 What we’ll discuss today ▸ Claim Check Pattern ▸ Content Enricher Pattern ▸ Message Translator Pattern ▸ Outbox Pattern
4. EDA Terminology 4 … is a way of designing loosely coupled services that respond (in near real-time) based on the sending and receiving of events. 🤔 Event-Driven Architecture (EDA) ?
5. EDA Terminology 5 Multiple flavours https://martinfowler.com/articles/201701-event-driven.html 1. Event Notification 2. Event-Carried State Transfer 3. Event Sourcing 4. CQRS (selectively combined with the above)
6. EDA Terminology 6 This talk’s meaning of Event-Driven Architecture https://martinfowler.com/articles/201701-event-driven.html 1. Event Notification 2. Event-Carried State Transfer 3. Event Sourcing 4. CQRS (selectively combined with the above) this talk’s meaning
7. It’s all about events … 7 EVENTS EVENTS EVERYWHERE
8. 8 One seeks bullets of silver … … but only finds trade-offs.
9. Agenda 9 Let’s explore the patterns… ▸ Claim Check Pattern ▸ Content Enricher Pattern ▸ Message Translator Pattern ▸ Outbox Pattern
10. Claim Check Pattern 10 ▸ inherently binary data graphics, documents, audio, video, … ▸ embedded data (e.g. Base64 encoded) in otherwise textual formats (json, xml, …) unnaturally large / unreasonably heavy event payloads Challenge
11. Claim Check Pattern 11 ▸ offload large (binary) payload into separate storage system ▸ replace actual payload with reference to claim check ▸ use claim check to fetch data when needed at consumer side externalize large event payloads … or parts thereof Claim Check Pattern
12. Claim Check Pattern 12
13. Claim Check Pattern 13
14. Claim Check Pattern 14
15. Claim Check Pattern 15
16. Claim Check Pattern 16
17. Claim Check Pattern 17
18. 18 Claim Check Pattern
19. Agenda 19 Let’s explore the patterns… ▸ Claim Check Pattern ▸ Content Enricher Pattern ▸ Message Translator Pattern ▸ Outbox Pattern
20. Content Enricher Pattern 20 ▸ payloads are lacking additional details ▸ certain consumers need to know more ▸ example: very compact notification events unnaturally small / unreasonably light event payloads Challenge
21. Content Enricher Pattern 21 ▸ add missing data e.g. include contextual information ▸ do lookups / perform joins against reference data ▸ take reference data to augment original payload increase the utility of sparse event payloads Content Enricher Pattern
22. Content Enricher Pattern 22
23. Content Enricher Pattern 23
24. Content Enricher Pattern 24
25. Content Enricher Pattern 25
26. Content Enricher Pattern 26
27. 27 Content Enricher Pattern
28. Agenda 28 Let’s explore the patterns… ▸ Claim Check Pattern ▸ Content Enricher Pattern ▸ Message Translator Pattern ▸ Outbox Pattern
29. Message Translator Pattern 29 ▸ examples: ･ integrate legacy systems ･ work with proprietary data formats ▸ minimize or at least reduce the contract-related coupling incompatible or inconvenient event payloads Challenge
30. Message Translator Pattern 30 ▸ perform a wide range of payload modifications: ･ basic field type conversions ･ rename, redact, or mask fields ･ switch format / protocol ･ … ▸ can also be used to build an anti-corruption layer establish compatibility between disparate systems Message Translator Pattern
31. Message Translator Pattern 31
32. Message Translator Pattern 32
33. Message Translator Pattern 33
34. Message Translator Pattern 34
35. Message Translator Pattern 35
36. Message Translator Pattern 36
37. 37 Message Translator Pattern
38. Agenda 38 Let’s explore the patterns… ▸ Claim Check Pattern ▸ Content Enricher Pattern ▸ Message Translator Pattern ▸ Outbox Pattern
39. Outbox Pattern 39 ▸ XA transactions ? potentially unreasonable choice or sometimes not available at all ▸ example: (micro)service updates its own database and sends event(s) to messaging/event streaming infra consistently update data in 2 different systems Challenge
40. Outbox Pattern 40 ▸ perform single database transaction spanning: ･ update(s) to affected DB tables (“domain data”) ･ update(s) to outbox DB table (“event data”) ▸ setup separate CDC pipeline for outbox table ･ transfer event data to messaging/event streaming infra ▸ consume CDC stream on-demand in near real-time avoid the dual-writes issue Outbox Pattern
41. Outbox Pattern 41
42. Outbox Pattern 42
43. Outbox Pattern 43
44. Outbox Pattern 44
45. Outbox Pattern 45
46. 46 Outbox Pattern
47. ▸ no silver bullets for event design / communication → only trade-offs ▸ patterns help to deal with implications and cope with trade-offs Wrap Up 47 Summary of discussed EDA patterns Claim Check Pattern Content Enricher Pattern Message Translator Pattern Outbox Pattern
48. Demo Repository 48 bit.ly/eda-patterns
49. Upcoming Talk 49 From edge to cloud, creating data pipelines using open-source with Strimzi on Kubernetes. Date: Wednesday, September 27 Time: 4:30 pm Location: Breakout Room 7 Speakers: Chris Cranford, Carles Arnal There will be a raffle with gifts at the end of the session!
50. linkedin.com/showcase/red-hat-developer youtube.com/RedHatDevelopers facebook.com/RedHatDeveloper twitter.com/rhdevelopers 50 Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you ! Questions ?
51. 51 https://unsplash.com/photos/DRzYMtae-vA https://pixabay.com/illustrations/night-sky-stars-moon-wolf-garou-4726445/ https://unsplash.com/photos/HTIduwcMMfQ https://unsplash.com/photos/w9rYOScXf6E https://unsplash.com/photos/iar-afB0QQw https://unsplash.com/photos/q65bNe9fW-w https://pixabay.com/photos/video-production-video-movie-film-4223885/ https://pixabay.com/photos/feather-bird-plumage-flying-4380861/ https://unsplash.com/photos/MceA9kSze0U https://pixabay.com/photos/travel-adapter-plug-charging-plug-3320763/ https://unsplash.com/photos/g97bviyFhvQ https://pixabay.com/illustrations/post-office-mailbox-old-yellow-2438804/ https://pixabay.com/photos/post-office-letter-box-mail-box-2455580/ https://pixabay.com/photos/ecommerce-shopping-credit-card-2607114/ in order of appearance Photo Credits