4 Patterns to Jumpstart your Event-Driven Architecture Journey @ Current 2023, San Jose California

Abstract:

The shift from monolithic applications to microservices is anything but easy. Since services usually don't operate in isolation, it's vital to implement proper communication models among them. A crucial aspect in this regard is to avoid tight coupling and numerous point-to-point connections between any two services. One effective approach is to build upon messaging infrastructure as a decoupling element and employ an event-driven application architecture.

During this session, we explore selected event-driven architecture patterns commonly found in the field: the claim-check pattern, the content enricher pattern, the message translator pattern, and the outbox pattern. For each of the four patterns, we look into a live demo scenario based on Apache Kafka and discuss some variations and trade-offs regarding the chosen implementation.

You will walk away with a solid understanding of how the discussed event-driven architecture patterns help you with building robust and decoupled service-to-service communication and how to apply them in your next Apache Kafka-based project.

Demo Repository: https://github.com/hpgrahsl/eda-pattern-examples

Recording: https://www.confluent.io/events/current/2023/4-patterns-to-jumpstart-your-event-driven-architecture-journey/

Transcript

1. 4 Patterns
   to Jumpstart your Event-Driven
   Architecture Journey
   Hans-Peter Grahsl
   Developer 🥑 Advocate
   1
   

   View Slide

2. Who am I ?
   2
   Developer 🥑 Advocate @ Red Hat
   based in Graz, Austria 󰎈
   open-source enthusiast
   5x Confluent Community Catalyst
   MongoDB Champion since 2020
   speaker at dev & tech conferences
   @hpgrahsl@mastodon.social
   Hans-Peter Grahsl
   “I’m passionate about event-driven
   architectures, distributed stream
   processing systems and data
   engineering.”
   

   View Slide

3. Agenda
   3
   What we’ll
   discuss today
   ▸ Claim Check Pattern
   ▸ Content Enricher Pattern
   ▸ Message Translator Pattern
   ▸ Outbox Pattern
   

   View Slide

4. EDA Terminology
   4
   … is a way of designing loosely coupled services
   that respond (in near real-time) based on the
   sending and receiving of events.
   🤔 Event-Driven Architecture (EDA) ?
   

   View Slide

5. EDA Terminology
   5
   Multiple flavours
   https://martinfowler.com/articles/201701-event-driven.html
   1. Event Notification
   2. Event-Carried State Transfer
   3. Event Sourcing
   4. CQRS (selectively combined with the above)
   

   View Slide

6. EDA Terminology
   6
   This talk’s meaning of Event-Driven Architecture
   https://martinfowler.com/articles/201701-event-driven.html
   1. Event Notification
   2. Event-Carried State Transfer
   3. Event Sourcing
   4. CQRS (selectively combined with the above)
   this
   talk’s
   meaning
   

   View Slide

7. It’s all about events …
   7
   EVENTS
   EVENTS EVERYWHERE
   

   View Slide

8. 8
   One seeks bullets of silver …
   … but only finds trade-offs.
   

   View Slide

9. Agenda
   9
   Let’s explore
   the patterns…
   ▸ Claim Check Pattern
   ▸ Content Enricher Pattern
   ▸ Message Translator Pattern
   ▸ Outbox Pattern
   
   

   View Slide

10. Claim Check Pattern
    10
    ▸ inherently binary data
    graphics, documents, audio, video, …
    ▸ embedded data (e.g. Base64 encoded)
    in otherwise textual formats (json, xml, …)
    unnaturally large / unreasonably heavy event payloads
    Challenge
    

    View Slide

11. Claim Check Pattern
    11
    ▸ offload large (binary) payload
    into separate storage system
    ▸ replace actual payload
    with reference to claim check
    ▸ use claim check to fetch data
    when needed at consumer side
    externalize large event payloads … or parts thereof
    Claim Check Pattern
    

    View Slide

12. Claim Check Pattern
    12
    

    View Slide

13. Claim Check Pattern
    13
    

    View Slide

14. Claim Check Pattern
    14
    

    View Slide

15. Claim Check Pattern
    15
    

    View Slide

16. Claim Check Pattern
    16
    

    View Slide

17. Claim Check Pattern
    17
    

    View Slide

18. 18
    Claim Check Pattern
    

    View Slide

19. Agenda
    19
    Let’s explore
    the patterns…
    ▸ Claim Check Pattern
    ▸ Content Enricher Pattern
    ▸ Message Translator Pattern
    ▸ Outbox Pattern
    
    

    View Slide

20. Content Enricher Pattern
    20
    ▸ payloads are lacking additional details
    ▸ certain consumers need to know more
    ▸ example: very compact notification events
    unnaturally small / unreasonably light event payloads
    Challenge
    

    View Slide

21. Content Enricher Pattern
    21
    ▸ add missing data e.g. include contextual information
    ▸ do lookups / perform joins
    against reference data
    ▸ take reference data to augment original payload
    increase the utility of sparse event payloads
    Content Enricher Pattern
    

    View Slide

22. Content Enricher Pattern
    22
    

    View Slide

23. Content Enricher Pattern
    23
    

    View Slide

24. Content Enricher Pattern
    24
    

    View Slide

25. Content Enricher Pattern
    25
    

    View Slide

26. Content Enricher Pattern
    26
    

    View Slide

27. 27
    Content Enricher Pattern
    

    View Slide

28. Agenda
    28
    Let’s explore
    the patterns…
    ▸ Claim Check Pattern
    ▸ Content Enricher Pattern
    ▸ Message Translator Pattern
    ▸ Outbox Pattern
    
    

    View Slide

29. Message Translator Pattern
    29
    ▸ examples:
    ･ integrate legacy systems
    ･ work with proprietary data formats
    ▸ minimize or at least reduce
    the contract-related coupling
    incompatible or inconvenient event payloads
    Challenge
    

    View Slide

30. Message Translator Pattern
    30
    ▸ perform a wide range of payload modifications:
    ･ basic field type conversions
    ･ rename, redact, or mask fields
    ･ switch format / protocol
    ･ …
    ▸ can also be used to build an anti-corruption layer
    establish compatibility between disparate systems
    Message Translator Pattern
    

    View Slide

31. Message Translator Pattern
    31
    

    View Slide

32. Message Translator Pattern
    32
    

    View Slide

33. Message Translator Pattern
    33
    

    View Slide

34. Message Translator Pattern
    34
    

    View Slide

35. Message Translator Pattern
    35
    

    View Slide

36. Message Translator Pattern
    36
    

    View Slide

37. 37
    Message Translator Pattern
    

    View Slide

38. Agenda
    38
    Let’s explore
    the patterns…
    ▸ Claim Check Pattern
    ▸ Content Enricher Pattern
    ▸ Message Translator Pattern
    ▸ Outbox Pattern
    
    

    View Slide

39. Outbox Pattern
    39
    ▸ XA transactions ?
    potentially unreasonable choice
    or sometimes not available at all
    ▸ example: (micro)service updates its own database
    and sends event(s) to messaging/event streaming infra
    consistently update data in 2 different systems
    Challenge
    

    View Slide

40. Outbox Pattern
    40
    ▸ perform single database transaction spanning:
    ･ update(s) to affected DB tables (“domain data”)
    ･ update(s) to outbox DB table (“event data”)
    ▸ setup separate CDC pipeline for outbox table
    ･ transfer event data to messaging/event streaming infra
    ▸ consume CDC stream on-demand in near real-time
    avoid the dual-writes issue
    Outbox Pattern
    

    View Slide

41. Outbox Pattern
    41
    

    View Slide

42. Outbox Pattern
    42
    

    View Slide

43. Outbox Pattern
    43
    

    View Slide

44. Outbox Pattern
    44
    

    View Slide

45. Outbox Pattern
    45
    

    View Slide

46. 46
    Outbox Pattern
    

    View Slide

47. ▸ no silver bullets for event design / communication → only trade-offs
    ▸ patterns help to deal with implications and cope with trade-offs
    Wrap Up
    47
    Summary of discussed EDA patterns
    Claim Check Pattern Content Enricher Pattern Message Translator Pattern Outbox Pattern
    

    View Slide

48. Demo Repository
    48 bit.ly/eda-patterns
    

    View Slide

49. Upcoming Talk
    49
    From edge to cloud, creating data pipelines
    using open-source with Strimzi on Kubernetes.
    Date: Wednesday, September 27
    Time: 4:30 pm
    Location: Breakout Room 7
    Speakers: Chris Cranford, Carles Arnal
    There will be a raffle with gifts
    at the end of the session!
    

    View Slide

50. linkedin.com/showcase/red-hat-developer
    youtube.com/RedHatDevelopers
    facebook.com/RedHatDeveloper
    twitter.com/rhdevelopers
    50
    Red Hat is the world’s leading provider of enterprise
    open source software solutions. Award-winning
    support, training, and consulting services make
    Red Hat a trusted adviser to the Fortune 500.
    Thank you !
    Questions ?
    

    View Slide

51. 51
    https://unsplash.com/photos/DRzYMtae-vA
    https://pixabay.com/illustrations/night-sky-stars-moon-wolf-garou-4726445/
    https://unsplash.com/photos/HTIduwcMMfQ
    https://unsplash.com/photos/w9rYOScXf6E
    https://unsplash.com/photos/iar-afB0QQw
    https://unsplash.com/photos/q65bNe9fW-w
    https://pixabay.com/photos/video-production-video-movie-film-4223885/
    https://pixabay.com/photos/feather-bird-plumage-flying-4380861/
    https://unsplash.com/photos/MceA9kSze0U
    https://pixabay.com/photos/travel-adapter-plug-charging-plug-3320763/
    https://unsplash.com/photos/g97bviyFhvQ
    https://pixabay.com/illustrations/post-office-mailbox-old-yellow-2438804/
    https://pixabay.com/photos/post-office-letter-box-mail-box-2455580/
    https://pixabay.com/photos/ecommerce-shopping-credit-card-2607114/
    in order of appearance
    Photo Credits
    

    View Slide