3 Event-Driven Architecture Patterns in Action Hans-Peter Grahsl Developer Advocate Red Hat @hpgrahsl

What we’ll be discussing today AGENDA 2 EDA and Events Claim Check Pattern Content Enricher Pattern Message Translator Pattern

OpenShift Developer Sandbox! Get free access for renewable 30 days to a self-service, cloud-hosted Kubernetes experience with Developer Sandbox for Red Hat OpenShift. red.ht/sandbox4all

OpenShift Developer Sandbox! red.ht/sandbox4all 👉 When signing up, use this code: 2zecv 👈

🤔 Event-Driven Architecture (EDA) ? EDA Intro 5 Event-Driven Architecture (EDA) is a way of designing loosely coupled applications and services that respond in near real-time based on the sending and receiving of events.

This talk’s meaning of EDA EDA Intro 6

Multiple “flavours” of Event-Driven Architecture EDA Intro 7 https://martinfowler.com/articles/201701-event-driven.html

8 EVENTS EVENTS EVERYWHERE

Events and Event Design Events … or “something has happened (somewhere)” 9 ● order created (order service) ● order created with id ‘123’ (order service) ● order created with id ‘123’, and N products (order service) ● order created with id ‘123’, N products, and total amount of M (order service) ● … ● order created with id ‘123’, product A, product B,... , total amount of M, , customer with id ‘XYZ’, and address ‘foo blah’, … (order service)

Events and Event Design 10 🤔 Where does this end? 🤔 🛑 When should this stop? 🛑

Purposeful Event Design → Event Type Examples Events and Event Design 11

Events and Event Design 12 One seeks bullets of silver … … but only finds trade-offs.

Up next… AGENDA 13 Claim Check Pattern Content Enricher Pattern Message Translator Pattern

Challenge Claim Check Pattern 14 ● (unnaturally / too) heavy payloads ? raw / binary files graphics, documents, audio, video, … embedded data (base64) within structured text formats (xml, json,…)

Claim Check Pattern Claim Check Pattern 15 ● offload large (binary) payload into separate storage system ● replace actual payload with reference to claim check ● use claim check to fetch data when needed at consumer side

Demo Scenario Claim Check Pattern 16 ● context: media processing pipeline ● images sent between services ● communication via Kafka topics ● external storage system MinIO

Claim Check Pattern 17 Claim Check Pattern

Claim Check Pattern 18 Claim Check Pattern

Claim Check Pattern 19 Claim Check Pattern

Claim Check Pattern 20 Claim Check Pattern

Claim Check Pattern 21 Claim Check Pattern

Claim Check Pattern 22 Claim Check Pattern

Implementation Details Claim Check Pattern 23 ● Kafka producer/consumer apps written with Quarkus ● offloading binary data delegated to specific Serde ● MinIO (S3-compatible) acts as storage system → reasonably-sized payloads in order not to pollute event streaming platform with tons of binary data

24 Claim Check Pattern

Discussion Claim Check Pattern 25 ● Serde for configurable data offloading with ease ● in Kafka Connect context → apply Converter ● think about object storage id generation choices ● currently full payload only → no field-level scope ● to some degree it’s a dual-writes problem

Up next… AGENDA 26 Claim Check Pattern Content Enricher Pattern Message Translator Pattern

Challenge Content Enricher Pattern 27 ● (unnaturally / too) light payloads ? payloads lacking additional details certain consumers need to know more e.g. due to very compact notification events

Content Enricher Pattern Content Enricher Pattern 28 ● add data to original payload ● perform lookups or do joins against reference data ● take reference data to augment original payload with additional information

Demo Scenario Content Enricher Pattern 29 ● context: IoT sensor data processing ● raw stream contains sensor measurements only ● certain consumers need device meta-data ● sensor stream is enriched with missing data

Content Enricher Pattern 30

Content Enricher Pattern 31

Content Enricher Pattern 32

Content Enricher Pattern 33

Content Enricher Pattern 34

Content Enricher Pattern 35

Implementation Details Content Enricher Pattern 36 ● device data resides in MySQL table ● Kafka data ingestion by means of CDC → Debezium ● Quarkus producer app simulates random sensor data ● KStreams app joins sensor data with device data ● enriched data continuously written to separate topic → consumers can access and directly work with enriched data

Content Enricher Pattern

Discussion Content Enricher Pattern 38 ● augmentation possible with different tech stacks ● stream processing lib/framework recommended ● alternatively SQL abstractions might be beneficial ● enrichment kept separate from consumer ● reference data local in/close to processing app

Up next… AGENDA 39 Claim Check Pattern Content Enricher Pattern Message Translator Pattern

Challenge Message Translator Pattern 40 inconvenient / incompatible payloads ? ● minimize or at least reduce the contract-related coupling ● examples: ○ integrate legacy systems ○ work with proprietary data formats ○ prevent leakage of domain models

Message Translator Pattern Message Translator Pattern 41 ● perform a wide range of payload modifications ○ basic field type conversions ○ rename, redact, or mask fields ○ complete format/protocol change ○ … ● entity modifications between bounded contexts ● ideally “self-contained” operations only

Demo Scenario Message Translator Pattern ● context: point-of-sales record processing ● files (source) with data in proprietary CSV format ● web service (sink) expecting specific JSON format ● streaming data pipeline with in-flight transformations 42

Message Translator Pattern 43

Message Translator Pattern 44

Message Translator Pattern 45

Message Translator Pattern 46

Message Translator Pattern 47

Message Translator Pattern 48

Implementation Details Message Translator Pattern 49 ● streaming data pipeline with Kafka Connect ○ file source connector processing CSV files ○ http sink connector feeding JSON into web API ● message translation with Kafka Connect SMT chain ○ drop and rename fields ○ field type conversions → enabler for data integration between disparate systems

Message Translator Pattern 50

Discussion Message Translator Pattern 51 ● based on Kafka Connect → config only ● supports many different data sources / sinks ● various on-the-fly transformations → SMT chains ● translations on either/both sides of the data flow ● no direct support for stateful processing

Summary of Observations in EDA EDA Summary 52 ● looser coupling, better extensibility, higher independence ● good event design needs collaboration + domain expertise ● no silver bullets for event design → trade-offs ● patterns help to deal with implications / to cope with trade-offs

Demo Repository red.ht/eda-patterns

OpenShift Developer Sandbox! red.ht/sandbox4all 👉 When signing up, use this code: 2zecv 👈

linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Red Hat is the world’s leading provider of enterprise open source software solutions. Award-winning support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500. Thank you ! 55

Photo Credits in order of appearance 56 https://unsplash.com/photos/DRzYMtae-vA https://pixabay.com/illustrations/night-sky-stars-moon-wolf-garou-4726445/ https://unsplash.com/photos/HTIduwcMMfQ https://unsplash.com/photos/w9rYOScXf6E https://unsplash.com/photos/iar-afB0QQw https://unsplash.com/photos/q65bNe9fW-w https://pixabay.com/photos/video-production-video-movie-film-4223885/ https://pixabay.com/photos/feather-bird-plumage-flying-4380861/ https://unsplash.com/photos/MceA9kSze0U https://pixabay.com/photos/travel-adapter-plug-charging-plug-3320763/ https://unsplash.com/photos/g97bviyFhvQ