Apidays Paris 2023 - Kafka in Event-Driven Architectures, Hassane Moustapha, ENGIE

Apidays Paris 2023 - Software and APIs for Smart, Sustainable and Sovereign Societies
December 6, 7 & 8, 2023

Kafka in Event-Driven Architectures
Hassane Moustapha, Senior Technical Architect, ENGIE

------

Check out our conferences at https://www.apidays.global/

Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8

Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io

Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/

Transcript

1. Kafka in Event Driven Architectures Hassane Moustapha 2023
2. Agenda EDA - Kesako ? EDA - Component, Benefits & Implementations EDA - Real-life use case EDA - How Kafka fits in 2023
3. Hassane Moustapha @_hkairi www.goscreencasts.io
4. Career path Developer NIC, XONE-TECH, ORANGE CTO NELAMSERVICES, Dakar Sr Software Architect NETICOA, Dakar 2018 HEAD OF DIGITAL DEVELOPMENT BANK OF AFRICA GROUP, Dakar - Casablanca 2023 Sr Technical Architect Engie Solutions France, Paris
5. What is Event-Driven Architecture (EDA)? Event-Driven Architecture (EDA) is a software design paradigm that emphasizes the production, detection, and reaction to events. An event is any significant occurrence or change in the state of a system. In EDA, events drive the flow of information and control between components, allowing for a more dynamic and responsive system.
6. Key components of Event-Driven Architecture Event producers: These are the components responsible for generating events. They can be user interfaces, sensors, or other external systems that produce events when their state changes or when they perform a particular action. Event consumers: These components listen for and react to events. They can be responsible for processing the events, storing data, or triggering further actions. Event channels: These are the communication channels that enable the flow of events between producers and consumers. They can be message brokers, queues, or other middleware that ensure the reliable and efficient delivery of events. Event store: An optional component in some EDA systems, it is responsible for persisting the event data. This can be useful for auditing, debugging, or analytics purposes.
7. Producer Consumer Channel Store Key components of Event-Driven Architecture
8. Real life example
9. “We want our customers to be able to send money to their loved ones, families and friends even if they don't have a bank account.”
10. “We want our customers to be able to send money to their loved ones, families and friends even if they don't have a bank account.”
11. Use case: Cash out - v1 Core Cash out Notific- ations Analytics Web Banking Event Bounded Context Command / Message
12. Use case: Cash out - v1 1 - Cash out Order Core Cash out Notific- ations Analytics Web Banking Event Bounded Context Command / Message
13. Use case: Cash out - v1 1 - Cash out Order 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking Event Bounded Context Command / Message
14. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking Event Bounded Context Command / Message
15. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 4 - withdrawal Ref generated Event Bounded Context Command / Message
16. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 4 - withdrawal Ref generated Command / Message Event 5 - Send Ref to Beneficiary Bounded Context
17. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 6 - Send Email 4 - withdrawal Ref generated Event 5 - Send Ref to Beneficiary Bounded Context Command / Message
18. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 6 - Send Email 7 - In-App Notification 4 - withdrawal Ref generated Event 5 - Send Ref to Beneficiary Bounded Context Command / Message
19. Use case: Cash out - v1 1 - Cash out Order 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 6 - Send Email 7 - In-App Notification 8 - Money Withdrawn 4 - withdrawal Ref generated Event 5 - Send Ref to Beneficiary Bounded Context Command / Message
20. Implementing EDA in your organization: The Process 1. Identify the events: Determine the significant events that occur within your system and decide which components should produce and consume them. 2. Design the event schema: Define the structure and format of your events to ensure consistency and compatibility between components. 3. Select the right event channels: Choose the appropriate communication channels for your events, such as message brokers or streaming platforms. 4. Implement the components: Develop the components that will produce and consume events, following best practices for decoupling and fault tolerance. 5. Test and monitor: Test your EDA implementation to ensure it meets your performance and reliability requirements, and implement monitoring and alerting to detect and resolve issues quickly. 6. Remember that you can enrich the events later if necessary ;-)
21. How Kafka fits in
22. Producer Consumer Channel Store Key components of Event-Driven Architecture
23. Producer Consumer Channel Store Key components of Event-Driven Architecture
24. Producer Consumer Channel Store Key components of Event-Driven Architecture Producer API
25. Producer Consumer Channel Store Key components of Event-Driven Architecture Producer API
26. Producer Consumer Channel Store Key components of Event-Driven Architecture Producer API Consumer API
27. Producer Consumer Channel Store Key components of Event-Driven Architecture Producer API Consumer API
28. Producer Consumer Channel Store Key components of Event-Driven Architecture Producer API Consumer API Kafka Broker
29. Producer Consumer Key components of Event-Driven Architecture Producer API Consumer API Kafka Broker
30. Use case: Cash out - v2 3- Cash out Order Created 2 - Cash out Order Core Cash out Notific- ations Analytics Web Banking 6 - Send Email 7 - In-App Notification 8 - Money Withdrawn 4 - withdrawal Ref generated Event 5 - Send Ref to Beneficiary Bounded Context 1 - Cash out Order Command / Message
32. Producer Consumer Close to the objective Producer API Consumer API Kafka Broker Business Process Monitoring Data enrichment Fraud Detection Monitoring Kafka Streams
33. Some advantages of Event-Driven Architecture • Decoupling ◦ Adding / changing components is easier • Scale better ◦ Each component can be scaled independently • Fault tolerance ◦ If one of the components fails, we can reprocess the event and the response to the client will be less affected • Close to Real time ◦ Things are running asynchronously
34. Other aspects • Dead Letter Queues • Topics • Event Notification • Event-Carried State Transfer • Serialization & Schema evolution ◦ Avro, Json, Protobuf • Security & Monitoring
35. Thank you @_hkairi | www.goscreencasts.io