Slide 1

Slide 1 text

Learn Kafka and event-driven architecture Vinay Kumar @vinaykuma201

Slide 2

Slide 2 text

• ORACLE ACE • Global Integration Architect • Author of “Beginning Oracle WebCenter portal 12c” • Blogger- http://www.techartifact.com/blogs • https://www.linkedin.com/in/vinaykumar2/ 2 About me

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Agenda 4 • Interaction Style • Traditional SOA approach • Event driven Architecture • Event with Microservices • Event streaming with event Hub • Integration with legacy. • Kafka • Kafka internals • Async APIs

Slide 5

Slide 5 text

Communication Styles 5 Type of Interaction Initiator Participants Time-driven Time The specifice system Request-driven Client Client & Server Event-driven event Open-ended

Slide 6

Slide 6 text

Time Driven 6 Store System Run in every 30 min to check the inventory

Slide 7

Slide 7 text

Request-driven 7

Slide 8

Slide 8 text

Event-driven 8 Store System “Inventory updated” “Inventory is low”

Slide 9

Slide 9 text

9 Document Title - Name - Function - Business Unit DD/MM/YYYY Event Notification

Slide 10

Slide 10 text

• Anything happened (or didnt happen). • A change in the state. • An event is always named in the past tense and is immutabled • A condition that triggers a notification. CustomerAddressChanged InventoryUpdated SalesOrderCreated PurschaseOrderCreated 10 Events

Slide 11

Slide 11 text

• “Real-time” events as they happen at the producer • Push notifications • One-way “fire-and-forget” • Immediate action at the consumers • Informational (“someone logged in”), not commands (“audit this”) 11 Characterstics of Events

Slide 12

Slide 12 text

Typical EDA Architecture Event Bus System System System System System System Event Producers Event Transport Event Consumer

Slide 13

Slide 13 text

• Supports the business demands for better service (no batch, less waiting) • No point-to-point integrations (fire & forget) • Fault tolerance, scalability, versatility, and other benefits of loose coupling. • Powerful real-time response and analytics. • Greater operational efficiencies 13 benefits of EDA

Slide 14

Slide 14 text

Where we do come from? 1 4

Slide 15

Slide 15 text

15 Document Title - Name - Function - Business Unit DD/MM/YYYY Monolithic Shop Customer Customer Inventory Payment Might be individual bounded context in new world

Slide 16

Slide 16 text

16 Document Title - Name - Function - Business Unit DD/MM/YYYY Traditional Archiecture (Point to Point) Shop Customer Marketing Inventory Payment Reporting Inventory

Slide 17

Slide 17 text

17 Document Title - Name - Function - Business Unit DD/MM/YYYY Traditional Archiecture- ESB Shop Customer Marketing Inventory Payment Reporting Enterprise Service Bus

Slide 18

Slide 18 text

18 Document Title - Name - Function - Business Unit DD/MM/YYYY Traditional Archiecture- ESB Shop Customer Marketing Inventory Payment Reporting Lets add some fraud check and new version V1 V2 Enterprise Service Bus fraud fraud

Slide 19

Slide 19 text

19 Document Title - Name - Function - Business Unit DD/MM/YYYY Lets add loyalty features Shop Customer Marketing Inventory Payment Reporting V1 V2 Enterprise Service Bus fraud fraud Loyalty

Slide 20

Slide 20 text

20 Document Title - Name - Function - Business Unit DD/MM/YYYY When we scale up and see Integration ripple effect Shop Customer Marketing Inventory Payment Reporting V1 V2 Enterprise Service Bus fraud fraud Loyalty Loyalty

Slide 21

Slide 21 text

• SOA is all about dividing domain logic into separate systems and exposing it as services • In some cases business logic will, by its very nature, be spread out over many systems or across various domain (cross domain). • The result is domain pollution and bloat in the SOA systems. Whats the problem

Slide 22

Slide 22 text

22 Document Title - Name - Function - Business Unit DD/MM/YYYY

Slide 23

Slide 23 text

• Domain driven design promote the business logic to expose as a service and focus should be on domain and domain logic. • Domain event is an activity happened that domain expert is concerned. • By exposing relevant Domain Events on a shared event bus we can isolate cross cutting functions to separate systems SOA and Domain events

Slide 24

Slide 24 text

SOA + Domain events Shop Customer Marketing Reporting Inventory Payment Event Bus Loyalty Fraud BAM

Slide 25

Slide 25 text

“You make me complete” SOA EDA

Slide 26

Slide 26 text

Event Driven (Async) in Microservices Shop DB Shop logic Shop API Customer DB Customer logic Customer API Payment DB Payment logic Payment API Event Hub Shop Microservices Payment Microservices Customer Microservices Producer, Consumer Consumer Consumer

Slide 27

Slide 27 text

27 Document Title - Name - Function - Business Unit DD/MM/YYYY Event Streaming Data Source Shop DB Shop logic Shop API Customer DB Customer logic Customer API Payment DB Payment logic Payment API Event Hub Shop Microservices Payment Microservices Customer Microservices Producer, Consumer Consumer Consumer Mobile Apps Social Media Stocks Blockchain Location IOT Events Streaming

Slide 28

Slide 28 text

28 Document Title - Name - Function - Business Unit DD/MM/YYYY Microservice events and Streaming processing State Microservice API Microservices Cluster Mobile Apps Social Media Stocks Blockchain Location IOT Events Stream Event Hub Events Stream Stream Processing Cluster Stream Analytics dashboard Events Stream Reference Model Results BI tools Search/Discover Mobile & online apps SQL Service Service Service

Slide 29

Slide 29 text

• Domain event- In domain-driven design, domain events are described as something that happens in the domain and is important to domain experts. - A user has registered - An order has been cancelled. - The payment has been received Domain events are relevant both within a bounded context and across bounded contexts for implementing processes within the domain. Best for communication between bounded context. 29 Document Title - Name - Function - Business Unit DD/MM/YYYY Domain event and event sourcing ■ Event Sourcing - Event Sourcing ensures that all changes to application state are stored as a sequence of events. It store the events that lead to specific state and state too. - MobileNumberProvided (MobileNumber) - VerificationCodeGenerated (VerificationCode) - MobileNumberValidated (no additional state) - UserDetailsProvided (FullName, Address, …) These events are sufficient to reconstruct the current state of the UserRegistration aggregate at any time. Event Sourcing is for persistent strategy. Event Sourcing makes it easier to fix inconsistencies. Event Sourcing is local for a domain.

Slide 30

Slide 30 text

Why Kafka for event-driven? 3 0

Slide 31

Slide 31 text

31 Document Title - Name - Function - Business Unit DD/MM/YYYY

Slide 32

Slide 32 text

32 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka Overview • Distributed publish-subscribe messaging system. • Designed for processing of real time activity stream data (log, metrics, collections, social media streams,…..) • Does not use JMS API and standards • Kafka maintains feeds of message in topics • Initially developed at Linkedin, now part of Apache.

Slide 33

Slide 33 text

33 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka History

Slide 34

Slide 34 text

• Reliability. Kafka is distributed, partitioned, replicated, and fault tolerant. Kafka replicates data and is able to support multiple subscribers. Additionally, it automatically balances consumers in the event of failure. • Scalability. Kafka is a distributed system that scales quickly and easily without incurring any downtime. • Durability. Kafka uses a distributed commit log, which means messages persists on disk as fast as possible providing intra-cluster replication, hence it is durable. • Performance. Kafka has high throughput for both publishing and subscribing messages. It maintains stable performance even when dealing with many terabytes of stored messages. 34 Document Title - Name - Function - Business Unit DD/MM/YYYY Benefits of Kafka

Slide 35

Slide 35 text

• Kafka is a messaging system that is designed to be fast, scalable, and durable. • A producer is an entity/application that publishes data to a Kafka cluster, which is made up of brokers. • A Broker is responsible for receiving and storing the data when a producer publishes. • A consumer then consumes data from a broker at a specified offset, i.e. position. • A Topic is a category/feed name to which records are stored and published. Topics have partitions and order guaranteed per partitions • All Kafka records are organized into topics. Producer applications write data to topics and consumer applications read from topics. 35 Document Title - Name - Function - Business Unit DD/MM/YYYY What is kafka

Slide 36

Slide 36 text

36 Document Title - Name - Function - Business Unit DD/MM/YYYY

Slide 37

Slide 37 text

37 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka Architecture.

Slide 38

Slide 38 text

• Topic is divided in partitions. • The message order is only guarantee inside a partition • Consumer offsets are persisted by Kafka with a commit/auto-commit mechanism. • Consumers subscribes to topics • Consumers with different group-id receives all messages of the topics they subscribe. They consume the messages at their own speed. • Consumers sharing the same group-id will be assigned to one (or several) partition of the topics they subscribe. They only receive messages from their partitions. So a constraint appears here: the number of partitions in a topic gives the maximum number of parallel consumers. • The assignment of partitions to consumer can be automatic and performed by Kafka (through Zookeeper). If a consumer stops polling or is too slow, a process call “re- balancing” is performed and the partitions are re-assigned to other consumers. 38 Document Title - Name - Function - Business Unit DD/MM/YYYY Key Concepts of Kafka

Slide 39

Slide 39 text

• Kafka normally divides topic in multiply partitions. • Each partition is an ordered, immutable sequence of messages that is continually appended to. • A message in a partition is identified by a sequence number called offset. • The FIFO is only guarantee inside a partition. • When a topic is created, the number of partitions should be given • The producer can choose which partition will get the message or let Kafka decides for him based on a hash of the message key (recommended). So the message key is important and will be the used to ensure the message order. • Moreover, as the consumer will be assigned to one or several partition, the key will also “group” messages to a same consumer. 39 Document Title - Name - Function - Business Unit DD/MM/YYYY Key Concepts of Kafka - continued

Slide 40

Slide 40 text

• A data source writes messages to the log and one or more consumers reads from the log at the point in time they choose. • In the diagram below a data source is writing to the log and consumers A and B are reading from the log at different offsets. 40 Document Title - Name - Function - Business Unit DD/MM/YYYY Log Anatomy

Slide 41

Slide 41 text

• We have a broker with three topics, where each topic has 8 partitions. • The producer sends a record to partition 1 in topic 1 and since the partition is empty the record ends up at offset 0. 41 Document Title - Name - Function - Business Unit DD/MM/YYYY Record flow in Apache Kafka

Slide 42

Slide 42 text

• Next record is added to partition 1 will and up at offset 1, and the next record at offset 2 and so on. • This is a commit log, each record is appended to the log and there is no way to change the existing records in the log(immutable). This is also the same offset that the consumer uses to specify where to start reading. 42 Document Title - Name - Function - Business Unit DD/MM/YYYY Record flow in Apache Kafka - continued

Slide 43

Slide 43 text

43 Document Title - Name - Function - Business Unit DD/MM/YYYY Apache Kafka Architecture

Slide 44

Slide 44 text

• Each broker holds a number of partitions and each of these partitions can be either a leader or a replica for a topic. • All writes and reads to a topic go through the leader and the leader coordinates updating replicas with new data. If a leader fails, a replica takes over as the new leader. 44 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka - Partitions and Brokers

Slide 45

Slide 45 text

• Producers write to a single leader, this provides a means of load balancing production so that each write can be serviced by a separate broker and machine. • In the image, the producer is writing to partition 0 of the topic and partition 0 replicates that write to the available replicas. 45 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka – Producers writing to broker

Slide 46

Slide 46 text

46 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka Architecture- Topic Replication factor

Slide 47

Slide 47 text

• Producer is process that can publish a message to a topic. • Consumer is a process that can subscribe to one or more topics and consume messages published to topics. • Topic category is the name of the feed to which messages are published. • Broker is a process running on single machine • Cluster is a group of brokers working together. • Broker management done by Zookeeper. 47 Document Title - Name - Function - Business Unit DD/MM/YYYY Flow of a record in Kafka

Slide 48

Slide 48 text

• Auto Scalable infrastructure. • Multi language support (SDKs) • Event Streaming Database • GUI driven management and monitoring • Enterprise Grade security • Diagnostic Logs • Data Monitor • Global Resilience • Disaster Recovery • Connector with legacy application • Retention management • Flexible DevOps • ….. Document Title - Name - Function - Business Unit DD/MM/YYYY What are capabilites of events hub

Slide 49

Slide 49 text

• Traditional message broker (not really event driven) 49 Document Title - Name - Function - Business Unit DD/MM/YYYY Alternatives of event-hub or kafka?

Slide 50

Slide 50 text

50 Document Title - Name - Function - Business Unit DD/MM/YYYY What about legacy App? RDBMS Existing App Event Hub New APP Change Data capture

Slide 51

Slide 51 text

• Attunity Replicate • Debezium (open source) • IBM IIDR • Oracle GoldenGate for Big Data • SQ Data 51 Document Title - Name - Function - Business Unit DD/MM/YYYY Change data capture tools

Slide 52

Slide 52 text

52 Document Title - Name - Function - Business Unit DD/MM/YYYY Microservice events and Streaming processing and legacy application State Microservice API Microservices Cluster Mobile Apps Social Media Stocks Blockchain Location IOT Events Stream Event Hub Events Stream Stream Processing Cluster Stream Analytics dashboard Events Stream Reference Model Results BI tools Search/Discover Mobile & online apps SQL Service Service Service State State Finance Sales Audit Change Data capture Big Data

Slide 53

Slide 53 text

53 Document Title - Name - Function - Business Unit DD/MM/YYYY

Slide 54

Slide 54 text

Integrate kafka with legacy • JDBC connector for Kafka connect • Use CDC (Change data capture) tool which integrates with kafka connect.

Slide 55

Slide 55 text

• Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other data systems. Runs separately from Kafka brokers. 55 Document Title - Name - Function - Business Unit DD/MM/YYYY Kafka connect

Slide 56

Slide 56 text

• How do manage the event lifecycle? • We have API management platform for APIs. 56 Document Title - Name - Function - Business Unit DD/MM/YYYY Async API - https://www.asyncapi.com/ Event lifecycle - Design - Documentation - Code generation - Event management - Test - Monitoring An Async API document is a file that defines and annotates the different components of a specific Event-Driven API.

Slide 57

Slide 57 text

57 Document Title - Name - Function - Business Unit DD/MM/YYYY OpenAPI- AsyncAPI comparison

Slide 58

Slide 58 text

Summary 5 8 • Make the split right – Bounded context. • Events are the communication between bounded context. • Event can be Async communication b/w microsevices. • Kafka is great source for messaging. • Event hub is key in new enterprise integration world. • Use CDC for legacy integrationn. • Try Async API for event documentation.

Slide 59

Slide 59 text

• https://www.slideshare.net/gschmutz/building-event-driven-microservices-with-apache- kafka-208145957 • https://www.slideshare.net/jeppec/soa-and-event-driven-architecture-soa- 20?qid=604f3115-642b-48d4-b7ef-66ce11ab9b0b&v=&b=&from_search=65 • https://docs.confluent.io/current/connect/index.html • https://data-flair.training/blogs/kafka-architecture/ • https://martinfowler.com/bliki/BoundedContext.html • https://insidebigdata.com/2018/04/12/developing-deeper-understanding-apache-kafka- architecture/ • https://www.asyncapi.com/ 59 Document Title - Name - Function - Business Unit DD/MM/YYYY References

Slide 60

Slide 60 text

Questions? 6 0