4
Today’s ecosystems get pretty big
4
• 2.2 trillion messages per day (6
Petabytes)
• Up to 400 Microservices pre
cluster.
• 20-200 Brokers per cluster
Slide 5
Slide 5 text
5
Today’s ecosystems get pretty big
5
• 1 billion messages per day
• 20,000 messages per second
• 100 teams
7
KAFKA
Serving
Layer
(Cassandra etc.)
Kafka Streams /
KSQL
Streaming Platforms
Data is embedded in
each engine
High Throughput
Messaging
Clustered
Java App
Slide 8
Slide 8 text
8
apps_opened
app_crashes
unstable_apps
crashed_per_day
opened_per_day
(b) Crashes,
Per App, Per day
(a) Apps Opened,
Per App, Per day
(c) Unstable
Applications
Streaming Pipeline
Slide 9
Slide 9 text
9
KAFKA
Serving
Layer
(Cassandra etc.)
Kafka Streams /
KSQL
Streaming Platforms
Data is embedded in
each engine
High Throughput
Messaging
Clustered
Java App
Slide 10
Slide 10 text
10
An event log is a simple idea
Messages are added at the end of the log
Old New
Slide 11
Slide 11 text
11
Readers have a position all of their own
Sally
is here
George
is here
Fred
is here
Old New
Scan Scan
Scan
Slide 12
Slide 12 text
12
You can rewind and replay, just like Tivo!
Old New
Sally
is here Scan
Slide 13
Slide 13 text
13
The hard part: Tying it all together!
Slide 14
Slide 14 text
14
Many ”logs” over many machines
Producing
Services
Kafka
Consuming
Services
Slide 15
Slide 15 text
15
Resistant to Failure
Producing
Services
Kafka
Consuming
Services
Slide 16
Slide 16 text
16
KAFKA
Serving
Layer
(Cassandra etc.)
Kafka Streams /
KSQL
Streaming Platforms
Data is embedded in
each engine
High Throughput
Messaging
Clustered
Java App
Slide 17
Slide 17 text
17
Streaming Example
apps_opened opened_per_day
Slide 18
Slide 18 text
18
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 19
Slide 19 text
19
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 20
Slide 20 text
20
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 21
Slide 21 text
21
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 22
Slide 22 text
22
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 23
Slide 23 text
23
CREATE TABLE opened_per_day AS
SELECT app_id, count(*)
FROM apps_opened
WINDOW TUMBLING (SIZE 1 DAY)
GROUP BY app_id;
apps_opened opened_per_day
Slide 24
Slide 24 text
24
Streaming is manipulating events in flight,
at scale.
26
Ecosystems
App
Increasingly we build ecosystems
Slide 27
Slide 27 text
27
SOA / Microservices / EDA
Customer
Service
Shipping
Service
Slide 28
Slide 28 text
28
The Problem is DATA
Slide 29
Slide 29 text
29
Most services share the same core facts.
Catalog
Most services live
in here
Slide 30
Slide 30 text
30
Events have two hats
Notification Data
replication
Slide 31
Slide 31 text
31
Buying an iPad (with REST/RPC)
Submit
Order
shipOrder() getCustomer()
Orders
Service
Shipping
Service
Customer
Service
Webserver
Slide 32
Slide 32 text
32
Events for Notification Only
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST
Notification
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
Slide 33
Slide 33 text
33
Pluggability
Submit
Order
Order
Created
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
Repricing
getCustomer()
REST
Notification
Slide 34
Slide 34 text
34
Events for Data Locality
Customer
Updated
Submit
Order
Order
Created
Data is
replicated
Orders
Service
Shipping
Service
Customer
Service
Webserver
KAFKA
Slide 35
Slide 35 text
35
Events have two hats
Notification Data
replication
Slide 36
Slide 36 text
36
Stateless / Stateful Stream Processing
Relates to these hats
Slide 37
Slide 37 text
37
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
REST/RPC
Orders
Service
Customer
Service
Webserver
KAFKA
KStreams
Shipping Service
Stateless Stream Processing
Notification
Kafka Steams
/ KSQL
Slide 38
Slide 38 text
38
Message Broker (Kafka)
Submit
Order
Order
Created
Orders
Service
Customer
Service
Webserver
KAFKA
KStreams
Shipping Service
KTable
Customer
Updated
Stateful Stream Processing
Data
replication
41
Streaming is about
1. Joining & Operating on
Streams
2. Joining & Operating on
Materialized Tables
On Notification
Data Replication
Slide 42
Slide 42 text
42
Kafka: a Streaming Platform
The Log Connectors
Connectors
Producer Consumer
Streaming Engine
Slide 43
Slide 43 text
43
8 Steps to Streaming Services
Slide 44
Slide 44 text
44
1. Use events to decouple and collaborate
Slide 45
Slide 45 text
45
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Event Collaboration
Order
Completed
CQRS
Slide 46
Slide 46 text
46
2. Use Connect (& CDC) to evolve away
from legacy
Slide 47
Slide 47 text
47
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Make Legacy Datasets Available via the Log
Order
Completed
CQRS
Connect
Products
Slide 48
Slide 48 text
48
3. Use the Single Writer Principal
Slide 49
Slide 49 text
49
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
State changes to a topic owned by one service
Connect
Products
Schema Registry
Order
Completed
Slide 50
Slide 50 text
50
Orders
Service
Email
Service
T1 T2
T3
T4
REST
Service
T5
Local consistency points in the absence
of Global Consistency
Slide 51
Slide 51 text
51
4. Use Kafka as a Shared Source of Truth
(Messaging that Remembers)
Slide 52
Slide 52 text
52
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Shared Source of Truth
Connect
Products
Schema Registry
Order
Completed Reporting
Slide 53
Slide 53 text
53
KAFKA
Order
Requested
Order
Validated
Order
Received
Browser
Webserver
Orders
Service
Product Catalogue stored in 3 places
Connect
Products
Schema Registry
Order
Completed Reporting
Reporting view
may be “thinner”
Slide 54
Slide 54 text
54
5. Move Data to Code
Slide 55
Slide 55 text
55
Slide 56
Slide 56 text
56
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service Stock
Stock
Materialize Stock ‘View’ Inside Service
KAFKA
Slide 57
Slide 57 text
57
Kafka has several features for reducing
the need to move data on startup
- Standby Replicas
- Disk Checkpoints
- Compacted topics
Slide 58
Slide 58 text
58
6. Use State Stores as local ‘database’,
backed up in Kafka
Slide 59
Slide 59 text
59
Connect
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
State stores behave like local databases
KAFKA
State Store
Slide 60
Slide 60 text
60
7. Use Transactions to tie All
Interactions Together
Slide 61
Slide 61 text
61
Connect
TRANSACTION
Order
Requested
Order
Validated
Order
Completed
Order
Received
Products
Browser
Webserver
Schema Registry
Orders
Service
Reserved Stocks
Stock
Stock
Reserved Stocks
Transactions
KAFKA
Slide 62
Slide 62 text
62
8. Evolve and Grow
Slide 63
Slide 63 text
63
Kafka
KAFKA
Finance
Front Office
Operations
Tiered Contexts
Slide 64
Slide 64 text
64
Span regions or clouds
Slide 65
Slide 65 text
65
Handle Disconnectedness
Slide 66
Slide 66 text
66
So…
Slide 67
Slide 67 text
67
Stream
Processing
Optimize for complexity vs optimize for scale
Event Driven
Architectures
Slide 68
Slide 68 text
68
Events provide the key to evolutionary
architectures
Notification Data
replication
Slide 69
Slide 69 text
69
Spectrum of use cases
Finer Grained,
Collaborative,
Connected
Courser Grained,
Non-collaborative,
Disconnected
Notification Data Replication
Slide 70
Slide 70 text
70
Streaming is the
toolset for dealing with
events at scale
Slide 71
Slide 71 text
71
• Broadcast events
• Retain them in the log
• Evolve the event-stream with
streaming functions
• Recasting the event stream into
views when you need to query.
Event Driven Services
Slide 72
Slide 72 text
72
Find out more
Book: http://bit.ly/designing-event-driven-systems
Software: https://confluent.io/download/
Code: http://bit.ly/kafka-microservice-examples
Twitter: @benstopford