Events, Microservices and Data
Ben Stopford
Office of the CTO, Confluent
Slide 2
Slide 2 text
A new category of infrastructure for data that moves
Relational Database
Data warehouse
Messaging
Complex Event Processing
NoSQL & Big Data
Event Streaming
1970 1980 1990 2000 2010 2020
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Event Storage
Kafka stores
petabytes of data
Stream Processing
Real-time processing
over streams and tables
Scalability
Clusters of hundreds
of machines. Global.
+ + +
Roots in big data messaging
Slide 5
Slide 5 text
THREE TRENDS IN STREAMING
Events become first-class citizens
Microservices: ‘provision data’ as a service
Stream Processing becomes a ‘database’ for the event-driven world
Slide 6
Slide 6 text
THREE TRENDS IN STREAMING
Events become first-class citizens
Microservices: ‘provision data’ as a service
Stream Processing becomes a ‘database’ for the event-driven world
Slide 7
Slide 7 text
Event First: what changes?
• The way we think about data
• The way we architect
• The way we evolve an organization
• The way we program
Slide 8
Slide 8 text
What is an Event?
A fact. A real-world incident, captured by
a system.
An payment
A page view
A log line
A sensor reading
Slide 9
Slide 9 text
Events come in streams
Apps
M
onitoring
Security
Apps
Apps
L
A
T
F
O
R
M
Event
Stream
Order of events is important
Apps
M
onitoring
Apps
Apps
O
R
M
Older
Newer
Increasing Complexity
Apps
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps
Search
NoSQL
Mon
Sec
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Apps App
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
S T R E A M I N
Apps
Search
NoSQL
Apps
DWH
S T R E A M I N G P L AT
App
Apps
Search
NoSQL
Apps
S T R E A M I N G P L AT
Apps
Search
NoSQL
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
S T
Apps
Search
NoSQL
DWH
S T R E
App
Apps
Apps
Search
Apps
App
Apps Apps
Apps
Search
Apps Apps
App Apps
Apps
Search
App
Monolith Distributed Monolith Microservices
Event-Driven
Microservices
Evolution of software systems
Slide 12
Slide 12 text
Increasing Complexity
Apps
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps
Search
NoSQL
Mon
Sec
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Apps App
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
S T R E A M I N
Apps
Search
NoSQL
Apps
DWH
S T R E A M I N G P L AT
App
Apps
Search
NoSQL
Apps
S T R E A M I N G P L AT
Apps
Search
NoSQL
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
S T
Apps
Search
NoSQL
DWH
S T R E
App
Apps
Apps
Search
Apps
App
Apps Apps
Apps
Search
Apps Apps
App Apps
Apps
Search
App
Kafka
Evolution of software systems
Monolith Distributed Monolith Microservices
Event-Driven
Microservices
Slide 13
Slide 13 text
Apps
Search
NoSQL
Mo
Se
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App Apps
Search
NoSQL
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
DWH
Hadoop
S T R E A M I N G P L AT F O R M
App
Apps Apps Apps
Apps Apps Apps
App
Apps Apps Apps
Apps Apps Apps
App Apps Apps Apps
Apps Apps Apps
App Apps Apps Ap
Apps Apps Apps
App Apps Apps
Apps Apps Apps
App
Apps Apps Apps
Apps Apps Apps
App
Apps Apps Apps
Apps Apps Apps
App
Backend services grow faster than the front end
Tightly
coupled
Loosely
coupled
Slide 14
Slide 14 text
Event shift the way we think
Monolithic Approach
-a database
-a variable
-a RPC
-a user clicking buttons
Event First Approach
- an event stream
- a “data” flow
- a stream processor
- software talking to software
Slide 15
Slide 15 text
Event-first changes how data
is stored
Slide 16
Slide 16 text
Traditional systems use mutable state
DB
This isn’t wrong, it’s just lossy
Apps
Search Mon
Apps Apps
S T R E A M I N G P L A T F O R M
Slide 17
Slide 17 text
Events record the a timeseries of what
actually happened.
Shopping Cart Events
2 Trousers added
1 Jumper added
1 Trousers removed
1 Hat added
Checkout
Shopping Cart
Event
User
Journey
12.42
12.44
12.49
12.50
12.59
Slide 18
Slide 18 text
Stored as a stream Stored statefully (think DB)
12.42
12.44
12.49
12.50
12.59 Information lost!
Event
User
Journey
Slide 19
Slide 19 text
12.42
12.44
12.49
12.50
12.59
We can derive the current state
(but not the other way around)
Apps Apps
DERIVE
Stream Processor
Slide 20
Slide 20 text
Observe, Store, Project
a form of “Event Sourcing”
Familiar
Stateful
View
LOSSY
PROJECTION
Stream = Exactly
what happened
Slide 21
Slide 21 text
Events tell us not just what happened
but also how it happened
Slide 22
Slide 22 text
The End State
The sequencing of moves is often more
important than the end state
The Game
Slide 23
Slide 23 text
A stock price: observe the game, not just the current state
Slide 24
Slide 24 text
A customer journey: observe everything
Slide 25
Slide 25 text
Formula 1
Slide 26
Slide 26 text
Formula 1: Observe the game, optimize the end state
now and in the future
End state
Slide 27
Slide 27 text
Formulae 1 – High-Level Architecture
• 400 Sensors on car
• 70,000 derivative
measures
• Analyzed in real time
• Tire modelling
• Racing line
• Aerodynamics
• Machine Learning and
Physics Models.
• Replayed later for post
race analysis.
Race Track HQ
e.g. Tire modelling:
- Temp
- Pressure
- Suspension compression
Stream Processing
Slide 28
Slide 28 text
Post race analysis
Slide 29
Slide 29 text
ML
Source of Truth
Retain events, rewind and replay the stream processor
New
derivative
measures
Slide 30
Slide 30 text
Another form of “Event Sourcing”
- Record what happened
- Rewind, replay and rederive (View, App, ML, Physics Model etc.)
Slide 31
Slide 31 text
New York Times
Source of Truth
Every article since
1851
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Normalized assets
(images, articles, bylines, tags
all separate messages)
Denormalized into
“Content View”
Slide 32
Slide 32 text
Billing Shipping
Fraud Fraud
CUSTOMER
ANALYTICS
STORED EVENTS
Stored Events used to create many different ‘views’
Slide 33
Slide 33 text
Event Streams
Orders
Payments
Customers
Distinct Visits
Destination
Spark
Postgres
Lambda
Other Kafka
Select Event Source
Stream Processing
SELECT *
FROM ORDERS O, CUSTOMERS C
WHERE O.REGION = ‘EU’
AND C.TYPE = ‘Platinum’
Msgs/Day
Customers
Stream Processing
Postgres
Lambda
Orders
History
1w
All
Event Streaming Platforms make data self service (via events)
Slide 34
Slide 34 text
THREE TRENDS IN STREAMING
Events become first-class citizens
Microservices: provision data as a service
Stream Processing becomes a ‘database’ for the event-driven world
Slide 35
Slide 35 text
THREE TRENDS IN STREAMING
Events become first-class citizens
Microservices: provision data as a service
Stream Processing becomes a ‘database’ for the event-driven world
Slide 36
Slide 36 text
A future of
Aside: Serverless Functions
Slide 37
Slide 37 text
Bad Analogy
Serverless Functions are like very small microservices
Slide 38
Slide 38 text
Serverless Functions (FaaS)
• Write a function
• Upload
• Configure a trigger (HTTP, Messaging, Object Store, Database, Timer etc.)
Request Respond Event Source
Slide 39
Slide 39 text
FaaS in a Nutshell
• Fully managed
• Pay for execution time
• Auto-scales with load
• Short lived
Slide 40
Slide 40 text
Better Analogy
Serverless Functions small event-driven containers
Slide 41
Slide 41 text
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS
More events created ->
More “containers” are spun up in response
More images
stored
More functions are
triggered to create
thumbnails
Slide 42
Slide 42 text
In the future it seems
unlikely that infrastructure
will respond to events.
But where will we manage
our data?
Slide 43
Slide 43 text
Mr Microservices
Slide 44
Slide 44 text
Event-Streaming approaches this
from a different angle
Slide 45
Slide 45 text
FaaS is event-driven
But it isn’t event-streaming
Slide 46
Slide 46 text
Customers
Event Source
Orders
Event Source
Payments
Event Source
Serverless Way: event driven but not streaming
FaaS/μS
FaaS/μS
FaaS/μS
Use Case: Email a customer the details of their order
Payments
Event Source
What if we’re only interested in platinum clients?
-> 100 functions, 100 DB calls, 2 emails.
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
Slide 49
Slide 49 text
FaaS
FaaS
FaaS
KSQL
KSQL in-lines data processing into the flow
FaaS
FaaS
Stateless “Business Logic”
Filters what isn’t
needed
Orders
Payments
Customers
Everything now enriched,
summarized in the events
Slide 50
Slide 50 text
FaaS
Traditional
Application
Event-Driven
Application
Application
Database
KSQL
Stateful
“Database”
FaaS
FaaS
FaaS
FaaS
FaaS
Streaming
Stateless
Stateless
Stateless
Compute Layer
Stream processors acting as the ‘database’ of the event driven world.
Slide 51
Slide 51 text
THREE TRENDS
• Events as first-class citizens
• Microservices: provision data as a service
• Stream Processing becomes a ‘database’ for the event-driven world
Slide 52
Slide 52 text
Thank you
@benstopford
Book:
https://www.confluent.io/designing-event-driven-systems