Slide 1

Slide 1 text

Events, Microservices and Data Ben Stopford Office of the CTO, Confluent

Slide 2

Slide 2 text

A new category of infrastructure for data that moves Relational Database Data warehouse Messaging Complex Event Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging

Slide 5

Slide 5 text

THREE TRENDS IN STREAMING Events become first-class citizens Microservices: ‘provision data’ as a service Stream Processing becomes a ‘database’ for the event-driven world

Slide 6

Slide 6 text

THREE TRENDS IN STREAMING Events become first-class citizens Microservices: ‘provision data’ as a service Stream Processing becomes a ‘database’ for the event-driven world

Slide 7

Slide 7 text

Event First: what changes? • The way we think about data • The way we architect • The way we evolve an organization • The way we program

Slide 8

Slide 8 text

What is an Event? A fact. A real-world incident, captured by a system. An payment A page view A log line A sensor reading

Slide 9

Slide 9 text

Events come in streams Apps M onitoring Security Apps Apps L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer

Slide 10

Slide 10 text

Stream Processors Combine, Filter & Summarize Event Streams KSQL

Slide 11

Slide 11 text

Increasing Complexity Apps Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Monolith Distributed Monolith Microservices Event-Driven Microservices Evolution of software systems

Slide 12

Slide 12 text

Increasing Complexity Apps Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices

Slide 13

Slide 13 text

Apps Search NoSQL Mo Se Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Backend services grow faster than the front end Tightly coupled Loosely coupled

Slide 14

Slide 14 text

Event shift the way we think Monolithic Approach -a database -a variable -a RPC -a user clicking buttons Event First Approach - an event stream - a “data” flow - a stream processor - software talking to software

Slide 15

Slide 15 text

Event-first changes how data is stored

Slide 16

Slide 16 text

Traditional systems use mutable state DB This isn’t wrong, it’s just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M

Slide 17

Slide 17 text

Events record the a timeseries of what actually happened. Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59

Slide 18

Slide 18 text

Stored as a stream Stored statefully (think DB) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey

Slide 19

Slide 19 text

12.42 12.44 12.49 12.50 12.59 We can derive the current state (but not the other way around) Apps Apps DERIVE Stream Processor

Slide 20

Slide 20 text

Observe, Store, Project a form of “Event Sourcing” Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened

Slide 21

Slide 21 text

Events tell us not just what happened but also how it happened

Slide 22

Slide 22 text

The End State The sequencing of moves is often more important than the end state The Game

Slide 23

Slide 23 text

A stock price: observe the game, not just the current state

Slide 24

Slide 24 text

A customer journey: observe everything

Slide 25

Slide 25 text

Formula 1

Slide 26

Slide 26 text

Formula 1: Observe the game, optimize the end state now and in the future End state

Slide 27

Slide 27 text

Formulae 1 – High-Level Architecture • 400 Sensors on car • 70,000 derivative measures • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing

Slide 28

Slide 28 text

Post race analysis

Slide 29

Slide 29 text

ML Source of Truth Retain events, rewind and replay the stream processor New derivative measures

Slide 30

Slide 30 text

Another form of “Event Sourcing” - Record what happened - Rewind, replay and rederive (View, App, ML, Physics Model etc.)

Slide 31

Slide 31 text

New York Times Source of Truth Every article since 1851 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”

Slide 32

Slide 32 text

Billing Shipping Fraud Fraud CUSTOMER ANALYTICS STORED EVENTS Stored Events used to create many different ‘views’

Slide 33

Slide 33 text

Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres Lambda Other Kafka Select Event Source Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Postgres Lambda Orders History 1w All Event Streaming Platforms make data self service (via events)

Slide 34

Slide 34 text

THREE TRENDS IN STREAMING Events become first-class citizens Microservices: provision data as a service Stream Processing becomes a ‘database’ for the event-driven world

Slide 35

Slide 35 text

THREE TRENDS IN STREAMING Events become first-class citizens Microservices: provision data as a service Stream Processing becomes a ‘database’ for the event-driven world

Slide 36

Slide 36 text

A future of Aside: Serverless Functions

Slide 37

Slide 37 text

Bad Analogy Serverless Functions are like very small microservices

Slide 38

Slide 38 text

Serverless Functions (FaaS) • Write a function • Upload • Configure a trigger (HTTP, Messaging, Object Store, Database, Timer etc.) Request Respond Event Source

Slide 39

Slide 39 text

FaaS in a Nutshell • Fully managed • Pay for execution time • Auto-scales with load • Short lived

Slide 40

Slide 40 text

Better Analogy Serverless Functions small event-driven containers

Slide 41

Slide 41 text

FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS More events created -> More “containers” are spun up in response More images stored More functions are triggered to create thumbnails

Slide 42

Slide 42 text

In the future it seems unlikely that infrastructure will respond to events. But where will we manage our data?

Slide 43

Slide 43 text

Mr Microservices

Slide 44

Slide 44 text

Event-Streaming approaches this from a different angle

Slide 45

Slide 45 text

FaaS is event-driven But it isn’t event-streaming

Slide 46

Slide 46 text

Customers Event Source Orders Event Source Payments Event Source Serverless Way: event driven but not streaming FaaS/μS FaaS/μS FaaS/μS Use Case: Email a customer the details of their order

Slide 47

Slide 47 text

Payments Event Source Increase Load: 100 concurrent functions doing IO. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS

Slide 48

Slide 48 text

Payments Event Source What if we’re only interested in platinum clients? -> 100 functions, 100 DB calls, 2 emails. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS

Slide 49

Slide 49 text

FaaS FaaS FaaS KSQL KSQL in-lines data processing into the flow FaaS FaaS Stateless “Business Logic” Filters what isn’t needed Orders Payments Customers Everything now enriched, summarized in the events

Slide 50

Slide 50 text

FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful “Database” FaaS FaaS FaaS FaaS FaaS Streaming Stateless Stateless Stateless Compute Layer Stream processors acting as the ‘database’ of the event driven world.

Slide 51

Slide 51 text

THREE TRENDS • Events as first-class citizens • Microservices: provision data as a service • Stream Processing becomes a ‘database’ for the event-driven world

Slide 52

Slide 52 text

Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems