Tweet
Tweet

Slide 1

Slide 1 text

Event Sourcing, Stream Processing & Serverless Ben Stopford Office of the CTO, Confluent

Slide 2

Slide 2 text

What we’re going to talk about • Event Sourcing • What it is and how does it relate to Event Streaming? • Stream Processing as a kind of “Database” • What does this mean? • Serverless Functions • How do this relate?

Slide 3

Slide 3 text

Can you do event sourcing with Kafka?

Slide 4

Slide 4 text

Traditional Event Sourcing

Slide 5

Slide 5 text

Popular example: Shopping Cart DB Apps Search Apps Apps Database Table matches what the user sees.

Slide 6

Slide 6 text

12.42 12.44 12.49 12.50 12.59 Event Sourcing stores events, then derives the ‘current state view’ Apps Apps DERIVE Chronological Reduce Event Timeseries of user activity

Slide 7

Slide 7 text

Traditional Event Sourcing (Store immutable events in a database in time order) Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Table of events Persist events Apps Apps

Slide 8

Slide 8 text

Traditional Event Sourcing (Read) Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search Monitoring Apps Apps Chronological Reduce on read (done inside the app) Query by customer Id (+session?) - No schema migration - Similar to ’schema on read’

Slide 9

Slide 9 text

3 Benefits

Slide 10

Slide 10 text

Evidentiary Accountants don’t use erasers (e.g. audit, ledger, git)

Slide 11

Slide 11 text

Replayability Recover corrupted data after a programmatic bug

Slide 12

Slide 12 text

Analytics Keep the data needed to extract trends and behaviors i.e. non-lossy (e.g. insight, metrics, ML)

Slide 13

Slide 13 text

Traditional Event Sourcing • Use a database (any one will do) • Create a table and insert events as they occur • Query all the events associated with your problem* • Reduce them chronologically to get the current state *Aggregate ID in DDD parlance

Slide 14

Slide 14 text

Traditional Event Sourcing with Kafka • Use a database Kafka • Create a table topic insert events as they occur • Query all the events associated with your problem* • Reduce them chronologically to get the current state *Aggregate ID in DDD parlance

Slide 15

Slide 15 text

Confusion: You can’t query Kafka by say Customer Id* *Aggregate ID in DDD parlance

Slide 16

Slide 16 text

If we can’t query by Customer ID then what do we do?

Slide 17

Slide 17 text

CQRS is a tonic: Cache the projection in a ‘View’ Apps Search Monitoring Apps Apps S T R E A M I N G P L AT F O R M Query by customer Id Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M View Events/Command Events are the Storage Model Stream Processor Cache/DB/Ktable etc. Regenerate the view rather than doing schema migration

Slide 18

Slide 18 text

CQRS provides the benefits of event sourcing using a “Materialized View”

Slide 19

Slide 19 text

Even with CQRS, Event Sourcing is Hard CQRS helps, but it’s still quite hard if you’re a CRUD app

Slide 20

Slide 20 text

What’s the problem? Harder: • Eventually Consistent • Multi-model (Complexity ∝ #Schemas in the log) • More moving parts Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L A T F O R M CRUD System CQRS

Slide 21

Slide 21 text

New York Times Website Source of Truth Every article since 1851 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”

Slide 22

Slide 22 text

If CRUD makes sense there are other ways: audit tables, CDC, etc. Trigger Evidentiary Replayable N/A to web app Analytics CDC

Slide 23

Slide 23 text

More advanced: Use a Bi-Temporal Database

Slide 24

Slide 24 text

Events make most sense where data has to move

Slide 25

Slide 25 text

This is where CQRS comes into its own!

Slide 26

Slide 26 text

Online Transaction Processing: e.g. a Flight Booking System - Flight price served 10,000 x #bookings - Consistency required only at booking time

Slide 27

Slide 27 text

CQRS with event movement Apps Search Monitoring Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M View Book Flight Apps Search Apps S T R E A M I N G P L A View Apps Search NoSQL Apps DWH S T R E A M I N G P L A View Get Flights Get Flights Get Flights Global Read Central Write

Slide 28

Slide 28 text

The exact same logic applies to microservices

Slide 29

Slide 29 text

Event Sourcing for Microservices Basket Service Fraud Service Billing Service Email Service Basket Events

Slide 30

Slide 30 text

Event Sourcing for Microservices Basket Service Fraud Service Billing Service Email Service Basket Events Events are the storage model Each microservice creates a view that suits its use case

Slide 31

Slide 31 text

Event Sourcing “with a DB” for monoliths. Event Streaming for Microservices & Scale. (Often via. CQRS)

Slide 32

Slide 32 text

Event Streaming

Slide 33

Slide 33 text

Event Streaming is a more general form of Event Sourcing/CQRS Event Streaming • Events as shared data model • Many microservices • Polyglot persistence • Event-Driven processing Traditional Event Sourcing • Events as a storage model • Single microservice • Single DB • data-at-rest

Slide 34

Slide 34 text

Event Streams is about many event sources (Join, Filter, Transform and Summarize) Fraud Service Orders Service Payment Service Customer Service Event Log Projection created in Kafka Streams API

Slide 35

Slide 35 text

KStreams & KSQL have different positioning • KStreams is a library for Dataflow programming: • App Logic & Stream Processor (including state) are combined. • Apps are stateful. • JVM only. • KSQL is a ‘database’ for event preparation: • App sends SQL to a separate process • Apps are stateless • Connect from any language

Slide 36

Slide 36 text

This difference makes most sense if we we look to the future.

Slide 37

Slide 37 text

Cloud & Serverless

Slide 38

Slide 38 text

Thesis • Serverless provides event-driven infrastructure • KSQL is the corollary: an event-driven database

Slide 39

Slide 39 text

Serverless Functions (FaaS) • Write a function • Upload • Configure a trigger (HTTP, Messaging, Object Store, Database, Timer etc.) Request Respond Event Source

Slide 40

Slide 40 text

FaaS in a Nutshell • Fully managed (Runs in a container pool) • Pay for execution time (not resources used) • Auto-scales with load • 0-1000+ concurrent functions • Stateless • Short lived (limit 5-15 mins) • Weak ordering guarantees • Cold start’s can be (very) slow: 100ms – 45s (AWS 250ms-7s)

Slide 41

Slide 41 text

Where is FaaS useful? • Spikey workloads and ‘occasional’ use cases • Use cases that don’t typically warrant massive parallelism e.g. CI systems. • General purpose programming paradigm?

Slide 42

Slide 42 text

But there are open questions

Slide 43

Slide 43 text

Serverless Developer Ecosystem • Runtime diagnostics • Monitoring • Deploy loop • Testing • IDE integration Currently quite poor

Slide 44

Slide 44 text

Harder than current approaches Easier than current approaches Amazon Google Microsoft Serverless programming will likely become prevalent

Slide 45

Slide 45 text

In the future it seems unlikely we’ll manage our own infrastructure. But where will we manage our data?

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

Event-Streaming approaches this from a different angle

Slide 48

Slide 48 text

FaaS is event-driven But it isn’t streaming

Slide 49

Slide 49 text

Complex, Timing issues, Scaling limits Customers Event Source Orders Event Source Payments Event Source Serverless functions handle only one event source FaaS/μS FaaS/μS FaaS/μS

Slide 50

Slide 50 text

A slightly more complex example: Send email only to platinum customers

Slide 51

Slide 51 text

Payments Event Source Event is received by serverless function FaaS/μS

Slide 52

Slide 52 text

Payments Event Source Block and calls the database to get customer+order FaaS/μS Get customer Get order

Slide 53

Slide 53 text

Payments Event Source Is it a ‘Platinum’ customer? FaaS/μS Get customer Get order Is the customer platinum?

Slide 54

Slide 54 text

Payments Event Source Send email if ‘Platinum’ FaaS/μS Get customer Get order Maybe send email

Slide 55

Slide 55 text

Payments Event Source Increase Load: 100 concurrant functions doing IO. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS

Slide 56

Slide 56 text

Payments Event Source Only send 2 emails. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS

Slide 57

Slide 57 text

Send SQL Process boundary Orders Payments KSQL Customers Table Customers KSQL simplifies: App Logic CREATE STREAM foo AS SELECT * FROM orders, payments, customers LEFT JOIN… WHERE customer.type = ‘PLATINUM’ Order Payment Customer KSQL - Handle timing issues - No “per-event” IO. - Price efficient

Slide 58

Slide 58 text

Functions have no additional data dependencies: Everything is in the event!

Slide 59

Slide 59 text

Queries filter out the events you need (much like you filter rows in a database query)

Slide 60

Slide 60 text

FaaS FaaS FaaS KSQL Customers Table KSQL as a “Database” for Event-Driven Infrastructure FaaS FaaS Stateless, elastic compute Prepare the events we need (Sateful) Orders Payments Customers Autoscale with load

Slide 61

Slide 61 text

FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful Data Layer FaaS FaaS FaaS FaaS FaaS Streaming Stateless Stateless Stateless Compute Layer Massive linear scalability with elasticity

Slide 62

Slide 62 text

Event-Driven vs. Event Streaming Event Driven Event Streaming Multiple Event Sources Use Database + ETL + Code Handles automatically Efficiency Extract data from DB in the FaaS (IO) Only the data you need Logic-driven data requests. Call DB from the FaaS (IO) DB/KStreams KqlDB?

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

Event Streaming Platform

Slide 65

Slide 65 text

Summary • Event Streaming provides the benefits of Event Sourcing to microservices and data pipelines. • Events are the data model. • Projections are the serving model: matching to each specific use case • Serving layer can be regenerated from the log (CQRS) • KSQL provides the same benefits for event-driven programs: e.g. preparing the event streams each FaaS application’s specific needs • In serverless architectures this drives efficiency: a ‘database- equivalent’ for event-driven infrastructure.

Slide 66

Slide 66 text

FaaS FaaS FaaS KSQL Can I Build This? FaaS FaaS AWS Lambda / Azure Functions Connectors (in Preview) Hosted KSQL In Preview Confluent Cloud

Slide 67

Slide 67 text

Things I didn’t tell you • Tools like KSQL provide data provisioning, not state mutation. • Use single writers. Try KSQL DB? • Can KSQL handle large state? • Unintended rebalance can stall processing • Static membership (KIP-345) – name the list of stream processors • Increase the timeout for rebalance after node removal (group.max.session.timeout.ms) • Worst case reload: RocksDB ~GbE speed • Can Kafka be used for long term storage? • Log files are immutable once they roll (unless compacted) • Jun spent a decade working on DB2 • Careful: • Historical reads can stall real-time requests (cached) • ZFS has several page cache optimizations • Tiered storage will help

Slide 68

Slide 68 text

Find out More • Peeking Behind the Curtains of Serverless Platforms, Wang et al. • Cloud Programming Simplified: A Berkeley View on Serverless Compute • Neil Avery’s Journey to Event Driven Part 3. The Affinity Between Events, Streams and Serverless. • Designing Event Driven Systems, Ben Stopford

Slide 69

Slide 69 text

Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems