Trends in Event Streaming Ben Stopford Office of the CTO, Confluent

Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + An Event Streaming Platform is a Data Platform for Data in Motion

Why should I care?

Logos

1. Batch to Real-time

No content

No content

2. Contextual Event Driven Applications

Geospatial Matching Route Re- planning Business Events Business Events Customer Driver Example: Taxi

No content

Event Streaming Platform

3. Businesses processes becoming increasingly automated

Loan Application Human centric: 1-2 Weeks Software centric: Seconds

Databases are designed to aid human- computer interaction Saving inputs, returning inputs to the screen, etc.

Fundamental Assumption: Data is Passive

Event Streaming Database

As business processes become increasingly automated they need a “database” designed for machineàmachine interaction

4. More copies of data, less deviation

Data in one place vs data in many places Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps h L Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App App Apps Search Monitoring Apps Apps App Apps Search Monitoring Apps Apps Apps h Monitoring Apps Apps App App Data is accurate. Very tightly coupled. Easy for different apps to evolve independently How do I join? Data gets out of sync

Three options • Single database: all data in one place • Doesn’t scale in people terms • Microservices: Data spread in many “golden sources” • Opperationanal issues ensue • Data get’s copied everywhere as teams attempt to hit deadlines • Event Streaming • Single source of truth • Applications take a copy, that can be re-sourced when necessary

Apps Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka: Source of truth Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices

Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Event Streaming

5. Self-Service Data

Trade Surveillance Project • 9 months sourcing 16 data sets • Different formats (including for historical extracts) • Batch based approach

Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing DB App Orders History 1w All Making Data Self Service

Self-Service • Netflix • Monsanto • Nordea • …

Event Streaming Database is the abstraction for this

6. Migration to cloud

Confluent Cloud • Dramatically cheaper than self managed for most use cases • No operational burden • No need to size clusters or pre- purchase hardware • Scale from zero (i.e. free) to whatever you need.

Trends • Batch to real-time • Contextual event driven applications • Increasingly automated business processes • More copies, less deviation • Self-service data • Migration to cloud

Evolution

Thank you