Trends in Event Streaming

A look at a number of key trends in Event Streaming for 2020 including:
- Batch to realtime
- Contextual Event-Driven applications
- Business processes becoming more realtime
- More copies of data, less deviation
- Self-service data
- Migration to cloud

Transcript

1. Trends in Event Streaming Ben Stopford Office of the CTO, Confluent
2. Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + An Event Streaming Platform is a Data Platform for Data in Motion
3. Why should I care?
4. Logos
5. 1. Batch to Real-time
8. 2. Contextual Event Driven Applications
9. Geospatial Matching Route Re- planning Business Events Business Events Customer Driver Example: Taxi
11. Event Streaming Platform
12. 3. Businesses processes becoming increasingly automated
13. Loan Application Human centric: 1-2 Weeks Software centric: Seconds
14. Databases are designed to aid human- computer interaction Saving inputs, returning inputs to the screen, etc.
15. Fundamental Assumption: Data is Passive
16. Event Streaming Database
17. As business processes become increasingly automated they need a “database” designed for machineàmachine interaction
18. 4. More copies of data, less deviation
19. Data in one place vs data in many places Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps h L Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App App Apps Search Monitoring Apps Apps App Apps Search Monitoring Apps Apps Apps h Monitoring Apps Apps App App Data is accurate. Very tightly coupled. Easy for different apps to evolve independently How do I join? Data gets out of sync
20. Three options • Single database: all data in one place • Doesn’t scale in people terms • Microservices: Data spread in many “golden sources” • Opperationanal issues ensue • Data get’s copied everywhere as teams attempt to hit deadlines • Event Streaming • Single source of truth • Applications take a copy, that can be re-sourced when necessary
21. Apps Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka: Source of truth Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
22. Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Event Streaming
23. 5. Self-Service Data
24. Trade Surveillance Project • 9 months sourcing 16 data sets • Different formats (including for historical extracts) • Batch based approach
25. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing DB App Orders History 1w All Making Data Self Service
26. Self-Service • Netflix • Monsanto • Nordea • …
27. Event Streaming Database is the abstraction for this
28. 6. Migration to cloud
29. Confluent Cloud • Dramatically cheaper than self managed for most use cases • No operational burden • No need to size clusters or pre- purchase hardware • Scale from zero (i.e. free) to whatever you need.
30. Trends • Batch to real-time • Contextual event driven applications • Increasingly automated business processes • More copies, less deviation • Self-service data • Migration to cloud
31. Evolution
32. Thank you