Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Trends in Event Streaming

C6598a8b085d0c720cde07f89768a80c?s=47 benstopford
February 03, 2020

Trends in Event Streaming

A look at a number of key trends in Event Streaming for 2020 including:
- Batch to realtime
- Contextual Event-Driven applications
- Business processes becoming more realtime
- More copies of data, less deviation
- Self-service data
- Migration to cloud

C6598a8b085d0c720cde07f89768a80c?s=128

benstopford

February 03, 2020
Tweet

Transcript

  1. Trends in Event Streaming Ben Stopford Office of the CTO,

    Confluent
  2. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + An Event Streaming Platform is a Data Platform for Data in Motion
  3. Why should I care?

  4. Logos

  5. 1. Batch to Real-time

  6. None
  7. None
  8. 2. Contextual Event Driven Applications

  9. Geospatial Matching Route Re- planning Business Events Business Events Customer

    Driver Example: Taxi
  10. None
  11. Event Streaming Platform

  12. 3. Businesses processes becoming increasingly automated

  13. Loan Application Human centric: 1-2 Weeks Software centric: Seconds

  14. Databases are designed to aid human- computer interaction Saving inputs,

    returning inputs to the screen, etc.
  15. Fundamental Assumption: Data is Passive

  16. Event Streaming Database

  17. As business processes become increasingly automated they need a “database”

    designed for machineàmachine interaction
  18. 4. More copies of data, less deviation

  19. Data in one place vs data in many places Apps

    Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps h L Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App App Apps Search Monitoring Apps Apps App Apps Search Monitoring Apps Apps Apps h Monitoring Apps Apps App App Data is accurate. Very tightly coupled. Easy for different apps to evolve independently How do I join? Data gets out of sync
  20. Three options • Single database: all data in one place

    • Doesn’t scale in people terms • Microservices: Data spread in many “golden sources” • Opperationanal issues ensue • Data get’s copied everywhere as teams attempt to hit deadlines • Event Streaming • Single source of truth • Applications take a copy, that can be re-sourced when necessary
  21. Apps Monitoring Security Apps Apps S T R E A

    M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka: Source of truth Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  22. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Event Streaming
  23. 5. Self-Service Data

  24. Trade Surveillance Project • 9 months sourcing 16 data sets

    • Different formats (including for historical extracts) • Batch based approach
  25. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing DB App Orders History 1w All Making Data Self Service
  26. Self-Service • Netflix • Monsanto • Nordea • …

  27. Event Streaming Database is the abstraction for this

  28. 6. Migration to cloud

  29. Confluent Cloud • Dramatically cheaper than self managed for most

    use cases • No operational burden • No need to size clusters or pre- purchase hardware • Scale from zero (i.e. free) to whatever you need.
  30. Trends • Batch to real-time • Contextual event driven applications

    • Increasingly automated business processes • More copies, less deviation • Self-service data • Migration to cloud
  31. Evolution

  32. Thank you