The Future of Streaming: Global Apps, Event Stores and Serverless

Stream processing affects a wide range of industries today: capturing sensor data, connecting microservices, processing the workloads of internet giants and giving us a real-time alternative to batch analytics.
While these use cases are exciting and valuable they are only a taste of what is to come. In this talk we look at three areas that are likely to become more prominent: Global Apps, Event Stores and Serverless Stream Processing

Transcript

1. The Future of Streaming: Global Apps, Event Stores and Serverless Ben Stopford Office of the CTO, Confluent
2. Streaming sits at the intersection of how we deal with data and how we write programs
3. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
4. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Streaming Platform
5. Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
6. > 2 trillion messages per day
7. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
8. Events change our thinking Monolithic Approach -A database -a variable -a singleton -a RPC Event-First Approach - An event - A stream - A log - A stream processor
9. Event-driven programs have location transparency
10. They take us on journeys
11. Events let us run anywhere Interconnecting these separate worlds as real-time ecosystems
12. The future lies in integrated global streaming
13. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
14. Events change the way we observe the world around us
15. Events: A fact. An observation of the world. An payment A page view A log line A sensor reading
16. Events come in streams Apps M onitoring Security Apps Apps L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M
17. Events record what happened. Streams record how it happened
18. Traditional systems use mutable state DB This isn’t wrong, it’s just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
19. Events record the user’s journey Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
20. Stored as a stream Stored statefully (think DB) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
21. 12.42 12.44 12.49 12.50 12.59 We can derive the current state (but not the other way around) Apps Apps DERIVE Stream Processor
22. Streaming is a form of Event Sourcing The current state is a projection of the recording Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened
23. Streams let us “observe the game” one event at a time
24. The End State Often the game is more important than the end state The Game
25. A stock price: observe the game, not just the current state
26. A customer journey: observe everything
27. Formula 1
28. Formula 1: Observe the game, optimize the end state now and in the future End state
29. Formulae 1 – High-Level Architecture • 400 Sensors on car • 70,000 derivative measures • Events streamed back to base • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
30. Post race analysis
31. ML Source of Truth Retain events, rewind and replay the stream processor
32. Another form of “Event Sourcing” - Record what happened - Rewind, replay and rederive (View, App, ML, Physics Model etc.)
33. New York Times Store of Every article since 1851 (Source of Truth) https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
34. Billing Shipping Fraud Fraud CUSTOMER ANALYSIS EVENT STORE Rich, real-time recordings of customers and companies
35. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres KSQL Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Spark KSQL Orders History 1w All Event stores make data self service (real time & historical)
36. Rich recordings of customers and companies Real-time Historical Self Service
37. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
38. A future of Streaming changes how we observe the game. Cloud changes how we play it.
39. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Confluent Cloud 2019 2019
40. Serverless and Stream Processing are closely related
41. Using FaaS • Write a function • Upload • Configure a trigger (HTTP, Event, Object Store, Database, Timer etc.)
42. FaaS in a Nutshell • Short lived (max ~5 mins) • Pay as you use • 0-1000 concurrent functions, autoscales with load • Interesting for spikey compute • Interesting for low priority use cases e.g. CI systems.
43. But there are open questions
44. Serverless Developer Ecosystem • Runtime diagnostics • Monitoring • Deploy loop • Testing • IDE integration Currently quite poor
45. Harder than current approaches Easier than current approaches Amazon Google Microsoft
47. FaaS is event-driven But it isn’t streaming
48. Serverless Way: event driven but not streaming Orders Customers Payments FaaS FaaS FaaS
49. STREAMING: Event-first - how we think Event-sourced - how we store Event-driven - how we combine data and interact
50. Transaction Orders Payments KSQL Customers Streaming is Event-First, Event-Sourced & Event-Driven Stateful or Stateless
51. FaaS FaaS FaaS Transaction KSQL Stream processors can act as a “data layer” for FaaS ? FaaS FaaS Stateless Stateful (slower elasticity) Orders Payments Customers
52. FaaS FaaS FaaS Transaction Orders Payments KSQL Customers Stateless Stateful Inherit Kafka’s Rich Feature Set? FaaS FaaS
53. FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful Data Layer FaaS FaaS FaaS FaaS FaaS Streaming Event-first Event-sourced Event-driven Stateless Stateless Stateless Compute Layer Auto-scaling, correctness, pluggability
54. THREE TASTES OF THE FUTURE Global Apps: Location independent applications Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
55. GLOBAL SYSTEMS, STORED EVENTS, CLOUD NATIVE STREAM PROCESSING Data Layer FaaS FaaS FaaS FaaS FaaS
56. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems