Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Future of Applications is Streaming (40mins)

C6598a8b085d0c720cde07f89768a80c?s=47 benstopford
February 13, 2019
120

The Future of Applications is Streaming (40mins)

40min version of the original 50 min talk which takes a forward-leaning view on Event Streaming with topics including Events, Event Sourcing and Serverless.

C6598a8b085d0c720cde07f89768a80c?s=128

benstopford

February 13, 2019
Tweet

Transcript

  1. The Future of Applications is Streaming Ben Stopford Office of

    the CTO, Confluent
  2. None
  3. A new category Relational Database Data warehouse Messaging Complex Event

    Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020
  4. None
  5. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
  6. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps

    Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Event Streaming Platform
  7. Event Streaming Events are first class citizens (rows in tables

    are derivative structures) What changes? • The way we think about data • The way we architect (globally) • The way we evolve an organization • The way we program
  8. > 2 trillion messages per day

  9. THREE TASTES OF THE FUTURE The Event First Approach: The

    unification of data and communication. Self Service Data: Self service data, based on raw events Serverless Stream Processing: A data layer for the event-driven world
  10. Events: A fact. A real world incident, captured by a

    system. An payment A page view A log line A sensor reading
  11. Events come in streams Apps M onitoring Security Apps Apps

    L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer
  12. Stream Processors Combine, Filter & Summarize Event Streams KSQL

  13. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Monolith Distributed Monolith Microservices Event-Driven Microservices Evolution of software systems
  14. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  15. Apps Search NoSQL Mo Se Apps Apps S T R

    E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Backend services grow faster than the front end Tightly coupled Loosely coupled
  16. Event shift the way we think Monolithic Approach -A database

    -a variable -a singleton -a RPC Event First Approach - An event - A stream - A “data” flow - A stream processor
  17. THREE TASTES OF THE FUTURE The Event First Approach: The

    unification of data and communication. Self Service Data: Self service data, based on raw events Serverless Stream Processing: A data layer for the event-driven world
  18. Systems built on events retain important information otherwise lost.

  19. Traditional systems use mutable state DB This isn’t wrong, it’s

    just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
  20. Events record the user’s journey Shopping Cart Events 2 Trousers

    added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  21. Stored as a stream Stored statefully (think DB) 12.42 12.44

    12.49 12.50 12.59 Information lost! Event User Journey
  22. 12.42 12.44 12.49 12.50 12.59 We can derive the current

    state (but not the other way around) Apps Apps DERIVE Stream Processor
  23. Streaming is a form of Event Sourcing The current state

    is a projection of the recording Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened
  24. Events describe what happened. Sequences of events (streams) describe how

    it happened
  25. The End State The sequencing of moves is often more

    important than the end state The Game
  26. A stock price: observe the game, not just the current

    state
  27. A customer journey: observe everything

  28. Formula 1

  29. Formula 1: Observe the game, optimize the end state now

    and in the future End state
  30. Formulae 1 – High-Level Architecture • 400 Sensors on car

    • 70,000 derivative measures • Events streamed back to base • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
  31. Post race analysis

  32. ML Source of Truth Retain events, rewind and replay the

    stream processor
  33. Another form of “Event Sourcing” - Record what happened -

    Rewind, replay and rederive (View, App, ML, Physics Model etc.)
  34. New York Times Source of Truth Every article since 1851

    https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
  35. Billing Shipping Fraud Fraud CUSTOMER ANALYTICS EVENT STORE A Database

    turned Inside Out, Unbundled!
  36. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Spark Lambda Orders History 1w All Event Streaming Platforms make data self service (via events)
  37. Contextual Event Driven Applications Real-time events with historical Context Defined

    Self Service
  38. THREE TASTES OF THE FUTURE The Event First Approach: The

    unification of data and communication. Self Service Data: Self service data, based on raw events Serverless Stream Processing: The data layer for the event-driven world
  39. A future of The cloud changes the game

  40. Serverless and Stream Processing are closely related

  41. Using FaaS • Write a function • Upload • Configure

    an event trigger (HTTP, Event, Object Store, Database, Timer etc.)
  42. FaaS in a Nutshell • Fully managed (Runs in a

    container pool) • Pay for execution time not resources used (100ms granularity, 7.5x DIY VM cost) • Auto-scales with load ~ 0-1000 concurrent functions • Short lived (max ~5 mins) • Weak ordering guarantees • Cold start’s can be (very) slow: 100ms – 45s (AWS 250ms-7s) • Toy or new paradigm for general purpose programming?
  43. But there are open questions

  44. Serverless Developer Ecosystem • Runtime diagnostics • Monitoring • Deploy

    loop • Testing • IDE integration Currently quite poor
  45. Harder than current approaches Easier than current approaches Amazon Google

    Microsoft
  46. None
  47. FaaS is event-driven But it isn’t streaming

  48. Serverless Way: event driven but not streaming Orders Customers Payments

    FaaS FaaS FaaS
  49. Orders Payments KSQL Customers Streaming is Event-First, Event-Sourced & Event-Driven

    Stateful or Stateless
  50. FaaS FaaS FaaS KSQL Stream processors can act as a

    “data layer” for FaaS ? FaaS FaaS Stateless “Business Logic” (high elasticity) Stateful (slower elasticity) Orders Payments Customers All relevant data available in “enriched, summarized” events
  51. FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful “Database”

    FaaS FaaS FaaS FaaS FaaS Streaming Stateless Stateless Stateless Compute Layer Stream processors acting as the ‘database’ of the event driven world.
  52. FaaS FaaS FaaS Transaction Orders Payments KSQL Customers Stateless Stateful

    Inherit Kafka’s Rich Feature Set? FaaS FaaS
  53. THREE TASTES OF THE FUTURE The Event First Approach: The

    unification of data and communication. Self Service Data: Self service data, based on raw events Serverless Stream Processing: A data layer for the event-driven world
  54. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems