Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Event Streaming Fundamentals

C6598a8b085d0c720cde07f89768a80c?s=47 benstopford
September 17, 2019

Event Streaming Fundamentals

Event Streaming Fundamentals

C6598a8b085d0c720cde07f89768a80c?s=128

benstopford

September 17, 2019
Tweet

Transcript

  1. Event Streaming Fundamentals Ben Stopford Office of the CTO, Confluent

  2. A new category Relational Database Data warehouse Messaging Complex Event

    Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020
  3. None
  4. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps

    Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Event Streaming Platform Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Schema Registry
  5. Hang on

  6. Isn’t this an ESB?

  7. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
  8. The log An append-only, time-ordered sequence of immutable records Append

    writes Read cursor Latest events Oldest events
  9. The log can be rewound and replayed Read cursor Latest

    events Oldest events
  10. Solves problems seen in traditional messaging systems • Queue depth

    problem • Add subscribers at near 0 cost • Load balances multiple consumers with strict ordering • Replay (and store) • 1st class distributed system
  11. Event Streaming Events are first class citizens (rows in tables

    are derivative structures) What changes? • The way we think about data • The way we architect (globally) • The way we evolve an organization • The way we program
  12. ESB on steroids?

  13. KAFKA IPhone App Stability Query ~once a day

  14. Serving Layer (Cassandra etc.) IPhone App Stability KAFKA KStreams Data

    is summarized on the fly Volume
  15. ESB on steroids?

  16. THE EVENT-FIRST APPROACH The unification of data and communication.

  17. Events: A fact. A real world incident, captured by a

    system. An payment A page view A log line A sensor reading
  18. Events come in streams Apps M onitoring Security Apps Apps

    L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer
  19. Stream Processors Combine, Filter, Buffer & Summarize Event Streams

  20. Code can be embedded in the stream processing layer Business

    Rules Machine Learning
  21. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  22. Apps Search NoSQL Mon Se Apps Apps S T R

    E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Search Monitoring Apps Apps App Apps Apps Apps Apps Search Monitoring Apps Apps App Apps Apps Apps Apps Search Monitoring Apps Apps App Kafka Where do you start? Back end tasks Event-Driven Microservices Services • Fraud detection • Recommendations • Billing • Settlement • Analytics Microservices Front End
  23. Event shift the way we think Monolithic Approach -A database

    -a variable -a singleton -a RPC Event First Approach - An event - A stream - A “data” flow - A stream processor
  24. “Events are facts that evolve and tell a story. A

    narrative that describes your system’s evolution as a whole; a day in the life of your business, day after day, forever.”
  25. Event Sourcing

  26. In Event Sourcing events are immutable, stateless and truthful. Events

    retain important information otherwise lost.
  27. Traditional systems use mutable state DB This isn’t wrong, it’s

    just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
  28. Events record the user’s journey Shopping Cart Events 2 Trousers

    added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  29. Stored as a stream Stored statefully (think DB) 12.42 12.44

    12.49 12.50 12.59 Information lost! Event User Journey
  30. 12.42 12.44 12.49 12.50 12.59 We can derive the current

    state (but not the other way around) Apps Apps DERIVE Stream Processor
  31. Streaming is a form of Event Sourcing The current state

    is a projection of the recording Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened
  32. Events describe what happened. Sequences of events (streams) describe how

    it happened
  33. The End State The sequencing of moves is often more

    important than the end state The Game
  34. A stock price: observe the game, not just the current

    state
  35. A customer journey: observe everything

  36. Formula 1

  37. Formula 1: Observe the game, optimize the end state now

    and in the future End state
  38. Formulae 1 – High-Level Architecture • 400 Sensors on car

    • 70,000 derivative measures • Events streamed back to base • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
  39. Post race analysis

  40. ML Source of Truth Retain events, rewind and replay the

    stream processor
  41. Another form of “Event Sourcing” - Record what happened -

    Rewind, replay and rederive (View, App, ML, Physics Model etc.)
  42. New York Times Source of Truth Every article since 1851

    https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View” Serving Layer
  43. Billing Shipping Fraud Fraud CUSTOMER ANALYTICS EVENT STORE A Database

    turned Inside Out, Unbundled! Microservice specific materialized views Apps write to the log
  44. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Spark Lambda Orders History 1w All Event Streaming Platforms make data self service (via events)
  45. We’ve seen Kafka as an Event Store • Loose coupling

    • Data decoupling • Pluggability • Receiver driven routing • Reload old events
  46. What about applications?

  47. In the Event Driven World KSQL is the Database

  48. Event Driven but not Streaming Orders Customers Payments App ETL

    ETL
  49. Orders Payments KStreams Customers Streaming is Event-First, Event-Sourced & Event-Driven

    Streaming as an API
  50. App KSQL Streaming as a “data layer” for an Event

    Driven App Stateless “Business Logic” in any language Stateful Orders Payments Customers All relevant data available in “enriched, summarized” events
  51. Traditional Application Event-Driven Application Application Database KSQL Stateful “Database” FaaS

    FaaS FaaS FaaS App Streaming Stateless Stateless Stateless App Layer Stream processors acting as the ‘database’ of the event driven world. App
  52. Let’s Summarise: - Event Streaming is more than Messaging or

    ESBs - Events are truthful, the narrative of your business - Event storage leads to self service evolution - Stream Processors become the databases of the event driven world
  53. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems