Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Event Streaming and the Future of Applications

Event Streaming and the Future of Applications

Events fundamentally change the way we build systems. As a means of communication they loosely couple microservices together, meaning sections of an architecture become isolated from the failure semantics of the rest. As a data model they tie communication and storage together via a technique called event sourcing, which retains information exactly as it happened, primed for future use cases or for analytics. Events also imply motion: Data moves from one application to another, with stream processors used to manipulate these data flows as they progress. Finally, events are the backbone of serverless architectures. A programming model that is likely to be prevalent in our future.
In this talk, we’ll look at the relationship between these different aspects of the humble event, providing insight into why the paradigm is important now, as well as reflecting on its inevitable positioning in our future.

C6598a8b085d0c720cde07f89768a80c?s=128

benstopford

May 08, 2019
Tweet

Transcript

  1. Event Streaming and the Future of Applications Ben Stopford Office

    of the CTO, Confluent
  2. Old and emerging categories Relational Database Data warehouse Messaging Complex

    Event Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020 Cloud & Serverless
  3. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
  4. Events: A fact. An observation about the world, captured at

    a point in time. A payment An order A page view A log line A sensor reading
  5. Events come in streams Apps M onitoring Security Apps Apps

    L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer
  6. Streams let us react to the world as it changes

    over time Order Created Order Confirmed Order Paid Order Shipped
  7. Stream Processors React, Combine, Filter & Summarize Event Streams KSQL

  8. If you embrace events it changes your programs in two

    main ways: 1. The way you store data 2. The way your programs communicate
  9. Let’s start with a simple example DB What’s wrong with

    this architecture? Apps Search Apps Apps
  10. Events create a timeseries of things a user did Shopping

    Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  11. Stored as a stream Stored as “Current State” (i.e. CRUD)

    12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
  12. 12.42 12.44 12.49 12.50 12.59 We can derive the ‘current

    state view’ (but not the other way around) Apps Apps DERIVE Stream Processor
  13. Events have 3 main benefits as a storage model

  14. Business Lens Evidentiary Accountants don’t use erasers (e.g. audit, ledger,

    git)
  15. Ops Lens Replayability (e.g. recovering data after programmatic error)

  16. Analytics Lens • Trends and behaviors • (e.g. insight, metrics,

    ML)
  17. What do such systems look like?

  18. New York Times – Event Based Storage in Kafka. (“View”

    continuously derived from the log) Source of Truth Every article since 1851 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
  19. If you embrace events it changes your programs in two

    main ways: 1. The way you store data 2. The way your programs communicate
  20. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  21. Apps Search NoSQL Mo Se Apps Apps S T R

    E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Architectures grow Loosely coupled
  22. Apps Search NoSQL Mo Se Apps Apps DWH Hadoop S

    T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Event Preparation • Join • Filter • Transform • Enrich (from tables) • Summarize
  23. Make the consumption of real-time event streams easy, even if

    the data comes from many different places.
  24. KSQL is a kind of Database for Creating new Event

    Streams Orders Service Payment Service Customer Service KSQL Aggregate Events Apps Search Ap Order Payment Customer SELECT * FROM…
  25. Serverless functions, being inherently event-driven, are a great example

  26. What is a Serverless Function? • Write a function •

    Upload • Configure an event trigger (HTTP, Event, Object Store, Database, Timer etc.) • Fully managed (Runs in a container pool)
  27. The benefits of Serverless Functions stem from being event-driven •

    Auto-scales with load ~ 0-1000 concurrent functions • Event backlog drives autoscaling • Pay for execution time not resources used
  28. Harder than current approaches Easier than current approaches Amazon Google

    Microsoft Serverless programming will likely become prevalent
  29. Serverless Functions are event-driven They aren’t “streaming”

  30. Default: Complex, Timing issues, Scaling limits Customers Event Source FaaS

    FaaS FaaS Orders Event Source Payments Event Source
  31. FaaS FaaS FaaS KSQL Stream processors can act as a

    “data layer” for FaaS ? FaaS FaaS High elasticity Stateful operations Orders Payments Customers All data is in the “enriched” event
  32. Summary • Events underpin the storage models of truthful/factful systems.

    • Events decouple architectures and provide real-time data. • Serverless is infrastructure made event-driven. • Stream Processing is the natural corollary, for data.
  33. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems Signing at Confluent booth immediately

    after this talk