Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Saxo Events, Microservices and Data

C6598a8b085d0c720cde07f89768a80c?s=47 benstopford
February 18, 2020

Saxo Events, Microservices and Data

Saxo deck

C6598a8b085d0c720cde07f89768a80c?s=128

benstopford

February 18, 2020
Tweet

Transcript

  1. Events, Microservices and Data Ben Stopford Office of the CTO,

    Confluent
  2. A new category of infrastructure for data that moves Relational

    Database Data warehouse Messaging Complex Event Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020
  3. None
  4. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
  5. THREE TRENDS IN STREAMING Events become first-class citizens Microservices: ‘provision

    data’ as a service Stream Processing becomes a ‘database’ for the event-driven world
  6. THREE TRENDS IN STREAMING Events become first-class citizens Microservices: ‘provision

    data’ as a service Stream Processing becomes a ‘database’ for the event-driven world
  7. Event First: what changes? • The way we think about

    data • The way we architect • The way we evolve an organization • The way we program
  8. What is an Event? A fact. A real-world incident, captured

    by a system. An payment A page view A log line A sensor reading
  9. Events come in streams Apps M onitoring Security Apps Apps

    L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer
  10. Stream Processors Combine, Filter & Summarize Event Streams KSQL

  11. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Monolith Distributed Monolith Microservices Event-Driven Microservices Evolution of software systems
  12. Increasing Complexity Apps Monitoring Security Apps Apps S T R

    E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  13. Apps Search NoSQL Mo Se Apps Apps S T R

    E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Backend services grow faster than the front end Tightly coupled Loosely coupled
  14. Event shift the way we think Monolithic Approach -a database

    -a variable -a RPC -a user clicking buttons Event First Approach - an event stream - a “data” flow - a stream processor - software talking to software
  15. Event-first changes how data is stored

  16. Traditional systems use mutable state DB This isn’t wrong, it’s

    just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
  17. Events record the a timeseries of what actually happened. Shopping

    Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  18. Stored as a stream Stored statefully (think DB) 12.42 12.44

    12.49 12.50 12.59 Information lost! Event User Journey
  19. 12.42 12.44 12.49 12.50 12.59 We can derive the current

    state (but not the other way around) Apps Apps DERIVE Stream Processor
  20. Observe, Store, Project a form of “Event Sourcing” Familiar Stateful

    View LOSSY PROJECTION Stream = Exactly what happened
  21. Events tell us not just what happened but also how

    it happened
  22. The End State The sequencing of moves is often more

    important than the end state The Game
  23. A stock price: observe the game, not just the current

    state
  24. A customer journey: observe everything

  25. Formula 1

  26. Formula 1: Observe the game, optimize the end state now

    and in the future End state
  27. Formulae 1 – High-Level Architecture • 400 Sensors on car

    • 70,000 derivative measures • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
  28. Post race analysis

  29. ML Source of Truth Retain events, rewind and replay the

    stream processor New derivative measures
  30. Another form of “Event Sourcing” - Record what happened -

    Rewind, replay and rederive (View, App, ML, Physics Model etc.)
  31. New York Times Source of Truth Every article since 1851

    https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
  32. Billing Shipping Fraud Fraud CUSTOMER ANALYTICS STORED EVENTS Stored Events

    used to create many different ‘views’
  33. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    Lambda Other Kafka Select Event Source Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Postgres Lambda Orders History 1w All Event Streaming Platforms make data self service (via events)
  34. THREE TRENDS IN STREAMING Events become first-class citizens Microservices: provision

    data as a service Stream Processing becomes a ‘database’ for the event-driven world
  35. THREE TRENDS IN STREAMING Events become first-class citizens Microservices: provision

    data as a service Stream Processing becomes a ‘database’ for the event-driven world
  36. A future of Aside: Serverless Functions

  37. Bad Analogy Serverless Functions are like very small microservices

  38. Serverless Functions (FaaS) • Write a function • Upload •

    Configure a trigger (HTTP, Messaging, Object Store, Database, Timer etc.) Request Respond Event Source
  39. FaaS in a Nutshell • Fully managed • Pay for

    execution time • Auto-scales with load • Short lived
  40. Better Analogy Serverless Functions small event-driven containers

  41. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS

    FaaS/μS FaaS/μS FaaS/μS FaaS More events created -> More “containers” are spun up in response More images stored More functions are triggered to create thumbnails
  42. In the future it seems unlikely that infrastructure will respond

    to events. But where will we manage our data?
  43. Mr Microservices

  44. Event-Streaming approaches this from a different angle

  45. FaaS is event-driven But it isn’t event-streaming

  46. Customers Event Source Orders Event Source Payments Event Source Serverless

    Way: event driven but not streaming FaaS/μS FaaS/μS FaaS/μS Use Case: Email a customer the details of their order
  47. Payments Event Source Increase Load: 100 concurrent functions doing IO.

    FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS
  48. Payments Event Source What if we’re only interested in platinum

    clients? -> 100 functions, 100 DB calls, 2 emails. FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS FaaS/μS
  49. FaaS FaaS FaaS KSQL KSQL in-lines data processing into the

    flow FaaS FaaS Stateless “Business Logic” Filters what isn’t needed Orders Payments Customers Everything now enriched, summarized in the events
  50. FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful “Database”

    FaaS FaaS FaaS FaaS FaaS Streaming Stateless Stateless Stateless Compute Layer Stream processors acting as the ‘database’ of the event driven world.
  51. THREE TRENDS • Events as first-class citizens • Microservices: provision

    data as a service • Stream Processing becomes a ‘database’ for the event-driven world
  52. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems