Event Streaming Rises Lee Dongjin | [email protected]

Contents ● Background: Modern Enterprise Architecture ○ Problems ● Event Streaming ○ Approaches ○ Log ● Related Projects ● Conclusion

Background: Modern Enterprise Architecture MS MS Application Data Integration Data Processing

Problems: Application Layer ● Polyglot Persistence ○ ACID Property ○ Atomicity or Abortability - but how? ● Heterogeneous Distributed Transaction ○ cf: Homogeneous Distributed Transaction ○ ex: VoltDB, Google Spanner ● Limitations of 2PC ○ Blocking Protocol ○ Not well supported Process

Problems: Data Integration Layer ● ETL: Extract, Transform, Load ● Low coverage ● Too slow ○ Can’t handle real-time data! ● Not scalable ○ Hard to Extract ○ Repeated Load ○ Complexity: O(m * n)

Problems: Data Processing Layer ● All problems in Data Integration Layer AND... ● Processing Logic = DAG of Processors ○ No reliable real-time input / output ○ Complex to connect storages ■ Schema: Easy to break ○ Processor w/ Internal State ■ How to recover on failure? ● In short: A TOTAL MESS. Filter Enrich Agg

Event Streaming ● Storage as a Mutable State ○ … of final snapshot ○ Bound collection of records ● Storage as a Sequence of Immutable Events ○ … being accumulated ○ Unbound collection of immutable events

Distributed Transaction w/ Event Streaming (1) ● Reactive approach (detects events from & emits events to event store) Process Event Store Storage A Storage B Storage C

Distributed Transaction w/ Event Streaming (2) ● Process: emits transaction request Process Event Store Storage A Storage B Storage C TX 1

Distributed Transaction w/ Event Streaming (3) ● Storage A: detects transaction request & emits success event on A Process Event Store Storage A Storage B Storage C TX 1 TX 1/A

Distributed Transaction w/ Event Streaming (4) ● Storage B: detects success event on A & emits success event on B Process Event Store Storage A Storage B Storage C TX 1 TX 1/A TX 1/B …

Distributed Transaction w/ Event Streaming (5) ● Storage C: detects success event on B & emits success event on C Process Storage A Storage B Storage C TX 1 TX 1/A TX 1/B … … … TX 1/C

Distributed Transaction w/ Event Streaming (6) ● Process: detects success event on C & emits transaction success Process Storage A Storage B Storage C TX 1 TX 1/A TX 1/B … … … TX 1/C TX 1 …

Distributed Transaction w/ Event Streaming (7) ● If failure: for example, Storage B emits failure event on B Process Storage A Storage B Storage C TX 1 TX 1/A TX 1/B … TX 1/C TX 1 … TX 2 TX 2/A TX 2/B

Distributed Transaction w/ Event Streaming (8) ● Process: detects failure event on A, B, or C and emits transaction failure Process Storage A Storage B Storage C TX 1 TX 1/A TX 1/B … TX 1/C TX 1 TX 2 TX 2/A TX 2/B TX 2

Distributed Transaction w/ Event Streaming (9) ● Storage A, B or C: detects transaction failure, rollbacks transaction Process Storage A Storage B Storage C TX 1 TX 1/A TX 1/B … TX 1/C TX 1 TX 2 TX 2/A TX 2/B TX 2

Conditions for Event Store ● Optimized to write ● Ordering guarantee ○ Read events in stored order ● Subscribable ● Scalability & Fault tolerance Event Store

Log: A data structure (1) ● Append-only Data Structure ○ Log Sequence No. (=id), Payload ○ No update or delete ● Provides… ○ Optimized to write ○ Ordering guarantee ○ Logical timestamp ■ Independent to physical time

Log: A data structure (2) ● Not seen, but Everywhere ○ At the heart of the data system ● From Traditional Databases to Distributed Systems ○ A Tool to achieve Atomicity and Durability ■ Write Ahead Log (WAL), Commit Log, Transaction Log, Replication Log, ... ○ A Storage itself ■ Log Structured Merge tree ■ LevelDB (by Google), RocksDB (by Facebook) ■ Google BigTable, HBase HFile, Cassandra SSTable

Log as a Distributed Storage (1) ● Log as an application programming model ○ Append Only ○ Sequential Read w/ Subscribing ○ Durable ● Log as a Distributed Storage ○ Scalability by Partitioning ■ Partition Count = Maximum Parallelism ■ Ordering guarantee only in Partition ■ Key for partitioning ○ Fault Tolerance by Replication ■ Maintains multiple copies of each partition

Log as a Distributed Storage (2) ● Implementations ○ Apache Kafka (by Linkedin) ○ Apache Bookeeper (by Yahoo! Research) ○ Apache Pulsar ○ CORFU (by Microsoft Research) ○ LogDevice (by Facebook) ○ AWS Kinesis, GCP Pub/Sub, Azure Event Hub

OLEP: OnLine Event Processing ● Advantages ○ Scalable Distributed Transaction ● Limitations ○ Eventual Consistency ○ Isolation Level ■ Read Committed only ■ Snapshot Isolation is under research P1 P2 P3 t1 t2

What type of Streams? ● Event Stream ○ Accumulates events ○ A way to communicate between distributed services ■ Mainly microservices ○ Key for partition assignment (optional) ● Update Stream (AKA Changelog stream) ○ Represents modification history of a state ○ A way to represent the logical state of a storage ■ Including (but not restricted to) databases ○ Key for record identity (required) Update Stream Event Stream

Update Stream (AKA Changelog Stream) (1) ● Example Key Value k1 v1 Key Value k1 v1 k2 v2 Key Value k1 v1-1 k2 v2 Key Value k1 v1-1 (k2, v2) (k1, v1-1) (k2, null) State Update Stream

Update Stream (AKA Changelog Stream) (2) ● Stream-Table Duality ○ Table → Stream ■ All states (including storages) can be represented as a form of update stream ○ Stream → Table ■ … and Vice Versa ■ State as a Materialized View w/ timestamp ● Examples ○ Replication Log ○ CDC (Change Data Capture) feature like Oracle GoldenGate

Data Integration with Update Stream (1) ● Example: replicating MySQL Mysql (original) Mysql (replicated) Mysql (w/ Index) BigQuery Changelog

Data Integration with Update Stream (2) ● Extract once ○ w/ Predefined Schema ○ w/ Cleansing ● Transform as many times you need ● Load per storage type ● Advantages: Scalable & Real-time! Original Storage Changelog Stream Derived Stream Destination Storages

● In event-streaming world, the schema of events stored in Log works as a contract between different microservices or teams. ○ … dislike to RPC world, where API works as a Contract in RPC. ● Each devops team that publishes event stream takes the responsibility of maintaining the stream Clean and Forward-compatible. ● Example: A dedicated service to register, document, evolve, and validate schemas ○ https://issues.apache.org/jira/browse/AVRO-1124 Schema as a Contract

Machine Replication Principle ● “Updating state deterministically based on an ordered events” ● “A processor with internal state can be serialized into changelog, and vice versa.” Processor w/ State

Real-time Data Processing Model w/ Log ● Data Processing Task ○ A Dag of Processors ○ Read from Log, Write to Log ● Processor ○ Stateless Processor ○ Stateful Processor ■ Fault Tolerance w/ Internal State Changelog ■ Crash: Can recover from Changelog! Filter Enrich Aggre gation

Data Processing Architecture (1) ● Lambda Architecture

Data Processing Architecture (2) ● Kappa Architecture

Related Projects: Extract and Load (1) ● Kafka Connect ○ De-facto Standard ○ Supports both of Table → Stream and Stream → Table ○ Provides ‘Extract and Load’ functionality (w/ little Transformation) ● Brooklin (by Linkedin) ○ Provides a generic bridge between heterogeneous Streaming / Storage Systems ■ Table → Table, Table → Stream, Stream → Table, Stream → Stream ○ Deployed as a dedicated system ○ Open-sourced in 2019

Related Projects: Extract and Load (2) ● Debezium ○ Focuses CDC only (Table → Stream) ○ Supports MySQL, PostgreSQL, Oracle, SQL Server, MongoDB, Cassandra, ... ○ Deployed as Kafka Connect plugin or Embedded Engine ● DBLog (by Netflix) ○ Focuses CDC only (Table → Stream) ○ Supports MySQL and PostgreSQL ○ Deployed as a dedicated system ○ Planning to be open-sourced in 2020

Related Projects: Processing Framework ● Approaches ○ Real-time MapReduce ○ Event-Driven Microservices

Related Projects: Processing Framework ● Real-time MapReduce ○ Spark ○ Flink ● Event-Driven Microservices ○ Kafka Streams ○ ksqlDB ■ “Streaming application as a Query”

Conclusion (1) ● Event streaming is a paradigm shift in storing & processing data. ● Log is a primary programming model for Event streaming.

Conclusion (2) ● Log as a primary programming model ○ … provides an excellent abstraction for a wide range of problems ● Application ○ Distributed Heterogeneous Transaction ● Data Integration ○ Scalable ETL w/ great coverage in real-time ● Data Processing ○ Stateful processor w/ fault tolerance

Conclusion (3) ● Traditional concepts in enterprise application are changing ○ No boundaries, No orders anymore ○ “All processes are just a kind of processor w/ state store interconnected via Log.”

… And one more thing (1) Distributed Application Database Distributed Log Log Changelog stream → State Table updates → Materialized View State per access pattern Table with Index State to Changelog Stream Table to Transaction Log Log to Log processing WAL to Transaction Log processing

… And one more thing (2) “Any sufficiently advanced distributed application is indistinguishable from database.”

Questions?

References (1) ● Jay Kreps, ‘The Log: What every software engineer should know about real-time data's unifying abstraction’, Linkedin Engineering, 2013 ○ https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should- know-about-real-time-datas-unifying ● Adam Jacobs, ‘The Pathologies of Big Data’, acmqueue volume 7, issue 6, 2009 ○ https://queue.acm.org/detail.cfm?id=1563874 ● Martin Kleppmann et al, ‘Online Event Processing’, acmqueue volume 17, issue 1, 2019 ○ https://queue.acm.org/detail.cfm?id=3321612 ● O’Neil et al, ‘The log-structured merge-tree (LSM-tree)’, Acta Informatica volume 33, 1996 ○ https://dl.acm.org/doi/10.1007/s002360050048

References (2) ● Jeffrey Dean et al, ‘Bigtable: A Distributed Storage System for Structured Data’, Usenix, 2006 ○ https://research.google/pubs/pub27898/ ● Jay Kreps, ‘I ♥ Logs: Apache Kafka, Stream Processing, and Real-time Data’, Stanford Seminar, 2014 ○ https://www.youtube.com/watch?v=SU8LaHLh6Ng ● Jay Kreps, ‘Questioning Lambda Architecture’, oreilly.com, 2014 ○ https://www.oreilly.com/radar/questioning-the-lambda-architecture/ ● Matthias J. Sax et al, 'Streams and Tables: Two Sides of the Same Coin', Proceedings of the international workshop on Real-time business intelligence and Analytics, 2018 ○ https://dl.acm.org/doi/10.1145/3242153.3242155

References (3) ● Martin Kleppmann and Jay Kreps, ‘Kafka, Samza and the Unix philosophy of distributed data’, IEEE Data Engineering Bulletin 38, 2015 ○ https://martin.kleppmann.com/2015/12/15/stream-processing-data-engineering-bulletin.html ● Haruna Isah et al, ‘A Survey of Distributed Data Stream Processing Frameworks’, IEEE Access volume 7, 2019 ○ https://ieeexplore.ieee.org/document/8864052 ● Samarth Shetty, ‘Streaming Data Pipelines with Brooklin’, Linkedin Engineering, 2017 ○ https://engineering.linkedin.com/blog/2017/10/streaming-data-pipelines-with-brooklin ● Celia Kung, ‘Open Sourcing Brooklin: Near Real-Time Data Streaming at Scale’, Linkedin Engineering, 2019 ○ https://engineering.linkedin.com/blog/2019/brooklin-open-source

References (4) ● Andreas Andreakis and Ioannis Papapanagiotou, ‘DBLog: A Generic Change-Data-Capture Framework’, Netflix Technology, 2019 ○ https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b ● Jay Kreps, ‘Introducing ksqlDB’, Confluent, 2019 ○ https://ksqldb.io/