Business Logic Payments Business Logic DB RPC Call RPC Response RPC Call RPC Response RPC Call RPC Response REST Call REST Call ▪ Perform an order iff there is stock available and the payment is cleared. ▪ Services are stateless ▪ Database does the heavy- liVing ▪ High latency, costly coordina?on calls
business logic. ▪ Services now are stateful ▪ Low-latency access to local state ▪ Service calls s?ll expensive ▪ Not obvious how to scale this out. Services Architecture (2): Embedded State/DB Order DB Business Logic Stock DB Business Logic Payments DB Business Logic REST Call REST Call
an event-log. ▪ Services are asynchronous/reac?ve. ▪ If we lose state, we replay the log and rebuild it. ▪ Time-travel debugging, audits, etc. are trivial. Services Architecture (3): Event Sourcing Order DB Business Logic Stock DB Business Logic Payments DB Business Logic REST Call REST Call event-log event-log
Responses event-log event-log Order 1 Business Logic Order 2 Business Logic Order 3 Business Logic Stock 1 Business Logic Stock 2 Business Logic Payment 1 Business Logic event-log DB DB DB DB DB DB
of machines ▪ Consistent snapshots of state, with exactly-once guarantees ▪ We can scale-in/out operators to handle varying workloads Does it ring a bell? OP1 OP4 OP2 OP5 Input Message Queues OP3 Output Message Queue Apache Flink
and message broker Guaranteed message delivery and exactly-once processing Each operator executes a group of functions that share the same state Operator-local state partitioned on key input for scalability and fault-tolerance
= { case (Pong(_, s), PingCounter(n, i, allRequests)) ⇒ PingCounter(n, i + 1, s +: allRequests) case (_, state) ⇒ state // Do nothing with the State } // rest of the implementation } 42 FnNamespace λ1 λ2 λn type EventHandler[S] = PartialFunction[(FnResponse, S), S]
central place in service-oriented architectures, taking over the execution of acid transactions, ensuring message delivery and processing, in order to perform scalable execution of services.
15th, covering all fundamental stream processing concepts (?me, order, windows, joins, etc.). We are using Apache Flink for assignments & include invited talks from industry & academia. Enrollment is open. tudelft.nl/taming-big-data-streams Shameless plug by Asterios Katsifodimos @kasterios
Akhter, Marios Fragkoulis, Asterios Katsifodimos. In the Proceedings of the 45th Interna?onal Conference on Very Large Data Bases (VLDB) 2019. 2. OperaConal Stream Processing: Towards Scalable and Consistent Event-Driven ApplicaCons: Asterios Katsifodimos, Marios Fragkoulis. In the Proceedings of the 22nd Interna?onal Conference on Extending Database Technology (EDBT) 2019. 3. "Benchmarking Distributed Stream Data Processing Systems": Jeyhun Karimov, Tilmann Rabl, Asterios Katsifodimos, Roman Samarev, Henri Heiskanen, Volker Markl. In the Proceedings of the Interna?onal Conference on Data Engineering (ICDE) 2018. 4. “Efficient Window AggregaCon with General Stream Slicing”: Jonas Traub, Philipp M. Grulich, Alejandro Rodriguez Cuellar, Sebas?an Breß, Asterios Katsifodimos, Tilmann Rabl and Volker Markl. In the Proceedings of the 22nd Interna?onal Conference on Extending Database Technology (EDBT) 2019. 74