Saga pattern Powered by Kogito A solution for distributed architectures

Traditional applications (JTA)

Traditional applications (JTA) ● Transaction Manager coordinates the transaction across multiple resources ● Shared domain and database ● Two-phase commit (2PC) ○ Atomic ○ Consistent ○ Isolated ○ Durable

Distributed services - Tradeoffs ● Isolated domains and databases ● Network (bandwidth, latency, etc.) ● Remote service availability ● Transports (HTTP, JMS, gRPC, etc.) ● Languages (Go, Java, Python, etc.) ● Sync, Async or Event driven

Distributed services - Avoid transactions Batch processing ● Use timestamp for sorting commands / actions ● Process batch periodically Good for ● Long lived actions ● Event driven architectures ● Consistency as a requirement Not suitable for ● Short lived actions ● Performance as a requirement

Distributed services - Strong consistency Two-Phase Commit ● XA Standard ● WS-AtomicTransactions ● REST-AT Strong consistency puts an upper bound on scalability, throughput, low latency and availability. Good for ● Tightly coupled transactions ● Consistency as a requirement ● Short lived actions Not suitable for ● Event Driven Architectures ● Performance as a requirement ● Long running actions https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/5/html/transactions_development_guide/ch17s05#two-phase-commit-overview

Distributed services - Eventual consistency Sagas ● Design pattern to manage data consistency across participants in distributed transaction scenarios ● Based on compensating transactions (compensation ≠ rollback) ● Actions are idempotent ● Dirty reads (semantic lock, i.e. _Pending states) ● Handle timeouts and retries Good for ● Performance as a requirement ● Short lived actions ● Long lived actions ● Event Driven Architectures Not suitable for ● Consistency as a requirement

The Saga Pattern https://enmilocalfunciona.io/el-lado-oscuro-de-los-microservicios-transacciones-cross-service/

Choreography Orchestration Saga architectural styles

● Services reacting to events ● No workflow logic centralization ● Distribute the decision making and sequencing among the participants ● There is no SPOF ● May introduce cyclic dependencies between the services ● Difficult to monitor ● Difficult to evolve and change the workflow ● Simple to implement with no extra service to maintain ● Good for simple use cases (few number of services and events) Choreography

Orchestration Saga Execution Coordinator (SEC) ● Centralize the workflow logic ● Controls the steps ● Controls the compensations ● Introduces a potential SPOF ● SEC with high-availablity ● Allows an easier monitoring

Orchestration Avoid spreading control logic (failure management) in the whole codebase Teams can focus on each service domain logic Architecture evolution is easier Suitable for complex use cases (huge number of events and services) Avoid coupling events in multiple services

Choreography Orchestration Services need to know about other services events Services know only about their domain events PAYMENT_FAILED SHIPPING_CANCELLED RELEASE_STOCK

Adding a new Service Require changes in many places Easier to do with few changes

“The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years” (Martin Fowler) https://martinfowler.com/articles/201701-event-driven.html

Kogito as a SAGA orchestrator? ● Workflow engine ● Compensations ● Timers to control async response (send an event and wait for response) ○ Stateful timeouts/retries (e.g. 1 min, 1 hour, 1 day...) ● Saga can be modeled as a process under the hood ● Integration with Decisions, Rules, … ● Leverage existing Kogito features (data-index, job-service, management-console) ● Monitoring and tracing ● Distributed scalable architecture

Kogito already in previous KieLives [KieLive#18] Kogito - Powering Business Automation in the Cloud [KieLive#16] Kogito: The past, present and future

Workflow and compensation coordination ● Declarative (BPMN, Serverless Workflow, …) ● Simple to change and evolve ● Saga Log (states of each step) ● Control what should be compensated ● How to compensate ● Order to execute compensations ● Monitoring and management

Timeout control ● Non-blocking ● Stateful ● Retries / Replay ● Not trivial to implement ● OOTB in Kogito ● Monitoring and management

High level architecture

Simple Order Saga

Simple Order Saga

Simple Order Saga

Hands-on: Simple Order Saga Repository: https://github.com/tiagodolphine/kogito-examples/tree/process-saga-example/process-saga-quarkus mvn clean compile quarkus:dev curl -XPOST 'localhost:8080/orders' \ -H 'Content-Type: application/json' \ -D '{ "orderId": "order-123456", "failService" : "ShippingService" }'

Console Log

Advanced example

Cloud Native Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. https://github.com/cncf/foundation/blob/master/charter.md *Emphasis mine

Cloud Native Management Console Data Index Service Jobs Service

SAGA Architecture

Orders Saga - Success event flow

Orders Saga - Compensation event flow

Orders Saga - Compensation event flow

Demo - Stock process

Demo - Stock process (cont.)

Demo - Success

Demo - Compensation

Scalability, performance, high-throughput Eventual consistency is acceptable Kogito providing Saga orchestration in a cloud native way Leverage OOTB tools already provided by Kogito Conclusions about Sagas

Goals and future work ● Increase BPMN constructs ● Serverless Workflow with compensation ● Architecture improvements (persistence, jobs, data-index,…) ● LRA addon ● More to come on next sessions ○ Cancelation, confirmation and compensation ack

References ● Sagas - Hector Garcia. Princeton University (1987) ● Coordinating Multi-Transaction Activities - Hector Garcia. Princeton University (1990) ● Saga: How to implement complex business transactions without two phase commit - Bernd Rücker ● Saga - Microservices.io - Chris Richardson ● Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey ● Modern day service orchestration using DSLs - Thoughtworks ● Saga distributed transactions pattern - Microsoft Azure ● Patterns for distributed transactions within a microservices architecture - Keyang Xiang - Red Hat Developers ● Kogito Documentation