KieLive#20: Saga pattern powered by Kogito

Let's talk about the Saga pattern, why it is useful in microservices and distributed architectures and how to use Kogito to implement it.

Link to the live streaming: http://red.ht/KieLive20

KieLive#20: Saga pattern powered by Kogito

Microservices, event-driven and serverless architectures are the reality in a huge number of companies nowadays. This architecture favours domain and data isolation. Services are easier to maintain, replace, test, monitor and operate. With all the good parts it comes with the downsides, as we know there is no free lunch. One of the biggest concerns in the distributed world is how to control and manage long-lived transactions that can be simply correlated to business operations, for example, an order fulfillment. Therefore, keeping consistency in the whole architecture is a challenge since there is no single DBMS in charge of this hard work. Enter the Saga pattern! That is not a new solution per-se, in fact, it was coined in the '80s. Let's see how Sagas can be used as a solution for this problem and mainly how we can leverage Kogito to make things easier and robust.

About the invited speakers:
Tiago Dolphine is a senior Software Engineer at Red Hat working with business automation and open-source in Kogito and jBPM projects, providing solutions and tools for customers and community.

He also has a background in the production environment, with experience of monolith to microservices migration, besides he is an enthusiast in distributed systems, mainly based in reactive event-driven architectures aiming for scalability and performance.

Ruben Romero is a Software Engineer part of the OpenShift Middleware Solutions Engineering team at Red Hat. Ruben has been contributing to Kogito and jBPM for the last two years and enjoys Open Source and Business Automation.

Transcript

1. Saga pattern Powered by Kogito A solution for distributed architectures
2. Traditional applications (JTA)
3. Traditional applications (JTA) • Transaction Manager coordinates the transaction across multiple resources • Shared domain and database • Two-phase commit (2PC) ◦ Atomic ◦ Consistent ◦ Isolated ◦ Durable
4. Distributed services - Tradeoffs • Isolated domains and databases • Network (bandwidth, latency, etc.) • Remote service availability • Transports (HTTP, JMS, gRPC, etc.) • Languages (Go, Java, Python, etc.) • Sync, Async or Event driven
5. Distributed services - Avoid transactions Batch processing • Use timestamp for sorting commands / actions • Process batch periodically Good for • Long lived actions • Event driven architectures • Consistency as a requirement Not suitable for • Short lived actions • Performance as a requirement
6. Distributed services - Strong consistency Two-Phase Commit • XA Standard • WS-AtomicTransactions • REST-AT Strong consistency puts an upper bound on scalability, throughput, low latency and availability. Good for • Tightly coupled transactions • Consistency as a requirement • Short lived actions Not suitable for • Event Driven Architectures • Performance as a requirement • Long running actions https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/5/html/transactions_development_guide/ch17s05#two-phase-commit-overview
7. Distributed services - Eventual consistency Sagas • Design pattern to manage data consistency across participants in distributed transaction scenarios • Based on compensating transactions (compensation ≠ rollback) • Actions are idempotent • Dirty reads (semantic lock, i.e. _Pending states) • Handle timeouts and retries Good for • Performance as a requirement • Short lived actions • Long lived actions • Event Driven Architectures Not suitable for • Consistency as a requirement
8. The Saga Pattern https://enmilocalfunciona.io/el-lado-oscuro-de-los-microservicios-transacciones-cross-service/
9. Choreography Orchestration Saga architectural styles
10. • Services reacting to events • No workflow logic centralization • Distribute the decision making and sequencing among the participants • There is no SPOF • May introduce cyclic dependencies between the services • Difficult to monitor • Difficult to evolve and change the workflow • Simple to implement with no extra service to maintain • Good for simple use cases (few number of services and events) Choreography
11. Orchestration Saga Execution Coordinator (SEC) • Centralize the workflow logic • Controls the steps • Controls the compensations • Introduces a potential SPOF • SEC with high-availablity • Allows an easier monitoring
12. Orchestration Avoid spreading control logic (failure management) in the whole codebase Teams can focus on each service domain logic Architecture evolution is easier Suitable for complex use cases (huge number of events and services) Avoid coupling events in multiple services
13. Choreography Orchestration Services need to know about other services events Services know only about their domain events PAYMENT_FAILED SHIPPING_CANCELLED RELEASE_STOCK
14. Adding a new Service Require changes in many places Easier to do with few changes
15. “The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years” (Martin Fowler) https://martinfowler.com/articles/201701-event-driven.html
16. Kogito as a SAGA orchestrator? • Workflow engine • Compensations • Timers to control async response (send an event and wait for response) ◦ Stateful timeouts/retries (e.g. 1 min, 1 hour, 1 day...) • Saga can be modeled as a process under the hood • Integration with Decisions, Rules, … • Leverage existing Kogito features (data-index, job-service, management-console) • Monitoring and tracing • Distributed scalable architecture
17. Kogito already in previous KieLives [KieLive#18] Kogito - Powering Business Automation in the Cloud [KieLive#16] Kogito: The past, present and future
18. Workflow and compensation coordination • Declarative (BPMN, Serverless Workflow, …) • Simple to change and evolve • Saga Log (states of each step) • Control what should be compensated • How to compensate • Order to execute compensations • Monitoring and management
19. Timeout control • Non-blocking • Stateful • Retries / Replay • Not trivial to implement • OOTB in Kogito • Monitoring and management
20. High level architecture
21. Simple Order Saga
22. Simple Order Saga
23. Simple Order Saga
24. Hands-on: Simple Order Saga Repository: https://github.com/tiagodolphine/kogito-examples/tree/process-saga-example/process-saga-quarkus mvn clean compile quarkus:dev curl -XPOST 'localhost:8080/orders' \ -H 'Content-Type: application/json' \ -D '{ "orderId": "order-123456", "failService" : "ShippingService" }'
25. Console Log
26. Advanced example
27. Cloud Native Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach. These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. https://github.com/cncf/foundation/blob/master/charter.md *Emphasis mine
28. Cloud Native Management Console Data Index Service Jobs Service
29. SAGA Architecture
30. Orders Saga - Success event flow
31. Orders Saga - Compensation event flow
32. Orders Saga - Compensation event flow
33. Demo - Stock process
34. Demo - Stock process (cont.)
35. Demo - Success
36. Demo - Compensation
37. Scalability, performance, high-throughput Eventual consistency is acceptable Kogito providing Saga orchestration in a cloud native way Leverage OOTB tools already provided by Kogito Conclusions about Sagas
38. Goals and future work • Increase BPMN constructs • Serverless Workflow with compensation • Architecture improvements (persistence, jobs, data-index,…) • LRA addon • More to come on next sessions ◦ Cancelation, confirmation and compensation ack
39. References • Sagas - Hector Garcia. Princeton University (1987) • Coordinating Multi-Transaction Activities - Hector Garcia. Princeton University (1990) • Saga: How to implement complex business transactions without two phase commit - Bernd Rücker • Saga - Microservices.io - Chris Richardson • Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey • Modern day service orchestration using DSLs - Thoughtworks • Saga distributed transactions pattern - Microsoft Azure • Patterns for distributed transactions within a microservices architecture - Keyang Xiang - Red Hat Developers • Kogito Documentation