Microservices - Where are my Transactions and my Consitency????

Transcript

1. Microservices: Where Are My Transactions And My Consistency??? Eberhard Wolff Head of Architecture https://swaglab.rocks/ https://ewolff.com/
2. Microservices are Distributed Systems
3. Distributed Systems Are Hard: Consistency & Reliability
4. Happened-before ordering in distributed systems earned me a reward. “Time, Clocks and the Ordering of Events in a Distributed System” Leslie Lamport
5. A distributed system is one in which the failure Leslie Lamport of a computer you didn't even know existed can render your own computer unusable.
6. Databases have a hard time Kyle Kingsbury implementing distributed algorithms correctly https://aphyr.com/tag s/jepsen
7. I am not nearly as smart as Leslie Lamport or Kyle Kingsbury.
8. Avoid Consistency Issues! •Store each piece of data in one microservice •Microservice uses a database •Database solves the problem …and Kyle Kingsbury helps me to choose the right one.
9. Entity Service Customer Service Product Service Order Process Delivery Process Invoicing Database for scaling …and consistency
10. Entity Service • Every call goes through three services. • Performance • Latency
11. Entity Service • Failure can easily propagate. • How can you make the system resilient? • Eberhard = default customer, PS5 = default product ? Customer Service Product Service Order Process Delivery Process Invoicing
12. Transactions • Order process changes customer and product • Probably need a transaction Customer Service Product Service Order Process Delivery Process Invoicing
14. ACID • Properties of classic transactions (tx) • Atomicity: Either do full tx full or nothing • Change customer and product • Consistency: No violated constraints • No negative number of products in stock • Otherwise: rollback
15. ACID • Isolation: Parallel tx isolated • Other orders either processed before or after • i.e. no concurrent decrease of stock • Durability: Result of tx stored • e.g. # in stock stored
16. ACID • Solves parallelism • …and increases reliability • Very desirable!
17. Distributed Tx?
18. Two Phase Commit (2PC): Phase 1 Coordinator Resource Resource Coordinator identifies resources during transaction execution Can commit? Can commit? Yes/No Yes/No
19. Two Phase Commit (2PC): Phase 2 Coordinator Resource Resource Commit/Rollback Commit/Rollback Done Done Coordinator
20. 2PC Customer Service Product Service Order Process Delivery Process Invoicing Transfer Tx ID via network Pass Tx ID to database Start commit / rollback Execute commit / rollback
21. 2PC •Outdated solutions existed •Borland / Inprise ITS •CORBA OTS, JTA, C++ interfaces •Oracle, IMS, CICS, Tuxedo, MQ Series interfaces •🤔 Can I make a fortune implementing that for HTTP, gRPC…?
22. 2PC •Problem: slows down the happy path •2nd phase + additional overhead
23. 2PC •Problem: Blocks resource •Resource must be blocked until 2PC is fully done. •Potentially long •What if client crashes? …or if there are network problems?
24. 2PC •Problem: no fully reliable Commit phase 2: What if resource is now unable to commit? Crashes of client / coordinator? •Tx should prevent problems •I.e. reliability is key
25. 2PC •Not possible with current technologies •For good reason!
26. 2PC = less performance + no 100% reliability
27. 2PC: No option nowadays for good reasons
28. Entity Service: DON’T • Bad performance • Bad reliability • Needs distributed tx – impossible Customer Service Product Service Order Process Delivery Process Invoicing
29. Entity Service: DON’T
30. ACID Transactions: DON’T and CAN’T
31. Loose Coupling & Modules • Put everything for a set of features in one microservice • Including all data • Result: bounded context with loose coupling Order Process Including all customer & product data for orders Delivery Process All data for delivery Invoicing Process All data for invoicing
32. Saga •Saga: Sequence of local transactions •Local transaction triggers next local transaction •Process done stepwise
33. Saga •Failure: compensation •Compensation undoes the original transaction •e.g. cancel an order •Canceled order are stored •i.e. the state change is not undone
34. Saga Order Process Delivery Process Invoicing Process
35. Saga Order Process Delivery Process Invoicing Process Compensate! Reimburse
36. Saga •Closer to what happens in the real world •No atomicity •No isolation •Also: not all steps have received the order •i.e. transaction in flight: some services have not received it yet •WS-Transaction standard (2002)
37. Saga •Requires reliable communication •E.g. messaging + retransmission + idempotency
38. 2PC instead of Saga Order Process Delivery Process Invoicing Process Rollback! Atomicity: All information about the order is gone! Rollback!
39. What is the status of my order #42? WAT????? WAT????????????? There is no order #42. That happens if the order was not paid. It’s safer and more consistent that way. But we have no data - can’t tell you whether that was the problem. Transactions
42. Saga vs 2PC •Saga might catch business logic better www.enterpriseintegrationpatterns.com/ramblings/18 _starbucks.html •Sagas cause less problems if distributed •Don’t do 2PC! •If ACID is needed: Put it in one microservice!
43. Consistency in DDD •Entity: Object distinguished by identity •Value object: distinguished by values •Aggregate: Cluster of entities and value object •Service: Process or transformation outside entity or value object •Transactions / ACID consistency: where?
44. Consistency in DDD •Use the same aggregate boundaries to govern transactions and distribution. •Within an aggregate boundary, apply consistency rules synchronously. Across boundaries, handle updates asynchronously. •I.e. aggregates order, delivery, invoice inconsistent
45. Consistency in DDD •Entities / value objects inside order aggregate are consistent •Aggregate in order process bounded context might be inconsistent •Bounded contexts might be inconsistent
46. CAP
47. CAP Theorem: Chose 2 of C, A, P Consistency Partition Tolerance Availability Quorum Replication DNS NOT POSSIBLE!* * https://codahale.com/you-cant-sacrifice-partition-tolerance
48. BASE Basically Available Soft state Eventually consistent AP = Available not consistent
49. ACID Consistency is not CAP Consistency!
50. ACID vs CAP Consistency • ACID Consistency: No violated constraints • i.e. invariants, referential integrity etc • CAP consistency: All nodes have the same data
51. What Did I Order? Order Process Invoicing Process Delivery Process
52. Has the Invoice been Payed? Order Process Invoicing Process Delivery Process
53. Where is my Delivery? Order Process Invoicing Process Delivery Process
54. Where is the distribution?
55. Order Accepted •Start delivery & invoicing •Possible inconsistency: invoice, no delivery yet Order Process Delivery Process Invoicing Process
56. Delivery Not Possible •Cancel order •Reimburse! •Possible inconsistence: no reimbursement yet Order Process Delivery Process Invoicing Process
57. Conclusion
58. Conclusion •No ACID transactions between microservices •Saga: String of transactions + compensation •Saga might be the better option to model domain logic
59. Conclusion •Aggregates are supposed to be ACID consistent, not bounded contexts. •CAP shouldn’t be a huge issue •All depends on good split of business logic!
60. https://software-architektur.tv/2021/02/09/folge40.html
62. https://software-architektur.tv/2020/11/13/folge025.html
63. https://www.youtube.com/watch?v=1ELpfc4JOC0
64. https://www.socreatory.com/de/trainers/eberhard-wolff 19.2.2024 Software Architektur Kickstart