The Shell Game Called Eventual Consistency

As we build distributed highly scalable systems the central data store and transactions are no longer a safety net we can afford. In the world of event sourcing and CQRS (Command Query Responsibility Segregation) we need to design clever systems that don't show cracks and seams where eventual consistency is at play. We will tackle those unpleasant invariants and race conditions head on to investigate some technical and non technical smoke and mirror solutions that we can use to deliver a positive experience to end-users while finding the performance sweetspot.

We are utilizing the various PaaS/Serverless solutions to build more and more distributed systems. Often these systems need to work together to produce a result. When performance and scalability is of high priority, consistency (CAP theorem) takes a back seat. We still need to find ways to shelter the end-user from these design realities. The aim of this talk is to find ways of doing it. Be it through changing the business process or by doing clever tricks on the front end while giving the backend has a heartbeat to catch up. There are countless ways to do it. My goal is to investigate a few of them and get the conversations happening.

Transcript

1. The Shell Game Called Eventual Consistency 24 Feb 2021 Dasith Wijesiriwardena @dasiths
2. I am here because I have a love hate relationship with distributed systems. HELLO! I AM Dasith @dasiths dasith.me
3. Dealing with Eventual Consistency Introduction to CAP theorem Agenda Recap and Conclusion Strong Consistency vs Availability @dasiths
4. A Primer on CAP Theorem @dasiths
5. https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 @dasiths
6. Write Read @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Read
7. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read
8. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read
9. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read
10. C-P Systems Write Example 1: All nodes are connected @dasiths Consistent – Partition Tolerant
11. C-P Systems Write Example 1: All nodes are connected The system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant
12. C-P Systems Write Example 1: All nodes are connected Read The system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant
13. C-P Systems Write Example 1: All nodes are connected Read The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
14. C-P Systems Write Read Example 1: All nodes are connected Write Example 2: Connectivity intermittent The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
15. C-P Systems Write Read Example 1: All nodes are connected Write Example 2: Connectivity intermittent The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
16. C-P Systems Write Read Example 1: All nodes are connected Write Example 2: Connectivity intermittent The system isn’t available until connectivity is re-established. The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
17. • Strong consistency is critical. • Supports transactions that cover all nodes. • Generally easier to develop than A-P systems. C-P Systems @dasiths Consistent – Partition Tolerant
18. A-P Systems Write Example 1: All nodes are connected @dasiths Available – Partition Tolerant
19. A-P Systems Write Example 1: All nodes are connected Read The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant
20. A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. V1 V2 @dasiths Available – Partition Tolerant
21. A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
22. A-P Systems Write Example 1: All nodes are connected Read Read Write Example 2: Connectivity intermittent The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
23. A-P Systems Write Example 1: All nodes are connected Read Read Write Example 2: Connectivity intermittent The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
24. A-P Systems Write Example 1: All nodes are connected Read Read Write Example 2: Connectivity intermittent Read The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant
25. Read A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent Read The system is available but won’t be consistent until connectivity is re-established. V1 V2 @dasiths Available – Partition Tolerant
26. A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent The system is available but won’t be consistent until connectivity is re-established. @dasiths “Eventual Consistency” Available – Partition Tolerant Read Read V1 V2
27. • Availability is critical. • No all node covering transactions. (i.e. 2-phase commit). • Introduces “lesser forms of consistency”. • Developing can be complicated. A-P Systems @dasiths Available – Partition Tolerant
28. Looking at the Modern Landscape @dasiths
29. • Availability is a must. • RDBMS are not the only choice. • Strong consistency is expensive. •Storage cost is less than CPU/Memory costs. • Separate Read and Write optimised stores. In Hyperscale… @dasiths
30. The problem with Eventual Consistency @dasiths
31. Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem
32. Example: Strong Consistency Order requested @dasiths (C)onsistent-(P)artition TolerantSystem
33. Example: Strong Consistency Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem
34. Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem
35. Transaction Boundary BEGIN TRANSACTION SELECT @QtyAvailable = Quantity FROM Products WHERE ProductId = @ProductId IF @QtyAvailable > @QtyOrdered UPDATE Products SET Quantity = Quantity - @QtyOrdered WHERE ProductId = @ProductId ELSE THROW 51000, 'The ordered qty is more than available.', 1; COMMIT TRANSACTION Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem
36. Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem
37. Example: Strong Consistency Transaction Boundary View orders Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem
38. @dasiths
39. @dasiths
40. @dasiths
41. Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem
42. Order requested Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem
43. Order requested Acknowledged Example: Eventual Consistency Out of sync @dasiths (A)available-(P)artition TolerantSystem
44. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync @dasiths (A)available-(P)artition TolerantSystem
45. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync View orders @dasiths (A)available-(P)artition TolerantSystem
46. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync View orders @dasiths Order Does Not Exist!!! (A)available-(P)artition TolerantSystem
47. Order requested Acknowledged Propagate to other nodes Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem
48. Order requested Acknowledged Propagated to other partitions Example: Eventual Consistency Order confirmed @dasiths (A)available-(P)artition TolerantSystem
49. Order requested Acknowledged Propagated to other partitions Order confirmed Example: Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem
50. Order requested Acknowledged Propagated to other partitions Order confirmed Example: Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem
51. IT’S NOT THAT BAD. NO REALLY. @dasiths
52. Quick Thoughts on Eventual Consistency … Unless you are using pessimistic locking, all data is stale, there are possibilities of optimistic concurrency failures. There is some period of time that it takes to build the DTOs, put them on the wire and for the client to receive them and draw them on the screen. Greg Young http://codebetter.com/gregyoung/2010/04/14/quick-thoughts-on-eventual-consistency/ @dasiths
53. Starbucks Does Not Use Two-Phase Commit … In summary we can see that the real world is often asynchronous. Our daily lives consists of many coordinated, but asynchronous interactions (reading and replying to e-mail, buying coffee etc) … It also means that often we can look at daily life to help design successful messaging solutions. Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html @dasiths
54. Race Conditions Don’t Exist … Any time you see requirements that indicate a race condition, dig deeper. What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes… Udi Dahan https://udidahan.com/2010/08/31/race-conditions-dont-exist/ @dasiths
55. DEALING WITH EVENTUAL CONSISTENCY @dasiths
56. • Language Matters. Communicate actual status. ▪ Terminology like Submitted, Queued, Pending. • Multiple mediums of communication (Email, SMS etc). ▪ Let the customer know ASAP. • Compensating actions as first class domain concepts. ▪ Treat it as a long running process/workflow. Managing Expectations @dasiths
57. UI SLIGHT OF HAND SOME @dasiths
58. Use local storage as a fake cache Ways To Handle Eventual Consistency on The UI Disable and long poll Use a thank you screen WebSockets to show live progress @dasiths
59. Disable And Long Poll Order requested OrderId: ABC123 @dasiths
60. Disable And Long Poll Order requested OrderId: ABC123 @dasiths UI Disabled
61. Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While result is 404 @dasiths UI Disabled
62. Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While result is 404 Order details @dasiths Redirect to order details
63. Thank You Screen Order requested OrderId: ABC123 @dasiths
64. Thank You Screen Order requested OrderId: ABC123 Order Details @dasiths
65. Thank You Screen Order requested OrderId: ABC123 Order details @dasiths
66. Fake Cache Comment posted CommentId: ABC123 This is a stupid comment @dasiths
67. Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! var localComments = JSON.parse(localStorage.getItem('comments')) || []; var newComment = { 'questionId': "blah";, 'commentId': "ABC123";, 'commentText': commentText, 'isFromLocal': true }; localComments.push(newComment); localStorage.setItem('comments', JSON.stringify(localComments)); @dasiths
68. Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment
69. Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! questionId = "blah"; httpClient.Get(`api/question/${questionId}/comments`) .Subscribe(comments => { var localComments = JSON.parse(localStorage.getItem('comments')) || []; var questionComments = localComments .filter(q => q.questionId == questionId); // todo: Remove duplicates from local storage return [...comments, ...questionComments]; }) @dasiths
70. I see my comment Fake Cache Comment posted CommentId: ABC123 @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment But, I don’t see it yet!
71. Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! /api/quesion/blah/comments @dasiths /api/quesion/blah/comments Includes Lisa’s comment Doesn’t includes Lisa’s comment I see my comment I see it now too!
72. WebSockets Live Progress Order requested OrderId: ABC123 @dasiths
73. WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Finalizing… @dasiths
74. WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Created Order confirmed @dasiths
75. FAIL WHEN THINGS @dasiths
76. • Show details on screen if possible. • Communicate via multiple mediums. • Have a plan. Act Fast @dasiths
77. Disable+Poll Thank You Fake Cache Live Progress • Easy to implement. • UX isn’t great. • Easy to implement. • Assumes the system processes the request before user navigates out. • Requires complicated logic. • Good UX if implemented properly. • It’s fake. Use for non- critical use cases. • Requires WebSockets/HTTP2 for async server calls. • UI needs to handle server events and display. • Good UX. Recap @dasiths
78. EMBRACE THE REAL WORLD Async business processes are the reality @dasiths
79. http://pjousselin.free.fr @dasiths
80. Any questions? THANKS! @dasiths dasith.me https://www.nationalgeographic.com/travel/destinations/asia/sri-lanka/
81. Presentation template designed by powerpointify.com Special thanks to all people who made and shared these awesome resources for free: CREDITS Photographs by unsplash.com Free Fonts used: https://www.fontsquirrel.com/fonts/oswald @dasiths