Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Shell Game Called Eventual Consistency

The Shell Game Called Eventual Consistency

As we build distributed highly scalable systems the central data store and transactions are no longer a safety net we can afford. In the world of event sourcing and CQRS (Command Query Responsibility Segregation) we need to design clever systems that don't show cracks and seams where eventual consistency is at play. We will tackle those unpleasant invariants and race conditions head on to investigate some technical and non technical smoke and mirror solutions that we can use to deliver a positive experience to end-users while finding the performance sweetspot.

We are utilizing the various PaaS/Serverless solutions to build more and more distributed systems. Often these systems need to work together to produce a result. When performance and scalability is of high priority, consistency (CAP theorem) takes a back seat. We still need to find ways to shelter the end-user from these design realities. The aim of this talk is to find ways of doing it. Be it through changing the business process or by doing clever tricks on the front end while giving the backend has a heartbeat to catch up. There are countless ways to do it. My goal is to investigate a few of them and get the conversations happening.

B4c7e56ec325c0c55d5e29c7ab42e49f?s=128

Dasith Wijesiriwardena

February 21, 2021
Tweet

Transcript

  1. The Shell Game Called Eventual Consistency 24 Feb 2021 Dasith

    Wijesiriwardena @dasiths
  2. I am here because I have a love hate relationship

    with distributed systems. HELLO! I AM Dasith @dasiths dasith.me
  3. Dealing with Eventual Consistency Introduction to CAP theorem Agenda Recap

    and Conclusion Strong Consistency vs Availability @dasiths
  4. A Primer on CAP Theorem @dasiths

  5. https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 @dasiths

  6. Write Read @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Read

  7. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read

  8. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read

  9. @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read

  10. C-P Systems Write Example 1: All nodes are connected @dasiths

    Consistent – Partition Tolerant
  11. C-P Systems Write Example 1: All nodes are connected The

    system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant
  12. C-P Systems Write Example 1: All nodes are connected Read

    The system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant
  13. C-P Systems Write Example 1: All nodes are connected Read

    The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
  14. C-P Systems Write Read Example 1: All nodes are connected

    Write Example 2: Connectivity intermittent The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
  15. C-P Systems Write Read Example 1: All nodes are connected

    Write Example 2: Connectivity intermittent The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
  16. C-P Systems Write Read Example 1: All nodes are connected

    Write Example 2: Connectivity intermittent The system isn’t available until connectivity is re-established. The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant
  17. • Strong consistency is critical. • Supports transactions that cover

    all nodes. • Generally easier to develop than A-P systems. C-P Systems @dasiths Consistent – Partition Tolerant
  18. A-P Systems Write Example 1: All nodes are connected @dasiths

    Available – Partition Tolerant
  19. A-P Systems Write Example 1: All nodes are connected Read

    The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant
  20. A-P Systems Write Example 1: All nodes are connected Read

    Read The system isn’t immediately consistent but immediately available. V1 V2 @dasiths Available – Partition Tolerant
  21. A-P Systems Write Example 1: All nodes are connected Read

    Read The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
  22. A-P Systems Write Example 1: All nodes are connected Read

    Read Write Example 2: Connectivity intermittent The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
  23. A-P Systems Write Example 1: All nodes are connected Read

    Read Write Example 2: Connectivity intermittent The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant
  24. A-P Systems Write Example 1: All nodes are connected Read

    Read Write Example 2: Connectivity intermittent Read The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant
  25. Read A-P Systems Write Example 1: All nodes are connected

    Read Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent Read The system is available but won’t be consistent until connectivity is re-established. V1 V2 @dasiths Available – Partition Tolerant
  26. A-P Systems Write Example 1: All nodes are connected Read

    Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent The system is available but won’t be consistent until connectivity is re-established. @dasiths “Eventual Consistency” Available – Partition Tolerant Read Read V1 V2
  27. • Availability is critical. • No all node covering transactions.

    (i.e. 2-phase commit). • Introduces “lesser forms of consistency”. • Developing can be complicated. A-P Systems @dasiths Available – Partition Tolerant
  28. Looking at the Modern Landscape @dasiths

  29. • Availability is a must. • RDBMS are not the

    only choice. • Strong consistency is expensive. •Storage cost is less than CPU/Memory costs. • Separate Read and Write optimised stores. In Hyperscale… @dasiths
  30. The problem with Eventual Consistency @dasiths

  31. Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem

  32. Example: Strong Consistency Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

  33. Example: Strong Consistency Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

  34. Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths

    (C)onsistent-(P)artition TolerantSystem
  35. Transaction Boundary BEGIN TRANSACTION SELECT @QtyAvailable = Quantity FROM Products

    WHERE ProductId = @ProductId IF @QtyAvailable > @QtyOrdered UPDATE Products SET Quantity = Quantity - @QtyOrdered WHERE ProductId = @ProductId ELSE THROW 51000, 'The ordered qty is more than available.', 1; COMMIT TRANSACTION Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem
  36. Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths

    (C)onsistent-(P)artition TolerantSystem
  37. Example: Strong Consistency Transaction Boundary View orders Order confirmed Order

    requested @dasiths (C)onsistent-(P)artition TolerantSystem
  38. @dasiths

  39. @dasiths

  40. @dasiths

  41. Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem

  42. Order requested Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem

  43. Order requested Acknowledged Example: Eventual Consistency Out of sync @dasiths

    (A)available-(P)artition TolerantSystem
  44. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency

    Out of sync @dasiths (A)available-(P)artition TolerantSystem
  45. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency

    Out of sync View orders @dasiths (A)available-(P)artition TolerantSystem
  46. Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency

    Out of sync View orders @dasiths Order Does Not Exist!!! (A)available-(P)artition TolerantSystem
  47. Order requested Acknowledged Propagate to other nodes Example: Eventual Consistency

    @dasiths (A)available-(P)artition TolerantSystem
  48. Order requested Acknowledged Propagated to other partitions Example: Eventual Consistency

    Order confirmed @dasiths (A)available-(P)artition TolerantSystem
  49. Order requested Acknowledged Propagated to other partitions Order confirmed Example:

    Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem
  50. Order requested Acknowledged Propagated to other partitions Order confirmed Example:

    Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem
  51. IT’S NOT THAT BAD. NO REALLY. @dasiths

  52. Quick Thoughts on Eventual Consistency … Unless you are using

    pessimistic locking, all data is stale, there are possibilities of optimistic concurrency failures. There is some period of time that it takes to build the DTOs, put them on the wire and for the client to receive them and draw them on the screen. Greg Young http://codebetter.com/gregyoung/2010/04/14/quick-thoughts-on-eventual-consistency/ @dasiths
  53. Starbucks Does Not Use Two-Phase Commit … In summary we

    can see that the real world is often asynchronous. Our daily lives consists of many coordinated, but asynchronous interactions (reading and replying to e-mail, buying coffee etc) … It also means that often we can look at daily life to help design successful messaging solutions. Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html @dasiths
  54. Race Conditions Don’t Exist … Any time you see requirements

    that indicate a race condition, dig deeper. What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes… Udi Dahan https://udidahan.com/2010/08/31/race-conditions-dont-exist/ @dasiths
  55. DEALING WITH EVENTUAL CONSISTENCY @dasiths

  56. • Language Matters. Communicate actual status. ▪ Terminology like Submitted,

    Queued, Pending. • Multiple mediums of communication (Email, SMS etc). ▪ Let the customer know ASAP. • Compensating actions as first class domain concepts. ▪ Treat it as a long running process/workflow. Managing Expectations @dasiths
  57. UI SLIGHT OF HAND SOME @dasiths

  58. Use local storage as a fake cache Ways To Handle

    Eventual Consistency on The UI Disable and long poll Use a thank you screen WebSockets to show live progress @dasiths
  59. Disable And Long Poll Order requested OrderId: ABC123 @dasiths

  60. Disable And Long Poll Order requested OrderId: ABC123 @dasiths UI

    Disabled
  61. Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While

    result is 404 @dasiths UI Disabled
  62. Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While

    result is 404 Order details @dasiths Redirect to order details
  63. Thank You Screen Order requested OrderId: ABC123 @dasiths

  64. Thank You Screen Order requested OrderId: ABC123 Order Details @dasiths

  65. Thank You Screen Order requested OrderId: ABC123 Order details @dasiths

  66. Fake Cache Comment posted CommentId: ABC123 This is a stupid

    comment @dasiths
  67. Fake Cache Comment posted CommentId: ABC123 This is a stupid

    question!!! var localComments = JSON.parse(localStorage.getItem('comments')) || []; var newComment = { 'questionId': "blah";, 'commentId': "ABC123";, 'commentText': commentText, 'isFromLocal': true }; localComments.push(newComment); localStorage.setItem('comments', JSON.stringify(localComments)); @dasiths
  68. Fake Cache Comment posted CommentId: ABC123 This is a stupid

    question!!! @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment
  69. Fake Cache Comment posted CommentId: ABC123 This is a stupid

    question!!! questionId = "blah"; httpClient.Get(`api/question/${questionId}/comments`) .Subscribe(comments => { var localComments = JSON.parse(localStorage.getItem('comments')) || []; var questionComments = localComments .filter(q => q.questionId == questionId); // todo: Remove duplicates from local storage return [...comments, ...questionComments]; }) @dasiths
  70. I see my comment Fake Cache Comment posted CommentId: ABC123

    @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment But, I don’t see it yet!
  71. Fake Cache Comment posted CommentId: ABC123 This is a stupid

    question!!! /api/quesion/blah/comments @dasiths /api/quesion/blah/comments Includes Lisa’s comment Doesn’t includes Lisa’s comment I see my comment I see it now too!
  72. WebSockets Live Progress Order requested OrderId: ABC123 @dasiths

  73. WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Finalizing…

    @dasiths
  74. WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Created

    Order confirmed @dasiths
  75. FAIL WHEN THINGS @dasiths

  76. • Show details on screen if possible. • Communicate via

    multiple mediums. • Have a plan. Act Fast @dasiths
  77. Disable+Poll Thank You Fake Cache Live Progress • Easy to

    implement. • UX isn’t great. • Easy to implement. • Assumes the system processes the request before user navigates out. • Requires complicated logic. • Good UX if implemented properly. • It’s fake. Use for non- critical use cases. • Requires WebSockets/HTTP2 for async server calls. • UI needs to handle server events and display. • Good UX. Recap @dasiths
  78. EMBRACE THE REAL WORLD Async business processes are the reality

    @dasiths
  79. http://pjousselin.free.fr @dasiths

  80. Any questions? THANKS! @dasiths dasith.me https://www.nationalgeographic.com/travel/destinations/asia/sri-lanka/

  81. Presentation template designed by powerpointify.com Special thanks to all people

    who made and shared these awesome resources for free: CREDITS Photographs by unsplash.com Free Fonts used: https://www.fontsquirrel.com/fonts/oswald @dasiths