The Shell Game Called Eventual Consistency

Slide 1

Slide 1 text

The Shell Game Called Eventual Consistency 24 Feb 2021 Dasith Wijesiriwardena @dasiths

Slide 2

Slide 2 text

I am here because I have a love hate relationship with distributed systems. HELLO! I AM Dasith @dasiths dasith.me

Slide 3

Slide 3 text

Dealing with Eventual Consistency Introduction to CAP theorem Agenda Recap and Conclusion Strong Consistency vs Availability @dasiths

Slide 4

Slide 4 text

A Primer on CAP Theorem @dasiths

Slide 5

Slide 5 text

https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 @dasiths

Slide 6

Slide 6 text

Write Read @dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Read

Slide 7

Slide 7 text

@dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read

Slide 8

Slide 8 text

@dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read

Slide 9

Slide 9 text

@dasiths https://www.researchgate.net/figure/Visualization-of-CAP-theorem_fig2_282679529 Write Read Read Write Read

Slide 10

Slide 10 text

C-P Systems Write Example 1: All nodes are connected @dasiths Consistent – Partition Tolerant

Slide 11

Slide 11 text

C-P Systems Write Example 1: All nodes are connected The system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant

Slide 12

Slide 12 text

C-P Systems Write Example 1: All nodes are connected Read The system isn’t available until the changes are propagated to all nodes. @dasiths Consistent – Partition Tolerant

Slide 13

Slide 13 text

C-P Systems Write Example 1: All nodes are connected Read The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant

Slide 14

Slide 14 text

C-P Systems Write Read Example 1: All nodes are connected Write Example 2: Connectivity intermittent The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant

Slide 15

Slide 15 text

Slide 16

Slide 16 text

C-P Systems Write Read Example 1: All nodes are connected Write Example 2: Connectivity intermittent The system isn’t available until connectivity is re-established. The system isn’t available until the changes are propagated to all nodes. @dasiths Speed Consistent – Partition Tolerant

Slide 17

Slide 17 text

• Strong consistency is critical. • Supports transactions that cover all nodes. • Generally easier to develop than A-P systems. C-P Systems @dasiths Consistent – Partition Tolerant

Slide 18

Slide 18 text

A-P Systems Write Example 1: All nodes are connected @dasiths Available – Partition Tolerant

Slide 19

Slide 19 text

A-P Systems Write Example 1: All nodes are connected Read The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant

Slide 20

Slide 20 text

A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. V1 V2 @dasiths Available – Partition Tolerant

Slide 21

Slide 21 text

A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant

Slide 22

Slide 22 text

A-P Systems Write Example 1: All nodes are connected Read Read Write Example 2: Connectivity intermittent The system isn’t immediately consistent but immediately available. @dasiths Available – Partition Tolerant

Slide 23

Slide 23 text

Slide 24

Slide 24 text

A-P Systems Write Example 1: All nodes are connected Read Read Write Example 2: Connectivity intermittent Read The system isn’t immediately consistent but immediately available. Stale @dasiths Available – Partition Tolerant

Slide 25

Slide 25 text

Read A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent Read The system is available but won’t be consistent until connectivity is re-established. V1 V2 @dasiths Available – Partition Tolerant

Slide 26

Slide 26 text

A-P Systems Write Example 1: All nodes are connected Read Read The system isn’t immediately consistent but immediately available. Write Example 2: Connectivity intermittent The system is available but won’t be consistent until connectivity is re-established. @dasiths “Eventual Consistency” Available – Partition Tolerant Read Read V1 V2

Slide 27

Slide 27 text

• Availability is critical. • No all node covering transactions. (i.e. 2-phase commit). • Introduces “lesser forms of consistency”. • Developing can be complicated. A-P Systems @dasiths Available – Partition Tolerant

Slide 28

Slide 28 text

Looking at the Modern Landscape @dasiths

Slide 29

Slide 29 text

• Availability is a must. • RDBMS are not the only choice. • Strong consistency is expensive. •Storage cost is less than CPU/Memory costs. • Separate Read and Write optimised stores. In Hyperscale… @dasiths

Slide 30

Slide 30 text

The problem with Eventual Consistency @dasiths

Slide 31

Slide 31 text

Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 32

Slide 32 text

Example: Strong Consistency Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 33

Slide 33 text

Example: Strong Consistency Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 34

Slide 34 text

Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 35

Slide 35 text

Transaction Boundary BEGIN TRANSACTION SELECT @QtyAvailable = Quantity FROM Products WHERE ProductId = @ProductId IF @QtyAvailable > @QtyOrdered UPDATE Products SET Quantity = Quantity - @QtyOrdered WHERE ProductId = @ProductId ELSE THROW 51000, 'The ordered qty is more than available.', 1; COMMIT TRANSACTION Example: Strong Consistency @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 36

Slide 36 text

Example: Strong Consistency Transaction Boundary Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 37

Slide 37 text

Example: Strong Consistency Transaction Boundary View orders Order confirmed Order requested @dasiths (C)onsistent-(P)artition TolerantSystem

Slide 38

Slide 38 text

@dasiths

Slide 39

Slide 39 text

@dasiths

Slide 40

Slide 40 text

@dasiths

Slide 41

Slide 41 text

Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem

Slide 42

Slide 42 text

Order requested Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem

Slide 43

Slide 43 text

Order requested Acknowledged Example: Eventual Consistency Out of sync @dasiths (A)available-(P)artition TolerantSystem

Slide 44

Slide 44 text

Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync @dasiths (A)available-(P)artition TolerantSystem

Slide 45

Slide 45 text

Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync View orders @dasiths (A)available-(P)artition TolerantSystem

Slide 46

Slide 46 text

Order requested Acknowledged Propagate to other partitions Example: Eventual Consistency Out of sync View orders @dasiths Order Does Not Exist!!! (A)available-(P)artition TolerantSystem

Slide 47

Slide 47 text

Order requested Acknowledged Propagate to other nodes Example: Eventual Consistency @dasiths (A)available-(P)artition TolerantSystem

Slide 48

Slide 48 text

Order requested Acknowledged Propagated to other partitions Example: Eventual Consistency Order confirmed @dasiths (A)available-(P)artition TolerantSystem

Slide 49

Slide 49 text

Order requested Acknowledged Propagated to other partitions Order confirmed Example: Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem

Slide 50

Slide 50 text

Order requested Acknowledged Propagated to other partitions Order confirmed Example: Eventual Consistency View orders @dasiths (A)available-(P)artition TolerantSystem

Slide 51

Slide 51 text

IT’S NOT THAT BAD. NO REALLY. @dasiths

Slide 52

Slide 52 text

Quick Thoughts on Eventual Consistency … Unless you are using pessimistic locking, all data is stale, there are possibilities of optimistic concurrency failures. There is some period of time that it takes to build the DTOs, put them on the wire and for the client to receive them and draw them on the screen. Greg Young http://codebetter.com/gregyoung/2010/04/14/quick-thoughts-on-eventual-consistency/ @dasiths

Slide 53

Slide 53 text

Starbucks Does Not Use Two-Phase Commit … In summary we can see that the real world is often asynchronous. Our daily lives consists of many coordinated, but asynchronous interactions (reading and replying to e-mail, buying coffee etc) … It also means that often we can look at daily life to help design successful messaging solutions. Gregor Hohpe https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html @dasiths

Slide 54

Slide 54 text

Race Conditions Don’t Exist … Any time you see requirements that indicate a race condition, dig deeper. What you’re likely to find are some additional business concepts as well as the introduction of time and the creation of long-running business processes… Udi Dahan https://udidahan.com/2010/08/31/race-conditions-dont-exist/ @dasiths

Slide 55

Slide 55 text

DEALING WITH EVENTUAL CONSISTENCY @dasiths

Slide 56

Slide 56 text

• Language Matters. Communicate actual status. ▪ Terminology like Submitted, Queued, Pending. • Multiple mediums of communication (Email, SMS etc). ▪ Let the customer know ASAP. • Compensating actions as first class domain concepts. ▪ Treat it as a long running process/workflow. Managing Expectations @dasiths

Slide 57

Slide 57 text

UI SLIGHT OF HAND SOME @dasiths

Slide 58

Slide 58 text

Use local storage as a fake cache Ways To Handle Eventual Consistency on The UI Disable and long poll Use a thank you screen WebSockets to show live progress @dasiths

Slide 59

Slide 59 text

Disable And Long Poll Order requested OrderId: ABC123 @dasiths

Slide 60

Slide 60 text

Disable And Long Poll Order requested OrderId: ABC123 @dasiths UI Disabled

Slide 61

Slide 61 text

Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While result is 404 @dasiths UI Disabled

Slide 62

Slide 62 text

Disable And Long Poll Order requested OrderId: ABC123 /api/order/ABC123 While result is 404 Order details @dasiths Redirect to order details

Slide 63

Slide 63 text

Thank You Screen Order requested OrderId: ABC123 @dasiths

Slide 64

Slide 64 text

Thank You Screen Order requested OrderId: ABC123 Order Details @dasiths

Slide 65

Slide 65 text

Thank You Screen Order requested OrderId: ABC123 Order details @dasiths

Slide 66

Slide 66 text

Fake Cache Comment posted CommentId: ABC123 This is a stupid comment @dasiths

Slide 67

Slide 67 text

Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! var localComments = JSON.parse(localStorage.getItem('comments')) || []; var newComment = { 'questionId': "blah";, 'commentId': "ABC123";, 'commentText': commentText, 'isFromLocal': true }; localComments.push(newComment); localStorage.setItem('comments', JSON.stringify(localComments)); @dasiths

Slide 68

Slide 68 text

Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment

Slide 69

Slide 69 text

Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! questionId = "blah"; httpClient.Get(`api/question/${questionId}/comments`) .Subscribe(comments => { var localComments = JSON.parse(localStorage.getItem('comments')) || []; var questionComments = localComments .filter(q => q.questionId == questionId); // todo: Remove duplicates from local storage return [...comments, ...questionComments]; }) @dasiths

Slide 70

Slide 70 text

I see my comment Fake Cache Comment posted CommentId: ABC123 @dasiths /api/quesion/blah/comments Doesn’t includes Lisa’s comment But, I don’t see it yet!

Slide 71

Slide 71 text

Fake Cache Comment posted CommentId: ABC123 This is a stupid question!!! /api/quesion/blah/comments @dasiths /api/quesion/blah/comments Includes Lisa’s comment Doesn’t includes Lisa’s comment I see my comment I see it now too!

Slide 72

Slide 72 text

WebSockets Live Progress Order requested OrderId: ABC123 @dasiths

Slide 73

Slide 73 text

WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Finalizing… @dasiths

Slide 74

Slide 74 text

WebSockets Live Progress Order requested OrderId: ABC123 Order ABC123 Created Order confirmed @dasiths

Slide 75

Slide 75 text

FAIL WHEN THINGS @dasiths

Slide 76

Slide 76 text

• Show details on screen if possible. • Communicate via multiple mediums. • Have a plan. Act Fast @dasiths

Slide 77

Slide 77 text

Disable+Poll Thank You Fake Cache Live Progress • Easy to implement. • UX isn’t great. • Easy to implement. • Assumes the system processes the request before user navigates out. • Requires complicated logic. • Good UX if implemented properly. • It’s fake. Use for non- critical use cases. • Requires WebSockets/HTTP2 for async server calls. • UI needs to handle server events and display. • Good UX. Recap @dasiths

Slide 78

Slide 78 text

EMBRACE THE REAL WORLD Async business processes are the reality @dasiths

Slide 79

Slide 79 text

http://pjousselin.free.fr @dasiths

Slide 80

Slide 80 text

Any questions? THANKS! @dasiths dasith.me https://www.nationalgeographic.com/travel/destinations/asia/sri-lanka/

Slide 81

Slide 81 text

Presentation template designed by powerpointify.com Special thanks to all people who made and shared these awesome resources for free: CREDITS Photographs by unsplash.com Free Fonts used: https://www.fontsquirrel.com/fonts/oswald @dasiths