Practical Data Synchronization using CRDTs QConSF 2016

Presented @ QConSF 2016: https://qconsf.com/sf2016/presentation/practical-data-synchronization-using-CRDTs

Abstract:

In a connected world, synchronising mutable information between different devices with different clock precision can be a difficult problem. A piece of data may have many out-of-sync replicas but all of those should eventually be in a consistent state. For example, TomTom users, having personal navigation devices, smartphones, MyDrive website accounts, expect their navigation information be synchronised properly even in the occasional absence of network connection. Conflict-free Replicated Data Types (CRDTs) provide robust data structures to achieve proper synchronisation in an unreliable network of devices. They enable the conflict resolution being done locally at the data type level while guaranteeing the eventual consistency between replicas.

In addition to an introduction to common CRDT types, the main focus is on the special subtype of CRDT-Set called OUR-Set (Observed, Updated, Removed), which we created to extend known CRDT sets with update functionality.

I will demonstrate basic implementations of various CRDTs in Scala and enumerate subtle considerations which should be taken into account. I will also explain the advantages of these data structures to solve many synchronisation problems as well as their limitations.

Transcript

1. Prac%cal Data Synchroniza%on & CRDTs Dmitry Ivanov @idajan0s 2016 1
3. NavCloud 3
4. Who We Are "Fool" stack developers hacking on: • Backend services • Client libraries • Infrastructure && DevOps 4
5. Backend stack 5
6. Client Libraries 6
7. NavCloud Nature • Unstable connec,ons • Limited data plans & bandwidth • Seamless edit/view in offline mode • Concurrent changes with poten8al conflicts • No guarantee on updates order • No data loss • Data convergence to expected value 7
8. How to Deal with this Nature? 8
9. Bad programmers worry about the code. Good programmers worry about data structures — Linus Torvalds 9
10. CRDT 10
11. CRDT DT: Data Type CRDT is a data type with its own algebra 11
12. CRDT R: Replicated CRDT is a family of data structures which has been designed to be distributed 12
13. CRDT C: Conflict Free Resolving conflicts is done automa2cally 13
14. How? 14
15. Merge 15
16. What is Merge? • A binary opera-on on two CRDTs • Commuta've: x • y = y • x • Associa've: ( x • y ) • z = x • ( y • z ) • Idempotent: x • x = x 16
17. How Does it Help? In Distributed Systems: • Order is not guaranteed: • No Problem: Merge is Commuta-ve and Associa-ve • Events can be delivered more than once: • No problem: Merge is Idempotent 17
18. What Does it Bring in Prac1ce? • Local updates • Local merge of receiving data • All local merges converge 18
19. Examples 19
20. G-Counter 20
21. G-Counter Merge: Max of corresponding elements: A:6 B:3 C:9 TotalValue: Sum of all elements: 6 + 3 + 9 = 18 21
22. Max Func)on • A binary opera-on on two CRDTs • Commuta've: x max y = y max x • Associa've: ( x max y ) max z = x max ( y max z ) • Idempotent: x max x = x 22
23. G-Set 23
24. Union Func)on • A binary opera-on on two CRDTs • Commuta've: x ∪ y = y ∪ x • Associa've: ( x ∪ y ) ∪ z = x ∪ ( y ∪ z ) • Idempotent: x ∪ x = x 24
25. G-Set Merge: Union of sets: { x, y, z, a, b, c } Total Value: The same as the merge result 25
26. CRDT in NavCloud 26
27. Favorite Loca,ons Synchroniza,on 27
28. Naive Approach? 28
29. Last Write Wins 29
30. Problems • Unstable connec-ons • Actual update -me < Sent -me • Network latency • Sent -me < Received -me • Unreliable clocks 30
31. Stale update may win! 31
32. So What? 32
33. CRDT 33
34. NavCloud Nature vs CRDT • Unstable connec,ons ✔ • Limited data plans & bandwidth ✔ • Seamless edit/view in offline mode ✔ • Concurrent changes with poten8al conflicts ✔ • No guarantee on updates order ✔ • No data loss ✔ • Data convergence to expected value ✔ 34
35. Same Data Model Everywhere • Server • Clients • Data store 35
36. Merging Conflicts in Riak 36
37. The data consistency is determined by 'the weakest link' in your pipeline 37
38. Implemen'ng a CRDT Set What do we want? • Support for addi-on and removal opera-ons. • Op-mized for element muta-ons. • Footprint as compact as possible. 38
39. 2-Phase-Set Supports addi,ons and removals. • G-Set for added elements • G-Set for removed elements aka Tombstones 39
40. 2-Phase-Set 40
41. 2-Phase-Set Merge: [ Add { "cat", "dog", "ape" }; Rem { "ape" } ] Lookup: { "cat", "dog" } 41
42. 2-Phase-Set Lookup def lookup: Set[E] = addSet.diff(removeSet).lookup Merge def merge(anotherSet: TwoPSet[E]): TwoPSet[E] = new TwoPSet( addset.merge(anotherSet.addSet), removeSet.merge(anotherSet.removeSet)) 42
43. 2-Phase-Set Doesn't work for us: • Removed element can't be added again • Immutable elements: no updates possible 43
44. LWW-Element-Set Supports addi,ons and removals, with !mestamps. • G-Set for added elements • G-Set for removed elements aka Tombstones • Each element has a 3mestamp • Supports re-adding removed element using a higher 3mestamp 44
45. LWW-Element-Set 45
46. LWW-Element-Set Merge Add { (1, "cat"), (5, "cat"), (1, "dog"), (1, "ape") } Rem { (1, "cat"), (3, "cat") } 46
47. LWW-Element-Set Merge Add { (1, "cat"), (5, "cat"), (1, "dog"), (1, "ape") } Rem { (1, "cat"), (3, "cat") } Lookup { "cat", "dog", "ape" } 47
48. LWW-Element-Set Lookup def lookup: Set[E] = addSet.lookup.filter { addElem => !removeSet.exists { removeElem => removeElem.value == addElem.value && removeElem.timestamp > addElem.timestamp } }.map(_.value) Merge def merge(LWWSet<E> anotherSet): LWWSet<E> = new LWWSet( addset.merge(anotherSet.addSet), removeSet.merge(anotherSet.removeSet)) 48
49. LWW-Element-Set Doesn't work for us: • Immutable elements: no updates possible. 49
50. OR-Set OR - Observed / Removed Supports addi,ons and removals, with tags. • G-Set for added elements • G-Set for removed elements aka Tombstones • Unique tag is associated with each element • Supports re-adding removed elements 50
51. OR-Set 51
52. OR-Set Merge Add { (#a, "cat"), (#c, "cat"), (#b, "dog"), (#d, "ape") } Rem { (#a, "cat") } 52
53. OR-Set Merge Add { (#a, "cat"), (#c, "cat"), (#b, "dog"), (#d, "ape") } Rem { (#a, "cat") } Lookup { "cat", "dog", "ape" } 53
54. OR-Set Lookup E exists iff it has in AddSet a tag that is not in the RemoveSet. def lookup(): Set<E> = addSet.filter { addElem => !removeSet.exists { remElem => addElem.value == remElem.value && remElem.tag.equals(addElem.tag) } } .map(_.value); 54
55. OR-Set Merge def merge(anotherSet: ORSet[E]): ORSet[E] = new ORSet( addset.merge(anotherSet.addSet), removeSet.merge(anotherSet.removeSet)) 55
56. OR-Set Doesn't work for us: • Immutable elements: no updates possible. 56
57. OUR-Set Our take on Observed-Updated-Removed Set • Each element has a unique iden%fier • Element can be changed if iden4fier remains the same • Each element has a %mestamp • Timestamp is updated on each element muta4on Iden%ty (immutable unique id) vs Value (mutable) 57
58. OUR-Set Contains a single underlying set of elements with metadata: • Each element has a unique id field (e.g. a UUID) • Each element has a "removed" boolean flag • Each element has a )mestamp • Set can only contain one element with a par'cular id 58
59. OUR-Set 59
60. OUR-Set Merge { (id1, 5, "*ger"), (id2, 2, "dog", removed), (id3, 1, "ape") } 60
61. OUR-Set Merge: { (id1, 5, "*ger"), (id2, 2, "dog", removed), (id3, 1, "ape") } Lookup { "$ger", "ape" } 61
62. OUR-Set Merge def merge(anotherSet: OURSet[E]]): OURSet[E] = OURSet[E]( elements ++ anotherSet.elements) .groupBy (_.id) .map (group => group._2.maxBy(_.timestamp)) .toSet) Lookup def lookup(ourSet: OURSet[E]): Set[E] = ourSet.filter (!_.removed) .map (_.value) 62
63. Implementa)on NavCloud CRDT Model: Favorites 63
64. CRDT Model: Favorites FavoriteState element: • ID (to uniquely iden.fy a favorite) • Timestamp (to indicate the last change .me) • Removed flag (to indicate if favorite has been removed) • Favorite data: ( Name, Loca2on, ... ) 64
65. Convergence in case of equal !mestamps Compare func-on checks all the fields in order of priority: • Timestamp • Removed flag (Add or Delete bias) • .. rest a6ributes .. 65
66. Using CRDT everywhere • Use the same algorithm everywhere As simple as calling the merge func8on 66
67. Using CRDT everywhere Client <-> Server <-> Database def update(fromClient: OURSet[E]): OURSet[E] = { val fromDatabase = database.fetch(...) val newSet = fromDatabase.merge(fromClient) database.store(..., newSet) newSet } 67
69. Considera*ons & Limita*ons 69
70. "What about garbage?" • CRDTs tend to grow because of tombstones. • Deleted Element in the Set == Tombstone. • A poten?ally unbounded growth. 70
71. Prune deleted elements But when? Requirement: All nodes holding a CRDT Set replica should have seen a deleted element before it can be pruned. Otherwise deleted elements can be resurrected. 71
72. Time-To-Live for tombstones Prune tombstones once TTL exceeded. if ((DateTime.now() - tombstone.timestamp) > TimeToLive) { crdtSet.remove(tombstone) } Requirement: all nodes holding a CRDT set should apply the same TTL rule independently. 72
73. Prune deleted elements Problem Synchroniza+on between all replicas is needed for correctness. 73
74. Distributed transac.ons 74
75. - Academia, help! 75
77. Op#mized OR-Set Introduces replica awareness 77
78. Op#mized OR-Set Addi$onal metadata is added to every transferred state. { (replica_id -> seq_nr) } where: - replica_id - is a unique & stable replica iden5fier. - seq_nr - monotonically growing (a=er each op) local counter. 78
79. Op#mized OR-Set Each local state maintains a map: { replica_A: 1, replica_B: 1, replica_C: 3 } If a received state has a seq_nr lower than the corresponding local value -> ignore. 79
80. Op#mized OR-Set No Tombstones, yay! ☺ (Slightly) more complicated API: stable replica_id needed. ☹ 80
81. Update & Reply with a Diff Client modifies and sends only updated elements (Diff). Before: Server responds with a full merge result. 81
82. Update & Reply with a Diff We introduced a 'Scoped Diff': Server responds only with the elements which have won against those sent by the client. 82
83. Server -> Client Diff 83
84. - Academia, help?.. 84
86. δ-CRDT Builds on replica awareness Introduces a Causal Context: map of (replica_id -> seq_nr). Introduces a Dot Store: CRDT state (No tombstones). 86
87. δ-CRDT A formalized way to compute a minimal δ-CRDT instances against a target replica. 87
88. δ-CRDT Adrian Colyer (The Morning Paper) wrote a great paper review: blog.acolyer.org/2016/04/25/delta-state-replicated-data-types 88
89. Trouble With Time 89
90. There is no such thing as reliable (me*. 90
91. Tracking *me is actually tracking causality. — Jonas Bonér, "Life Beyond the Illusion of Present" 91
92. Causality & Ordering of events. 92
93. Time can be just good enough. 93
94. Ordering updates within a single node Timestamp field as a logical clock. Absolute value is not important, but it should always grow monotonically. 94
95. Ordering updates within a single node "+1 Strategy" (aka ensure monotonicity): Long resolveNewTimestamp(ElementState<E> state) { return Math.max( retrieveTimestamp(), state.lastModified() + 1 ); } 95
96. Ordering updates from different nodes If GPS clock is available -> use it (mainly Naviga&on Devices case). Prefer the server &me to a client's local 0me. 96
97. Edge case Mul$ple Clients modify the same element (concurrently || without a reliable clock). 97
98. One "merge" to rule them all 98
99. Clients & Server MUST have same 'merge' behaviour. == Given the same input, their 'merge' func/ons emit the same results. 99
100. Divergence may lead to endless synchroniza1on loops! 100
101. Lazy (data) loading OURSet Element • Metadata: UUID, $mestamp, "removed" flag • Data: <Value> 101
102. Lazy (data) loading New OURSet Element • Metadata: UUID, $mestamp, "removed" flag, + tag / hash • (Op(onal) Data: <Value> Flexible synchroniza1on strategy Eager || Lazy Fetch 102
103. What have we learned? • Academia is not as scary as it some-mes seems to pragma,c devs. • We need be2er and simpler abstrac-ons to develop Offline-friendly apps. • CRDTs give a great value, but there are some caveats. • Things like Lasp (lasp-lang.org) also could be the answer (?). 103
104. Show me the code github.com/ajan/s/{scala | java}-crdt 104
105. Thanks! Slides: h*p:/ /bit.ly/2fBlroS Dmitry Ivanov @idajan0s 105