Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical Data Synchronization with CRDTs (StrangeLoop, 2016)

Dmitry Ivanov
September 16, 2016

Practical Data Synchronization with CRDTs (StrangeLoop, 2016)

In a connected world, synchronising mutable information between different devices with different clock precision can be a difficult problem. A piece of data may have many out-of-sync replicas but all of those should eventually be in a consistent state. For example, TomTom users, having personal navigation devices, smartphones, MyDrive website accounts, expect their navigation information be synchronised properly even in the occasional absence of network connection. Conflict-free Replicated Data Types (CRDTs) provide robust data structures to achieve proper synchronisation in an unreliable network of devices. They enable the conflict resolution being done locally at the data type level while guaranteeing the eventual consistency between replicas.

In addition to an introduction to common CRDT types, the main focus is on the special subtype of CRDT-Set called OUR-Set (Observed, Updated, Removed), which we created to extend known CRDT sets with update functionality.

I will demonstrate basic implementations of various CRDTs in Scala and enumerate subtle considerations which should be taken into account. I will also explain the advantages of these data structures to solve many synchronisation problems as well as their limitations.

Dmitry Ivanov

September 16, 2016
Tweet

More Decks by Dmitry Ivanov

Other Decks in Programming

Transcript

  1. Prac%cal Data Synchroniza%on
    with
    CRDTs
    Dmitry Ivanov @idajan0s
    St. Louis, 2016
    1

    View Slide

  2. Disclaimer
    I'm NOT:
    • Distributed systems expert.
    • Hardcore academia guy.
    Just a curious engineer hacking on real
    world problems.
    2

    View Slide

  3. NavCloud
    3

    View Slide

  4. Who We Are
    "Fool" stack developers hacking on:
    • Backend services
    • Mobile || SDKs
    • Infrastructure && AWS && DevOps
    4

    View Slide

  5. Server Development Stack
    5

    View Slide

  6. Client Libraries
    6

    View Slide

  7. NavCloud Nature
    • Unstable connec,ons
    • Limited bandwidth
    • Seamless edit/view in offline mode
    • Concurrent changes with poten7al
    conflicts
    • No guarantee on updates order
    • No data loss
    • Data convergence to expected value
    7

    View Slide

  8. How to Deal with this Nature?
    8

    View Slide

  9. Bad programmers worry about the
    code. Good programmers worry
    about data structures
    — Linus Torvalds
    9

    View Slide

  10. CRDT
    10

    View Slide

  11. CRDT
    DT: Data Type
    CRDT is a data type with its own algebra
    11

    View Slide

  12. CRDT
    R: Replicated
    CRDT is a family of data structures which
    has been designed to be distributed
    12

    View Slide

  13. CRDT
    C: Conflict Free
    Resolving conflicts is done automa2cally
    13

    View Slide

  14. How?
    14

    View Slide

  15. Merge
    15

    View Slide

  16. What is Merge?
    • A binary opera-on on two CRDTs
    • Commuta've: x • y = y • x
    • Associa've: ( x • y ) • z = x • ( y • z )
    • Idempotent: x • x = x
    16

    View Slide

  17. How Does it Help?
    In Distributed Systems:
    • Order is not guaranteed:
    • No Problem: Merge is Commuta-ve and Associa-ve
    • Events can be delivered more than once:
    • No problem: Merge is Idempotent
    17

    View Slide

  18. What Does it Bring in Prac1ce?
    • Local updates
    • Local merge of receiving data
    • All local merges converge
    18

    View Slide

  19. Examples
    19

    View Slide

  20. G-Counter
    20

    View Slide

  21. G-Counter
    Merge: Max of corresponding elements: A:6 B:3 C:9
    TotalValue: Sum of all elements: 6 + 3 + 9 = 18
    21

    View Slide

  22. Max Func)on
    • A binary opera-on on two CRDTs
    • Commuta've: x max y = y max x
    • Associa've: ( x max y ) max z = x max ( y max z )
    • Idempotent: x max x = x
    22

    View Slide

  23. G-Set
    23

    View Slide

  24. G-Set
    Merge: Union of sets: { x, y, z, a, b, c }
    Total Value: The same as the merge result
    24

    View Slide

  25. Union Func)on
    • A binary opera-on on two CRDTs
    • Commuta've: x ∪ y = y ∪ x
    • Associa've: ( x ∪ y ) ∪ z = x ∪ ( y ∪ z )
    • Idempotent: x ∪ x = x
    25

    View Slide

  26. CRDT in NavCloud
    26

    View Slide

  27. Favorite Loca,ons
    Synchroniza,on
    27

    View Slide

  28. Naive Approach?
    28

    View Slide

  29. Last Write Wins
    29

    View Slide

  30. Problems
    • Unstable connec-ons
    • Actual update -me < Sent -me
    • Network latency
    • Sent -me < Received -me
    • Unreliable clocks
    30

    View Slide

  31. Stale update may win!
    31

    View Slide

  32. So What?
    32

    View Slide

  33. CRDT
    33

    View Slide

  34. NavCloud Nature vs CRDT
    • Unstable connec,ons ✔
    • Limited bandwidth ✔
    • Seamless edit/view in offline mode ✔
    • Concurrent changes with poten7al
    conflicts ✔
    • No guarantee on updates order ✔
    • No data loss ✔
    • Data convergence to expected value ✔
    34

    View Slide

  35. Same Data Model Everywhere
    • Server
    • Clients
    • Data store
    35

    View Slide

  36. Merging Conflicts in Riak
    36

    View Slide

  37. Implemen'ng a CRDT Set
    What do we want?
    • Support for addi-on and removal opera-ons.
    • Op-mized for element muta-ons.
    • Footprint as compact as possible.
    37

    View Slide

  38. 2-Phase-Set
    Supports addi,ons and removals.
    • G-Set for added elements
    • G-Set for removed elements aka Tombstones
    38

    View Slide

  39. 2-Phase-Set
    39

    View Slide

  40. 2-Phase-Set
    Merge: [ Add { "cat", "dog", "ape" }; Rem { "ape" } ]
    Lookup: { "cat", "dog" }
    40

    View Slide

  41. 2-Phase-Set
    Lookup
    def lookup: Set[E] = addSet.diff(removeSet).lookup
    Merge
    def merge(anotherSet: TwoPSet[E]): TwoPSet[E] =
    new TwoPSet( union(addset, anotherSet.addSet ),
    union(removeSet, anotherSet.removeSet ))
    41

    View Slide

  42. 2-Phase-Set
    Doesn't work for us:
    • Removed element can't be added again
    • Immutable elements: no updates possible
    42

    View Slide

  43. LWW-Element-Set
    Supports addi,ons and removals, with !mestamps.
    • G-Set for added elements
    • G-Set for removed elements aka Tombstones
    • Each element has a 3mestamp
    • Supports re-adding removed element using a higher 3mestamp
    43

    View Slide

  44. LWW-Element-Set
    44

    View Slide

  45. LWW-Element-Set
    Merge
    Add { (1, "cat"), (5, "cat"), (1, "dog"), (1, "ape") }
    Rem { (1, "cat"), (3, "cat") }
    45

    View Slide

  46. LWW-Element-Set
    Merge
    Add { (1, "cat"), (5, "cat"), (1, "dog"), (1, "ape") }
    Rem { (1, "cat"), (3, "cat") }
    Lookup
    { "cat", "dog", "ape" }
    46

    View Slide

  47. LWW-Element-Set
    Lookup
    def lookup: Set[E] = addSet.lookup.filter { addElem =>
    !removeSet.exists { removeElem =>
    removeElem.value == addElem.value && removeElem.timestamp > addElem.timestamp
    }
    }.map(_.value)
    Merge
    def merge(LWWSet anotherSet): LWWSet =
    new LWWSet( union(addset, anotherSet.addSet ),
    union(removeSet, anotherSet.removeSet ))
    47

    View Slide

  48. LWW-Element-Set
    Doesn't work for us:
    • Immutable elements: no updates possible.
    48

    View Slide

  49. OR-Set
    OR - Observed / Removed
    Supports addi,ons and removals, with tags.
    • G-Set for added elements
    • G-Set for removed elements aka Tombstones
    • Unique tag is associated with each element
    • Supports re-adding removed elements
    49

    View Slide

  50. OR-Set
    50

    View Slide

  51. OR-Set
    Merge
    Add { (#a, "cat"), (#c, "cat"), (#b, "dog"), (#d, "ape") }
    Rem { (#a, "cat") }
    51

    View Slide

  52. OR-Set
    Merge
    Add { (#a, "cat"), (#c, "cat"), (#b, "dog"), (#d, "ape") }
    Rem { (#a, "cat") }
    Lookup
    { "cat", "dog", "ape" }
    52

    View Slide

  53. OR-Set
    Lookup
    E exists iff it has in AddSet a tag that is not in the RemoveSet.
    def lookup(): Set =
    addSet.filter { addElem =>
    !removeSet.exists { remElem =>
    addElem.value == remElem.value
    && remElem.tag.equals(addElem.tag) }
    }
    .map(_.value);
    53

    View Slide

  54. OR-Set
    Merge
    def merge(anotherSet: ORSet[E]): ORSet[E] =
    new ORSet( union(addset, anotherSet.addSet ),
    union(removeSet, anotherSet.removeSet))
    54

    View Slide

  55. OR-Set
    Doesn't work for us:
    • Immutable elements: no updates possible.
    55

    View Slide

  56. OUR-Set
    Our take on Observed-Updated-Removed Set
    • Each element has a unique iden%fier
    • Element can be changed if iden4fier remains the same
    • Each element has a %mestamp
    • Timestamp is updated on each element muta4on
    Iden%ty (immutable unique id) vs Value (mutable)
    56

    View Slide

  57. OUR-Set
    Contains a single underlying set of elements with metadata:
    • Each element has a unique id field (e.g. a UUID)
    • Each element has a "removed" boolean flag
    • Each element has a )mestamp
    • Set can only contain one element with a par'cular id
    57

    View Slide

  58. OUR-Set
    58

    View Slide

  59. OUR-Set
    Merge
    { (id1, 5, "*ger"), (id2, 2, "dog", removed), (id3, 1, "ape") }
    59

    View Slide

  60. OUR-Set
    Merge:
    { (id1, 5, "*ger"), (id2, 2, "dog", removed), (id3, 1, "ape") }
    Lookup
    { "$ger", "ape" }
    60

    View Slide

  61. OUR-Set
    Merge
    def merge(anotherSet: OURSet[E]]): OURSet[E] =
    OURSet[E]( elements ++ anotherSet.elements)
    .groupBy (_.id)
    .map (group => group._2.maxBy(_.timestamp))
    .toSet)
    Lookup
    def lookup(ourSet: OURSet[E]): Set[E] =
    ourSet.filter (!_.removed)
    .map (_.value)
    61

    View Slide

  62. Implementa)on
    NavCloud CRDT Model: Favorites
    62

    View Slide

  63. CRDT Model: Favorites
    FavoriteState element:
    • ID (to uniquely iden.fy a favorite)
    • Timestamp (to indicate the last change .me)
    • Removed flag (to indicate if favorite has been removed)
    • Favorite data: ( Name, Loca2on, ... )
    63

    View Slide

  64. Convergence in case of equal !mestamps
    Compare func-on checks all the fields in order of priority:
    • Timestamp
    • Removed flag (Add or Delete bias)
    • .. rest a6ributes ..
    64

    View Slide

  65. Using CRDT everywhere
    • Use the same algorithm everywhere
    As simple as calling the merge func8on
    65

    View Slide

  66. Using CRDT everywhere
    Client <-> Server <-> Database
    def update(fromClient: OURSet[E]): OURSet[E] = {
    val fromDatabase = database.fetch(...)
    val newSet = fromDatabase.merge(fromClient)
    database.store(..., newSet)
    newSet
    }
    66

    View Slide

  67. 67

    View Slide

  68. Considera*ons & Limita*ons
    68

    View Slide

  69. "What about garbage?"
    • CRDTs tend to grow because of tombstones.
    • Deleted Element in the Set == Tombstone.
    • A poten?ally unbounded growth.
    69

    View Slide

  70. Prune deleted elements
    But when?
    Requirement:
    All nodes holding a CRDT Set replica should have seen a deleted
    element before it can be pruned.
    Otherwise deleted elements can be resurrected.
    70

    View Slide

  71. Time-To-Live for tombstones
    Prune tombstones once TTL exceeded.
    if ((DateTime.now() - tombstone.timestamp) > TimeToLive) {
    crdtSet.remove(tombstone)
    }
    Requirement: all nodes holding a CRDT set should apply the same
    TTL rule independently.
    71

    View Slide

  72. Send and reply with a Diff
    Client modifies and sends only updated elements (Diff).
    Before: Server responds with a full merge result.
    72

    View Slide

  73. Send and reply with a Diff
    We introduced a 'Scoped Diff':
    Server responds only with the elements which have won against
    those sent by the client.
    73

    View Slide

  74. Server -> Client Diff
    74

    View Slide

  75. Trouble With Time
    75

    View Slide

  76. There is no such thing as reliable (me*.
    76

    View Slide

  77. Tracking *me is actually
    tracking causality.
    — Jonas Bonér, "Life Beyond the Illusion of Present"
    77

    View Slide

  78. Causality & Ordering of events.
    78

    View Slide

  79. Time can be just good enough.
    79

    View Slide

  80. Ordering updates within a single node
    Timestamp field as a logical clock.
    Absolute value is not important,
    but it should always grow monotonically.
    80

    View Slide

  81. Ordering updates within a single node
    "+1 Strategy" (aka ensure monotonicity):
    Long resolveNewTimestamp(ElementState state) {
    return Math.max( retrieveTimestamp(),
    state.lastModified() + 1 );
    }
    81

    View Slide

  82. Ordering updates from different nodes
    If GPS clock is available -> use it (mainly Naviga&on Devices case).
    Prefer the server &me to a client's local 0me.
    82

    View Slide

  83. Edge case
    Mul$ple Clients modify the same element
    (concurrently || without a reliable clock).
    83

    View Slide

  84. One "merge" to rule them all
    84

    View Slide

  85. Clients & Server MUST have same 'merge'
    behaviour.
    ==
    Given the same input, their 'merge' func/ons
    emit the same results.
    85

    View Slide

  86. Divergence may lead to endless synchroniza1on loops!
    86

    View Slide

  87. Lazy (data) loading
    OURSet Element
    • Metadata: UUID, $mestamp, "removed" flag
    • Data:
    87

    View Slide

  88. Lazy (data) loading
    New OURSet Element
    • Metadata: UUID, $mestamp, "removed" flag, + tag / hash
    • (Op(onal) Data:
    Flexible synchroniza1on strategy
    Eager || Lazy Fetch
    88

    View Slide

  89. What have we learned?
    • Academia is not as scary as it some-mes seems to pragma,c devs.
    • We need be2er and simpler abstrac-ons to develop
    Offline-friendly apps.
    • CRDTs give a great value, but there are some caveats.
    • Things like Lasp (lasp-lang.org) also could be the answer (?).
    89

    View Slide

  90. Show me the code
    github.com/ajan/s/{scala | java}-crdt
    90

    View Slide

  91. Want to know more? (Part 1)
    • CRDTs for fun and eventual profit, - Noel Welsh, 2013.
    • Readings in conflict-free replicated data types, - Christopher
    Meiklejohn, 2015.
    • A comprehensive study of Convergent and CommutaJve
    Replicated Data Types, - Marc Shapiro, Nuno Preguiça, Carlos
    Baquero, Marek Zawirski, 2011.
    91

    View Slide

  92. Want to know more? (Part 2)
    • Lasp: A language for distributed, coordina7on-free programming,
    - Meiklejohn & Van Roy, 2015.
    • Swarm.js+React — real-7me, offline-ready Holy Grail web apps, -
    Victor Grishchenko.
    92

    View Slide

  93. Thanks!
    Dmitry Ivanov @idajan0s
    93

    View Slide