Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Certified Mergeable Replicated Data Types

Certified Mergeable Replicated Data Types

Replicated data types (RDTs) are data structures that permit concurrent modification of multiple potentially geo-distributed replicas without coordination between them. RDTs are designed in such a way that conflicting operations are eventually deterministically reconciled ensuring convergence. Constructing correct RDTs remains a difficult endeavour due to the complexity of reasoning about independently evolving states of the replicas. With the focus on the correctness of RDTs (and rightly so), existing approaches to RDTs are less efficient compared to their sequential counterparts in terms of time and space complexity. This is unfortunate since RDTs are often used in an local-first setting where the local operations far outweigh remote communication.

In this paper, we present Peepul, a pragmatic approach to building and verifying efficient RDTs. To make reasoning about correctness easier, we cast RDTs in the mould of distributed version control system, and equip it with a threeway merge function for reconciling conflicting versions. Further, we go beyond just verifying convergence, and provide a methodology to verify arbitrarily complex specifications. We develop a replication-aware simulation relation based technique to relate RDT specifications to their efficient purely functional implementations. We have developed Peepul as an F* library that discharges proof obligations to an SMT solver. The verified efficient RDTs are extracted as OCaml code and used in Irmin, a Git-like distributed database.

KC Sivaramakrishnan

April 27, 2022
Tweet

More Decks by KC Sivaramakrishnan

Other Decks in Science

Transcript

  1. Certi fi ed Mergeable Replicated Data Types “KC” Sivaramakrishnan joint

    work with Vimala Soundarapandian, Adharsh Kamath and Kartik Nagar
  2. 4 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d end Sequential Counter
  3. • Written in idiomatic style • Composable 4 module Counter

    : sig type t val read : t -> int val add : t -> int -> t val sub : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d end type counter_list = Counter.t list Sequential Counter
  4. INTERNET 0 0 Replicated Counter +2 2 +3 3 •

    Idea: Apply the local operations at all replicas
  5. INTERNET Replicated Counter +2 +3 5 5 5 5 •

    Idea: Apply the local operations at all replicas
  6. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end
  7. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7
  8. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 +1
  9. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3
  10. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 24 *3
  11. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 24 22 *3 +1
  12. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 24 22 *3 +1 Diverges
  13. 7 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 24 22 *3 +1 Diverges Addition and multiplication do not commute
  14. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end
  15. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end • Idea: Capture the effect of multiplication through the commutative addition operation
  16. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 • Idea: Capture the effect of multiplication through the commutative addition operation
  17. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 +1 • Idea: Capture the effect of multiplication through the commutative addition operation
  18. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 • Idea: Capture the effect of multiplication through the commutative addition operation
  19. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 22 +14 • Idea: Capture the effect of multiplication through the commutative addition operation
  20. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 22 22 +14 +1 • Idea: Capture the effect of multiplication through the commutative addition operation
  21. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 22 22 +14 +1 Converges • Idea: Capture the effect of multiplication through the commutative addition operation
  22. 8 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n end 7 8 21 +1 *3 22 22 +14 +1 Converges • Idea: Capture the effect of multiplication through the commutative addition operation • CRDTs
  23. Convergent Replicated Data Types (CRDT) • CRDT is guaranteed to

    ensure strong eventual consistency (SEC) ★ G-counters, PN-counters, OR-Sets, Graphs, Ropes, docs, sheets ★ Simple interface for the clients of CRDTs 9
  24. Convergent Replicated Data Types (CRDT) • CRDT is guaranteed to

    ensure strong eventual consistency (SEC) ★ G-counters, PN-counters, OR-Sets, Graphs, Ropes, docs, sheets ★ Simple interface for the clients of CRDTs • Need to reengineer every datatype to ensure SEC (commutativity) ★ Do not mirror sequential counter parts => implementation & proof burden ★ Do not compose! ✦ counter set is not a composition of counter and set CRDTs 9
  25. Can we program & reason about replicated data types as

    an extension of their sequential counterparts?
  26. Can we program & reason about replicated data types as

    an extension of their sequential counterparts? MRDT
  27. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end
  28. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7
  29. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 +1
  30. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3
  31. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3 22
  32. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3 22 22
  33. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3 22 22 22 = 7 + (8-1) + (21 -7)
  34. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3 22 22 22 = 7 + (8-1) + (21 -7) • 3-way merge function makes the counter suitable for distribution
  35. 11 module Counter : sig type t val read :

    t -> int val add : t -> int -> t val sub : t -> int -> t val mult : t -> int -> t val merge : lca:t -> v1:t -> v2:t -> t end = struct type t = int let read x = x let add x d = x + d let sub x d = x - d let mult x n = x * n let merge ~lca ~v1 ~v2 = lca + (v1 - lca) + (v2 - lca) end 7 8 21 +1 *3 22 22 22 = 7 + (8-1) + (21 -7) • 3-way merge function makes the counter suitable for distribution • Does not appeal to individual operations => independently extend data-type
  36. 12 Systems ➞ PL • CRDTs need to take care

    of systems level concerns such as message loss, duplication and reordering
  37. 12 Systems ➞ PL • CRDTs need to take care

    of systems level concerns such as message loss, duplication and reordering • 3-way merge is oblivious to these ✦ By leaving those concerns to MRDT middleware
  38. 12 7 8 21 +1 *3 22 22 Systems ➞

    PL • CRDTs need to take care of systems level concerns such as message loss, duplication and reordering • 3-way merge is oblivious to these ✦ By leaving those concerns to MRDT middleware
  39. ?? 12 7 8 21 +1 *3 22 22 Systems

    ➞ PL • CRDTs need to take care of systems level concerns such as message loss, duplication and reordering • 3-way merge is oblivious to these ✦ By leaving those concerns to MRDT middleware
  40. ?? 12 7 8 21 +1 *3 22 22 22

    22 = 21 + (21-21) + (22 -21) Systems ➞ PL • CRDTs need to take care of systems level concerns such as message loss, duplication and reordering • 3-way merge is oblivious to these ✦ By leaving those concerns to MRDT middleware
  41. 14 • OR-set — add-wins when there is a concurrent

    add and remove of the same element Observed-Removed Set
  42. 14 • OR-set — add-wins when there is a concurrent

    add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  43. 14 {1} • OR-set — add-wins when there is a

    concurrent add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  44. 14 {1} {1} add(1) • OR-set — add-wins when there

    is a concurrent add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  45. 14 {1} {1} { } add(1) rem(1) • OR-set —

    add-wins when there is a concurrent add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  46. 14 {1} {1} { } { } { } add(1)

    rem(1) • OR-set — add-wins when there is a concurrent add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) { } ∪ ({1} - {1}) ∪ ({ } - {1}) = { } ∪ { } ∪ { } = { } (expected {1}) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  47. 14 {1} {1} { } { } { } add(1)

    rem(1) • Convergence is not suf fi cient; Intent is not preserved • OR-set — add-wins when there is a concurrent add and remove of the same element Observed-Removed Set let merge ~lca ~v1 ~v2 = (lca ∩ v1 ∩ v2) (* unmodified elements *) ∪ (v1 - lca) (* added in v1 *) ∪ (v2 - lca) (* added in v2 *) { } ∪ ({1} - {1}) ∪ ({ } - {1}) = { } ∪ { } ∪ { } = { } (expected {1}) Kaki et al. “Mergeable Replicated Data Types”, OOPSLA 2019
  48. Concretising Intent • Intent is a woolly term ★ How

    can we formalise the intent of operations on a data structure? 15 l v1 v2 v
  49. Concretising Intent • Intent is a woolly term ★ How

    can we formalise the intent of operations on a data structure? • We need ★ A formal language to specify the intent of an RDT ★ Mechanization to bridge the air gap between speci fi cation and implementation due to distributed system complexity 15 l v1 v2 v
  50. Peepul — Certi fi ed MRDTs • An F* library

    implementing and proving MRDTs ★ https://github.com/prismlab/peepul 16
  51. Peepul — Certi fi ed MRDTs • An F* library

    implementing and proving MRDTs ★ https://github.com/prismlab/peepul • Speci fi cation language is event-based ★ Burckhardt et al. “Replicated Data Types: Speci fi cation, Veri fi cation and Optimality”, POPL 2014 16
  52. Peepul — Certi fi ed MRDTs • An F* library

    implementing and proving MRDTs ★ https://github.com/prismlab/peepul • Speci fi cation language is event-based ★ Burckhardt et al. “Replicated Data Types: Speci fi cation, Veri fi cation and Optimality”, POPL 2014 • Replication-aware simulation to connect speci fi cation with implementation 16
  53. Peepul — Certi fi ed MRDTs • An F* library

    implementing and proving MRDTs ★ https://github.com/prismlab/peepul • Speci fi cation language is event-based ★ Burckhardt et al. “Replicated Data Types: Speci fi cation, Veri fi cation and Optimality”, POPL 2014 • Replication-aware simulation to connect speci fi cation with implementation • Composition of MRDTs and their proofs! 16
  54. Peepul — Certi fi ed MRDTs • An F* library

    implementing and proving MRDTs ★ https://github.com/prismlab/peepul • Speci fi cation language is event-based ★ Burckhardt et al. “Replicated Data Types: Speci fi cation, Veri fi cation and Optimality”, POPL 2014 • Replication-aware simulation to connect speci fi cation with implementation • Composition of MRDTs and their proofs! • Extracted RDTs are compatible with Irmin — a Git-like distributed database 16
  55. Fixing OR-Set • Discriminate duplicate additions by associating a unique

    id 17 { (a,1) } { (a,1); (a,2) } { } add(a) rem(a)
  56. Fixing OR-Set • Discriminate duplicate additions by associating a unique

    id 17 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { } ∪ ( { (a,1); (a,2) } - { (a,1) }) ∪ ( { } - { (a,1) } ) = { } ∪ { (a,2) } ∪ { } = { (a,2) }
  57. Fixing OR-Set • Discriminate duplicate additions by associating a unique

    id • MRDT implementation 17 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { } ∪ ( { (a,1); (a,2) } - { (a,1) }) ∪ ( { } - { (a,1) } ) = { } ∪ { (a,2) } ∪ { } = { (a,2) }
  58. Fixing OR-Set • Discriminate duplicate additions by associating a unique

    id • MRDT implementation 17 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { } ∪ ( { (a,1); (a,2) } - { (a,1) }) ∪ ( { } - { (a,1) } ) = { } ∪ { (a,2) } ∪ { } = { (a,2) }
  59. Fixing OR-Set • Discriminate duplicate additions by associating a unique

    id • MRDT implementation 17 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { } ∪ ( { (a,1); (a,2) } - { (a,1) }) ∪ ( { } - { (a,1) } ) = { } ∪ { (a,2) } ∪ { } = { (a,2) } Unique Lamport Timestamps
  60. 18 Specifying OR-Set Abstract state add(a) add(a) rem(a) rd vis

    vis vis vis { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  61. 18 Specifying OR-Set Abstract state add(a) add(a) rem(a) rd vis

    vis vis vis { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  62. 18 Specifying OR-Set Abstract state = { a } add(a)

    add(a) rem(a) rd vis vis vis vis { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  63. Verifying Operations 1. Show that the simulation holds for operations

    20 to prove Operation de fi nition de fi ned once-and-for-all Simulation relation
  64. Verifying Operations 1. Show that the simulation holds for operations

    20 to prove Operation de fi nition de fi ned once-and-for-all Simulation relation 2. Show that the simulation holds for merge
  65. Verifying Operations 1. Show that the simulation holds for operations

    20 to prove Operation de fi nition de fi ned once-and-for-all Simulation relation 2. Show that the simulation holds for merge
  66. Verifying Operations 1. Show that the simulation holds for operations

    20 to prove Operation de fi nition de fi ned once-and-for-all Simulation relation 2. Show that the simulation holds for merge Merge de fi nition Assume de fi ned once-and-for-all To prove
  67. Verifying Operations 3. Show that the speci fi cation and

    the implementation agree on the return values of operations 21
  68. Verifying Operations 3. Show that the speci fi cation and

    the implementation agree on the return values of operations 21 4. Convergence
  69. Verifying Operations 3. Show that the speci fi cation and

    the implementation agree on the return values of operations 21 4. Convergence ✦ Permits the different replicas to converge to states that are observationally equal but not structurally equal ✤ Example: differently balanced BSTs
  70. Verifying Operations 3. Show that the speci fi cation and

    the implementation agree on the return values of operations 21 4. Convergence ✦ Permits the different replicas to converge to states that are observationally equal but not structurally equal ✤ Example: differently balanced BSTs
  71. Space-ef fi cient OR-Set • Recall that the OR-set has

    duplicates • How can we remove them? 22 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  72. Space-ef fi cient OR-Set • Recall that the OR-set has

    duplicates • How can we remove them? • Idea ★ On addition, replace existing element’s timestamp with the new timestamp ★ On merge, pick the larger timestamp 22 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  73. Space-ef fi cient OR-Set • Recall that the OR-set has

    duplicates • How can we remove them? • Idea ★ On addition, replace existing element’s timestamp with the new timestamp ★ On merge, pick the larger timestamp 22 { (a,1) } { (a,1); (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) Correctness argument is tricky
  74. Space-ef fi cient OR-Set 23 { (a,1) } { (a,1);

    (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { (a,1) } { (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  75. Space-ef fi cient OR-Set 23 { (a,1) } { (a,1);

    (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { (a,1) } { (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a)
  76. Space-ef fi cient OR-Set 23 { (a,1) } { (a,1);

    (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) { (a,1) } { (a,2) } { } { (a,2) } { (a,2) } add(a) rem(a) Simulation relation is more intricate as one would expect
  77. Composing IRC-style chat • Build IRC-style group chat ★ Send

    and read messages in channels ★ For simplicity, channels and messages cannot be deleted • Represent application state as a grow-only map with string (channel name) keys and mergeable-log as values • Goal: ★ map and log proved correct separately ★ Use the proof of underlying RDTs to prove chat application correctness 26
  78. Generic Map MRDT • Speci fi cation • Project fi

    lters the abstract state of the map on the key k and returns an abstract state of the underlying data type ★ Provided by the user once for a generic MRDT 27 where
  79. Generic Map MRDT • Speci fi cation • Project fi

    lters the abstract state of the map on the key k and returns an abstract state of the underlying data type ★ Provided by the user once for a generic MRDT 27 where set (“general”, append (“hello”)) set (“compiler”, append (“error”)) set (“general”, append (“world”)) vis vis get (“general”, rd) [“world”; “hello”]
  80. 28 Generic Map MRDT Implementation Simulation Relation Get applies given

    operation on the value at key k and returns the value
  81. 28 Generic Map MRDT Implementation Simulation Relation Get applies given

    operation on the value at key k and returns the value Set is Get + update the map with the new state
  82. 28 Generic Map MRDT Implementation Simulation Relation Get applies given

    operation on the value at key k and returns the value Set is Get + update the map with the new state Merge uses the merge of the underlying value type!
  83. 28 Generic Map MRDT Implementation Simulation Relation Get applies given

    operation on the value at key k and returns the value Set is Get + update the map with the new state Merge uses the merge of the underlying value type! Simulation relation appeals to the value type’s simulation relation!
  84. • Program state is constructed by instantiating generic map with

    mergeable log ★ The proof of correctness of the chat application directly follows from the composition! 29 Composing IRC-style chat
  85. Mergeable Queues • Replicated queue with at-least-once dequeue semantics ★

    First veri fi ed queue RDT! • Our aim is to have O(1) enqueue and dequeue and O(n) merge 30
  86. Mergeable Queues • Implementation ★ Uses two-list functional queue implementation

    ✦ amortised O(1) enqueue and dequeue operations ★ Merge uses longest common contiguous subsequence algorithm — O(n) 31 M
  87. Mergeable Queues • Implementation ★ Uses two-list functional queue implementation

    ✦ amortised O(1) enqueue and dequeue operations ★ Merge uses longest common contiguous subsequence algorithm — O(n) • Speci fi cation 1.Any element popped in either A or B does not remain in M 2. Any element pushed into either A or B appears in M 3. An element that remains untouched in LCA, A, B remains in M 4. Order of pairs of elements in LCA, A, B must be preserved in M, if those elements are present in M. 31 M
  88. Mergeable Queues • Implementation ★ Uses two-list functional queue implementation

    ✦ amortised O(1) enqueue and dequeue operations ★ Merge uses longest common contiguous subsequence algorithm — O(n) • Speci fi cation 1.Any element popped in either A or B does not remain in M 2. Any element pushed into either A or B appears in M 3. An element that remains untouched in LCA, A, B remains in M 4. Order of pairs of elements in LCA, A, B must be preserved in M, if those elements are present in M. 31 M Implementation far removed from the specification!
  89. • Programming and proving with RDTs is complicated due to

    concurrency and the lack of suitable programming abstractions 33 Summary
  90. • Programming and proving with RDTs is complicated due to

    concurrency and the lack of suitable programming abstractions • MRDTs simplify RDTs by implementing them as extensions of sequential data types ★ Reasoning about correctness is still hard 33 Summary
  91. • Programming and proving with RDTs is complicated due to

    concurrency and the lack of suitable programming abstractions • MRDTs simplify RDTs by implementing them as extensions of sequential data types ★ Reasoning about correctness is still hard • Peepul is an F* library for certi fi ed MRDTs ★ Replication-aware simulation for proving complex MRDTs ★ Complex MRDTs can be constructed and proved using simpler MRDTs 33 Summary
  92. • Programming and proving with RDTs is complicated due to

    concurrency and the lack of suitable programming abstractions • MRDTs simplify RDTs by implementing them as extensions of sequential data types ★ Reasoning about correctness is still hard • Peepul is an F* library for certi fi ed MRDTs ★ Replication-aware simulation for proving complex MRDTs ★ Complex MRDTs can be constructed and proved using simpler MRDTs • F* allows us to strike a balance between automated and interactive proofs ★ Extract to OCaml and run on Irmin! 33 Summary