Verifying Strong Eventual Consistency in Distributed Systems

Slide 1

Slide 1 text

Verifying Strong Eventual Consistency in Distributed Systems VICTOR B. F. GOMES, MARTIN KLEPPMANN, DOMINIC P. MULLIGAN, ALASTAIR R. BERESFORD Read By: Raghav Roy

Slide 2

Slide 2 text

whoami

Slide 3

Slide 3 text

What I will be covering - Motivation - Foundation - Implementation

Slide 4

Slide 4 text

What I will be covering ● {Strong, Eventual, Strong Eventual} Consistency

Slide 5

Slide 5 text

What I will be covering ● {Strong, Eventual, Strong Eventual} Consistency ● Roles real world networks play

Slide 6

Slide 6 text

What I will be covering ● {Strong, Eventual, Strong Eventual} Consistency ● Roles real world networks play ● Prerequisite Information

Slide 7

Slide 7 text

What I will be covering ● {Strong, Eventual, Strong Eventual} Consistency ● Roles real world networks play ● Prerequisite Information ● Why other algorithms failed - Brief

Slide 8

Slide 8 text

What I will be covering ● {Strong, Eventual, Strong Eventual} Consistency ● Roles real world networks play ● Prerequisite Information ● Why other algorithms failed - Brief ● Proof Strategy and General Purpose Models

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

{Strong, Eventual, Strong Eventual} Consistency

Slide 12

Slide 12 text

Strong Consistency

Slide 13

Slide 13 text

Strong Consistency ● Replicas update in the same order

Slide 14

Slide 14 text

Strong Consistency ● Replicas update in the same order ● Consensus ○ Serialisation Bottleneck

Slide 15

Slide 15 text

Strong Consistency ● Replicas update in the same order ● Consensus ○ Serialisation Bottleneck ○ Tolerate n/2 faults

Slide 16

Slide 16 text

Strong Consistency ● Replicas update in the same order ● Consensus ○ Serialisation Bottleneck ○ Tolerate n/2 faults ● Sequential, Linearisable

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Eventual Consistency

Slide 21

Slide 21 text

Eventual Consistency ● Update local the propagate ○ No foreground sync

Slide 22

Slide 22 text

Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery

Slide 23

Slide 23 text

Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery ● On conﬂict ○ Arbitrate

Slide 24

Slide 24 text

Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery ● On conﬂict ○ Arbitrate ○ Roll-back ● Consensus moved to background

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Diverge

Slide 27

Slide 27 text

Conflict!

Slide 28

Slide 28 text

Strong Eventual Consistency

Slide 29

Slide 29 text

Strong Eventual Consistency ● Update local the propagate ○ No foreground sync

Slide 30

Slide 30 text

Strong Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery

Slide 31

Slide 31 text

Strong Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery ● No conﬂict ○ Unique outcome of concurrent updates

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Strong Eventual Consistency ● Update local the propagate ○ No foreground sync ○ Eventual, reliable delivery ● No conﬂict ○ Unique outcome of concurrent updates ● No consensus: n-1 faults ● Solves CAP theorem? (A critique of the CAP theorem)

Slide 34

Slide 34 text

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Op-Based Commute Necessary conditions for convergence

Slide 40

Slide 40 text

Op-Based Commute Necessary conditions for convergence ● Liveness: All replicas execute all operations in delivery order

Slide 41

Slide 41 text

Op-Based Commute Necessary conditions for convergence ● Liveness: All replicas execute all operations in delivery order ● Safety: Concurrent operations commute

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

Updates a and b should commute

Slide 45

Slide 45 text

Now Add Networks!

Slide 46

Slide 46 text

Networks Do Not Spark Joy ● Replication algorithm must operate across computer networks

Slide 47

Slide 47 text

Networks Do Not Spark Joy ● Replication algorithm must operate across computer networks ● These may arbitrarily delay, drop, or re-order messages

Slide 48

Slide 48 text

Networks Do Not Spark Joy ● Replication algorithm must operate across computer networks ● These may arbitrarily delay, drop, or re-order messages ● Experience temporary partitions of the nodes

Slide 49

Slide 49 text

Networks Do Not Spark Joy ● Replication algorithm must operate across computer networks ● These may arbitrarily delay, drop, or re-order messages ● Experience temporary partitions of the nodes ● Suffer node failures.

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Why Other Algorithms Failed

Slide 52

Slide 52 text

Why They Failed ● Wrong assumptions about the Network Infrastructure

Slide 53

Slide 53 text

Why They Failed ● Wrong assumptions about the Network Infrastructure ● The requirement for a central server for some of these increases the risk of faults

Slide 54

Slide 54 text

Why They Failed ● Wrong assumptions about the Network Infrastructure ● The requirement for a central server for some of these increases the risk of faults ● Their informal reasoning has produced plausible-looking, but incorrect algorithms. (The next two slides are borrowed from Martin Kleppmann’s talk)

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Isabelle/HOL

Slide 58

Slide 58 text

Relevant Semantics What is a Formal Proof?

Slide 59

Slide 59 text

Relevant Semantics What is a Formal Proof? ● A derivation in formal calculus ○ For example: A ∧ B → B ∧ A

Slide 60

Slide 60 text

Relevant Semantics What is a Formal Proof? ● A derivation in formal calculus ○ For example: A ∧ B → B ∧ A ○ Left as an exercise for the reader: (check it out here)

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Relevant Semantics What is a Formal Proof? ● A derivation in formal calculus ○ For example: A ∧ B → B ∧ A ○ Left as an exercise for the reader: (check it out here) What is a Theorem Prover? ● In the context of Isabelle - Automated proofs, and interactive

Slide 63

Slide 63 text

Slide 64

Slide 64 text

Relevant Semantics ● Other veriﬁcation tools: Model checking, static analysis (do not deliver proofs)

Slide 65

Slide 65 text

Relevant Semantics ● Other veriﬁcation tools: Model checking, static analysis (do not deliver proofs) ● Analyse systems thoroughly

Slide 66

Slide 66 text

Relevant Semantics ● Other veriﬁcation tools: Model checking, static analysis (do not deliver proofs) ● Analyse systems thoroughly ● Find design and speciﬁcation errors early

Slide 67

Slide 67 text

Relevant Semantics ● Other veriﬁcation tools: Model checking, static analysis (do not deliver proofs) ● Analyse systems thoroughly ● Find design and speciﬁcation errors early ● High assurance, etc

Slide 68

Slide 68 text

Relevant Semantics Logical Implications

Slide 69

Slide 69 text

Relevant Semantics Logical Implications ● A => B => C or [A;B] => C ○ Read: A and B implies C

Slide 70

Slide 70 text

Relevant Semantics Logical Implications ● A => B => C or [A;B] => C ○ Read: A and B implies C ● Used to write rules, theorems and proof states

Slide 71

Slide 71 text

Relevant Semantics Logical Implications ● A => B => C or [A;B] => C ○ Read: A and B implies C ● Used to write rules, theorems and proof states ● t → s for logical implication between formulae

Slide 72

Slide 72 text

Relevant Semantics ● λx . t ● Anonymous function mapping an argument x to t(x)

Slide 73

Slide 73 text

Relevant Semantics Lists ● [] or ‘nil’ empty list

Slide 74

Slide 74 text

Relevant Semantics Lists ● [] or ‘nil’ empty list ● # - “cons” - prepends an element to an existing list

Slide 75

Slide 75 text

Relevant Semantics Lists ● [] or ‘nil’ empty list ● # - “cons” - prepends an element to an existing list ● @ - concatenation/appending

Slide 76

Slide 76 text

Relevant Semantics Sets ● {} - empty set

Slide 77

Slide 77 text

Relevant Semantics Sets ● {} - empty set ● t ∪ u, t ∩ u, and x ∈ t have usual meanings

Slide 78

Slide 78 text

Relevant Semantics Deﬁnitions and theorems ● Inductive relations are deﬁned with the inductive keyword.

Slide 79

Slide 79 text

Relevant Semantics Definitions and theorems ● Inductive relations are defined with the inductive keyword. inductive only-fives :: nat list ⇒ bool where only-fives [] | [[ only-fives xs ]] ⇒ only-fives (5#xs)

Slide 80

Slide 80 text

Relevant Semantics Deﬁnitions and theorems ● Lemmas, theorems, and corollaries can be asserted using the lemma, theorem, and corollary keywords

Slide 81

Slide 81 text

Relevant Semantics Definitions and theorems ● Lemmas, theorems, and corollaries can be asserted using the lemma, theorem, and corollary keywords theorem only-fives-concat: assumes only-fives xs and only-fives ys shows only-fives (xs @ ys)

Slide 82

Slide 82 text

Relevant Semantics Deﬁnitions and theorems ● Locales: May be thought of as an interface with associated laws that implementations must obey

Slide 83

Slide 83 text

Slide 84

Slide 84 text

Relevant Semantics Definitions and theorems ● Locales: May be thought of as an interface with associated laws that implementations must obey locale semigroup = fixes f :: ′a ⇒ ′a ⇒ ′a assumes f x (f y z) = f (f x y) z ● Introduces a locale, with a fixed, typed constant f, and a law asserting that f is associative.

Slide 85

Slide 85 text

Slide 86

Slide 86 text

Proof Strategy

Slide 87

Slide 87 text

Proof Strategy Breather?

Slide 88

Slide 88 text

Proof Strategy ● The approach here breaks the proof into simple modules or Locales

Slide 89

Slide 89 text

Proof Strategy ● The approach here breaks the proof into simple modules or Locales ● More than half the code to construct a general purpose model of consistency and an axiomatic network model

Slide 90

Slide 90 text

Slide 91

Slide 91 text

Proof Strategy ● The approach here breaks the proof into simple modules or Locales ● More than half the code to construct a general purpose model of consistency and an axiomatic network model ● The remainder - Formalisation of three CRDTs and their proofs for correctness ● Keeping the general purpose modules abstract and independent

Slide 92

Slide 92 text

Slide 93

Slide 93 text

Proof Strategy ● Formalisation of Strong Eventual Consistency (SEC) ○ What they mean by convergence - prove an abstract convergence theorem

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Slide 96

Slide 96 text

Proof Strategy ● Formalisation of Strong Eventual Consistency (SEC) ○ What they mean by convergence - prove an abstract convergence theorem ● This is independent of networks or any particular CRDT ● Describing an axiomatic model of asynchronous networks ○ Only part of the proof with any axiomatic assumptions ● Prove that the network satisﬁes the ordering properties required by the abstract convergence theorem

Slide 97

Slide 97 text

Slide 98

Slide 98 text

No content

Slide 99

Slide 99 text

Abstract Convergence

Slide 100

Slide 100 text

Abstract Convergence ● SEC is stronger than Eventual Consistency ○ Whenever two nodes have received the same set of updates, they must be in the same state

Slide 101

Slide 101 text

Slide 102

Slide 102 text

Abstract Convergence ● SEC is stronger than Eventual Consistency ○ Whenever two nodes have received the same set of updates, they must be in the same state ○ Constrains the value ‘read’ can return at any time ● To Formalise this using Isabelle, no assumptions made about the network or the data structures ○ Abstract model of operations - can be reordered

Slide 103

Slide 103 text

You are here

Slide 104

Slide 104 text

Happens Before and Causality ● Simplest way to achieve convergence - All operations commute ○ Too strong to be useful

Slide 105

Slide 105 text

Happens Before and Causality ● Simplest way to achieve convergence - All operations commute ○ Too strong to be useful ● Better - Only “concurrent” operations commute

Slide 106

Slide 106 text

Slide 107

Slide 107 text

Slide 108

Slide 108 text

Happens Before and Causality ● Simplest way to achieve convergence - All operations commute ○ Too strong to be useful ● Better - Only “concurrent” operations commute ● Any other operations that “knew about” each other i.e. have a “happens-before” relationship (causal dependency) need not commute ○ x ≺ y to indicate that operation x happened before y ○ Type - ′oper ⇒ ′oper ⇒ bool ○ ≺ can be applied to two operations of some abstract type ′oper, returning either True or False

Slide 109

Slide 109 text

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Happens Before and Causality ● Inductive Deﬁnition of operations being consistent with happens-before (or simply hb-consistent)

Slide 112

Slide 112 text

Slide 113

Slide 113 text

Slide 114

Slide 114 text

Happens Before and Causality ● Inductive Deﬁnition of operations being consistent with happens-before (or simply hb-consistent) inductive hb-consistent :: ′oper list ⇒ bool where hb-consistent [] | [[ hb-consistent xs; ∀ x ∈ set xs. ¬ y ≺ x ]] ⇒ hb-consistent (xs @ [y]) ● The empty list is hb-consistent ● Furthermore, given an hb-consistent list xs, we can append an operation y ○ Provided that y does not happen-before any existing operation x in xs

Slide 115

Slide 115 text

Slide 116

Slide 116 text

Slide 117

Slide 117 text

Interpretation of operations ● Modeling state changes - “interpretation function” of type ○ interp :: ′oper ⇒ ′state ⇒ ′state option

Slide 118

Slide 118 text

Slide 119

Slide 119 text

Slide 120

Slide 120 text

Interpretation of operations ● Modeling state changes - “interpretation function” of type ○ interp :: ′oper ⇒ ′state ⇒ ′state option ● This can be looked at as a “state transformer” - function that maps an old state to a new state, or fails by returning “None” ● Capturing this in a Locale - locale happens-before = preorder hb-weak hb for hb-weak :: ′oper ⇒ ′oper ⇒ bool and hb :: ′oper ⇒ ′oper ⇒ bool + ﬁxes interp :: ′oper ⇒ ′state ⇒ ′state option ● This locale extends the “preorder” locale - useful lemmas

Slide 121

Slide 121 text

Slide 122

Slide 122 text

Slide 123

Slide 123 text

Interpretation of operations ● Given two operations x and y, we can now deﬁne the composition of state transformers

Slide 124

Slide 124 text

Interpretation of operations ● Given two operations x and y, we can now deﬁne the composition of state transformers ● 〈x〉 I> 〈y〉 to denote the state transformer that ﬁrst applies the effect of x to some state, and then applies the effect of y to the result.

Slide 125

Slide 125 text

Slide 126

Slide 126 text

Interpretation of operations ● Let’s deﬁne apply-operations

Slide 127

Slide 127 text

Slide 128

Slide 128 text

Interpretation of operations ● Let’s deﬁne apply-operations ● This will compose an arbitrary list of operations into a state transformer deﬁnition apply-operations :: ′oper list ⇒ ′state ⇒ ′state option where apply-operations ops ≡ foldl (op |>) Some (map interp ops) ● The result: state transformer that applies the interpretation of each of the operations in the list in left to right order to some initial state

Slide 129

Slide 129 text

Slide 130

Slide 130 text

Commutativity and Convergence

Slide 131

Slide 131 text

Commutativity and Convergence ● Operations x and y commute when 〈x〉 I> 〈y〉 = 〈y〉 I> 〈x〉 ○ We can swap the order of the interpretation without changing the resulting state

Slide 132

Slide 132 text

Commutativity and Convergence ● Operations x and y commute when 〈x〉 I> 〈y〉 = 〈y〉 I> 〈x〉 ○ We can swap the order of the interpretation without changing the resulting state ● Too strong to have this hold for all operations

Slide 133

Slide 133 text

Slide 134

Slide 134 text

Slide 135

Slide 135 text

Commutativity and Convergence ● Let’s show Convergence!

Slide 136

Slide 136 text

Commutativity and Convergence ● Let’s show Convergence! ● Two “hb-consistent” lists of “distinct” operations - Let these be permutations

Slide 137

Slide 137 text

Commutativity and Convergence ● Let’s show Convergence! ● Two “hb-consistent” lists of “distinct” operations - Let these be permutations ● If concurrent operations commute then they have the same interpretation

Slide 138

Slide 138 text

Slide 139

Slide 139 text

Formalising Strong Eventual Consistency ● The only thing left to consider is “progress”

Slide 140

Slide 140 text

Formalising Strong Eventual Consistency ● The only thing left to consider is “progress” ● Valid operations should not become stuck in an error state

Slide 141

Slide 141 text

Slide 142

Slide 142 text

Slide 143

Slide 143 text

Slide 144

Slide 144 text

Slide 145

Slide 145 text

Formalising Strong Eventual Consistency ● The only thing left to consider is “progress” ● Valid operations should not become stuck in an error state locale strong-eventual-consistency = happens-before + ﬁxes op-history :: ′oper list ⇒ bool and initial-state :: ′state assumes causality: [[ op-history xs ]] ⇒ hb-consistent xs and distinctness: [[ op-history xs ]] ⇒ distinct xs and trunc-history: [[ op-history(xs@[x]) ]] ⇒ op-history xs

Slide 146

Slide 146 text

Slide 147

Slide 147 text

Slide 148

Slide 148 text

Formalising Strong Eventual Consistency ● Some details on that,

Slide 149

Slide 149 text

Formalising Strong Eventual Consistency ● Some details on that, ● Concise summary of the properties that we require in order to achieve SEC

Slide 150

Slide 150 text

Formalising Strong Eventual Consistency ● Some details on that, ● Concise summary of the properties that we require in order to achieve SEC ● Op-history is an abstract predicate describing any valid operation history of some algorithm following: (concurrent-ops-commute, distinct, hb-consistent)

Slide 151

Slide 151 text

Slide 152

Slide 152 text

Formalising Strong Eventual Consistency ● Operations convergence

Slide 153

Slide 153 text

Formalising Strong Eventual Consistency ● Progress (no failure for valid ops)

Slide 154

Slide 154 text

Formalising Strong Eventual Consistency ● First three assumptions are satisﬁed by the network model (no algorithm speciﬁc proofs required)

Slide 155

Slide 155 text

Slide 156

Slide 156 text

Formalising Strong Eventual Consistency ● First three assumptions are satisfied by the network model (no algorithm specific proofs required) ● For individual algorithms we only need to prove commutativity and no-failure ● Note: trunc-history assumption requires that every prefix of a valid operation history is also valid ○ => Convergence theorem holds at every step of the execution.

Slide 157

Slide 157 text

Slide 158

Slide 158 text

Slide 159

Slide 159 text

You are here

Slide 160

Slide 160 text

Axiomatic Network Model

Slide 161

Slide 161 text

Axiomatic Network Model ● In this section, we develop a formal deﬁnition of an asynchronous unreliable causal broadcast network

Slide 162

Slide 162 text

Slide 163

Slide 163 text

Slide 164

Slide 164 text

Slide 165

Slide 165 text

Axiomatic Network Model ● In this section, we develop a formal deﬁnition of an asynchronous unreliable causal broadcast network ● This model will then satisfy the causal delivery requirements of many Op-based CRDTs ● Also, this makes it suitable for use in decentralised settings without the need of a central server, or a quorum of nodes (Stronger consistency models do not have this property). ● The asynchronous aspect means that we make no timing assumptions ○ Messages sent over the network may suffer unbounded delays before they are delivered

Slide 166

Slide 166 text

Slide 167

Slide 167 text

Slide 168

Slide 168 text

Slide 169

Slide 169 text

Modeling a Distributed System ● Aim: Model as an unbounded number of nodes

Slide 170

Slide 170 text

Modeling a Distributed System ● Aim: Model as an unbounded number of nodes ● No assumptions of the communication pattern

Slide 171

Slide 171 text

Slide 172

Slide 172 text

Slide 173

Slide 173 text

Slide 174

Slide 174 text

Modeling a Distributed System ● Aim: Model as an unbounded number of nodes ● No assumptions of the communication pattern ● We assume that each node is uniquely identiﬁed by a natural number (totally ordered) ● Every node’s history has every event (execution step) stored in it - Standard ● History of node i is obtained by “history” -> list of events ● “distinct” is an Isabelle library function that asserts that a list contains no duplicates

Slide 175

Slide 175 text

Slide 176

Slide 176 text

You are here

Slide 177

Slide 177 text

Modeling a Distributed System ● Node’s history is ﬁnite, at the end of the node history the node could have failed or successfully terminated

Slide 178

Slide 178 text

Modeling a Distributed System ● Node’s history is ﬁnite, at the end of the node history the node could have failed or successfully terminated ● Node failures are treated as permanent - Crash Stop abstraction

Slide 179

Slide 179 text

Slide 180

Slide 180 text

Asynchronous Broadcast Network ● We extend the node-histories locale

Slide 181

Slide 181 text

Slide 182

Slide 182 text

Slide 183

Slide 183 text

Asynchronous Broadcast Network ● We extend the node-histories locale ● We need to deﬁne how nodes communicate - Broadcast or Deliver ○ Deliver refers to message being received from the network and “delivered” to an application datatype ′msg event = Broadcast ′msg | Deliver ′msg ● Can be thought of as a Deterministic State Machine where each transition corresponds to a broadcast or a deliver event. ● Broadcast abstraction is the standard for op-based CRDTs because it best ﬁts the replication pattern

Slide 184

Slide 184 text

Slide 185

Slide 185 text

Slide 186

Slide 186 text

Asynchronous Broadcast Network ● Now we can start formally specifying the properties of a broadcast network

Slide 187

Slide 187 text

Asynchronous Broadcast Network ● Now we can start formally specifying the properties of a broadcast network ● Three Axioms: Delivery-Has-A-Cause, Deliver-Locally and Msg-Id-Unique

Slide 188

Slide 188 text

Slide 189

Slide 189 text

Slide 190

Slide 190 text

Slide 191

Slide 191 text

Slide 192

Slide 192 text

Asynchronous Broadcast Network ● Now we can start formally specifying the properties of a broadcast network ● Three Axioms: Delivery-Has-A-Cause, Deliver-Locally and Msg-Id-Unique locale network = node-histories history for history :: nat ⇒ ′msg event list + ﬁxes msg-id :: ′msg ⇒ ′msgid assumes delivery-has-a-cause: [[ Deliver m ∈ set (history i) ]] ⇒ ∃ j. Broadcast m ∈ set (history j) and deliver-locally: [[ Broadcast m ∈ set (history i) ]] ⇒ Broadcast m ⊏i Deliver m

Slide 193

Slide 193 text

Slide 194

Slide 194 text

Asynchronous Broadcast Network ● Delivery Has a Cause: No “out of thin air” values, if m was delivered at some node, then there exists a node where m was broadcast

Slide 195

Slide 195 text

Slide 196

Slide 196 text

Slide 197

Slide 197 text

Slide 198

Slide 198 text

Asynchronous Broadcast Network ● Delivery Has a Cause: No “out of thin air” values, if m was delivered at some node, then there exists a node where m was broadcast ● Deliver Locally: All broadcast messages are delivered to the node that broadcast the message as well ● Msg ID Unique: We assume the existence of msg-id :: ′msg ⇒ ′msgid that maps every message to some global identiﬁer (unique node IDs, sequence numbers, timestamps) ● Network Locale inherits “histories-distinct” from node-histories ○ Every message that is delivered on some node, there is exactly one broadcast event that created this message

Slide 199

Slide 199 text

Slide 200

Slide 200 text

Slide 201

Slide 201 text

You are here

Slide 202

Slide 202 text

Causally Ordered Delivery ● We need to deﬁne an instance of ordering relation ≺ on messages, and prove that is satisﬁes strict partial ordering

Slide 203

Slide 203 text

Causally Ordered Delivery ● We need to deﬁne an instance of ordering relation ≺ on messages, and prove that is satisﬁes strict partial ordering ● m1 happens before m2, if the node that generated m2 “knew about” m1 when m2 was generated

Slide 204

Slide 204 text

Slide 205

Slide 205 text

Causally Ordered Delivery Verbal deﬁnition

Slide 206

Slide 206 text

Causally Ordered Delivery Verbal deﬁnition ● m1 and m2 were broadcast by the same node, and m1 was broadcast before m2.

Slide 207

Slide 207 text

Causally Ordered Delivery Verbal deﬁnition ● m1 and m2 were broadcast by the same node, and m1 was broadcast before m2. ● The node that broadcast m2 had delivered m1 before it broadcast m2.

Slide 208

Slide 208 text

Causally Ordered Delivery Verbal deﬁnition ● m1 and m2 were broadcast by the same node, and m1 was broadcast before m2. ● The node that broadcast m2 had delivered m1 before it broadcast m2. ● There exists some operation m3 such that m1 ≺ m3 and m3 ≺ m2

Slide 209

Slide 209 text

Slide 210

Slide 210 text

Causally Ordered Delivery ● With this, we can create a restricted variant our broadcast network model by extending the network locale

Slide 211

Slide 211 text

Causally Ordered Delivery ● With this, we can create a restricted variant our broadcast network model by extending the network locale Assumptions ● if there are any happens-before dependencies between messages, they must be delivered in that order.

Slide 212

Slide 212 text

Slide 213

Slide 213 text

Slide 214

Slide 214 text

You are here

Slide 215

Slide 215 text

Using Operations in the Network ● So far we have only talked about order of “messages”, but we need to attach these messages to operations

Slide 216

Slide 216 text

Using Operations in the Network ● So far we have only talked about order of “messages”, but we need to attach these messages to operations (Spoiler: New Locale!)

Slide 217

Slide 217 text

Slide 218

Slide 218 text

Using Operations in the Network ● So far we have only talked about order of “messages”, but we need to attach these messages to operations (Spoiler: New Locale!) ● We can extend our convergence theorem into our network model by extending the causal-network locale ● All we need to do is specialise the variable of messages ‘msg to be a pair of ′msgid × ′oper, and we can ﬁx the msg-id functions to this locale

Slide 219

Slide 219 text

Slide 220

Slide 220 text

Using Operations in the Network

Slide 221

Slide 221 text

Using Operations in the Network ● Since this extends network

Slide 222

Slide 222 text

Using Operations in the Network ● Since this extends network ● It also meets its requirement of the happens-before locale

Slide 223

Slide 223 text

Slide 224

Slide 224 text

Using Operations in the Network ● Since this extends network ● It also meets its requirement of the happens-before locale ● So the lemmas and deﬁnitions of this locale can use the happens-before relations ≺ ● We can prove that the sequence of message deliveries at any node is consistent with hb-consistent (can show this by preﬁxing “hb-consistent” to the theorem)

Slide 225

Slide 225 text

Using Operations in the Network theorem hb.hb-consistent (node-deliver-messages (history i))

Slide 226

Slide 226 text

Slide 227

Slide 227 text

Using Operations in the Network theorem hb.hb-consistent (node-deliver-messages (history i)) ● Here, node-deliver-messages ﬁlters the history of events at some node to return only messages that were delivered, in order ● When the message is delivered, we can take the operation ‘oper from it and use it’s “interpretation” to update that node

Slide 228

Slide 228 text

Using Operations in the Network ● We can then define the state of some node by defining “apply-operations” (with msg-id) Remember this definition!

Slide 229

Slide 229 text

You are here

Slide 230

Slide 230 text

Only Valid Messages ● Messages that are broadcast can have some restrictions by an algorithm, we need a general purpose way of modelling this

Slide 231

Slide 231 text

Only Valid Messages ● Messages that are broadcast can have some restrictions by an algorithm, we need a general purpose way of modelling this ● As they may not be able to be expressed in Isabelle’s type system - New Locale!

Slide 232

Slide 232 text

Slide 233

Slide 233 text

Only Valid Messages ● Broadcast Only Valid Messages is the ﬁnal Axiom, all it requires is that ○ If a node broadcasts a message, it must be valid according to “valid-msg”

Slide 234

Slide 234 text

Only Valid Messages ● Broadcast Only Valid Messages is the ﬁnal Axiom, all it requires is that ○ If a node broadcasts a message, it must be valid according to “valid-msg” ● Algorithms embedded in this locale are the ones that deﬁnes this predicate for valid message.

Slide 235

Slide 235 text

You are here

Slide 236

Slide 236 text

Replication Algorithms

Slide 237

Slide 237 text

Replication Algorithms Breather?

Slide 238

Slide 238 text

Replicated Growable Array ● Replicated ordered list - supports insert and delete operations

Slide 239

Slide 239 text

Replicated Growable Array ● Replicated ordered list - supports insert and delete operations ● Here, every insert and delete must identify the position at which the modiﬁcation should take place

Slide 240

Slide 240 text

Slide 241

Slide 241 text

Slide 242

Slide 242 text

Replicated Growable Array ● Replicated ordered list - supports insert and delete operations ● Here, every insert and delete must identify the position at which the modiﬁcation should take place ● This is because unlike in a non-replicated case, index of list element can change if there are concurrent inserts or deletes ● Insertion - After an existing list element (with a given ID) or the head of the list if there is no ID ● Deletion - It’s not completely safe to remove a list element, concurrent insertions would not be able to locate this element ○ Retains tombstone - deletion merely sets a ﬂag to mark it as deleted

Slide 243

Slide 243 text

Slide 244

Slide 244 text

Replicated Growable Array

Slide 245

Slide 245 text

Replicated Growable Array ● RGA state at each node - list of elements

Slide 246

Slide 246 text

Replicated Growable Array ● RGA state at each node - list of elements ● Each element is a triple

Slide 247

Slide 247 text

Slide 248

Slide 248 text

Replicated Growable Array ● RGA state at each node - list of elements ● Each element is a triple ● Unique ID of the list element, value to inserted, ﬂag that indicates that the element as has been deleted type-synonym ( ′id, ′v) elt = ′id × ′v × bool ● Insert takes three params - Previous state of the list, new element to insert, ID of existing element after which value has to be inserted

Slide 249

Slide 249 text

Slide 250

Slide 250 text

Replicated Growable Array ● The function iterates over the list and compares the ID for each element

Slide 251

Slide 251 text

Replicated Growable Array ● The function iterates over the list and compares the ID for each element ● When the insertion position is found, “insert-body” is invoked to perform the actual insertion

Slide 252

Slide 252 text

Replicated Growable Array ● In a replicated datatype, several nodes could be inserting at the same location concurrently

Slide 253

Slide 253 text

Slide 254

Slide 254 text

Replicated Growable Array ● In a replicated datatype, several nodes could be inserting at the same location concurrently ● These insertions may be processed in a different order by different nodes ● How do we make it converge?

Slide 255

Slide 255 text

Slide 256

Slide 256 text

Replicated Growable Array ● The insert-body function skips over elements with an ID greater than that of the newly added element

Slide 257

Slide 257 text

Replicated Growable Array ● The insert-body function skips over elements with an ID greater than that of the newly added element ● IDs will be in total linear order (speciﬁed above as ‘id::{linorder})

Slide 258

Slide 258 text

Replicated Growable Array ● Implementing delete with the same idea

Slide 259

Slide 259 text

Replicated Growable Array ● Implementing delete with the same idea ● It searches for the element with a given ID and sets its ﬂag to True to mark it as deleted

Slide 260

Slide 260 text

Replicated Growable Array ● Implementing delete with the same idea ● It searches for the element with a given ID and sets its ﬂag to True to mark it as deleted

Slide 261

Slide 261 text

Are We Forgetting Something?

Slide 262

Slide 262 text

Reasoning Commutativity ● We discussed earlier that the only thing we need to show for particular algorithms is that all concurrent operations commute.

Slide 263

Slide 263 text

Reasoning Commutativity ● We discussed earlier that the only thing we need to show for particular algorithms is that all concurrent operations commute. ● Easy to show delete always commutes with itself

Slide 264

Slide 264 text

Slide 265

Slide 265 text

Reasoning Commutativity ● To show insert commutes with itself, we need to make a few assumptions

Slide 266

Slide 266 text

Reasoning Commutativity ● To show insert commutes with itself, we need to make a few assumptions ● e1 and e2 are of type ′id × ′v × bool

Slide 267

Slide 267 text

Slide 268

Slide 268 text

Reasoning Commutativity ● To show insert commutes with itself, we need to make a few assumptions ● e1 and e2 are of type ′id × ′v × bool ● i1 :: ′id is the position after which e1 should be inserted ● Similarly, i2 is the position where e2 should be inserted

Slide 269

Slide 269 text

Slide 270

Slide 270 text

Slide 271

Slide 271 text

Reasoning Commutativity ● Finally, we need to show that insert-delete commute

Slide 272

Slide 272 text

Reasoning Commutativity ● Finally, we need to show that insert-delete commute

Slide 273

Slide 273 text

Reasoning Commutativity ● Finally, we need to show that insert-delete commute ● Just the constraint that the element to be deleted is not the same as the element to be inserted (insert wins strategy)

Slide 274

Slide 274 text

Let’s Add This to The Network Model!

Slide 275

Slide 275 text

Embedding RGA in the network model ● To be able to prove SEC for RGA, we need to embed the insert and delete operations in the network model

Slide 276

Slide 276 text

Embedding RGA in the network model ● To be able to prove SEC for RGA, we need to embed the insert and delete operations in the network model ● We need to deﬁne a datatype for these operations, and an interpretation function (we saw this earlier)

Slide 277

Slide 277 text

Slide 278

Slide 278 text

Slide 279

Slide 279 text

Embedding RGA in the network model ● When are these operations valid? ○ IDs of the operations must be unique

Slide 280

Slide 280 text

Embedding RGA in the network model ● When are these operations valid? ○ IDs of the operations must be unique ○ Whenever an element is referred to be insert or delete, the element must exist

Slide 281

Slide 281 text

Embedding RGA in the network model ● When are these operations valid? ○ IDs of the operations must be unique ○ Whenever an element is referred to be insert or delete, the element must exist ● We can now deﬁne the “valid-rga-msg” predicate

Slide 282

Slide 282 text

Slide 283

Slide 283 text

Slide 284

Slide 284 text

Embedding RGA in the network model ● With these deﬁnitions, we can simply deﬁne the rga Locale by extending the network-with-constrained-ops

Slide 285

Slide 285 text

Embedding RGA in the network model ● With these deﬁnitions, we can simply deﬁne the rga Locale by extending the network-with-constrained-ops ● Initial state is the empty list

Slide 286

Slide 286 text

Embedding RGA in the network model ● With these deﬁnitions, we can simply deﬁne the rga Locale by extending the network-with-constrained-ops ● Initial state is the empty list ● The validity predicate we described above

Slide 287

Slide 287 text

Slide 288

Slide 288 text

Embedding RGA in the network model ● We also need to show that whenever an insert or delete refers to an existing element, there is always a prior insertion operation that created this element:

Slide 289

Slide 289 text

Embedding RGA in the network model ● We also need to show that whenever an insert or delete refers to an existing element, there is always a prior insertion operation that created this element:

Slide 290

Slide 290 text

Embedding RGA in the network model ● Since the network ensures causally ordered delivery, all nodes must deliver some insertion op1 before the dependent operation op2

Slide 291

Slide 291 text

Embedding RGA in the network model ● Since the network ensures causally ordered delivery, all nodes must deliver some insertion op1 before the dependent operation op2 ● Therefore, all cases where operations do not commute, one happens before another

Slide 292

Slide 292 text

Slide 293

Slide 293 text

Embedding RGA in the network model ● Finally, we need to show that Failure case for an interpretation operation is never reached.

Slide 294

Slide 294 text

Embedding RGA in the network model ● Finally, we need to show that Failure case for an interpretation operation is never reached. ● With this, it's easy to show that rga satisﬁes all the requirements of SEC (formally)

Slide 295

Slide 295 text

You are here

Slide 296

Slide 296 text

Two Other (Simpler) CRDTs

Slide 297

Slide 297 text

Increment-Decrement Counter ● We can now show that the proof framework provides reusable components that simplify the proofs for new algorithms

Slide 298

Slide 298 text

Increment-Decrement Counter ● We can now show that the proof framework provides reusable components that simplify the proofs for new algorithms ● Let’s start with the simplest CRDT

Slide 299

Slide 299 text

Increment-Decrement Counter ● We can now show that the proof framework provides reusable components that simplify the proofs for new algorithms ● Let’s start with the simplest CRDT ● Increment and Decrement a shared integer counter

Slide 300

Slide 300 text

Slide 301

Slide 301 text

Increment-Decrement Counter ● Interpretation function!

Slide 302

Slide 302 text

Increment-Decrement Counter ● It becomes an easy exercise to show commutativity of the operations

Slide 303

Slide 303 text

Slide 304

Slide 304 text

Increment-Decrement Counter ● It becomes an easy exercise to show commutativity of the operations ● We don’t need to extend the network-with-constrained-ops locale ○ The operations need not even be causally delivered ● Just with this, we can show that the Counter is a sublocale of strong-eventual-consistency (from which we can obtain convergence and progress theorems)

Slide 305

Slide 305 text

Increment-Decrement Counter

Slide 306

Slide 306 text

You are here

Slide 307

Slide 307 text

Observed Removed Set ● ORSet is a well known CRDT for implementing replicated sets

Slide 308

Slide 308 text

Observed Removed Set ● ORSet is a well known CRDT for implementing replicated sets ● Supports two operations - Adding and Removing arbitrary elements in the set

Slide 309

Slide 309 text

Observed Removed Set ● ORSet is a well known CRDT for implementing replicated sets ● Supports two operations - Adding and Removing arbitrary elements in the set ● Let’s deﬁne the datatype

Slide 310

Slide 310 text

Slide 311

Slide 311 text

Slide 312

Slide 312 text

Observed Removed Set ● ORSet is a well known CRDT for implementing replicated sets ● Supports two operations - Adding and Removing arbitrary elements in the set ● Let’s deﬁne the datatype ● ‘id - abstract type of message identiﬁers and ‘a refers to the type of the value that the application wants to add to the set ● When element e needs to be added, Add i e is tagged with “i” to distinguish it from other operations that may add the same element to the set ● When element e needs to be removed, Rem is e is called

Slide 313

Slide 313 text

Slide 314

Slide 314 text

Observed Removed Set ● Let’s deﬁne this using it’s datatype

Slide 315

Slide 315 text

Observed Removed Set ● Let’s deﬁne this using it’s datatype ● The name comes from the fact that the algorithm “observes” the state of the node when removing an element

Slide 316

Slide 316 text

Slide 317

Slide 317 text

Observed Removed Set ● Let’s deﬁne this using it’s datatype ● The name comes from the fact that the algorithm “observes” the state of the node when removing an element ● The state at each node is a function that maps each element ‘a to the set of ID’s of operations that have added to that element ● ‘a is part of the ORset if the set of IDs is non-empty; Init state - λx. {}, ○ The function that maps every possible element ‘a to the empty set of IDs

Slide 318

Slide 318 text

Observed Removed Set ● When interpreting Add - add the identiﬁer of that operation to the node state

Slide 319

Slide 319 text

Observed Removed Set ● When interpreting Add - add the identiﬁer of that operation to the node state ● When interpreting Remove - update the node to remove all causally prior Add identiﬁers

Slide 320

Slide 320 text

Observed Removed Set

Slide 321

Slide 321 text

Observed Removed Set ● Here, state((op-elem oper) := after) is Isabelle’s syntax for pointwise function update.

Slide 322

Slide 322 text

Observed Removed Set ● Here, state((op-elem oper) := after) is Isabelle’s syntax for pointwise function update. ● A remove operation effectively undoes the prior additions of that element of the set.

Slide 323

Slide 323 text

Slide 324

Slide 324 text

Observed Removed Set ● Finally, what’s left to specify the ORset locale, we need to show that Add and Rem use identiﬁers correctly.

Slide 325

Slide 325 text

Slide 326

Slide 326 text

Observed Removed Set ● Finally, what’s left to specify the ORset locale, we need to show that Add and Rem use identiﬁers correctly. ● Add operations should be globally unique (unique ID of the message) ● Rem operation must contain the set of addition identiﬁer in the node ○ At the moment the Rem operation was issued

Slide 327

Slide 327 text

Slide 328

Slide 328 text

Observed Removed Set ● With this, we can extend the network-with-constrained-ops locale to deﬁne the orset locale

Slide 329

Slide 329 text

Observed Removed Set ● With this, we can extend the network-with-constrained-ops locale to deﬁne the orset locale

Slide 330

Slide 330 text

Observed Removed Set ● With this, we can extend the network-with-constrained-ops locale to deﬁne the orset locale ● Now, for Strong Eventual Consistency, we must show that the “apply-operations” predicate never fails ○ Easy here since it never returns None

Slide 331

Slide 331 text

Slide 332

Slide 332 text

Observed Removed Set ● Finally, we need to show that concurrent operations commute (we’re almost there!)

Slide 333

Slide 333 text

Observed Removed Set ● Finally, we need to show that concurrent operations commute (we’re almost there!) ● Two concurrent adds or two removes are easily veriﬁable

Slide 334

Slide 334 text

Observed Removed Set ● But for add and remove operations, this is only if the identiﬁer of the addition is not one of the identiﬁers affected by the removal

Slide 335

Slide 335 text

Observed Removed Set ● But for add and remove operations, this is only if the identiﬁer of the addition is not one of the identiﬁers affected by the removal

Slide 336

Slide 336 text

Observed Removed Set ● But for add and remove operations, this is only if the identiﬁer of the addition is not one of the identiﬁers affected by the removal ● To show that holds for all concurrent Add and Rem is a bit more work

Slide 337

Slide 337 text

Observed Removed Set ● We deﬁne added-ids to be the identiﬁers of all Add operations in a list of delivery events (even if these are subsequently removed)

Slide 338

Slide 338 text

Observed Removed Set ● We define added-ids to be the identifiers of all Add operations in a list of delivery events (even if these are subsequently removed) ● Then, we can show that the set of identifiers in the node state is a subset of added-ids ○ Add only ever adds IDs to the node state, and Rem only ever remove IDs

Slide 339

Slide 339 text

Slide 340

Slide 340 text

Observed Removed Set ● From this, we can show that if Add and Rem are concurrent, ie, the identiﬁer of the Add cannot be in the set of identiﬁers removed by Rem

Slide 341

Slide 341 text

Observed Removed Set ● From this, we can show that if Add and Rem are concurrent, ie, the identiﬁer of the Add cannot be in the set of identiﬁers removed by Rem

Slide 342

Slide 342 text

Observed Removed Set ● Now that we have proved that the assumption of add-rem-commute holds for all concurrent operations

Slide 343

Slide 343 text

Observed Removed Set ● Now that we have proved that the assumption of add-rem-commute holds for all concurrent operations ● Let’s deduce that all concurrent operations commute:

Slide 344

Slide 344 text

Observed Removed Set ● Now that we have proved that the assumption of add-rem-commute holds for all concurrent operations ● Let’s deduce that all concurrent operations commute:

Slide 345

Slide 345 text

Observed Removed Set ● Now that we have proved that the assumption of add-rem-commute holds for all concurrent operations ● Let’s deduce that all concurrent operations commute: ● With this, (apply-operations-never-fails and concurrent-operations-commute), we can immediately prove that orset is a sublocale of strong-eventual-consistency.

Slide 346

Slide 346 text

You are here

Slide 347

Slide 347 text

Whew!

Slide 348

Slide 348 text

Whew! Some Final Remarks

Slide 349

Slide 349 text

Final Remarks ● When we have different nodes concurrently perform updates, without coordinating with each other (like with SEC)

Slide 350

Slide 350 text

Slide 351

Slide 351 text

Slide 352

Slide 352 text

Final Remarks ● When we have different nodes concurrently perform updates, without coordinating with each other (like with SEC) ● We need conflict resolution for concurrent updates at a single node ● User-defined conflict resolution - leave it for manual resolution by the user ● Last Write Wins - Pick the version with the highest timestamp (discard other versions)

Slide 353

Slide 353 text

Slide 354

Slide 354 text

Final Remarks ● Informal reasoning has repeatedly produced approaches that fail to converge in certain scenarios - several proofs turned out to be false

Slide 355

Slide 355 text

Final Remarks ● Informal reasoning has repeatedly produced approaches that fail to converge in certain scenarios - several proofs turned out to be false ● Formal veriﬁcation to distributed systems is an active area of research

Slide 356

Slide 356 text

Slide 357

Slide 357 text

Additional Papers That Can Be Read ● Formal design and veriﬁcation of operational transformation algorithms for copies convergence ● Failed Operational Transform Models ○ Concurrency control in groupware systems ○ An Integrating, Transformation-Oriented Approach to Concurrency Control and Undo in Group Editors ● Tutorial to Locales and Locale Interpretation ● Detecting causal relationships in distributed computations: In search of the holy grail

Slide 358

Slide 358 text

References ● Verifying Strong Eventual Consistency in Distributed Systems ● Github link to all the proofs ● CRDTs and the Quest for Distributed Consistency - Martin Kleppmann ● Strong Eventual Consistency and Conflict-free Replicated Data Types - Marc Shapiro ● Intro to formal verification using traffic signal controllers ● Course slides from TUM - For Isabelle ● A critique of the CAP theorem - Martin Kleppmann ● Operation-based CRDTs: arrays (part 2)

Slide 359

Slide 359 text

Thank you!