Just-Right Consistency - Closing the CAP Gap

Just-Right Consistency

Closing the CAP Gap
Christopher S. Meiklejohn (@cmeik),

Peter Lash
LIGHT ONE

View Slide

Outline:
Closing the CAP Gap
• Just-Right Consistency 
Available as possible, and consistent when
necessary
2

View Slide

Outline:
Closing the CAP Gap
necessary
• AntidoteDB 
The ﬁrst database that provides transactions with
strong semantics, targeted at the JRC approach
2

View Slide

Outline:
Closing the CAP Gap
necessary
• AntidoteDB 
The ﬁrst database that provides transactions with
strong semantics, targeted at the JRC approach
• Moving forward 
Antidote’s path forward from research to
company and product
2

View Slide

Motivation
Cloud Databases
3

View Slide

[Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]

View Slide

A
Centralized database.

View Slide

A
Clients read and write
against the primary copy.

View Slide

A
B
C
Geo-replicated for both
fault-tolerance and high-availability.

View Slide

A
B
C
Clients read and write locally
for low-latency.

View Slide

A
B
C
What happens if C
can’t communicate with other replicas?

View Slide

A
B
C
Choice 1: Consistent-Under-Partition (CP)
• Synchronize each operation 
Maintains “single system image”

View Slide

A
B
C
• Spanner/F1, serializability model 
Coordination is expensive; Spanner typically has to
wait 100ms to commit an update transaction

View Slide

A
B
C
• Spanner/F1, serializability model 
Coordination is expensive; Spanner typically has to
wait 100ms to commit an update transaction
Over-conservative,

but easy to program!

View Slide

A
B
C
Choice 2: Available-Under-Partition (AP)
• Riak, Cassandra, Dynamo 
Operations issued against local copy, and across the cluster in
parallel

View Slide

A
B
C
parallel
• Local operation only, asynchronous propagation 
Stale reads and write conﬂicts will occur without
synchronization

View Slide

A
B
C
parallel
• Local operation only, asynchronous propagation 
Stale reads and write conﬂicts will occur without
synchronization
Available,

but difﬁcult to program!

View Slide

A
B
C
CAP Theorem
CP AP

View Slide

A
B
C
CAP Theorem
High cost
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
High availability
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
High availability
Anomalies
CP AP

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
High availability
Anomalies
CP AP
False dichotomy!

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
High availability
Anomalies
CP AP
False dichotomy!
• No “one-size-ﬁts-all” consistency model 
Choosing either model will either be over-conservative or
risk anomalies

View Slide

A
B
C
CAP Theorem
High cost
Low availability
Synchronization
Low cost
High availability
Anomalies
CP AP
False dichotomy!
• No “one-size-ﬁts-all” consistency model 
Choosing either model will either be over-conservative or
risk anomalies
• Application-level invariants 
Instead, tailor consistency choices based on application-
level invariants for each operation

View Slide

Just Right Consistency
• Preserve sequential patterns 
Applications written sequentially that are correct should maintain
correctness under concurrency
13

View Slide

• AP-compatible invariants 
Strongest AP model; invariants that only require “one way”
communications
13

View Slide

communications
• CAP-sensitive invariants 
Transactions that require coordination; “two way” communication
invariants
13

View Slide

communications
• CAP-sensitive invariants 
Transactions that require coordination; “two way” communication
invariants
• Tools for analysis and veriﬁcation 
Identify and verify application has sufﬁcient synchronization to
ensure application invariants
13

View Slide

Example
Fælles Medicinkort
14

View Slide

Fælles Medicinkort
• FMK [production] / FMKe [synthetic workload] 
Danish National Joint Medicine Card; operating 24x7
since 2013 for 6 million Danish citizens
15

View Slide

Fælles Medicinkort
• Lifecycle management for prescriptions 
Involves patient, pharmacy, and doctor management
around active prescriptions in Denmark
15

View Slide

Fælles Medicinkort
• Assumed correct in isolation 
“Correct-Individually”, C in ACID, each operation
ensures application-level invariants
15

View Slide

Fælles Medicinkort
• Assumed correct in isolation 
“Correct-Individually”, C in ACID, each operation
ensures application-level invariants
15
• create-prescription 
Create prescription for patient, doctor, pharmacy

• update-prescription-medication 
Add or increase medication to prescription

• process-prescription 
Deliver a medication by a pharmacy

• get-*-prescriptions 
Query functions to return information about prescriptions

View Slide

FMKe Invariants
• Relative order [referential integrity] 
Create a prescription and reference it by a
patient
16

View Slide

FMKe Invariants
patient
• Joint update [atomicity] 
Create prescription, then update doctor,
patient, and pharmacy
16

View Slide

FMKe Invariants
patient
• Joint update [atomicity] 
Create prescription, then update doctor,
patient, and pharmacy
• Precondition check [if, then] 
Medication should not be over delivered
16

View Slide

Invariants
AP-compatible
17

View Slide

AP-compatible
• No synchronization 
Updates occur locally without blocking, no
synchronization in the critical path
18

View Slide

AP-compatible
• Asynchronous operation 
Updates are fast, available, and exploit
concurrency
18

View Slide

AP-compatible
• Asynchronous operation 
Updates are fast, available, and exploit
concurrency
• Compatible invariants 
Relative order and joint update invariants can
be preserved
18

View Slide

AP-compatibe
Data Model
19

View Slide

RA
RB

View Slide

RA
RB
1
set(1)

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
2
3
Concurrent assignments

don’t commute!

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
2
3
Concurrent assignments

don’t commute!
Assignment requires CP.

View Slide

24
Can we ﬁnd a suitable data model
for AP systems?

View Slide

Can we make non-commutative
updates commutative?
24
Can we ﬁnd a suitable data model
for AP systems?

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
?
?
How do we deterministically pick

a value to keep?

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
?
?

a value to keep?
Do we use a timestamp?

(like Cassandra, and drop a value?)

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
?
?

a value to keep?
Do we use a timestamp?

(like Cassandra, and drop a value?)
Timestamps make concurrent
operations commute

but fail to capture intent.

View Slide

Can we be smarter about
the merge function?
26

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
3
3
max(2,3)
max(2,3)
Deterministic
conﬂict resolution
function.

View Slide

RA
RB
1
set(1)
3
2
set(2)
set(3)
3
3
max(2,3)
max(2,3)
Deterministic
conﬂict resolution
function.
CRDTs
generalize
this framework.

View Slide

Conﬂict-Free  
Replicated Data Types
• Replicated abstract data types 
Extension of sequential data type that
encapsulates deterministic merge
function
28

View Slide

Conﬂict-Free  
Replicated Data Types
• Replicated abstract data types 
Extension of sequential data type that
encapsulates deterministic merge
function
• Many existing designs 
Sets, counters, registers, ﬂags, maps
28

View Slide

AP-compatibe
Relative Order
29

View Slide

RA
RB

View Slide

RA
RB
Maintain program order
implication invariant.

View Slide

RA
RB
Maintain program order
implication invariant.
For instance, P => Q.

View Slide

RA
RB
Q
true(Q)
Make Q true.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Make P true.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Program order implies ordering relationship.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Ordering is respected at other replicas.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Out of order propagation violates invariant!

View Slide

RA
RB
Q
true(Q)
P
true(P)
P is true,
Q is NOT true!

View Slide

Let’s look at a
concrete example.
37

View Slide

RA
RB

View Slide

RA
RB
Q
true(Q)
Change default administrator password.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Enable administrator login.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Replica A is secure.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Replica B is secure.

View Slide

RA
RB
Q
true(Q)
P
true(P)
Reordering allows default password
to be used to login!

View Slide

Causal Consistency
• Respect causality 
Ensure updates are delivered in the causal order 
[Lamport 78]
44

View Slide

Causal Consistency
[Lamport 78]
• Strongest available model 
Always able to return some compatible version
for an object
44

View Slide

Causal Consistency
[Lamport 78]
• Strongest available model 
Always able to return some compatible version
for an object
• Referential integrity 
Causal consistency is sufﬁcient for providing
referential integrity in an AP database
44

View Slide

…relative order invariants
are preserved transparently!
45
Causal consistency
means…

View Slide

AP-compatibe
Joint Update
46

View Slide

RA
RB
C1
Client performing reads.

View Slide

RA
RB
C1
Rx
create Rx
Create prescription.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Add reference in doctor record.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Pt
update Pt(Rx)
Add reference in patient record.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Pt
update Pt(Rx)
Ph
update Ph(Rx)
Add reference in pharmacy record.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Pt
update Pt(Rx)
Ph
update Ph(Rx)
Updates are causally consistent.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Pt
update Pt(Rx)
Ph
update Ph(Rx)
Client can read inconsistent state.

View Slide

RA
RB
C1
Rx
create Rx
Dr
update Dr(Rx)
Pt
update Pt(Rx)
Ph
update Ph(Rx)
Client is missing update to pharmacy.

View Slide

Can we ensure updates are
All-or-Nothing?
55

View Slide

RA
RB
C1
T1
create Rx
update Dr(Rx)
update Pt(Rx)
update Ph(Rx)
Group updates into an atomic transaction.

View Slide

RA
RB
C1
T1
create Rx
update Dr(Rx)
update Pt(Rx)
update Ph(Rx)
Updates reﬂect “All-Or-Nothing” property
through snapshots.

View Slide

RA
RB
C1
T1
create Rx
update Dr(Rx)
update Pt(Rx)
update Ph(Rx)
T2
Transactions are delivered in causal order.

View Slide

RA
RB
C1
T1
create Rx
update Dr(Rx)
update Pt(Rx)
update Ph(Rx)
T2
Therefore, snapshots are causally consistent.

View Slide

AP-compatible transactions
provide the “A” in ACID
60

View Slide

Transactional
Causal Consistency
61
Strongest model
that is available (AP)

View Slide

Invariants
CAP-sensitive
62

View Slide

What about preventing over
delivery of prescriptions?
63

View Slide

RA(2)
RB(2) ?
?
RC(2) ?
Three replicas each with
two available medications.

View Slide

RA(2)
RB(2) 1
1
1
pp(1)
RC(2) 1
Replica A checks precondition
and delivers medication.

View Slide

RA(2)
RB(2) 1
1
1
pp(1)
RC(2) 1
Correct outcome

where one medication remains.

View Slide

Is this safe
with concurrent operations?
67

View Slide

RA(2)
RB(2) ?
?
RC(2) ?

View Slide

RA(2)
RB(2) 4
4
1
pp(1)
RC(2) 4
4
add(3)

View Slide

RA(2)
RB(2) 4
4
1
pp(1)
RC(2) 4
4
add(3)
Replica C adds three medications

to the prescription.

View Slide

RA(2)
RB(2) 4
4
1
pp(1)
RC(2) 4
4
add(3)
Correct outcome
with four remaining medications.

View Slide

RA(2)
RB(2) 4
4
1
pp(1)
RC(2) 4
4
add(3)
Correct outcome
with four remaining medications.
Precondition is stable under
concurrent addition.

View Slide

Is this safe
with concurrent deliveries?
72

View Slide

RA(2)
RB(2) ?
?
RC(2) ?

View Slide

RA(2)
RB(2) -1
-1
1
pp(1)
RC(2) -1
0
pp(2)

View Slide

RA(2)
RB(2) -1
-1
1
pp(1)
RC(2) -1
0
pp(2)
Replica C concurrently checks precondition

and delivers two medications.

View Slide

RA(2)
RB(2) -1
-1
1
pp(1)
RC(2) -1
0
pp(2)
Incorrect outcome
violating non-negative invariant.

View Slide

RA(2)
RB(2) -1
-1
1
pp(1)
RC(2) -1
0
pp(2)
Incorrect outcome
Precondition is NOT stable
under concurrent fulﬁllment.

View Slide

RA(2)
RB(2) -1
-1
1
pp(1)
RC(2) -1
0
pp(2)
Incorrect outcome
Precondition is NOT stable
under concurrent fulﬁllment.
• Forbid concurrency 
Prevent operations from proceeding without synchronization to
enforce invariant
• Allow concurrency and remove invariant 
Allow operation to proceed, knowing that the invariant may be
violated under concurrent operations

View Slide

How do we know when it’s
safe?
77

View Slide

CISE Analysis
78

View Slide

RA
RB I?
I?
?
Upre?
RC I?
?
Vpre?
Analyze possible pairs
of concurrent operations…

View Slide

RA
RB I?
I?
?
Upre?
RC I?
?
Vpre?
…to identify operations where
the invariant can be violated.

View Slide

CISE Analysis
• Individually correct 
Individual operations never violate the
invariant
81

View Slide

CISE Analysis
invariant
• Convergence 
Concurrent effects commute
81

View Slide

CISE Analysis
invariant
• Convergence 
• Precondition stability 
Preconditions are stable under every pair
of concurrent operations
81

View Slide

CISE Analysis
invariant
• Convergence 
• Precondition stability 
Preconditions are stable under every pair
of concurrent operations
81
If satisﬁed, invariant is
guaranteed with concurrency.

View Slide

Database
AntidoteDB
82

View Slide

AntidoteDB
• Open-source Erlang database 
Developed in Erlang, on top of the Riak Core
distributed systems framework
83

View Slide

AntidoteDB
• Transactional Causal Consistency 
Only industrial-grade database providing both
causal consistency and all-or-nothing transactions
83

View Slide

AntidoteDB
• Transactional Causal Consistency 
Only industrial-grade database providing both
causal consistency and all-or-nothing transactions
• Alpha release available 
Currently under development, but an alpha
release of the product is available on GitHub
83

View Slide

A
B
N1
N2
TxnMgr
Materializer
Log
InterDC-Repl
Each data center…

View Slide

A
B
N1
N2
TxnMgr
Materializer
Log
InterDC-Repl
…contains multiple nodes…

View Slide

A
B
N1
N2
TxnMgr
Materializer
Log
InterDC-Repl
…each operating a transaction manager, materializers, log.

View Slide

A
B
N1
N2
TxnMgr
Materializer
Log
InterDC-Repl
Strong consistency inside of the data center…

View Slide

A
B
N1
N2
TxnMgr
Materializer
Log
InterDC-Repl
…with a causal consistency protocol running in the wide area.

View Slide

Data Model
89
Register

• Last-Writer Wins
• Multi-Value
Set

• Grow-Only
• Add-Wins
• Remove-Wins
Map
Counter

• Unlimited
• Restricted ≥ 0
Graph

• Directed
• Monotonic DAG
• Edit graph
Sequence

View Slide

Object API
90
User1 = {michel, antidote_crdt_mvreg, user_bucket},
{ok, Time2} = antidote:update_objects(ignore, [],
[{User1, assign,
{["Michel", “[email protected]”],
ClientIdentifier}}]),
{ok, Result, Time2} = antidote:read_objects(
ignore, [], [User1]).

View Slide

Object API
91
[{User1, assign,
Identify an object by object identiﬁer.

View Slide

Object API
92
[{User1, assign,
Use the update API to assign a value to this register.

View Slide

Object API
93
[{User1, assign,
Read the object, providing a minimum snapshot time.

View Slide

Object API
93
[{User1, assign,
Simple, operation-based API.
(think Redis, Riak CRDTs)

View Slide

Object API
93
[{User1, assign,
Simple, operation-based API.
(think Redis, Riak CRDTs)
Causal dependencies are
automatically captured by
execution order.

View Slide

Transaction API
94
{ok, TxId} = antidote:start_transaction(Timestamp, []),
{ok, _} = antidote:read_objects([Set], TxId),
ok = antidote:update_objects([{Set, add, "Java"}], TxId),
{ok, _} = antidote:commit_transaction(TxId).

View Slide

Transaction API
95
Start a transaction with the transaction API,
with a given snapshot time and return a transaction identiﬁer.

View Slide

Transaction API
96
Read objects using the interactive transaction API.

View Slide

Transaction API
97
Update objects using the interactive transaction API.

View Slide

Transaction API
98
Once ﬁnished updating, commit the transaction.

View Slide

Transaction API
98
Once ﬁnished updating, commit the transaction.
Transactions read
causally consistent snapshots
and updates are
applied atomically.

View Slide

Scalability
99
Kops / s
100
200
300
400
500
600
700
800
1 x 5
1 x 10
1 x 25
2 x 25
3 x 25
1 x 5
1 x 10
1 x 25
2 x 25
3 x 25
1 x 5
1 x 10
1 x 25
2 x 25
3 x 25
1 x 5
1 x 10
1 x 25
2 x 25
3 x 25
99(1) 90(10) 75(25) 50(50)
read(update) ratio
DCs ×
Servers
LWW registers
100k keys/partition
power law distribution

View Slide

Cure vs. SOA
100
Kops / s
0
100
200
300
400
500
600
700
800
900
1000
1100
Eiger
GR
Cure
EC
Eiger
GR
Cure
EC
Eiger
GR
Cure
EC
Eiger
GR
Cure
EC
99(1) 90(10) 75(25) 50(50)
read(update) ratio
3 DCs × 25 Servers
LWW registers

View Slide

Cure vs. EC
101
Kops / s
100
200
300
400
500
600
700
800
900
1000
1100
1200
Cure, 1KB
EC, 1KB
Cure, 10KB
EC, 10KB
Cure, 1KB
EC, 1KB
Cure, 10KB
EC, 10KB
Cure, 1KB
EC, 1KB
Cure, 10KB
EC, 10KB
Cure, 1KB
EC, 1KB
Cure, 10KB
EC, 10KB
99(1) 90(10) 75(25) 50(50)
read(update) ratio
3 DCs x 25 Servers
CRDT sets

View Slide

Future Features
• Intra-DC replication 
Antidote provides no replication within the
datacenter and assumes only geo-
replication at the moment
102

View Slide

Future Features
• Intra-DC replication 
Antidote provides no replication within the
datacenter and assumes only geo-
replication at the moment
• ACID transactions 
For Antidote to provide all of JRC, it needs
ACID transaction support: no research
needed, only implementation
102

View Slide

Moving Forward
• Research prototype 
Originally a research prototype to build a database
requiring reduced synchronization (SyncFree FP7)
with Basho, Rovio, and Trifork
103

View Slide

Moving Forward
• Research ahead 
LightKone (H2020) will investigate moving AntidoteDB
close to the edge to provide DDN services
103

View Slide

Moving Forward
• Research ahead 
LightKone (H2020) will investigate moving AntidoteDB
close to the edge to provide DDN services
• Industrialization 
Obtaining seed funding to start a company to
industrialize AntidoteDB
103

View Slide

Resources
• https://github.com/SyncFree/antidote 
AntidoteDB
104

View Slide

Resources
AntidoteDB
• http://syncfree.github.io/antidote/ 
Documentation for AntidoteDB
104

View Slide

Resources
AntidoteDB
• www.antidotedb.com 
Website
104

View Slide

Resources
AntidoteDB
• www.antidotedb.com 
Website
• docker pull antidotedb/antidote 
Try out Antidote!
104

View Slide

Thanks!
105
More questions?
Come visit us at the
Evolution bar!

View Slide

Just-Right Consistency - Closing the CAP Gap

Just-Right Consistency - Closing the CAP Gap

Christopher Meiklejohn

More Decks by Christopher Meiklejohn

Other Decks in Research

Featured

Transcript