Outline: Closing the CAP Gap • Just-Right Consistency Available as possible, and consistent when necessary • AntidoteDB The first database that provides transactions with strong semantics, targeted at the JRC approach 2
Outline: Closing the CAP Gap • Just-Right Consistency Available as possible, and consistent when necessary • AntidoteDB The first database that provides transactions with strong semantics, targeted at the JRC approach • Moving forward Antidote’s path forward from research to company and product 2
A Centralized database. [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A Clients read and write against the primary copy. [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Geo-replicated for both fault-tolerance and high-availability. [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Clients read and write locally for low-latency. [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C What happens if C can’t communicate with other replicas? [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 1: Consistent-Under-Partition (CP) • Synchronize each operation Maintains “single system image” [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 1: Consistent-Under-Partition (CP) • Synchronize each operation Maintains “single system image” • Spanner/F1, serializability model Coordination is expensive; Spanner typically has to wait 100ms to commit an update transaction [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 1: Consistent-Under-Partition (CP) • Synchronize each operation Maintains “single system image” • Spanner/F1, serializability model Coordination is expensive; Spanner typically has to wait 100ms to commit an update transaction Over-conservative,
but easy to program! [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 2: Available-Under-Partition (AP) • Riak, Cassandra, Dynamo Operations issued against local copy, and across the cluster in parallel [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 2: Available-Under-Partition (AP) • Riak, Cassandra, Dynamo Operations issued against local copy, and across the cluster in parallel • Local operation only, asynchronous propagation Stale reads and write conflicts will occur without synchronization [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C Choice 2: Available-Under-Partition (AP) • Riak, Cassandra, Dynamo Operations issued against local copy, and across the cluster in parallel • Local operation only, asynchronous propagation Stale reads and write conflicts will occur without synchronization Available,
but difficult to program! [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization Low cost CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization Low cost High availability CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization Low cost High availability Anomalies CP AP [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization Low cost High availability Anomalies CP AP False dichotomy! [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452]
A B C CAP Theorem High cost Low availability Synchronization Low cost High availability Anomalies CP AP False dichotomy! [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452] • No “one-size-fits-all” consistency model Choosing either model will either be over-conservative or risk anomalies
A B C CAP Theorem High cost Low availability Synchronization Low cost High availability Anomalies CP AP False dichotomy! [Photo: http://vignette3.wikia.nocookie.net/the-titans-rp-and-information/images/f/f5/Blank-World-map2.gif/revision/latest/scale-to-width-down/1280?cb=20141016203452] • No “one-size-fits-all” consistency model Choosing either model will either be over-conservative or risk anomalies • Application-level invariants Instead, tailor consistency choices based on application- level invariants for each operation
Just Right Consistency • Preserve sequential patterns Applications written sequentially that are correct should maintain correctness under concurrency 13
Just Right Consistency • Preserve sequential patterns Applications written sequentially that are correct should maintain correctness under concurrency • AP-compatible invariants Strongest AP model; invariants that only require “one way” communications 13
Just Right Consistency • Preserve sequential patterns Applications written sequentially that are correct should maintain correctness under concurrency • AP-compatible invariants Strongest AP model; invariants that only require “one way” communications • CAP-sensitive invariants Transactions that require coordination; “two way” communication invariants 13
Just Right Consistency • Preserve sequential patterns Applications written sequentially that are correct should maintain correctness under concurrency • AP-compatible invariants Strongest AP model; invariants that only require “one way” communications • CAP-sensitive invariants Transactions that require coordination; “two way” communication invariants • Tools for analysis and verification Identify and verify application has sufficient synchronization to ensure application invariants 13
Fælles Medicinkort • FMK [production] / FMKe [synthetic workload] Danish National Joint Medicine Card; operating 24x7 since 2013 for 6 million Danish citizens 15
Fælles Medicinkort • FMK [production] / FMKe [synthetic workload] Danish National Joint Medicine Card; operating 24x7 since 2013 for 6 million Danish citizens • Lifecycle management for prescriptions Involves patient, pharmacy, and doctor management around active prescriptions in Denmark 15
Fælles Medicinkort • FMK [production] / FMKe [synthetic workload] Danish National Joint Medicine Card; operating 24x7 since 2013 for 6 million Danish citizens • Lifecycle management for prescriptions Involves patient, pharmacy, and doctor management around active prescriptions in Denmark • Assumed correct in isolation “Correct-Individually”, C in ACID, each operation ensures application-level invariants 15
Fælles Medicinkort • FMK [production] / FMKe [synthetic workload] Danish National Joint Medicine Card; operating 24x7 since 2013 for 6 million Danish citizens • Lifecycle management for prescriptions Involves patient, pharmacy, and doctor management around active prescriptions in Denmark • Assumed correct in isolation “Correct-Individually”, C in ACID, each operation ensures application-level invariants 15 • create-prescription Create prescription for patient, doctor, pharmacy
• update-prescription-medication Add or increase medication to prescription
• process-prescription Deliver a medication by a pharmacy
• get-*-prescriptions Query functions to return information about prescriptions
FMKe Invariants • Relative order [secondary indexes] Create a prescription and reference it by a patient • Joint update [atomicity] Create prescription, then update doctor, patient, and pharmacy 16
FMKe Invariants • Relative order [secondary indexes] Create a prescription and reference it by a patient • Joint update [atomicity] Create prescription, then update doctor, patient, and pharmacy • Precondition check [if, then] Medication should not be over delivered 16
AP-compatible • No synchronization Updates occur locally without blocking, no synchronization in the critical path • Asynchronous operation Updates are fast, available, and exploit concurrency 18
AP-compatible • No synchronization Updates occur locally without blocking, no synchronization in the critical path • Asynchronous operation Updates are fast, available, and exploit concurrency • Compatible invariants Relative order and joint update invariants can be preserved 18
Conflict-Free Replicated Data Types • Replicated abstract data types Extension of sequential data type that encapsulates deterministic merge function 28
Conflict-Free Replicated Data Types • Replicated abstract data types Extension of sequential data type that encapsulates deterministic merge function • Many existing designs Sets, counters, registers, flags, maps 28
Causal Consistency • Respect causality Ensure updates are delivered in the causal order • Strongest available AP model Always able to return some compatible version for an object 44
Causal Consistency • Respect causality Ensure updates are delivered in the causal order • Strongest available AP model Always able to return some compatible version for an object • Secondary indexing Causal consistency is sufficient for providing secondary indexing in an AP database 44
RA(2) RB(2) -1 -1 1 pp(1) RC(2) -1 0 pp(2) Incorrect outcome violating non-negative invariant. Precondition is NOT stable under concurrent fulfillment. • Forbid concurrency Prevent operations from proceeding without synchronization to enforce invariant • Allow concurrency and remove invariant Allow operation to proceed, knowing that the invariant may be violated under concurrent operations
CISE Analysis • Individually correct Individual operations never violate the invariant • Convergence Concurrent effects commute • Precondition stability Preconditions are stable under every pair of concurrent operations 81
CISE Analysis • Individually correct Individual operations never violate the invariant • Convergence Concurrent effects commute • Precondition stability Preconditions are stable under every pair of concurrent operations 81 If satisfied, invariant is guaranteed with concurrency.
AntidoteDB • Open-source Erlang database Developed in Erlang, on top of the Riak Core distributed systems framework • Transactional Causal Consistency Only industrial-grade database providing both causal consistency and all-or-nothing transactions 83
AntidoteDB • Open-source Erlang database Developed in Erlang, on top of the Riak Core distributed systems framework • Transactional Causal Consistency Only industrial-grade database providing both causal consistency and all-or-nothing transactions • Alpha release available Currently under development, but an alpha release of the product is available on GitHub 83
Object API 92 User1 = {michel, antidote_crdt_mvreg, user_bucket}, {ok, Time2} = antidote:update_objects(ignore, [], [{User1, assign, {["Michel", “[email protected]”], ClientIdentifier}}]), {ok, Result, Time2} = antidote:read_objects( ignore, [], [User1]). Use the update API to assign a value to this register.
Transaction API 95 Start a transaction with the transaction API, with a given snapshot time and return a transaction identifier. {ok, TxId} = antidote:start_transaction(Timestamp, []), {ok, _} = antidote:read_objects([Set], TxId), ok = antidote:update_objects([{Set, add, "Java"}], TxId), {ok, _} = antidote:commit_transaction(TxId).
Scalability 99 Kops / s 100 200 300 400 500 600 700 800 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 1 x 5 1 x 10 1 x 25 2 x 25 3 x 25 99(1) 90(10) 75(25) 50(50) read(update) ratio DCs × Servers LWW registers 100k keys/partition power law distribution
Future Features • Intra-DC replication Antidote provides no replication within the datacenter and assumes only geo- replication at the moment • ACID transactions For Antidote to provide all of JRC, it needs ACID transaction support: no research needed, only implementation 102
Moving Forward • Research prototype Originally a research prototype to build a database requiring reduced synchronization (SyncFree FP7) with Basho, Rovio, and Trifork 103
Moving Forward • Research prototype Originally a research prototype to build a database requiring reduced synchronization (SyncFree FP7) with Basho, Rovio, and Trifork • Research ahead LightKone (H2020) will investigate moving AntidoteDB close to the edge to provide DDN services 103
Moving Forward • Research prototype Originally a research prototype to build a database requiring reduced synchronization (SyncFree FP7) with Basho, Rovio, and Trifork • Research ahead LightKone (H2020) will investigate moving AntidoteDB close to the edge to provide DDN services • Industrialization Obtaining seed funding to start a company to industrialize AntidoteDB 103
cmeiklejohn
smatsumori
rpc
.png){kind=avatar}
elith
ahspragu
kiyohiro8
masakat0
kosuken
kentaro
yumulab
nttcom
hpprc
yowata
tenderlove
dougneiner
tammielis
jeffersonlam
tammyeverts
searls
pauljervisheath
quramy
aarron
revolveconf
cherdarchuk
keathley