Eventual Consistency and Deterministic Dataflow Programming Christopher Meiklejohn Basho Technologies, Inc. / SyncFree Consortium LADIS Workshop ’14 Cambridge, United Kingdom October 23, 2014 Meiklejohn (Basho) DerflowL LADIS ’14 1 / 17 

Overview 1 Motivation 2 Our Approach 3 Conclusions and Future Work 4 References Meiklejohn (Basho) DerflowL LADIS ’14 2 / 17 

Motivation Riak [7] is a Dynamo-inspired [4] key-value store Querying by key main mechanism for data storage and retrieval Three mechanisms presently for more expressive data access: MapReduce-like system [3, 6], secondary indexing, integration with Apache Solr Each additional mechanism contains drawbacks Mechanisms are not fault-tolerant Structure is rigid: need to know schema when storing data Not possible to perform composition of queries Meiklejohn (Basho) DerflowL LADIS ’14 3 / 17 

The Goal Building large scale distributed applications with strong properties Users want to be able to compute with their data in an efficient and composable manner while guaranteeing strong properties Provide a framework for building eventually consistent materialized views that have strong convergence properties Data types which have strong convergence properties Deterministic language to compose data types and preserve these strong convergence properties Meiklejohn (Basho) DerflowL LADIS ’14 4 / 17 

Our Approach Leverage conflict-free replicated data types (CRDTs) Builds on top of Derflow [1], a deterministic dataflow language Extends Derflow with CRDTs, similar to LVars [8] and Bloom L [2] Applies techniques from C-CRDTs [10] for incremental computation Uses version vectors with exceptions for tracking program state [9] Uses Dynamo-style quorums for aggregating computation results Meiklejohn (Basho) DerflowL LADIS ’14 5 / 17 

CRDTs Conflict-free Replicated Data Types [11] Strong Eventual Consistency (SEC) Data structures which contain resolution logic Based on mathematical properties Figure : Set where element can be added and removed once. 1 2 1 Can be generalized to arbitrary adds and removes. 2 Iwan Briquemont, Optimising Client-side Geo-replication with Partially Replicated Data Structures, Master’s thesis, Université catholique de Louvain, Aug. 2014. Meiklejohn (Basho) DerflowL LADIS ’14 6 / 17 

Derflow / Derflow L Distributed deterministic programming language Similar to Mozart/Oz, Ozma [5] Relies on a distributed variable store Extend the model to state-based CRDTs and C-CRDTs Inputs and outputs of programs are CRDTs Meiklejohn (Basho) DerflowL LADIS ’14 7 / 17 

Computational CRDTs Defines CRDTs which observe partial state Allows for chaining, but requires explicit mapping function Defines a merge function to combine partial results Maps well to Dynamo data storage model Replicas within a replica set should have equal state, Disjoint between replica sets Meiklejohn (Basho) DerflowL LADIS ’14 8 / 17 

Version Vectors with Exceptions Each program tracks the versions of inputs used in computations Data is tracked in a version vector with exceptions Treat computations as values in Dynamo-style system Can be used for anti-entropy, read repair, etc. Meiklejohn (Basho) DerflowL LADIS ’14 9 / 17 

Dynamo Quorums Identify covering set of nodes for key space Can include multiple copies from the same replica set Merge (LUB) across replica sets Merge (Delta) across copies of the same replica Figure : Ring with 32 partitions and 3 nodes Meiklejohn (Basho) DerflowL LADIS ’14 10 / 17 

Dynamo Quorums Identify covering set of nodes for key space Can include multiple copies from the same replica set Merge (LUB) across replica sets Merge (Delta) across copies of the same replica Figure : Ring with 32 partitions and 3 nodes with covering set identified in red Meiklejohn (Basho) DerflowL LADIS ’14 11 / 17 

Dynamo Quorums Identify covering set of nodes for key space Can include multiple copies from the same replica set Merge (LUB) across replica sets Merge (Delta) across copies of the same replica Figure : Ring with 32 partitions and 3 nodes with fault-tolerant covering set identified in red and blue Meiklejohn (Basho) DerflowL LADIS ’14 12 / 17 

Conclusions and Future Work Current status: DerflowL implemented on Riak Core, extended with CRDTs Implemented a test harness for DerflowL using open source work from Basho to test distribution and operation in partitions Future work: Composing CRDTs currently is very explicit and cumbersome Will be solved as part of the SyncFree research project Program analysis for detection of incorrect programs Meiklejohn (Basho) DerflowL LADIS ’14 13 / 17 

References I M. Bravo, Z. Li, P. Van Roy, and C. Meiklejohn. Derflow: Distributed deterministic dataflow programming for erlang. In Proceedings of the Thirteenth ACM SIGPLAN Workshop on Erlang, Erlang ’14, pages 51–60, New York, NY, USA, 2014. ACM. N. Conway, W. Marczak, P. Alvaro, J. M. Hellerstein, and D. Maier. Logic and lattices for distributed programming. Technical Report UCB/EECS-2012-167, EECS Department, University of California, Berkeley, Jun 2012. J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. Commun. ACM, 51(1):107–113, Jan. 2008. Meiklejohn (Basho) DerflowL LADIS ’14 14 / 17 

References II G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels. Dynamo: Amazon’s highly available key-value store. In Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP ’07, pages 205–220, New York, NY, USA, 2007. ACM. S. Doeraene and P. Van Roy. A new concurrency model for scala based on a declarative dataflow core. In Proceedings of the 4th Workshop on Scala, SCALA ’13, pages 4:1–4:10, New York, NY, USA, 2013. ACM. B. Fink. Distributed computation on dynamo-style distributed storage: Riak pipe. In Proceedings of the Eleventh ACM SIGPLAN Workshop on Erlang Workshop, Erlang ’12, pages 43–50, New York, NY, USA, 2012. ACM. Meiklejohn (Basho) DerflowL LADIS ’14 15 / 17 

References III R. Klophaus. Riak core: Building distributed applications without shared state. In ACM SIGPLAN Commercial Users of Functional Programming, CUFP ’10, pages 14:1–14:1, New York, NY, USA, 2010. ACM. L. Kuper and R. R. Newton. Lvars: Lattice-based data structures for deterministic parallelism. In Proceedings of the 2Nd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC ’13, pages 71–84, New York, NY, USA, 2013. ACM. D. Malkhi and D. Terry. Concise version vectors in winfs. In Proceedings of the 19th International Conference on Distributed Computing, DISC’05, pages 339–353, Berlin, Heidelberg, 2005. Springer-Verlag. Meiklejohn (Basho) DerflowL LADIS ’14 16 / 17 

References IV D. Navalho, S. Duarte, N. Preguiça, and M. Shapiro. Incremental stream processing using computational conflict-free replicated data types. In Proceedings of the 3rd International Workshop on Cloud Data and Platforms, CloudDP ’13, pages 31–36, New York, NY, USA, 2013. ACM. M. Shapiro, N. Preguiça, C. Baquero, and M. Zawirski. A comprehensive study of Convergent and Commutative Replicated Data Types. Rapport de recherche RR-7506, INRIA, Jan. 2011. Meiklejohn (Basho) DerflowL LADIS ’14 17 / 17