1980s! Declarative The cloud is crying out for this! Heterogeneous, evolving, autoscaling Beautiful Growing body of work on distributed Logic Programming Asking and answering fundamental new questions! A Database/Logic Approach PODS POPL PODC
SIGMOD20, Eurosys20 Anna: Autoscaling multi-tier KVS ICDE18, VLDB19 f(x) f(x) f(x) f(x) f(x) f(x) f(x) f(x) f(x) f(x) Proving out the foundations in a traditional systems setting.
in NDlog illustrates surprising relationships between protocols. For example, we have shown that distance vector and dynamic source routing protocols differ only in a simple, traditional query optimization decision: the order in which a query’s predicates are evaluated. ” “ CACM 11/2009
Hadoop’s HDFS in Overlog Replacement of Hadoop Scheduler in Overlog 13 EuroSys ‘10 We began by creating an Overlog implementation of basic Paxos … Our first effort was clean and fairly simple: 22 Overlog rules in 53 lines of code, corresponding nearly line-for-line with the invariants from Lamport’s original paper [21]. ” “
more concise than Java (LOCs) Similar performance (within 10-20%) Separation of policy and mechanism Ease of evolution 1. High availability (failover + Paxos) 2. Scalability (hash partitioned FS master) 3. Monitoring as an aspect
set of machines to choose the same thing (Paxos) SMR: Get a (fixed) set of machines to choose the same sequence of things (Multipaxos) Commit Protocols: Get a (fixed) set of machines to agree on transaction commit (Two-Phase Commit)
to batter the consistency mechanisms down to a minimum… make it as hard as possible for application developers to get permission to use them —James Hamilton (IBM, MS, Amazon) in Birman, Chockler: “Toward a Cloud Computing Research Agenda”, LADIS 2009 ” “
Routing! We did need it for certain DHT overlays that were “racy” We implemented it for BOOM-FS (Multipaxos)! When MUST we coordinate, when can we find something more clever? What is coordination FOR?!
Lessons learned from early languages and prototypes • The need for theory Algebraic Approaches • Semantics: Dedalus & Bloom • Computability: The CALM Theorem Today: Lessons and Foundations
as the rightmost attribute of all predicates. Restrictions on rules to track state over time “Time is what keeps everything from happening at once.” Ray Cummings, The Girl in the Golden Atom, 1922 Datalog 2010
minimal model Which coincides with the least fixed point semantics! In Dedalus, each model is a trace 1. Infinite across time (hence minimality is tricky) 2. Non-deterministic across time, via the choice construct (hence uniqueness is tricky)
model: those facts that are eventually always true Programs with a unique ultimate model are confluent. Berkeley/Hasselt Tech Report, 2011 Datalog in Academia and Industry, LNCS 2012 TPLP 2015
coordination-free distributed implementation if and only if it is monotonic. Ameloot, Neven, Van den Bussche, PODS 2011 CALM: CONSISTENCY AS LOGICAL MONOTONICITY
it is a deterministic function from sets to sets, regardless of non-deterministic message ordering and duplication. Definition: A distributed program P is monotonic if for any input sets S, T if S ⊆ T, then P(S) ⊆ P(T).
it is a deterministic function from sets to sets, regardless of non-deterministic message ordering and duplication. Definition: A distributed program P is monotonic if for any input sets S, T if S ⊆ T, then P(S) ⊆ P(T).
consistent if it is a deterministic function from sets to sets, regardless of non-deterministic message ordering and duplication. Definition: A distributed program P is monotonic if for any input sets S, T if S ⊆ T, then P(S) ⊆ P(T). Definition: A distributed program P(T) uses coordination if it requires messages to be sent under all possible input partitionings (shardings) of T.
waits-for cycle, you can (autonomously) declare deadlock. More information will not change the result. Garbage Collection: Suspecting garbage (the non-existence of a path from root) is not enough; more information may change the result. Hence you are required to check all nodes for information (under any assignment of objects to nodes!) Two Canonical Examples Deadlock! Garbage?
program P has a consistent, coordination-free distributed implementation if and only if it is monotonic. Hellerstein JM. The declarative imperative: Experiences and conjectures in distributed logic. ACM SIGMOD Record, Sep 2010. Ameloot TJ, Neven F, Van den Bussche J. Relational transducers for declarative networking. JACM, Apr 2013. Ameloot TJ, Ketsman B, Neven F, Zinn D. Weaker forms of monotonicity for declarative networking: a more fine-grained answer to the CALM-conjecture. ACM TODS, Feb 2016. Hellerstein JM, Alvaro P. Keeping CALM: When Distributed Consistency is Easy. CACM, September 2020.
knows the policy for sharding Some “not exists” queries become coordination-free! A hierarchy of definitions of “weaker monotonicity” under “stronger policy knowledge” Deeper connections to Datalog variants Descriptive Complexity characterizations Is there more? PODS ‘14
Lessons learned from early languages and prototypes • The need for theory Algebraic Approaches • Join semilattices and CRDTs • Lattice composition Today: Lesson and Foundations • Semantics: Dedalus & Bloom • Computability: The CALM Theorem
in C++ Simple lattices a la BloomL for various consistency levels Choose any HAT , we tag your data with the right lattice Anna runs without any coordination, locks or atomics Every core uses private memory Versions are gossiped/merged across cores Extremely simple, crazy fast and scalable VLDB 2013 ICDE 2019
Lessons learned from early languages and prototypes • The need for theory Algebraic Approaches • Join semilattices and CRDTs • Lattice composition • CALM Advances • Towards a Programmable Cloud Today: Lessons and Foundations • Semantics: Dedalus & Bloom • Computability: The CALM Theorem
It does not help you achieve it. Constructive CALM Input: a monotonic input/output “problem” Desired output: a coordination-free implementation. Or some help designing it! Relaxed CALM Input: a non-monotonic problem specification Desired output: a “relaxed” specification that is monotonic and “similar”.
Datalog enhanced with: Negation on stored facts (only!) A successor relation (some source of order) Immerman-Vardi Theorem: Semipositive Datalog = PTIME A practical, weaker form of monotonicity corresponding to semi-positive Datalog?! What is our source of order? Is this simply coordination, or is there something weaker?
Availability C Consistency T Targets for Optimization A declaration of four independent facets Functionality modulo distribution Tolerate f independent failures Application-specific guarantees per API Multi-Objective Optimization
Storage ML Frameworks Actors (e.g. Orleans) Functional (e.g. Spark) Logic (e.g. Bloom) Futures (e.g. Ray) P Program Semantics A Availability New DSLs HYDROLOGIC HYDRAULIC Verified Lifting HYDROLYSIS Compiler HYDROFLOW Deployment HYDROFLOW Program C Consistency Targets for Optimization T Sequential Code HYDRO Polyglot programming Declarative PACT IR Data/Lattice/Eventflow IR Adaptive self-deploying exe’s
does not access the following information: All (“membership”: a table of all nodes in the system) Id (“self-awareness”: the name of the local node) For queries Q in language L, the following are all equivalent! Q can be computed by a coordination-free L-transducer Q is monotonic Q can be computed by an oblivious L-transducer Practical implications of (non-)obliviousness for highly dynamic systems! E.g. serverless computing, edge/fog computing, p2p
based on logic and lattices Monotone composition of “lego blocks” lattices into bigger programs. Allows non-monotonic expressions but they require “extra” syntax. Syntactic CALM analysis Alvaro P, Conway N, Hellerstein JM, Marczak WR. Consistency Analysis in Bloom: a CALM and Collected Approach. CIDR 2011. Conway N, Marczak WR, Alvaro P, Hellerstein JM, Maier D. Logic and lattices for distributed programming. ACM SoCC, 2012.
like S3 • CALM consistency levels via simple lattices • Autoscaling & multitier serverless storage • Won best-of-conference at ICDE, VLDB1, 2 1 Wu, Chenggang, et al. "Anna: A kvs for any scale." IEEE Transactions on Knowledge and Data Engineering (2019). 2 Wu, Chenggang, Vikram Sreekanti, and Joseph M. Hellerstein. "Autoscaling tiered cloud storage in Anna." PVLDB 12.6 (2019): 624-638.
Crazy fast under contention Up to 700x faster than Masstree within a multicore machine Up to 10x faster than Cassandra in a geo-distributed deployment Coordination-free consistency. No atomics, no locks, no waiting ever! 700x!
a fault tolerance shim layer between any FaaS and any object store • Currently evaluated between AWS Lambda and AWS S3! Multisite Transactional Causal Consistency (MTCC)2 Causal: Preserve Lamport’s happened before relation Multisite transactional: Nested functions running across multiple machines. 81 1Sreekanti, Vikram, et al. A Fault-Tolerance Shim for Serverless Computing. To appear, Eurosys (2020). 2Wu, Chenggang, et al. Transactional Causal Consistency for Serverless Computing. To appear, ACM SIGMOD (2020).