partial failure • Enforcing the “single system” illusion Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” 10
partial failure • Enforcing the “single system” illusion Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models Contract between the system and the programmer 10
partial failure • Enforcing the “single system” illusion Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models Contract between the system and the programmer • Correctness criteria For both distributed and concurrent programs 10
partial failure • Enforcing the “single system” illusion Traditional approach for gaining performance, scalability, fault- tolerance while still “easy to program” • Consistency models Contract between the system and the programmer • Correctness criteria For both distributed and concurrent programs • Synchronization as “knob” Models live along a spectrum requiring more or less synchronization 10
same state • Primary requirement “Eventual” replica-to-replica communication • Order insensitive! Operations are commutative • Duplicate insensitive! Operations are idempotent 33
streams of inputs into streams of outputs • Convergent data structures Data abstraction (inputs/outputs) is the CRDT • Confluence Provides composition that preserves the SEC property 60
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
elements to initial set and update. update(S1, {add, [1,2,3]}), %% Create second set. S2 = declare(set), %% Apply map operation between S1 and S2. map(S1, fun(X) -> X * 2 end, S2).
CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream 77
CRDT produces a monotonic stream of states • Monotonic processes Read from one or more input replica streams and produce a single output replica stream • Inflationary reads Read operation ensures that we only read inflationary updates to replicas 77
a minimum number of impressions • Clients will go offline Clients have limited connectivity and the system still needs to make progress while clients are offline 104
a minimum number of impressions • Clients will go offline Clients have limited connectivity and the system still needs to make progress while clients are offline • No lost updates All displayed advertisements should be accounted for, with no lost updates 104
all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed 127
all modeled through monotonic state growth • Arbitrary distribution Use of convergent data structures allows computational graph to be arbitrarily distributed • Divergence Divergence is a factor of synchronization period, concurrency, and throughput rate 127
large clusters • Delta-state synchronization Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants Lower-bound invariants, configurable at runtime 130
large clusters • Delta-state synchronization Efficient incremental state dissemination and anti-entropy mechanism [Almeida et al. 2016] • Epsilon-invariants Lower-bound invariants, configurable at runtime • Scalable Demonstrated high scalability in production Cloud environments 130
• Membership Configurable membership protocol which can operate in a client-server or peer-to-peer mode [Leitao et al. 2007] • Broadcast (via Gossip, Tree, etc.) Efficient dissemination of both program state and application state via gossip, broadcast tree, or hybrid mode [Leitao et al. 2007] 132
clients • Operate locally Objects are mutated locally; delta’s buffered locally and periodically gossiped • Only fixed number of clients Clients resort to full state synchronization when they’ve been partitioned too long 138
Amazon cloud computing environment • Modular approach Many of the components built and can be operated outside of Lasp to improve scalability of Erlang 145
Amazon cloud computing environment • Modular approach Many of the components built and can be operated outside of Lasp to improve scalability of Erlang • Automated and repeatable Fully automated deployment, scenario execution, log aggregation and archival of experimental results 145
Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming Convergent Programs: Lattice Processing [Meiklejohn et al. 2015] 149
Data Types [Shapiro et al. 2011] 2. Retain the properties of functional programming Convergent Programs: Lattice Processing [Meiklejohn et al. 2015] 3. Distributed, and fault-tolerant runtime Distributed Runtime: Anabranch [Meiklejohn et al. work-in-progress] 149
“Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required Global invariants, atomic visibility, etc. 150
“Internet of Things” applications make synchronization for replicated state impractical • Apply synchronization only where required Global invariants, atomic visibility, etc. • Holistic approach Taking a holistic approach to the design of distributed systems is important for building higher-availability applications 150
http://github.com/lasp-lang/lasp • Partisan TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols https://github.com/lasp-lang/partisan • Plumtree Epidemic broadcast trees for use with Partisan https://github.com/lasp-lang/plumtree 151
http://github.com/lasp-lang/lasp • Partisan TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols https://github.com/lasp-lang/partisan • Plumtree Epidemic broadcast trees for use with Partisan https://github.com/lasp-lang/plumtree • Sprinter Service discovery and deployment for Mesos and Kubernetes https://github.com/lasp-lang/sprinter 151
http://github.com/lasp-lang/lasp • Partisan TCP-based pluggable membership service offering client/server, static, and HyParView- based protocols https://github.com/lasp-lang/partisan • Plumtree Epidemic broadcast trees for use with Partisan https://github.com/lasp-lang/plumtree • Sprinter Service discovery and deployment for Mesos and Kubernetes https://github.com/lasp-lang/sprinter • Types CRDT implementations for Erlang https://github.com/lasp-lang/types 151
cmeiklejohn
harukakiyohara_
iwiwi
nttcom
daimoriwaki
gpeyre
kurita
monochromegane
usaito
nonsquared
jcasabona
zenorocha
philhawksworth
ddemaree
geoffreycrofte
zakiyama
jmmastey
maggiecrowley
jakevdp
mojombo
tammyeverts
