$30 off During Our Annual Pro Sale. View Details »

Declarative, Convergent, Edge Computation

Declarative, Convergent, Edge Computation

Craft Conference, 2017

Christopher Meiklejohn

April 28, 2017
Tweet

More Decks by Christopher Meiklejohn

Other Decks in Research

Transcript

  1. Declarative,
    Convergent,
    Edge Computation
    Christopher S. Meiklejohn
    Université catholique de Louvain, Belgium
    Instituto Superior Técnico, Portugal
    LIGHT ONE

    View Slide

  2. RA
    RB

    View Slide

  3. RA
    RB
    1
    set(1)

    View Slide

  4. RA
    RB
    1
    set(1)
    3
    2
    set(2)
    set(3)
    Concurrent

    operations.

    View Slide

  5. RA
    RB
    1
    set(1)
    3
    2
    set(2)
    set(3)
    ?
    ?
    Nondeterministic
    outcome.

    View Slide

  6. Synchronization
    • To enforce an order

    Makes programming easier
    6

    View Slide

  7. Synchronization
    • To enforce an order

    Makes programming easier
    • Eliminate accidental nondeterminism

    Prevent race conditions
    6

    View Slide

  8. Synchronization
    • To enforce an order

    Makes programming easier
    • Eliminate accidental nondeterminism

    Prevent race conditions
    • Techniques

    Locks, mutexes, semaphores, monitors,
    Paxos, Raft, etc.
    6

    View Slide

  9. Difficult Cases
    • “Internet of Things”, 

    Low power, limited memory and
    connectivity
    7

    View Slide

  10. Difficult Cases
    • “Internet of Things”, 

    Low power, limited memory and
    connectivity
    • Mobile Gaming

    Offline operation with replicated, shared
    state
    7

    View Slide

  11. 8

    View Slide

  12. The Fundamentals of
    Distributed Computation
    9

    View Slide

  13. Distributed Computation
    • Concurrent to distributed programming

    Consistency and now partial failure
    10

    View Slide

  14. Distributed Computation
    • Concurrent to distributed programming

    Consistency and now partial failure
    • Enforcing the “single system” illusion

    Traditional approach for gaining performance, scalability, fault-
    tolerance while still “easy to program”
    10

    View Slide

  15. Distributed Computation
    • Concurrent to distributed programming

    Consistency and now partial failure
    • Enforcing the “single system” illusion

    Traditional approach for gaining performance, scalability, fault-
    tolerance while still “easy to program”
    • Consistency models

    Contract between the system and the programmer
    10

    View Slide

  16. Distributed Computation
    • Concurrent to distributed programming

    Consistency and now partial failure
    • Enforcing the “single system” illusion

    Traditional approach for gaining performance, scalability, fault-
    tolerance while still “easy to program”
    • Consistency models

    Contract between the system and the programmer
    • Correctness criteria

    For both distributed and concurrent programs
    10

    View Slide

  17. Distributed Computation
    • Concurrent to distributed programming

    Consistency and now partial failure
    • Enforcing the “single system” illusion

    Traditional approach for gaining performance, scalability, fault-
    tolerance while still “easy to program”
    • Consistency models

    Contract between the system and the programmer
    • Correctness criteria

    For both distributed and concurrent programs
    • Synchronization as “knob”

    Models live along a spectrum requiring more or less
    synchronization
    10

    View Slide

  18. Databases
    Consistency Models
    11

    View Slide

  19. Databases
    Strong Consistency
    12

    View Slide

  20. R1
    C1 C2
    13

    View Slide

  21. R1
    C1 C2
    14

    View Slide

  22. R1
    C1 C2
    Read
    15

    View Slide

  23. R1
    C1 C2
    Read
    16

    View Slide

  24. R1
    C1 C2
    17

    View Slide

  25. R1
    C1 C2
    CAS
    18

    View Slide

  26. R1
    C1 C2
    19

    View Slide

  27. R1
    C1 C2
    CAS
    20
    Operation refused
    because value changed
    between read/write.

    View Slide

  28. I won’t diagram
    the Paxos protocol
    21

    View Slide

  29. R2
    C1 C2
    Value 2 Value 1
    Value 2
    R1 R3
    Paxos
    22
    Coordination is
    extremely expensive.

    View Slide

  30. Databases
    Eventual Consistency
    23

    View Slide

  31. R1 R2 R3
    C1 C2
    24

    View Slide

  32. R1 R2 R3
    C1 C2
    25

    View Slide

  33. R1 R2 R3
    C1 C2
    Read
    26

    View Slide

  34. R1 R2 R3
    C1 C2
    Read
    27

    View Slide

  35. R1 R2 R3
    C1 C2
    Write
    28

    View Slide

  36. R1 R2 R3
    C1 C2
    Write
    29

    View Slide

  37. R1 R2 R3
    C1 C2
    Write
    C1
    30
    Do I store

    both values

    or do I
    choose a value

    arbitrarily?

    View Slide

  38. R1 R2 R3
    C1 C2
    Read
    31
    If I store both,
    the programmer must
    expect multiple values…

    View Slide

  39. R1 R2 R3
    C1 C2
    Write
    32
    …and be able to
    semantically resolve

    them in application code.

    View Slide

  40. Strong Eventual Consistency
    (SEC)
    • Convergent

    Same updates realize the same state
    33

    View Slide

  41. Strong Eventual Consistency
    (SEC)
    • Convergent

    Same updates realize the same state
    • Primary requirement

    “Eventual” replica-to-replica communication
    33

    View Slide

  42. Strong Eventual Consistency
    (SEC)
    • Convergent

    Same updates realize the same state
    • Primary requirement

    “Eventual” replica-to-replica communication
    • Order insensitive!

    Operations are commutative
    33

    View Slide

  43. Strong Eventual Consistency
    (SEC)
    • Convergent

    Same updates realize the same state
    • Primary requirement

    “Eventual” replica-to-replica communication
    • Order insensitive!

    Operations are commutative
    • Duplicate insensitive!

    Operations are idempotent
    33

    View Slide

  44. RA
    RB

    View Slide

  45. RA
    RB
    1
    set(1)

    View Slide

  46. RA
    RB
    1
    set(1)
    3
    2
    set(2)
    set(3)

    View Slide

  47. RA
    RB
    1
    3
    2
    3
    3
    set(1) set(2)
    set(3)
    max(2,3)
    max(2,3)
    Automatic resolution
    of conflicting updates.

    View Slide

  48. How can we succeed with
    Strong Eventual
    Consistency?
    38

    View Slide

  49. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)
    39

    View Slide

  50. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)
    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    39

    View Slide

  51. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)
    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    39

    View Slide

  52. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)

    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    40

    View Slide

  53. Convergent Objects

    Conflict-Free 

    Replicated Data Types
    41
    SSS 2011

    View Slide

  54. Conflict-Free 

    Replicated Data Types
    • Many types exist with different properties

    Sets, counters, registers, flags, maps
    42

    View Slide

  55. Conflict-Free 

    Replicated Data Types
    • Many types exist with different properties

    Sets, counters, registers, flags, maps
    • Strong Eventual Consistency

    Instances satisfy SEC property per-object
    42

    View Slide

  56. Conflict-Free 

    Replicated Data Types
    • Many types exist with different properties

    Sets, counters, registers, flags, maps
    • Strong Eventual Consistency

    Instances satisfy SEC property per-object
    • Bounded join-semilattices

    Formalized using bounded join-semilattices
    where the merge operation is the join
    42

    View Slide

  57. Convergent Objects
    Observed-Remove Set
    43

    View Slide

  58. RA
    RB
    RC

    View Slide

  59. RA
    RB
    RC
    {1}
    (1, {a}, {})
    add(1)

    View Slide

  60. RA
    RB
    RC
    {1}
    (1, {a}, {})
    add(1)
    {1}
    (1, {b}, {})
    add(1)

    View Slide

  61. RA
    RB
    RC
    {1}
    (1, {a}, {})
    add(1)
    {1}
    (1, {b}, {})
    add(1)
    {}
    (1, {b}, {b})
    remove(1)

    View Slide

  62. RA
    RB
    RC
    {1}
    (1, {a}, {})
    add(1)
    {1}
    (1, {b}, {})
    add(1)
    {}
    (1, {b}, {b})
    remove(1)
    {1}
    {1}
    {1}
    (1, {a, b}, {b})
    (1, {a, b}, {b})
    (1, {a, b}, {b})
    Convergence

    reached.

    View Slide

  63. Convergent Objects
    Composition
    49

    View Slide

  64. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Replicated
    set of naturals
    across two nodes.

    View Slide

  65. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    Map \x.2x
    over a set of naturals.

    View Slide

  66. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    One node…

    View Slide

  67. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    …another node.

    View Slide

  68. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}

    View Slide

  69. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    Nondeterministic

    outcome.

    View Slide

  70. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    Correct output
    that’s seen all
    updates.

    View Slide

  71. RA {1}
    (1, {a}, {})
    {1}
    (1, {a, b}, {a})
    RB {1}
    (1, {b}, {})
    {1}
    (1, {a, b}, {a})
    {}
    (1, {a}, {a})
    add(1) remove(1)
    add(1)
    Map
    Process
    \x.2x
    F(RA) {2} {2}
    {}
    F(RB) {2} ?
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    {2}
    “Earlier” value
    that’s been
    delayed.

    View Slide

  72. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)

    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    58

    View Slide

  73. Convergent Programs
    Lattice Processing
    59
    PPDP 2015

    View Slide

  74. Lattice Processing
    • Asynchronous dataflow with streams

    Combine and transform streams of inputs
    into streams of outputs
    60

    View Slide

  75. Lattice Processing
    • Asynchronous dataflow with streams

    Combine and transform streams of inputs
    into streams of outputs
    • Convergent data structures

    Data abstraction (inputs/outputs) is the CRDT
    60

    View Slide

  76. Lattice Processing
    • Asynchronous dataflow with streams

    Combine and transform streams of inputs
    into streams of outputs
    • Convergent data structures

    Data abstraction (inputs/outputs) is the CRDT
    • Confluence

    Provides composition that preserves the SEC
    property
    60

    View Slide

  77. Lattice Processing
    Confluence
    61

    View Slide

  78. A B
    Map
    Process
    \x.2x
    62
    Sequential

    specification.

    View Slide

  79. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    63
    Replication

    per node.

    View Slide

  80. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    64
    Replication

    per node.

    View Slide

  81. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    65
    Replication

    per node.

    View Slide

  82. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    66
    One possible schedule….

    View Slide

  83. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    67
    …another possible schedule.

    View Slide

  84. A’
    A’’
    B’
    B’’
    Map
    Process
    \x.2x
    Map
    Process
    \x.2x
    A B
    Map
    Process
    \x.2x
    68

    View Slide

  85. A B
    Map
    Process
    \x.2x
    69
    All schedules equivalent
    to sequential schedule.

    View Slide

  86. A B
    F
    70
    Arbitrary

    application.

    View Slide

  87. Lattice Processing
    Example
    71

    View Slide

  88. 72
    %% Create initial set.
    S1 = declare(set),
    %% Add elements to initial set and update.
    update(S1, {add, [1,2,3]}),
    %% Create second set.
    S2 = declare(set),
    %% Apply map operation between S1 and S2.
    map(S1, fun(X) -> X * 2 end, S2).

    View Slide

  89. 73
    %% Create initial set.
    S1 = declare(set),
    %% Add elements to initial set and update.
    update(S1, {add, [1,2,3]}),
    %% Create second set.
    S2 = declare(set),
    %% Apply map operation between S1 and S2.
    map(S1, fun(X) -> X * 2 end, S2).

    View Slide

  90. 74
    %% Create initial set.
    S1 = declare(set),
    %% Add elements to initial set and update.
    update(S1, {add, [1,2,3]}),
    %% Create second set.
    S2 = declare(set),
    %% Apply map operation between S1 and S2.
    map(S1, fun(X) -> X * 2 end, S2).

    View Slide

  91. 75
    %% Create initial set.
    S1 = declare(set),
    %% Add elements to initial set and update.
    update(S1, {add, [1,2,3]}),
    %% Create second set.
    S2 = declare(set),
    %% Apply map operation between S1 and S2.
    map(S1, fun(X) -> X * 2 end, S2).

    View Slide

  92. 76
    %% Create initial set.
    S1 = declare(set),
    %% Add elements to initial set and update.
    update(S1, {add, [1,2,3]}),
    %% Create second set.
    S2 = declare(set),
    %% Apply map operation between S1 and S2.
    map(S1, fun(X) -> X * 2 end, S2).

    View Slide

  93. Processes
    • Replicas as monotonic streams

    Each replica of a CRDT produces a monotonic
    stream of states
    77

    View Slide

  94. Processes
    • Replicas as monotonic streams

    Each replica of a CRDT produces a monotonic
    stream of states
    • Monotonic processes

    Read from one or more input replica streams
    and produce a single output replica stream
    77

    View Slide

  95. Processes
    • Replicas as monotonic streams

    Each replica of a CRDT produces a monotonic
    stream of states
    • Monotonic processes

    Read from one or more input replica streams
    and produce a single output replica stream
    • Inflationary reads

    Read operation ensures that we only read
    inflationary updates to replicas
    77

    View Slide

  96. Lattice Processing
    Monotonic Streams
    78

    View Slide

  97. RA
    {}
    C1
    {}
    C2
    {}
    79

    View Slide

  98. RA
    {}
    (1, {a}, {})
    C1
    (1, {a}, {})
    {}
    C2
    {}
    80

    View Slide

  99. RA
    {}
    (1, {a}, {})
    (1, {a, b}, {})
    C1
    (1, {a}, {})
    {}
    C2
    (1, {b}, {})
    {}
    81

    View Slide

  100. RA
    {}
    (1, {a}, {})
    (1, {a, b}, {})
    (1, {a, b}, {a})
    C1
    (1, {a}, {})
    {}
    (1, {a}, {a})
    C2
    (1, {b}, {})
    {}
    82

    View Slide

  101. RA
    {}
    (1, {a}, {})
    (1, {a, b}, {})
    (1, {a, b}, {a})
    C1
    (1, {a}, {})
    {}
    (1, {a}, {a})
    C2
    (1, {b}, {})
    {}
    (1, {a, b}, {a})
    (1, {a, b}, {a}) (1, {a, b}, {a})
    83

    View Slide

  102. RA
    {}
    (1, {a}, {})
    (1, {a, b}, {})
    (1, {a, b}, {a})
    C1
    (1, {a}, {})
    {}
    (1, {a}, {a})
    C2
    (1, {b}, {})
    {}
    (1, {a, b}, {a})
    (1, {a, b}, {a}) (1, {a, b}, {a})
    84
    Clients can
    operate with partial state…

    View Slide

  103. RA
    {}
    (1, {a}, {})
    (1, {a, b}, {})
    (1, {a, b}, {a})
    C1
    (1, {a}, {})
    {}
    (1, {a}, {a})
    C2
    (1, {b}, {})
    {}
    (1, {a, b}, {a})
    (1, {a, b}, {a}) (1, {a, b}, {a})
    85
    … and synchronize
    with their local replica.

    View Slide

  104. Lattice Processing
    Monotonic Processes
    86

    View Slide

  105. RA
    {}
    P1 F(RA)
    {}
    87

    View Slide

  106. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    88

    View Slide

  107. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {})
    89

    View Slide

  108. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {}) F((1, {a}, {}))
    90

    View Slide

  109. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {}) F((1, {a}, {})) (2, {a}, {})
    91
    Every time replica changes…

    View Slide

  110. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {}) F((1, {a}, {})) (2, {a}, {})
    92
    ….the process will compute a new result.

    View Slide

  111. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {}) F((1, {a}, {})) (2, {a}, {})
    strict_read((1, {a}, {})
    (1, {a}, {})
    93

    View Slide

  112. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {}) F((1, {a}, {})) (2, {a}, {})
    strict_read((1, {a}, {})
    (1, {a}, {})
    (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a})
    strict_read((1, {a}, {a})
    (1, {a}, {a})
    94

    View Slide

  113. RA
    {}
    P1 F(RA)
    {}
    strict_read({})
    (1, {a}, {})
    (1, {a}, {})
    (1, {a}, {a}) F((1, {a}, {a})) (2, {a}, {a})
    strict_read((1, {a}, {a})
    (1, {a}, {a})
    95
    Omitted interleaving
    does not sacrifice correctness.

    View Slide

  114. Lattice Processing
    Map Example
    96

    View Slide

  115. RC
    F(RC)
    97

    View Slide

  116. RC
    F(RC) {2}
    Map
    Process
    \x.2x
    (2, {b}, {})
    (1, {b}, {})
    {1}
    98
    Transform element,
    map metadata.

    View Slide

  117. RC
    F(RC) {2}
    Map
    Process
    \x.2x
    (2, {b}, {})
    (1, {b}, {})
    {1} {}
    (1, {b}, {b})
    {}
    Map
    Process
    \x.2x
    (2, {b}, {b})
    99

    View Slide

  118. RC
    F(RC) {2}
    Map
    Process
    \x.2x
    (2, {b}, {})
    (1, {b}, {})
    {1} {}
    (1, {b}, {b})
    {}
    Map
    Process
    \x.2x
    (2, {b}, {b})
    {1}
    (1, {a, b}, {b})
    {2}
    Map
    Process
    \x.2x
    (2, {a, b}, {b})
    100

    View Slide

  119. RC
    F(RC) {2}
    Map
    Process
    \x.2x
    (2, {b}, {})
    (1, {b}, {})
    {1} {}
    (1, {b}, {b})
    {}
    Map
    Process
    \x.2x
    (2, {b}, {b})
    {1}
    (1, {a, b}, {b})
    {2}
    Map
    Process
    \x.2x
    (2, {a, b}, {b})
    {1}
    {2}
    101

    View Slide

  120. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)

    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    102

    View Slide

  121. Example Application
    Advertisement Counter
    103

    View Slide

  122. Advertisement Counter
    • Lower-bound invariant

    Advertisements are paid according to a minimum
    number of impressions
    104

    View Slide

  123. Advertisement Counter
    • Lower-bound invariant

    Advertisements are paid according to a minimum
    number of impressions
    • Clients will go offline

    Clients have limited connectivity and the system
    still needs to make progress while clients are offline
    104

    View Slide

  124. Advertisement Counter
    • Lower-bound invariant

    Advertisements are paid according to a minimum
    number of impressions
    • Clients will go offline

    Clients have limited connectivity and the system
    still needs to make progress while clients are offline
    • No lost updates

    All displayed advertisements should be accounted
    for, with no lost updates
    104

    View Slide

  125. Advertisement Counter
    Losing Updates
    105

    View Slide

  126. RA
    RB
    106

    View Slide

  127. RA
    RB
    1
    set(1)
    107

    View Slide

  128. RA
    RB
    1
    set(1)
    2
    2
    set(2)
    set(2)
    108

    View Slide

  129. RA
    RB
    1
    set(1)
    2
    2
    set(2)
    set(2)
    2
    2
    max(2,2)
    max(2,2)
    109

    View Slide

  130. RA
    RB
    1
    set(1)
    2
    2
    set(2)
    set(2)
    2
    2
    max(2,2)
    max(2,2)
    110
    Incorrect value
    is computed
    because of
    incompatible lattice.

    View Slide

  131. Advertisement Counter
    Application Flow
    111

    View Slide

  132. Server
    Client
    Client
    112

    View Slide

  133. Server
    Client
    Client
    113
    Client
    reads state
    from the server.

    View Slide

  134. Server
    Client
    Client
    114
    Client
    locally mutates

    state.

    View Slide

  135. Server
    Client
    Client
    115
    Client
    pushes changes

    back to
    the server.

    View Slide

  136. Server
    Client
    Client
    116

    View Slide

  137. Server
    Client
    Client
    117
    Server
    enforces invariants

    over state.

    View Slide

  138. Server
    Client
    Client
    118
    Client retrieves
    updated state
    periodically.

    View Slide

  139. Server
    Client
    Client
    119
    Clients
    unable to communicate

    may
    violate invariant.

    View Slide

  140. Advertisement Counter
    Application Design
    120

    View Slide

  141. Ads
    Contracts
    Product Filter Map
    Ads
    X
    Contracts
    Ads
    With
    Contracts
    Ads
    To
    Display
    Asynchronous Dataflow
    Counters
    Counters
    Counters
    Counter
    Ads
    Update
    (Insert)
    Application Initialization
    Counter
    Read
    > 50,000
    Ads
    Update
    (Remove)
    Epsilon-Invariant
    Ads Map
    Configure Invariant
    Enforce Invariant
    121

    View Slide

  142. Ads
    Contracts
    Product Filter Map
    Ads
    X
    Contracts
    Ads
    With
    Contracts
    Ads
    To
    Display
    Asynchronous Dataflow
    122

    View Slide

  143. Counters
    Counters
    Counters
    Counter
    Ads
    Update
    (Insert)
    Application Initialization
    123

    View Slide

  144. Counter
    Read
    > 50,000
    Ads
    Update
    (Remove)
    Epsilon-Invariant
    Ads Map
    Configure Invariant
    Enforce Invariant
    124

    View Slide

  145. Counter
    Read
    > 50,000
    Ads
    Update
    (Remove)
    Epsilon-Invariant
    Ads Map
    Configure Invariant
    Enforce Invariant
    125
    Configure
    invariants

    for all of the
    advertisements.

    View Slide

  146. Counter
    Read
    > 50,000
    Ads
    Update
    (Remove)
    Epsilon-Invariant
    Ads Map
    Configure Invariant
    Enforce Invariant
    126
    Remove the
    advertisement from
    the list.

    View Slide

  147. Advertisement Counter
    • Completely monotonic

    Disabling advertisements and contracts are all
    modeled through monotonic state growth
    127

    View Slide

  148. Advertisement Counter
    • Completely monotonic

    Disabling advertisements and contracts are all
    modeled through monotonic state growth
    • Arbitrary distribution

    Use of convergent data structures allows
    computational graph to be arbitrarily distributed
    127

    View Slide

  149. Advertisement Counter
    • Completely monotonic

    Disabling advertisements and contracts are all
    modeled through monotonic state growth
    • Arbitrary distribution

    Use of convergent data structures allows
    computational graph to be arbitrarily distributed
    • Divergence

    Divergence is a factor of synchronization
    period, concurrency, and throughput rate
    127

    View Slide

  150. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)

    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    128

    View Slide

  151. Distributed Runtime
    Anabranch
    129
    work-in-progress

    View Slide

  152. Anabranch
    • Layered approach

    Cluster membership and state dissemination for large clusters
    130

    View Slide

  153. Anabranch
    • Layered approach

    Cluster membership and state dissemination for large clusters
    • Delta-state synchronization 

    Efficient incremental state dissemination and anti-entropy
    mechanism [Almeida et al. 2016]
    130

    View Slide

  154. Anabranch
    • Layered approach

    Cluster membership and state dissemination for large clusters
    • Delta-state synchronization 

    Efficient incremental state dissemination and anti-entropy
    mechanism [Almeida et al. 2016]
    • Epsilon-invariants

    Lower-bound invariants, configurable at runtime
    130

    View Slide

  155. Anabranch
    • Layered approach

    Cluster membership and state dissemination for large clusters
    • Delta-state synchronization 

    Efficient incremental state dissemination and anti-entropy
    mechanism [Almeida et al. 2016]
    • Epsilon-invariants

    Lower-bound invariants, configurable at runtime
    • Scalable

    Demonstrated high scalability in production Cloud
    environments
    130

    View Slide

  156. Anabranch
    Layered Approach
    131

    View Slide

  157. Layered Approach
    132

    View Slide

  158. Layered Approach
    • Backend

    Configurable persistence layer depending on
    application.
    132

    View Slide

  159. Layered Approach
    • Backend

    Configurable persistence layer depending on
    application.
    • Membership

    Configurable membership protocol which can operate in
    a client-server or peer-to-peer mode [Leitao et al. 2007]
    132

    View Slide

  160. Layered Approach
    • Backend

    Configurable persistence layer depending on
    application.
    • Membership

    Configurable membership protocol which can operate in
    a client-server or peer-to-peer mode [Leitao et al. 2007]
    • Broadcast (via Gossip, Tree, etc.)

    Efficient dissemination of both program state and
    application state via gossip, broadcast tree, or hybrid
    mode [Leitao et al. 2007]
    132

    View Slide

  161. Mobile Phone
    Distributed Hash Table
    Application
    Language Execution
    KV Store
    KV Backend Membership
    Broadcast Layer
    Client/Server Peer-to-Peer

    View Slide

  162. Mobile Phone
    Distributed Hash Table
    Application
    Language Execution
    KV Store
    KV Backend Membership
    Broadcast Layer
    Client/Server Peer-to-Peer
    Language
    and applications.

    View Slide

  163. Mobile Phone
    Distributed Hash Table
    Application
    Language Execution
    KV Store
    KV Backend Membership
    Broadcast Layer
    Client/Server Peer-to-Peer
    Storage
    for CRDT state.

    View Slide

  164. Mobile Phone
    Distributed Hash Table
    Application
    Language Execution
    KV Store
    KV Backend Membership
    Broadcast Layer
    Client/Server Peer-to-Peer
    State
    dissemination.

    View Slide

  165. Anabranch
    Delta-state CRDTs
    137

    View Slide

  166. Delta-based Dissemination
    • Delta-state based CRDTs

    Reduces state transmission for clients
    138

    View Slide

  167. Delta-based Dissemination
    • Delta-state based CRDTs

    Reduces state transmission for clients
    • Operate locally

    Objects are mutated locally; delta’s buffered
    locally and periodically gossiped
    138

    View Slide

  168. Delta-based Dissemination
    • Delta-state based CRDTs

    Reduces state transmission for clients
    • Operate locally

    Objects are mutated locally; delta’s buffered
    locally and periodically gossiped
    • Only fixed number of clients

    Clients resort to full state synchronization
    when they’ve been partitioned too long
    138

    View Slide

  169. RC
    BufferA
    BufferB

    View Slide

  170. RC
    BufferA
    BufferB
    (1, {b}, {})
    {1}
    (1, {b}, {})
    (1, {b}, {})
    Buffer minimal
    change representation…

    View Slide

  171. RC
    BufferA
    BufferB
    (1, {b}, {})
    {1}
    (1, {b}, {})
    (1, {b}, {})
    {}
    (1, {b}, {b})
    (1, {b}, {b})
    (1, {b}, {b})

    View Slide

  172. RC
    BufferA
    BufferB
    (1, {b}, {})
    {1}
    (1, {b}, {})
    (1, {b}, {})
    {}
    (1, {b}, {b})
    (1, {b}, {b})
    (1, {b}, {b})
    {2}
    (1, {b}, {b})
    (2, {c}, {})
    (2, {c}, {})
    (2, {c}, {})
    …then, disseminate state
    in causal order
    to neighbors.

    View Slide

  173. RC
    BufferA
    BufferB
    (1, {b}, {})
    {1}
    (1, {b}, {})
    (1, {b}, {})
    {}
    (1, {b}, {b})
    (1, {b}, {b})
    (1, {b}, {b})
    {2}
    (1, {b}, {b})
    (2, {c}, {})
    (2, {c}, {})
    (2, {c}, {})
    {1, 2}
    (1, {a, b}, {b})
    (2, {c}, {})
    {1, 2}
    (1, {a}, {})
    (1, {a}, {})
    Only ship inflation
    from incoming state.

    View Slide

  174. Anabranch
    Scalability
    144

    View Slide

  175. Scalability
    • 1024+ nodes

    Demonstrated scalability to 1024 nodes in Amazon
    cloud computing environment
    145

    View Slide

  176. Scalability
    • 1024+ nodes

    Demonstrated scalability to 1024 nodes in Amazon
    cloud computing environment
    • Modular approach

    Many of the components built and can be operated
    outside of Lasp to improve scalability of Erlang
    145

    View Slide

  177. Scalability
    • 1024+ nodes

    Demonstrated scalability to 1024 nodes in Amazon
    cloud computing environment
    • Modular approach

    Many of the components built and can be operated
    outside of Lasp to improve scalability of Erlang
    • Automated and repeatable

    Fully automated deployment, scenario execution,
    log aggregation and archival of experimental results
    145

    View Slide

  178. Programming SEC
    1. Eliminate accidental nondeterminism

    (ex. deterministic, modeling non-monotonic operations
    monotonically)

    2. Retain the properties of functional
    programming

    (ex. confluence, “correct-by-construction” applications)
    3. Distributed, and fault-tolerant runtime

    (ex. replication, membership, dissemination)
    146

    View Slide

  179. What have we
    learned?
    147

    View Slide

  180. We’ve build up from zero
    synchronization
    148

    View Slide

  181. We’ve build up from zero
    synchronization
    148
    Instead of working to
    remove synchronization

    View Slide

  182. Programming SEC
    1. Eliminate accidental nondeterminism

    Convergent Objects: Conflict-Free Replicated Data Types
    [Shapiro et al. 2011]
    149

    View Slide

  183. Programming SEC
    1. Eliminate accidental nondeterminism

    Convergent Objects: Conflict-Free Replicated Data Types
    [Shapiro et al. 2011]
    2. Retain the properties of functional
    programming

    Convergent Programs: Lattice Processing

    [Meiklejohn et al. 2015]
    149

    View Slide

  184. Programming SEC
    1. Eliminate accidental nondeterminism

    Convergent Objects: Conflict-Free Replicated Data Types
    [Shapiro et al. 2011]
    2. Retain the properties of functional
    programming

    Convergent Programs: Lattice Processing

    [Meiklejohn et al. 2015]
    3. Distributed, and fault-tolerant runtime

    Distributed Runtime: Anabranch

    [Meiklejohn et al. work-in-progress]
    149

    View Slide

  185. Key Points
    • Synchronization is sometimes not possible

    Mobile and “Internet of Things” applications make
    synchronization for replicated state impractical
    150

    View Slide

  186. Key Points
    • Synchronization is sometimes not possible

    Mobile and “Internet of Things” applications make
    synchronization for replicated state impractical
    • Apply synchronization only where required

    Global invariants, atomic visibility, etc.
    150

    View Slide

  187. Key Points
    • Synchronization is sometimes not possible

    Mobile and “Internet of Things” applications make
    synchronization for replicated state impractical
    • Apply synchronization only where required

    Global invariants, atomic visibility, etc.
    • Holistic approach

    Taking a holistic approach to the design of
    distributed systems is important for building
    higher-availability applications
    150

    View Slide

  188. Artifacts
    • Lasp

    CRDT based programming model: testbed for Loquat

    http://github.com/lasp-lang/lasp
    151

    View Slide

  189. Artifacts
    • Lasp

    CRDT based programming model: testbed for Loquat

    http://github.com/lasp-lang/lasp
    • Partisan

    TCP-based pluggable membership service offering client/server, static, and HyParView-
    based protocols

    https://github.com/lasp-lang/partisan
    151

    View Slide

  190. Artifacts
    • Lasp

    CRDT based programming model: testbed for Loquat

    http://github.com/lasp-lang/lasp
    • Partisan

    TCP-based pluggable membership service offering client/server, static, and HyParView-
    based protocols

    https://github.com/lasp-lang/partisan
    • Plumtree

    Epidemic broadcast trees for use with Partisan

    https://github.com/lasp-lang/plumtree
    151

    View Slide

  191. Artifacts
    • Lasp

    CRDT based programming model: testbed for Loquat

    http://github.com/lasp-lang/lasp
    • Partisan

    TCP-based pluggable membership service offering client/server, static, and HyParView-
    based protocols

    https://github.com/lasp-lang/partisan
    • Plumtree

    Epidemic broadcast trees for use with Partisan

    https://github.com/lasp-lang/plumtree
    • Sprinter

    Service discovery and deployment for Mesos and Kubernetes

    https://github.com/lasp-lang/sprinter
    151

    View Slide

  192. Artifacts
    • Lasp

    CRDT based programming model: testbed for Loquat

    http://github.com/lasp-lang/lasp
    • Partisan

    TCP-based pluggable membership service offering client/server, static, and HyParView-
    based protocols

    https://github.com/lasp-lang/partisan
    • Plumtree

    Epidemic broadcast trees for use with Partisan

    https://github.com/lasp-lang/plumtree
    • Sprinter

    Service discovery and deployment for Mesos and Kubernetes

    https://github.com/lasp-lang/sprinter
    • Types

    CRDT implementations for Erlang

    https://github.com/lasp-lang/types
    151

    View Slide

  193. 152
    Christopher Meiklejohn

    @cmeik
    http://www.lasp-lang.org
    Thanks!

    View Slide