$30 off During Our Annual Pro Sale. View Details »

Mayday, we're syncing!

Mayday, we're syncing!

We live in the time of mobile-first development, where many developers keep facing the same problem over and over again: How do we synchronize users’ data across their multitude of devices? Phones can be temporarily offline, tablets only work with WiFi, and browsers may crash. Under no circumstances should users have to resolve merge conflicts, but they do expect history and rollbacks. Can we satisfy all these constraints? “Conflict-free replicated datatypes” are the solution. A rather young technology, poised to solve synchronisation issues once and for all. Come along to a stroll through research & practice of this emerging ecosystem.

Lars Hupel

June 21, 2022
Tweet

More Decks by Lars Hupel

Other Decks in Programming

Transcript

  1. Mayday,
    we're syncing!
    REACT VIENNA / 2022-06-21
    LARS HUPEL
    @LARSR_H
    LUCAS DOHMEN
    @MOONBEAMLABS

    View Slide

  2. View Slide

  3. View Slide

  4. When everything's
    in the cloud



    ... we are dependent on the Internet
    ... we are limited in how we can work
    ... we are excluding people

    View Slide

  5. Working Offline?


    in the Olden Days, we didn't
    always have a connection
    local file storage, manual sync

    View Slide

  6. Offline-first is the standard for
    development



    we like working offline
    no one wants to use Subversion nowadays
    why do we ask our users for connectivity?

    View Slide

  7. View Slide

  8. Hello
    world!
    Hello
    world
    Hello,
    world!

    View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. Lattices


    mathematical structure
    abstract definition of ordering

    View Slide

  13. interface Lattice {
    join(left: T, right: T): T;
    lteq(left: T, right: T): boolean;
    }
    // 1. join(x, y) == join(y, x)
    // 2. join(x, x) == x
    // 3. join(x, join(y, z)) == join(join(x, y), z)
    // ...

    View Slide

  14. https://commons.wikimedia.org/wiki/File:Lattice_of_the_divisibility_of_60.svg

    View Slide

  15. const divisionLattice: Lattice = {
    join(left: number, right: number) {
    return leastCommonMultiple(left, right);
    },
    lteq(left: number, right: number) {
    return right % left == 0;
    }
    }

    View Slide

  16. CRDTs
    Conflict-Free Replicated Datatypes

    View Slide

  17. Example: G-Sets
    {}
    {Skyr}
    {Cereal}
    {Skyr,
    Grapefruit}
    {Skyr, Cereal,
    Grapefruit}

    View Slide

  18. Tombstones


    deletion is not monotonic
    ☹️
    solution: mark as deleted

    View Slide

  19. Example: 2P-Sets
    {Skyr, Cereal,
    Grapefruit}
    {Skyr, Cereal,
    Grapefruit}
    {Skyr, Apple,
    Cereal, Grapefruit}
    {Skyr, Cereal,
    Grapefruit, Tea}
    {Skyr, Apple, Cereal,
    Grapefruit, Tea}

    View Slide

  20. Vector Clocks



    each of the n devices gets its own timestamp
    the global timestamp for all devices is
    composed of those n timestamps
    it represents the last known state

    View Slide

  21. Vector Clocks
    [1, 0] [2, 0]
    [0, 1]
    [0, 0] [2, 1]
    [T₁, T₂]

    View Slide

  22. T₁ = 0
    T₂ = 0
    Example: MV-Register
    x = 0 x = 1 T₁ = 1
    T₂ = 0
    👩
    x = 0 T₁ = 0
    T₂ = 0
    x = 1 T₁ = 1
    T₂ = 0
    x = 1 T₁ = 1
    T₂ = 0
    x = 2 T₁ = 2
    T₂ = 0
    x = 3 T₁ = 1
    T₂ = 1
    💣
    update
    sync
    update
    update
    x = 0
    🧑

    View Slide

  23. “Any two object replicas of a CvRDT
    eventually converge, assuming the system
    transmits payload infinitely often between
    pairs of replicas over eventually-reliable
    point-to-point channels.”

    View Slide

  24. View Slide

  25. View Slide

  26. Automerge



    JavaScript Library for Browser & Node
    offers JSON documents as datatype
    💡they are CRDTs

    View Slide

  27. Usage
    Requests pro
    Sekunde
    doc1 = Automerge.from({ todos: [] })
    doc2 = Automerge.change(doc1, 'Add cereal', doc => {
    doc.todos.push({ text: 'Cereal', done: false })
    })
    doc3 = Automerge.change(doc1, 'Add skyr', doc => {
    doc.todos.push({ text: 'Skyr', done: false })
    })
    doc4 = Automerge.merge(doc2, doc3)
    // contains traces of cereal & skyr

    View Slide

  28. Demo
    Shopping List

    View Slide

  29. View Slide

  30. https://github.com/innoq/groceries

    View Slide

  31. Server CRDT
    Browser
    List
    as HTML
    POST
    the new item

    View Slide

  32. Server CRDT
    Browser
    Changes
    via SSE
    POST
    the changeset
    CRDT

    View Slide

  33. Subscribe via SSE
    const source = new EventSource("/stream");
    source.addEventListener("message", (ev) => {
    list.applyChanges(ev.data);
    });

    View Slide

  34. Push Changes
    class List {
    constructor() {
    // ...
    this.changes = [];
    this.interval =
    setInterval(this.pushChanges.bind(this), 1000);
    }
    }

    View Slide

  35. Push Changes
    async pushChanges() {
    try {
    await fetch("/", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ changes: this.changes }),
    });
    this.changes = [];
    } catch(e) { /* ... */ }
    }

    View Slide

  36. More Features






    other datatypes
    Counter
    Text (collaborative editing)
    Peritext
    table
    P2P with Hypermerge

    View Slide

  37. Should the app work offline
    Do whatever
    you want <3
    Read or write?
    Check out
    Service Workers
    There is no way
    around
    CSR & CRDTs
    No Yes
    only read both

    View Slide

  38. Q&A
    Lucas Dohmen
    [email protected]
    @moonbeamlabs
    www.innoq.com
    innoQ Deutschland GmbH
    Krischerstr. 100
    40789 Monheim
    +49 2173 333660
    Ohlauer Str. 43
    10999 Berlin
    Ludwigstr. 180E
    63067 Offenbach
    Kreuzstr. 16
    80331 München
    Hermannstr. 13
    20095 Hamburg
    Erftstr. 15-17
    50672 Köln
    Königstorgraben 11
    90402 Nürnberg
    Lars Hupel
    [email protected]
    @larsr_h

    View Slide