Mayday, we're syncing!

We live in the time of mobile-first development, where many developers keep facing the same problem over and over again: How do we synchronize users’ data across their multitude of devices? Phones can be temporarily offline, tablets only work with WiFi, and browsers may crash. Under no circumstances should users have to resolve merge conflicts, but they do expect history and rollbacks. Can we satisfy all these constraints? “Conflict-free replicated datatypes” are the solution. A rather young technology, poised to solve synchronisation issues once and for all. Come along to a stroll through research & practice of this emerging ecosystem.

Transcript

1. Mayday,
   we're syncing!
   REACT VIENNA / 2022-06-21
   LARS HUPEL
   @LARSR_H
   LUCAS DOHMEN
   @MOONBEAMLABS
   

   View Slide

2. View Slide

3. View Slide

4. When everything's
   in the cloud
   •
   •
   •
   ... we are dependent on the Internet
   ... we are limited in how we can work
   ... we are excluding people
   

   View Slide

5. Working Offline?
   •
   •
   in the Olden Days, we didn't
   always have a connection
   local file storage, manual sync
   

   View Slide

6. Offline-first is the standard for
   development
   •
   •
   •
   we like working offline
   no one wants to use Subversion nowadays
   why do we ask our users for connectivity?
   

   View Slide

7. View Slide

8. Hello
   world!
   Hello
   world
   Hello,
   world!
   

   View Slide

9. View Slide

10. View Slide

11. View Slide

12. Lattices
    •
    •
    mathematical structure
    abstract definition of ordering
    

    View Slide

13. interface Lattice {
    join(left: T, right: T): T;
    lteq(left: T, right: T): boolean;
    }
    // 1. join(x, y) == join(y, x)
    // 2. join(x, x) == x
    // 3. join(x, join(y, z)) == join(join(x, y), z)
    // ...
    

    View Slide

14. https://commons.wikimedia.org/wiki/File:Lattice_of_the_divisibility_of_60.svg
    

    View Slide

15. const divisionLattice: Lattice = {
    join(left: number, right: number) {
    return leastCommonMultiple(left, right);
    },
    lteq(left: number, right: number) {
    return right % left == 0;
    }
    }
    

    View Slide

16. CRDTs
    Conflict-Free Replicated Datatypes
    

    View Slide

17. Example: G-Sets
    {}
    {Skyr}
    {Cereal}
    {Skyr,
    Grapefruit}
    {Skyr, Cereal,
    Grapefruit}
    

    View Slide

18. Tombstones
    •
    •
    deletion is not monotonic
    ☹️
    solution: mark as deleted
    

    View Slide

19. Example: 2P-Sets
    {Skyr, Cereal,
    Grapefruit}
    {Skyr, Cereal,
    Grapefruit}
    {Skyr, Apple,
    Cereal, Grapefruit}
    {Skyr, Cereal,
    Grapefruit, Tea}
    {Skyr, Apple, Cereal,
    Grapefruit, Tea}
    

    View Slide

20. Vector Clocks
    •
    •
    •
    each of the n devices gets its own timestamp
    the global timestamp for all devices is
    composed of those n timestamps
    it represents the last known state
    

    View Slide

21. Vector Clocks
    [1, 0] [2, 0]
    [0, 1]
    [0, 0] [2, 1]
    [T₁, T₂]
    

    View Slide

22. T₁ = 0
    T₂ = 0
    Example: MV-Register
    x = 0 x = 1 T₁ = 1
    T₂ = 0
    👩
    x = 0 T₁ = 0
    T₂ = 0
    x = 1 T₁ = 1
    T₂ = 0
    x = 1 T₁ = 1
    T₂ = 0
    x = 2 T₁ = 2
    T₂ = 0
    x = 3 T₁ = 1
    T₂ = 1
    💣
    update
    sync
    update
    update
    x = 0
    🧑
    

    View Slide

23. “Any two object replicas of a CvRDT
    eventually converge, assuming the system
    transmits payload infinitely often between
    pairs of replicas over eventually-reliable
    point-to-point channels.”
    

    View Slide

24. View Slide

25. View Slide

26. Automerge
    •
    •
    •
    JavaScript Library for Browser & Node
    offers JSON documents as datatype
    💡they are CRDTs
    

    View Slide

27. Usage
    Requests pro
    Sekunde
    doc1 = Automerge.from({ todos: [] })
    doc2 = Automerge.change(doc1, 'Add cereal', doc => {
    doc.todos.push({ text: 'Cereal', done: false })
    })
    doc3 = Automerge.change(doc1, 'Add skyr', doc => {
    doc.todos.push({ text: 'Skyr', done: false })
    })
    doc4 = Automerge.merge(doc2, doc3)
    // contains traces of cereal & skyr
    

    View Slide

28. Demo
    Shopping List
    

    View Slide

29. View Slide

30. https://github.com/innoq/groceries
    

    View Slide

31. Server CRDT
    Browser
    List
    as HTML
    POST
    the new item
    

    View Slide

32. Server CRDT
    Browser
    Changes
    via SSE
    POST
    the changeset
    CRDT
    

    View Slide

33. Subscribe via SSE
    const source = new EventSource("/stream");
    source.addEventListener("message", (ev) => {
    list.applyChanges(ev.data);
    });
    

    View Slide

34. Push Changes
    class List {
    constructor() {
    // ...
    this.changes = [];
    this.interval =
    setInterval(this.pushChanges.bind(this), 1000);
    }
    }
    

    View Slide

35. Push Changes
    async pushChanges() {
    try {
    await fetch("/", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ changes: this.changes }),
    });
    this.changes = [];
    } catch(e) { /* ... */ }
    }
    

    View Slide

36. More Features
    •
    •
    •
    •
    •
    •
    other datatypes
    Counter
    Text (collaborative editing)
    Peritext
    table
    P2P with Hypermerge
    

    View Slide

37. Should the app work offline
    Do whatever
    you want <3
    Read or write?
    Check out
    Service Workers
    There is no way
    around
    CSR & CRDTs
    No Yes
    only read both
    

    View Slide

38. Q&A
    Lucas Dohmen
    lucas.dohmen@innoq.com
    @moonbeamlabs
    www.innoq.com
    innoQ Deutschland GmbH
    Krischerstr. 100
    40789 Monheim
    +49 2173 333660
    Ohlauer Str. 43
    10999 Berlin
    Ludwigstr. 180E
    63067 Offenbach
    Kreuzstr. 16
    80331 München
    Hermannstr. 13
    20095 Hamburg
    Erftstr. 15-17
    50672 Köln
    Königstorgraben 11
    90402 Nürnberg
    Lars Hupel
    lars.hupel@innoq.com
    @larsr_h
    

    View Slide