Syncing data across user devices for distributed collaboration

Syncing data across user devices for distributed collaboration

Slides from a presentation given on 12 July 2019 at the Hydra distributed computing conference, St. Petersburg, Russia.
https://martin.kleppmann.com/2019/07/11/hydra-distributed.html

Abstract:

Google Docs and similar web apps are very convenient if several people need to work together on a document: all users can simultaneously edit the shared document without having to send files back and forth by email. From a distributed systems point of view, real-time collaboration is a replication problem: each user has a replica of the shared document on their device, which they can modify locally, and any changes are sent over the network to their collaborators.

Users may edit a document while offline, causing their replica to become temporarily inconsistent with the others. As they re-synchronise with others, conflicts need to be resolved so that all collaborators end up in a consistent state. Conflict-free Replicated Data Types (CRDTs), first proposed in 2011, are increasingly being used to merge different users’ versions of a document in collaborative software.

This talk will give an update on the latest research on CRDTs, and our open source implementation Automerge. We will also discuss efforts to build “local-first software”, a new generation of collaborative software, on top of these technologies.

0d4ef9af8e4f0cf5c162b48ba24faea6?s=128

Martin Kleppmann

July 12, 2019
Tweet

Transcript

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 27.
  27. 29.
  28. 30.
  29. 31.
  30. 32.
  31. 33.
  32. 34.
  33. 35.
  34. 36.
  35. 37.
  36. 38.
  37. 39.
  38. 40.
  39. 41.
  40. 42.
  41. 43.
  42. 44.
  43. 45.
  44. 46.
  45. 47.
  46. 48.
  47. 49.
  48. 50.

    For details, see our paper at https://doi.org/10.1145/3133933 Victor B. F.

    Gomes, Martin Kleppmann, Dominic P. Mulligan, and Alastair R. Beresford: Verifying Strong Eventual Consistency in Distributed Systems. PACMPL 1(OOPSLA), 2017.
  49. 51.
  50. 52.
  51. 53.
  52. 54.
  53. 55.
  54. 56.
  55. 57.
  56. 58.
  57. 59.
  58. 60.
  59. 61.
  60. 62.
  61. 63.
  62. 64.
  63. 65.
  64. 66.
  65. 67.
  66. 68.
  67. 69.
  68. 70.
  69. 71.
  70. 72.
  71. 73.
  72. 74.
  73. 75.
  74. 76.
  75. 77.
  76. 78.
  77. 79.
  78. 80.
  79. 81.
  80. 82.
  81. 83.
  82. 84.
  83. 85.
  84. 86.
  85. 87.
  86. 88.
  87. 89.

    Trellis, a Trello clone based on Automerge: https://github.com/automerge/trellis Joint work

    with Orion Henry, Peter van Hardenberg, Roshan Choxi, and Adam Wiggins.
  88. 90.
  89. 92.
  90. 93.
  91. 94.
  92. 95.

    MPL, a WebRTC network layer for Automerge: https://github.com/automerge/mpl Joint work

    with Orion Henry, Peter van Hardenberg, Roshan Choxi, and Adam Wiggins.
  93. 97.
  94. 98.

    Resources • Martin’s email: mk428@cl.cam.ac.uk • Martin on Twitter: @martinkl

    • Martin’s book: http://dataintensive.net/ • Local-first software: https://www.inkandswitch.com/local-first.html • Automerge: https://github.com/automerge/automerge • Trellis: https://github.com/automerge/trellis • Pixelpusher: https://github.com/automerge/pixelpusher • PushPin: https://inkandswitch.github.io/pushpin/ • Capstone: https://www.inkandswitch.com/capstone-manuscript.html • Hypermerge: https://github.com/automerge/hypermerge • Dat / Hypercore: https://datproject.org/ • Proving CRDTs correct: https://doi.org/10.1145/3133933 • JSON CRDT: http://arxiv.org/abs/1608.03960