Convergence versus Consensus: CRDTs and the Quest for Distributed Consistency

Slides from a talk given on 5 March 2018 at QCon London.
https://qconlondon.com/london2018/presentation/crdts-and-quest-distributed-consistency

Abstract:

We all know how to build applications that rely on a central server. However, such centralisation is not always desirable, and recently there has been new interest in developing decentralised applications. Blockchains inevitably come up in that conversation, but when you examine them critically, not every problem is best solved by a blockchain.

In this talk we will explore how to ensure data consistency in distributed systems, especially in systems that don't have an authoritative leader. We will see how to sync data between your phone and your laptop without sending it via a remote server. We will explore algorithms that allow several people to collaborate on a shared document, communicating via a peer-to-peer network.

Conflict-Free Replicated Datatypes (CRDTs) are a set of algorithms that ensure data consistency in such settings. Recent research on CRDTs has enabled us to better understand their consistency guarantees and design richer datatypes. On the practical side, CRDTs are making their way into more and more applications. This talk will examine that research and its uses, showing where we are now and where we are heading in the future.

Transcript

4. https://www.ucu.org.uk/why-we-are-taking-action-over-USS
47. For details, see our paper at https://doi.org/10.1145/3133933 Victor B. F. Gomes, Martin Kleppmann, Dominic P. Mulligan, and Alastair R. Beresford: Verifying Strong Eventual Consistency in Distributed Systems. PACMPL 1(OOPSLA), 2017.
51. Trellis, a Trello clone based on Automerge: https://github.com/automerge/trellis Joint work with Orion Henry, Peter van Hardenberg, Roshan Choxi, and Adam Wiggins.
53. Pixelpusher, a collaborative pixel art editor: https://github.com/automerge/pixelpusher Created by Javier Valencia, Jeff Peterson, Peter van Hardenberg, and Jim Pick.
56. MPL, a WebRTC network layer for Automerge: https://github.com/automerge/mpl Joint work with Orion Henry, Peter van Hardenberg, Roshan Choxi, and Adam Wiggins.
57. Hypermerge, a peer-to-peer network layer: https://github.com/automerge/hypermerge Created by Jim Pick, Jeff Peterson, and Peter van Hardenberg.
100. Resources • Automerge: https://github.com/automerge/automerge • Trellis: https://github.com/automerge/trellis • Pixelpusher: https://github.com/automerge/pixelpusher • MPL (WebRTC layer): https://github.com/automerge/mpl • Hypermerge: https://github.com/automerge/hypermerge • Dat / Hypercore: https://datproject.org/ • Proving CRDTs correct: https://doi.org/10.1145/3133933 • JSON CRDT: http://arxiv.org/abs/1608.03960 • My book: http://dataintensive.net/