CRDTs: The Hard Parts

CRDTs: The Hard Parts

Slides from a talk given at the Hydra distributed computing conference on 6 July 2020.


Conflict-free Replicated Data Types (CRDTs) are an increasingly popular family of algorithms for optimistic replication. They allow data to be concurrently updated on several replicas, even while those replicas are offline, and provide a robust way of merging those updates back into a consistent state. CRDTs are used in geo-replicated databases, multi-user collaboration software, distributed processing frameworks, and various other systems.

However, while the basic principles of CRDTs are now quite well known, many challenging problems are lurking below the surface. It turns out that CRDTs are easy to implement badly. Many published algorithms have anomalies that cause them to behave strangely in some situations. Simple implementations often have terrible performance, and making the performance good is challenging.

In this talk Martin goes beyond the introductory material on CRDTs, and discusses some of the hard-won lessons from years of research on making CRDTs work in practice.


Dr Martin Kleppmann is a researcher in distributed systems at the University of Cambridge, and author of the acclaimed "Designing Data-Intensive Applications" (O'Reilly Media, 2017). He mainly works on collaboration software, CRDTs, and formal verification of distributed algorithms. Previously he was a software engineer and entrepreneur at Internet companies including LinkedIn and Rapportive, where he worked on large-scale data infrastructure.


Martin Kleppmann

July 06, 2020


  1. None
  2. None
  3. None
  4. None
  5. None
  6. None
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. List (sequence, array) CRDTs: WOOT, Treedoc, Logoot, RGA, Causal Trees,

    LSEQ, …
  35. List (sequence, array) CRDTs: WOOT, Treedoc, Logoot, RGA, Causal Trees,

    LSEQ, … Emulating “move” as “delete-and-reinsert”: concurrent moves of the same item à duplication
  36. Concurrent move of the same item to different positions —

    what should happen?
  37. Converge to one of the destination positions (pick one arbitrarily

    but deterministically)
  38. “pick one arbitrarily” = last-writer wins register! posphone joe :=

    “head of the list” merge posphone joe == “head of the list” posphone joe := “after buy milk”
  39. List CRDT with move operation posphone joe := “after buy

    milk” need one register per list item state = AWSet({ (v1 , LWWRegister(p1 )), (v2 , LWWRegister(p2 )), … }) need a stable way of referencing list positions Treedoc: path through binary tree Logoot: list of (integer, replicaID) pairs RGA: s4vector Causal Trees: logical timestamp etc… Composition of any list CRDT + AWSet + LWWRegister = another CRDT
  40. Moving ranges of elements

  41. Moving ranges of elements

  42. Desired outcome

  43. Actual outcome Fixing this: an open problem!

  44. None
  45. None
  46. Concurrent moves of same node

  47. Concurrent moves of same node

  48. None
  49. None
  50. Moving A into B, and B into A

  51. Moving A into B, and B into A

  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. None
  59. None
  60. None
  61. None
  62. None
  63. None
  64. None
  65. None
  66. None
  67. File size File size (gzipped) Full document history, JSON format

    146,406,415 bytes 6,132,895 bytes Full document history, custom binary format 695,298 bytes 302,067 bytes Document history with cursor movements omitted 570,992 bytes 214,889 bytes CRDT document with editing history omitted 228,153 bytes 114,821 bytes CRDT document with tombstones removed 154,418 bytes 63,249 bytes Baseline: plain text with no CRDT metadata 104,852 bytes 27,569 bytes 48% 48% 150% 22% 211 x Benchmark data: keystroke-by-keystroke editing trace of a text file (LaTeX source of a research paper) containing 182,315 single-character insertions, 77,463 single- character deletions, and 102,049 cursor movements. Compressing CRDT metadata in Automerge
  68. None
  69. None
  70. None
  71. None
  72. None
  73. None
  74. None
  75. Text editing CRDTs: Logoot: Stéphane Weiss, Pascal Urso, and Pascal

    Molli: “Logoot: A Scalable Optimistic Replication Algorithm for Collaborative Editing on P2P Networks,” ICDCS 2009. LSEQ: Brice Nédelec, Pascal Molli, Achour Mostefaoui, and Emmanuel Desmontils: “LSEQ: an Adaptive Structure for Sequences in Distributed Collaborative Editing,” DocEng 2013. RGA: Hyun-Gul Roh, Myeongjae Jeon, Jin-Soo Kim, and Joonwon Lee: “Replicated abstract data types: Building blocks for collaborative applications,” Journal of Parallel and Distributed Computing, 71(3):354–368, 2011. Treedoc: Nuno Preguiça, Joan Manuel Marques, Marc Shapiro, and Mihai Letia: “A Commutative Replicated Data Type for Cooperative Editing,” ICDCS 2009. WOOT: Gérald Oster, Pascal Urso, Pascal Molli, and Abdessamad Imine: “Data consistency for P2P collaborative editing,” CSCW 2006. Astrong : Hagit Attiya, Sebastian Burckhardt, Alexey Gotsman, Adam Morrison, Hongseok Yang, and Marek Zawirski: “Specification and Complexity of Collaborative Text Editing,” PODC 2016.
  76. More details in these related publications: Interleaving anomaly: Martin Kleppmann,

    Victor B. F. Gomes, Dominic P. Mulligan, and Alastair R. Beresford: “Interleaving anomalies in collaborative text editors”. PaPoC 2019. Proof of no interleaving in RGA: Martin Kleppmann, Victor B F Gomes, Dominic P Mulligan, and Alastair R Beresford: “OpSets: Sequential Specifications for Replicated Datatypes,”, May 2018. Moving list items: Martin Kleppmann: “Moving Elements in List CRDTs”. PaPoC 2020. Move operation in CRDT trees: Martin Kleppmann, Dominic P. Mulligan, Victor B. F. Gomes, and Alastair R. Beresford: “A highly-available move operation for replicated trees and distributed filesystems”. Preprint, Reducing metadata overhead: Martin Kleppmann: “Experiment: columnar data encoding for Automerge”, 2019. perf/blob/master/columnar/ Local-first software: Martin Kleppmann, Adam Wiggins, Peter van Hardenberg, and Mark McGranaghan: “Local-first software: You own your data, in spite of the cloud”. Onward! 2019.
  77. Thanks! • Martin’s email: • Martin on Twitter:

    • Martin’s book: • CRDT resources: • Automerge: Thank you to these organisations for supporting this work!