CRDTs and Eventual Consistency

Presentation from QCon London 2014

Transcript

1. CRDTs Data Types for EC Systems
2. Problem?
3. Eventual Consistency Eventual consistency is a consistency model used in distributed computing that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value. " " --Wikipedia"
4. Trade Off
5. Scale Up $$$Big Iron (still fails)
6. Scale Out Commodity Servers CDNs, App servers Expertise
7. Fault Tolerance
8. Low Latency
9. Low Latency Amazon found every 100ms of latency cost them 1% in sales.
10. Low Latency Google found an extra 0.5 seconds in search page generation time dropped traffic by 20%.
11. CAP
12. C A
13. EC Causal RYOW Session Monotonic Read/Write
14. Pick Your Own http://basho.com/understanding-riaks- configurable-behaviors-part-1/ " http://www.bailis.org/blog/when-is-acid-acid- rarely/
15. Who Pays?
16. Developers But how?
17. Google F1 “We have a lot of experience with eventual consistency systems at Google.”" " “We find developers spend a significant fraction of their time building extremely complex and error-prone mechanisms to cope with eventual consistency”
18. Riak Overview
19. Riak Overview Erlang implementation of Dynamo
20. {“key”: “value”}
21. {“key”: “value”} key value key value key value key value key value key value key value key value key value Keys are namespaced into Buckets
22. Riak Overview Consistent Hashing
26. Riak Overview Dynamic Membership
27. Riak Overview Replication factor
28. Replica Replica Replica
29. High Availability Any non-failing node can respond to any request" " --Gilbert & Lynch
30. Riak Overview Two Writes: {Writer, Value, Time}
31. [{a, v1, a1}] [{b, v2, b1}] [{a, v1, a1}]
32. Riak Overview Last Writer Wins Allow Mult
33. Riak Overview Last Writer Wins [{b, v1, t2}] [{b, v1, t2}] [{b, v1, t2}]
34. Riak Overview Allow Mult [{a, v1, a1}, {b, v2, b1}] [{a, v1, a1}, {b, v2, b1}] [{a, v1, a1}, {b, v2, b1}]
35. User specified Merge
36. Semantic Resolution
37. DB IS DOWN Fill in Form P17QR-35" File for input
38. DB IS DOWN Fill in Form P17QR-35" File for input OOPS
39. Dynamo The Shopping Cart
40. A B HAIRDRYER
41. A B HAIRDRYER
42. A B PENCIL CASE HAIRDRYER
43. A B PENCIL CASE HAIRDRYER
44. A B [HAIRDRYER], [PENCIL CASE]
45. Merge Set Union of Values Simples, right?
46. Deterministic Merge
47. Deterministic Merge Ipempotent
48. Deterministic Merge Ipempotent Associative
49. Deterministic Merge Ipempotent Associative Commutative
50. Set Union? Removes?
51. Absence How can you tell if X is missing from A and not B because A hasn’t seen the addition, or if A has removed X?
52. Complexity
53. Ad Hoc
54. CRDTs
55. CRDTs Convergent Replicated Data Types
56. CRDTs Commutative Replicated Data Types
57. CRDTs Conflict Free Data Structures
58. Theory
62. Bounded Join Semilattices
63. Bounded Join Semilattices Partially ordered set; least upper bound; ACI.
64. Bounded Join Semmilattices Associativity: (X Y) Z = X (Y Z)
65. Bounded Join Semmilattices Commutativity: X Y = Y X
66. Bounded Join Semmilattices Idempotence: X X = X
67. Bounded Join Semmilattices Objects grow over time; merge computes LUB
68. Bounded Join Semmilattices Examples
69. b a c a, b a, c a, b, c Set; merge function: union. b, c
70. 3 5 7 5 7 7 Increasing natural; merge function: max.
71. F F T F T T Booleans; merge function: or.
72. Deterministic Merge Ipempotent Associative Commutative
73. Riak 2.0
74. Pick your semantic Add wins Remove wins Keep both
75. Trade Off More metadata == bigger objects
76. Actors? Version Vectors Entry Per Actor (Charron-Bost)
77. Riak 1.4 Counters: non-idempotent; 0 (Actors)
78. Riak 2.0 Sets: Add, Remove, Membership; Idempotent
79. Riak 2.0 Sets: Add wins 0(Actors + Elements)
80. Riak 2.0 Maps: Recursive; Associative Array; Nestable
81. Riak 2.0 Maps: Update wins; 0(Actors + Elements)
82. Riak 2.0 Maps: LWW-Register, Booleans, Sets and Maps
83. Riak 2.0 LWW-Register: last writer wins
84. Riak 2.0 Boolean: Enabled, Disabled; 0(Actors)
85. riak_dt git clone git@github.com:basho/riak_dt.git
86. Evolution of a Set
87. [{a, 1}, {b, 3}, {c, 2}] Causality Version Vectors
88. [{a, 2}, {b, 3}, {c, 2}] [{a, 1}, {b, 3}, {c, 2}] > Causality Version Vectors
89. [{a, 2}, {b, 3}, {c, 2}] [{a, 1}, {b, 4}, {c, 2}] [{a, 2}, {d, 1}, {c, 2}] [{a, 2}, {b, 4}, {c, 2}] Causality Version Vectors
90. Causality Version Vectors ‘Dots’ are Events
91. [{a, 2}, {b, 3}, {c, 2}] {b, 1} {b, 2} {b, 3} ‘Dots’ are Events Causality
92. Evolution of a Set G-SET
93. Evolution of a Set G-SET
94. Evolution of a Set G-SET 2P-SET
95. Evolution of a Set U-SET
96. Evolution of a Set U-SET OR-SET
97. Evolution of a Set U-SET OR-SET
98. Evolution of a Set U-SET OR-SET OR-SWAT
99. [{a, 1}] {a, 1} Shelly
100. [{a, 1}] [{a, 1} {a, 1} Shelly {a, 1} Shelly
101. [{a, 1}] [{a, 1}, {b, 3}] {a, 1} Shelly {b, 1} {b, 2} {b, 3} Bob Pete Phil {a, 1} Shelly
102. [{a, 1}, {b,3}] [{a, 1}, {b, 3}] {a, 1} Shelly {b, 1} {b, 2} {b, 3} Bob Pete Phil {a, 1} Shelly {b, 1} {b, 2} {b, 3} Bob Pete Phil
103. [{a, 2}, {b, 3}] {b, 1} {b, 3} [{a, 1}, {b, 3}] Bob Pete {a, 1} Shelly {b, 1} {b, 2} {b, 3} Bob Pete Phil {a, 1} Shelly {a, 2} Anna
104. [{a, 2}, {b, 3}] {b, 1} {b, 3} [{a, 2}, {b, 3}] Bob Pete {a, 1} Shelly {b, 1} {a, 2} {b, 3} Bob Pete Anna {a, 1} Shelly {a, 2} Anna
105. GARBAGE Tombstones Dead Actors
106. GARBAGE Tombstones Dead Actors
107. No More Siblings!
109. Riak 2.0 Dev Preview http://docs.basho.com/riak/ 2.0.0pre11/downloads/
110. Bucket Types
111. Strong Consistency
112. Security
113. Search 2.0
114. http://bashojobs.resumator.com
115. Questions? joel@basho.com