Riak -- Seattle Scalability

Presentation I about Riak/future-of-Riak at the Seattle Scalability meetup in August 2013. Note: slides are designed for presentation and miss context without me talking.

Transcript

1. Joseph Blomstedt (@jtuple)
   Basho Technologies
   Seattle Scalability Meetup
   August 2013
   

   View Slide

2. 2
   Joe Blomstedt
   

   View Slide

3. 3
   Senior Engineer
   

   View Slide

4. 4
   joe@basho.com
   

   View Slide

5. 5
   @jtuple
   

   View Slide

6. View Slide

7. View Slide

8. 8
   Open Source
   

   View Slide

9. 9
   Open Source
   Apache 2
   github.com/basho
   

   View Slide

10. View Slide

11. 11
    Multi-DC Replication
    Support
    

    View Slide

12. Riak Basics
    12
    

    View Slide

13. View Slide

14. 14
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

15. 15
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

16. 16
    PUT
    GET
    DELETE
    

    View Slide

17. 17
    users/joe
    PUT
    

    View Slide

18. 18
    users/joe
    GET
    

    View Slide

19. 19
    users/joe
    DELETE
    

    View Slide

20. 20
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

21. 21
    user/1234
    PUT
    {first: “John”,
    last: “Doe”,
    age: 40}
    

    View Slide

22. 22
    user/1234
    PUT
    {first: “John”,
    last: “Doe”,
    age: 40}
    first_bin: “John”
    last_bin: “Doe”
    age_int: 40
    

    View Slide

23. 23
    pics/pic42
    PUT
    animal_bin: cat
    cuteness_int: 10
    

    View Slide

24. 24
    Tagged Fields
    Exact/Range
    Single Index
    Streaming
    Pagination
    

    View Slide

25. 25
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

26. 26
    Automatic Indexing
    JSON
    XML
    Text
    Custom
    

    View Slide

27. 27
    Querying
    Exact Match
    Wildcard
    Range
    

    View Slide

28. 28
    Querying
    Grouping
    Prefix matching
    Term Boosting
    

    View Slide

29. 29
    Querying
    AND
    OR
    NOT
    

    View Slide

30. 30
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

31. 31
    Use Cases
    Filtering
    Aggregation
    Extraction
    

    View Slide

32. 32
    M/R Input
    Known Keys
    2nd Index Query
    Full Text Query
    

    View Slide

33. 33
    Distributed
    Mappers: data nodes
    Reducers: coordinator
    

    View Slide

34. 34
    Key / Value
    Secondary Indexes
    Full Text Search
    Map/Reduce
    Counters
    

    View Slide

35. 35
    Why do people
    use Riak?
    

    View Slide

36. Big Data
    36
    

    View Slide

37. Big Data
    37
    Scalable
    Distributed
    

    View Slide

38. Big Data
    38
    5 nodes
    100 nodes
    

    View Slide

39. Critical Data
    39
    

    View Slide

40. Critical Data
    40
    Big
    Medium
    small
    

    View Slide

41. Fault Tolerance
    41
    

    View Slide

42. Availability
    42
    

    View Slide

43. CAP Theorem
    43
    

    View Slide

44. 44
    Partition-tolerance
    Consistency
    Availability
    

    View Slide

45. 45
    Partition-tolerance
    Consistency Availability
    

    View Slide

46. 46
    Partition-tolerance
    Consistency Availability
    CP AP
    

    View Slide

47. 47
    Partition-tolerance
    Consistency Availability
    CP AP
    

    View Slide

48. 48
    C/P
    Strict Quorum A/P
    Sloppy Quorum A/P
    

    View Slide

49. 49
    C/P
    Strict Quorum A/P
    Sloppy Quorum A/P
    

    View Slide

50. 50
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

51. 51
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

52. 52
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

53. 53
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

54. 54
    C/P
    Strict Quorum A/P
    Sloppy Quorum A/P
    

    View Slide

55. 55
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

56. 56
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

57. 57
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

58. 58
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

59. 59
    C/P
    Strict Quorum A/P
    Sloppy Quorum A/P
    

    View Slide

60. 60
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

61. 61
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

62. 62
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

63. 63
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client
    

    View Slide

64. 64
    Node 1 Node 2 Node 3 Node 4 Node 5
    client client
    client client client
    

    View Slide

65. Eventual Consistency
    65
    

    View Slide

66. 66
    A A A
    

    View Slide

67. 67
    A A A
    

    View Slide

68. 68
    A A A
    B
    

    View Slide

69. 69
    A A A
    B
    

    View Slide

70. 70
    A A A
    B
    B B B
    

    View Slide

71. 71
    A A A
    

    View Slide

72. 72
    A A A
    B C
    

    View Slide

73. 73
    A A A
    B C
    

    View Slide

74. 74
    A A A
    B
    {B,C} {B,C} {B,C}
    C
    

    View Slide

75. 75
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

76. 76
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

77. 77
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

78. 78
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

79. 79
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

80. 80
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    CRDTs/Monotonicity
    

    View Slide

81. 81
    Write Once
    Immutable
    Last Write Wins
    Business Rules
    “Set Union”
    

    View Slide

82. 82
    {1} {1} {1}
    

    View Slide

83. 83
    {1} {1} {1}
    

    View Slide

84. 84
    {1} {1} {1}
    {1,2} {1,3} {1,2,4}
    

    View Slide

85. 85
    {1} {1} {1}
    {1,2} {1,3} {1,2,4}
    

    View Slide

86. 86
    {1} {1} {1}
    {1,2}
    {1,2,3,4} {1,2,3,4} {1,2,3,4}
    {1,3} {1,2,4}
    

    View Slide

87. Clustering
    87
    

    View Slide

88. 88
    Consistent Hashing
    Sloppy Quorums
    Hinted Handoff
    

    View Slide

89. 89
    0 2160
    

    View Slide

90. View Slide

91. View Slide

92. 92
    Node 2
    Node 1
    Node 3
    

    View Slide

93. 93
    Node 2
    Node 1
    Node 3
    Node 4
    

    View Slide

94. 94
    Node 2
    Node 1
    Node 3
    Node 4
    SHA1(Key)
    

    View Slide

95. 95
    Node 2
    Node 1
    Node 3
    Node 4
    SHA1(Key)
    primary
    replicas
    

    View Slide

96. 96
    Node 2
    Node 1
    Node 3
    Node 4
    primary
    +
    fallback
    replicas
    

    View Slide

97. 97
    Node 2
    Node 1
    Node 3
    Node 4
    handoff
    

    View Slide

98. Self Healing
    98
    

    View Slide

99. 99
    Here Today
    !=
    Here Tomorrow
    

    View Slide

100. 100
     Read Repair
     

     View Slide

101. 101
     Active Anti Entropy
     

     View Slide

102. 102
     Node 2
     Node 1
     Node 3
     Node 4
     AAE exchange
     

     View Slide

103. 103
     Active Anti Entropy
     Realtime
     Persistent
     Rebuilt Weekly
     

     View Slide

104. Upcoming
     104
     

     View Slide

105. 105
     Data Types (CRDTs)
     Strong Consistency (C/P)
     Solr Integration
     

     View Slide

106. CRDTs
     106
     

     View Slide

107. View Slide

108. View Slide

109. 109
     Counters
     Sets
     Maps
     

     View Slide

110. 110
     Counters
     Sets
     Maps
     

     View Slide

111. 111
     1 1 1
     

     View Slide

112. 112
     1 1 1
     

     View Slide

113. 113
     1 1 1
     2 4
     +1 +3
     

     View Slide

114. 114
     1 1 1
     2 4
     +1 +3
     

     View Slide

115. 115
     1 1 1
     2
     {2,4} {2,4} {2,4}
     4
     +1 +3
     

     View Slide

116. 116
     (a,1) (a,1) (a,1)
     

     View Slide

117. 117
     (a,1) (a,1) (a,1)
     

     View Slide

118. 118
     (a,1)
     +1 +3
     (a,1) (a,1)
     (a,2) (a,1),(b,3)
     

     View Slide

119. 119
     (a,1)
     +1 +3
     (a,1) (a,1)
     (a,2) (a,1),(b,3)
     

     View Slide

120. 120
     (a,1)
     +1 +3
     (a,1) (a,1)
     (a,2) (a,1),(b,3)
     (a,2),(b,3) (a,2),(b,3) (a,1),(b,3)
     

     View Slide

121. 121
     (a,1)
     +1 +3
     (a,1) (a,1)
     (a,2) (a,1),(b,3)
     (a,2),(b,3) (a,2),(b,3) (a,1),(b,3)
     =5
     

     View Slide

122. 122
     Counters
     Sets
     Maps
     

     View Slide

123. 123
     Counters
     Sets
     Maps
     

     View Slide

124. Consistency
     124
     

     View Slide

125. 125
     A/P or C/P
     Per Bucket Choice
     

     View Slide

126. 126
     Single Key
     Optimistic CAS
     Counters/Sets/Maps
     

     View Slide

127. 127
     Multi Paxos
     Disjoint Ensembles
     Scalable
     

     View Slide

128. Node 2
     Node 1
     Node 3
     

     View Slide

129. 129
     1 2
     3
     4
     5
     6
     7
     123
     

     View Slide

130. 130
     1 2
     3
     4
     5
     6
     7
     123
     234
     

     View Slide

131. 131
     1 2
     3
     4
     5
     6
     7
     123
     234
     345
     

     View Slide

132. 132
     1 2
     3
     4
     5
     6
     7
     123
     234
     345
     456
     

     View Slide

133. 133
     1 2
     3
     4
     5
     6
     7
     123
     234
     345
     456
     567...
     

     View Slide

134. 134
     1 2
     3
     4
     5
     6
     7
     123
     Ensembles
     234
     345
     456
     567...
     

     View Slide

135. 135
     Ensemble
     Indepedent Paxos Group
     Leader Election
     Serializability
     

     View Slide

136. 136
     write 2
     1
     1
     1
     write 2
     leader
     

     View Slide

137. 137
     write 2
     2
     2
     2
     write 2
     leader
     2
     X
     

     View Slide

138. 138
     3
     3
     3
     write 3
     leader
     3
     

     View Slide

139. 139
     3
     3
     3
     leader
     

     View Slide

140. 140
     +1
     1
     1
     1
     +1
     leader
     

     View Slide

141. 141
     2
     2
     2
     +1
     leader
     2
     

     View Slide

142. 142
     3
     3
     3
     leader
     3
     

     View Slide

143. 143
     3
     3
     3
     leader
     

     View Slide

144. Riak + Solr
     144
     

     View Slide

145. 145
     Next Generation Search
     Full Distributed Solr
     Embedded in Riak
     Active Anti Entropy
     

     View Slide

146. End
     146
     

     View Slide

147. 147
     We hiring!
     

     View Slide

148. View Slide

149. View Slide

150. ricon.io
     

     View Slide

151. Questions?
     151
     

     View Slide