Infinispan in the world dominated by RAFT

Slides for a talk I gave at Geecon Prague 2017. Source code for the demo is available at https://github.com/jfojtl/geecon17

1. in the world dominated by @jakubfojtl Infinispan RAFT @ysoftdevs
4. Enterprise workflow system
5. Enterprise workflow system “software for printers”
6. Enterprise workflow system “software for printers”
7. Really cool
8. Really cool we build Highly Available™ distributed system …
9. Really cool we build Highly Available™ distributed system … … which runs outside of our control
10. Agenda
11. Agenda Brief intro to Infinispan
12. Agenda Brief intro to Infinispan How we use it
13. Agenda Brief intro to Infinispan How we use it Sample application - demo
14. Infinispan Distributed in-memory datagrid
15. Infinispan Distributed in-memory datagrid
16. Infinispan Distributed in-memory datagrid Distributed Cache
17. Infinispan Distributed in-memory datagrid Distributed Cache NoSQL
 database
18. Infinispan Distributed in-memory datagrid Distributed Cache NoSQL
 database Event Broker
19. Infinispan Distributed in-memory datagrid Distributed Cache NoSQL
 database Event Broker Data
 Analytics
21. Hosting
22. Hosting in-process
23. Hosting in-process server
 distribution
24. Hosting Connecting in-process server
 distribution
25. Hosting Connecting in-process server
 distribution Java API
26. Hosting Connecting in-process server
 distribution Java API HotRod
27. Hosting Connecting in-process server
 distribution Java API REST HotRod
28. Building blocks
29. JGroups Building blocks
30. Lucene JGroups Building blocks
31. Lucene JBossMarshalling JGroups Building blocks
32. Lucene JBossMarshalling Proto buffers JGroups Building blocks
33. Lucene JBossMarshalling Proto buffers JGroups Push events Building blocks
34. Lucene JBossMarshalling Proto buffers Transactions JGroups Push events Building blocks
35. Lucene JBossMarshalling Proto buffers Transactions JGroups Push events Building blocks mostly optional
36. 4 modes of data distribution
37. 4 modes of data distribution
38. 4 modes of data distribution Local
39. 4 modes of data distribution Local Invalidation
40. 4 modes of data distribution Local Replicated Invalidation
41. 4 modes of data distribution Local Replicated Invalidation Distributed
42. Distributed mode
43. Distributed mode ISPN cluster
44. Distributed mode ISPN cluster Client
45. Distributed mode W(k,v) ISPN cluster Client
46. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] ISPN cluster Client
47. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] ISPN cluster Client Consistent hashing function
48. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] numOwners ISPN cluster Client Consistent hashing function
49. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] numOwners W op requires fixed # of nodes ISPN cluster Client Consistent hashing function
50. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] numOwners scales linearly W op requires fixed # of nodes ISPN cluster Client Consistent hashing function
51. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] numOwners scales linearly W op requires fixed # of nodes sync & async versions ISPN cluster Client Consistent hashing function
52. Distributed mode W(k,v) f(k) -> [nodeX,nodeY] numOwners scales linearly W op requires fixed # of nodes sync & async versions ISPN cluster W op is always accepted - AP system Client Consistent hashing function
53. RAFT
54. RAFT
55. RAFT Client
56. RAFT W(k,v) Client
57. RAFT W(k,v) Leader Client
58. RAFT W(k,v) Leader Client
59. RAFT W(k,v) Single partially-ordered replicated log Leader Client
60. RAFT W(k,v) Single partially-ordered replicated log Requires n/2+1 nodes for W ops (works best with odd # of nodes) - CP system Leader Client
61. RAFT W(k,v) Single partially-ordered replicated log Requires n/2+1 nodes for W ops (works best with odd # of nodes) - CP system Available as long as n/2+1 of cluster is healthy Leader Client
62. RAFT W(k,v) Single partially-ordered replicated log Requires n/2+1 nodes for W ops (works best with odd # of nodes) - CP system Available as long as n/2+1 of cluster is healthy Complicated recovery Leader Client
63. RAFT
64. RAFT
65. RAFT #R >> #W
66. RAFT #R >> #W secretlifeofdata.com
67. How we use it
68. ISPN cluster So you have a cluster…
69. ISPN cluster So you have a cluster… Split may
 will happen
70. ISPN cluster So you have a cluster… Split may
 will happen
71. And suddenly you have 2 clusters ISPN cluster ISPN cluster
72. And suddenly you have 2 clusters ISPN cluster ISPN cluster rebalance happens
73. And suddenly you have 2 clusters ISPN cluster ISPN cluster both clusters are available rebalance happens
74. And suddenly you have 2 clusters ISPN cluster ISPN cluster Client Client both clusters are available rebalance happens
75. And suddenly you have 2 clusters ISPN cluster ISPN cluster Client Client W(k,v) W(k,v) both clusters are available rebalance happens
76. ISPN cluster ISPN cluster + When the problem is solved
77. ISPN cluster ISPN cluster + When the problem is solved there are accumulated data on both sides of split brain
78. ISPN cluster ISPN cluster + When the problem is solved there are accumulated data on both sides of split brain Conflict resolution
79. Merging
80. Merging
81. Merging Merge needs to happen when you have 2 different values on the same key
82. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies
83. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies MergePolicies.PREFERRED_ALWAYS
84. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies MergePolicies.PREFERRED_ALWAYS MergePolicies.PREFERRED_NON_NULL
85. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies MergePolicies.PREFERRED_ALWAYS MergePolicies.PREFERRED_NON_NULL MergePolicies.REMOVE_ALL
86. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies MergePolicies.PREFERRED_ALWAYS MergePolicies.PREFERRED_NON_NULL MergePolicies.REMOVE_ALL Manual
87. Merging Merge needs to happen when you have 2 different values on the same key Infinispan has merging policies MergePolicies.PREFERRED_ALWAYS MergePolicies.PREFERRED_NON_NULL MergePolicies.REMOVE_ALL … you need to decide what happened first Manual
88. Time
89. Time
90. Time T1
91. Time T1 T2
92. Time T1 T2 <
93. Time T1 T2 <
94. Time T1 T2 5 s <
95. Time
96. Time is not continuous
97. Time is not continuous very hard to synchronise clocks across cluster
98. Time is not continuous very hard to synchronise clocks across cluster do not use time for coordination
99. Time is not continuous very hard to synchronise clocks across cluster do not use time for coordination Talk: ordering of events in Systems by @kavya
100. Append-only approach
101. Append-only approach we only add data to DB
102. Append-only approach we only add data to DB
103. Append-only approach we only add data to DB model changes of data as events on one entity
104. Append-only approach we only add data to DB model changes of data as events on one entity 3 kinds of events
105. Append-only approach we only add data to DB model changes of data as events on one entity 1. Order of changes is determined based on the type of the event 3 kinds of events
106. Append-only approach we only add data to DB model changes of data as events on one entity 1. Order of changes is determined based on the type of the event airplane journey: take off < landing < taxi 3 kinds of events
107. Append-only approach
108. Append-only approach
109. Append-only approach 2. commutative & idempotent events
110. Append-only approach 2. commutative & idempotent events e1 + e2 = e2 + e1
111. Append-only approach 2. commutative & idempotent events e1 + e2 + e2 = e2 + e1 e1 + e2 = e2 + e1
112. Append-only approach inspired by some properties of Commutative replicated data structures (CRDTs) 2. commutative & idempotent events e1 + e2 + e2 = e2 + e1 e1 + e2 = e2 + e1
113. Append-only approach inspired by some properties of Commutative replicated data structures (CRDTs) 2. commutative & idempotent events e1 + e2 + e2 = e2 + e1 3. Order of events does not matter e1 + e2 = e2 + e1
114. Write operations do not require coordination
 at all
115. Op requiring coordination
116. Op requiring coordination sequence number generation
117. Op requiring coordination sequence number generation
118. Op requiring coordination sequence number generation Client
119. Op requiring coordination sequence number generation Client S: 4
120. Op requiring coordination sequence number generation Client S: 4 5?
121. Op requiring coordination sequence number generation Client S: 4
122. Op requiring coordination sequence number generation Client S: 4 OK
123. Op requiring coordination sequence number generation Client S: 4
124. Op requiring coordination sequence number generation Client S: 4 5!
125. Op requiring coordination sequence number generation Client S: 4
126. Op requiring coordination sequence number generation Client S: 4 OK
127. Op requiring coordination sequence number generation Client S: 5
128. Op requiring coordination sequence number generation Client S: 5 No coordination vs
129. Op requiring coordination sequence number generation Client S: 5 No coordination vs Store operation can be done only on one node without direct involvement of other nodes
130. Op requiring coordination sequence number generation Client S: 5 No coordination no messages - no problem :) vs Store operation can be done only on one node without direct involvement of other nodes
131. Transactions
132. Transactions
133. Transactions For order of operations
134. Transactions For order of operations Roll-back incomplete writes to not damage data
135. Transactions W For order of operations Roll-back incomplete writes to not damage data
136. Transactions W For order of operations Roll-back incomplete writes to not damage data
137. Transactions W For order of operations Roll-back incomplete writes to not damage data tx
138. Transactions W For order of operations Roll-back incomplete writes to not damage data tx no update to existing data
139. Transactions W For order of operations Roll-back incomplete writes to not damage data tx no update to existing data NoSQL - no foreign keys
140. Transactions W For order of operations Roll-back incomplete writes to not damage data tx no update to existing data NoSQL - no foreign keys data cannot be damaged
141. Transactions W For order of operations Roll-back incomplete writes to not damage data no update to existing data NoSQL - no foreign keys data cannot be damaged
142. Transactions W For order of operations Roll-back incomplete writes to not damage data R no update to existing data NoSQL - no foreign keys data cannot be damaged read query must be designed to handle missing parts
143. Transactions W For order of operations Roll-back incomplete writes to not damage data R no update to existing data NoSQL - no foreign keys data cannot be damaged read query must be designed to handle missing parts leaves unwanted artefacts - requires cleaning
144. Client Client service service service ISPN cluster
145. Client Client microservice microservice microservice ISPN cluster
146. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes
147. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes op
148. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes op
149. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes op
150. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes
151. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes
152. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover
153. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover op
154. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover op
155. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover op
156. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover UUID for keys op
157. Client Client microservice microservice microservice ISPN cluster client responsible for finishing
 users’ wishes failover UUID for keys when client is down expectation of the user is that everything is lost op
158. Client PUT /resource { … “op-id” : “Resource.05c1b930- add8-40cd-840c-920ed77429ed” … }
159. Client PUT /resource { … “op-id” : “Resource.05c1b930- add8-40cd-840c-920ed77429ed” … } Idempotent
160. Client PUT /resource { … “op-id” : “Resource.05c1b930- add8-40cd-840c-920ed77429ed” … } Idempotent Distinquish between 2 wishes and 1 repeated wish
161. Client PUT /resource { … “op-id” : “Resource.05c1b930- add8-40cd-840c-920ed77429ed” … } Idempotent Distinquish between 2 wishes and 1 repeated wish Monitoring
162. demo
163. Take away
164. Take away
165. Take away RAFT is not bad
166. Take away RAFT is not bad Try to avoid coordination
167. Take away RAFT is not bad Distributed system is hard to encapsulate Try to avoid coordination
168. Take away RAFT is not bad Distributed system is hard to encapsulate The more resilient your application is the easier your life becomes once in production Try to avoid coordination
169. Take away RAFT is not bad Distributed system is hard to encapsulate The more resilient your application is the easier your life becomes once in production Know the properties and limitations of the building blocks you use Try to avoid coordination
170. @jakubfojtl | jfojtl.com Thank you Used resources can be found at https://pastebin.com/pqeD6M9n source code at https://github.com/jfojtl/geecon17