Data Dissemination: From Academia to Industry

Disseminating information across a large set of processes is a classic problem in distributed systems. In this talk we will present and discuss several classes of epidemic solutions for achieving reliable broadcast across a large number of processes that can be found in the scientific literature, and briefly explore their strong and weak points. Then we will focus on a concrete approach based on the combination of different gossiping techniques such that implicit, albeit robust, spanning trees can be leveraged to disseminate information both reliably and efficiently. We further discuss how this technique which spawned from academia, ended up being used in Industry, as a part of the Cluster Metadata Subsystem of Basho's Riak. We'll explore some of the key differences between the original academic proposal and its applications on Riak and discuss the practical reasons that motivated these modifications. Finally, we will discuss some future techniques that Basho is currently exploring.

Transcript

1. Data Dissemination: from Academia to Industry Jo˜ ao Leit˜ ao NOVA-LINCS & NOVA University of Lisbon Jordan West BASHO Inc. RICON Las Vegas October 2014
2. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
3. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
4. Motivation Data Dissemination Classical Distributed Systems Challenge How to disseminate information across a large number of participants? Some intuitive requirements: Reliable. Efficient. Scalable.
5. Motivation Data Dissemination Classical Distributed Systems Challenge How to disseminate information across a large number of participants? Some intuitive requirements: Reliable. Efficient. Scalable.
6. Motivation Data Dissemination Applications Notification systems. Streaming multimedia content. Cluster Management. In practice... When I started to think about this problem, I was mostly focused on peer-to-peer systems.
7. Motivation Data Dissemination Applications Notification systems. Streaming multimedia content. Cluster Management. In practice... When I started to think about this problem, I was mostly focused on peer-to-peer systems.
8. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
9. The Academic View: Data Dissemination Design Alternatives: One to All Lets start simple... If you have information to disseminate, send it to everyone! One to All
10. The Academic View: Data Dissemination Design Alternatives: One to All Lets start simple... If you have information to disseminate, send it to everyone! One to All
11. The Academic View: Data Dissemination Design Alternatives: One to All
12. The Academic View: Data Dissemination Design Alternatives: One to All
13. The Academic View: Data Dissemination Design Alternatives: One to All
14. The Academic View: Data Dissemination Design Alternatives: One to All
15. The Academic View: Data Dissemination Design Alternatives: One to All
16. The Academic View: Data Dissemination Design Alternatives: One to All
17. The Academic View: Data Dissemination Design Alternatives: One to All
18. The Academic View: Data Dissemination Design Alternatives: One to All One to All Positive Aspects: Straight forward. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Requires each node to know the full membership. Not Scalable (no load distribution). Not Resilient to Faults.
19. The Academic View: Data Dissemination Design Alternatives: One to All One to All Positive Aspects: Straight forward. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Requires each node to know the full membership. Not Scalable (no load distribution). Not Resilient to Faults.
20. The Academic View: Data Dissemination Design Alternatives: One to All One to All Positive Aspects: Straight forward. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Requires each node to know the full membership. Not Scalable (no load distribution). Not Resilient to Faults.
21. The Academic View: Data Dissemination Design Alternatives: One to All
22. The Academic View: Data Dissemination Design Alternatives: One to All
23. The Academic View: Data Dissemination Design Alternatives: One to All
24. The Academic View: Data Dissemination Design Alternatives: Spanning Tree Dealing with Load Distribution Organize participants/nodes in a tree and forward messages across the tree. Spanning Tree
25. The Academic View: Data Dissemination Design Alternatives: Spanning Tree Dealing with Load Distribution Organize participants/nodes in a tree and forward messages across the tree. Spanning Tree
26. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
27. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
28. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
29. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
30. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
31. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
32. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
33. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
34. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
35. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
36. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
37. The Academic View: Data Dissemination Design Alternatives: Spanning Tree Spanning Tree Positive Aspects: Load Distribution. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Complexity in Managing the Topology (Hinders Scalability). (Still) Not Resilient to Faults.
38. The Academic View: Data Dissemination Design Alternatives: Spanning Tree Spanning Tree Positive Aspects: Load Distribution. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Complexity in Managing the Topology (Hinders Scalability). (Still) Not Resilient to Faults.
39. The Academic View: Data Dissemination Design Alternatives: Spanning Tree Spanning Tree Positive Aspects: Load Distribution. No redundancy (i.e, each node receives each message a single time). Negative Aspects: Complexity in Managing the Topology (Hinders Scalability). (Still) Not Resilient to Faults.
40. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
41. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
42. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
43. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
44. The Academic View: Data Dissemination Design Alternatives: Spanning Tree
45. The Academic View: Data Dissemination Design Alternatives: Flood Dealing with Fault Tolerance Organize participants/nodes in a random, highly connected, graph and forward messages across all links. Flood
46. The Academic View: Data Dissemination Design Alternatives: Flood Dealing with Fault Tolerance Organize participants/nodes in a random, highly connected, graph and forward messages across all links. Flood
47. The Academic View: Data Dissemination Design Alternatives: Flood
48. The Academic View: Data Dissemination Design Alternatives: Flood
49. The Academic View: Data Dissemination Design Alternatives: Flood
50. The Academic View: Data Dissemination Design Alternatives: Flood
51. The Academic View: Data Dissemination Design Alternatives: Flood
52. The Academic View: Data Dissemination Design Alternatives: Flood
53. The Academic View: Data Dissemination Design Alternatives: Flood
54. The Academic View: Data Dissemination Design Alternatives: Flood
55. The Academic View: Data Dissemination Design Alternatives: Flood
56. The Academic View: Data Dissemination Design Alternatives: Flood
57. The Academic View: Data Dissemination Design Alternatives: Flood Flood Positive Aspects: Load Distribution. Simple Design, and a random overlay is easier to maintain than a tree. Robust (due to inherent redudancy). Negative Aspects: No longer that efficient (nodes receive and are required to process several copies of each message).
58. The Academic View: Data Dissemination Design Alternatives: Flood Flood Positive Aspects: Load Distribution. Simple Design, and a random overlay is easier to maintain than a tree. Robust (due to inherent redudancy). Negative Aspects: No longer that efficient (nodes receive and are required to process several copies of each message).
59. The Academic View: Data Dissemination Design Alternatives: Gossip Addressing Efficiency Organize participants/nodes in a random, highly connected, graph and have nodes forward each message across a subset (f ) of the links. Gossip
60. The Academic View: Data Dissemination Design Alternatives: Gossip Addressing Efficiency Organize participants/nodes in a random, highly connected, graph and have nodes forward each message across a subset (f ) of the links. Gossip
61. The Academic View: Data Dissemination Design Alternatives: Gossip
62. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
63. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
64. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
65. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
66. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
67. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
68. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
69. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
70. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
71. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
72. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip"fanout"="2"
73. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip Positive Aspects: Load Distribution. Simple Design, and a random overlay is easier to maintain than a tree. Robust (due to inherent redudancy). Negative Aspects: Slightly inefficient (but better than flood). No longer predictable (each message will be disseminated across different links).
74. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip Positive Aspects: Load Distribution. Simple Design, and a random overlay is easier to maintain than a tree. Robust (due to inherent redudancy). Negative Aspects: Slightly inefficient (but better than flood). No longer predictable (each message will be disseminated across different links).
75. The Academic View: Data Dissemination Design Alternatives: Gossip Gossip Positive Aspects: Load Distribution. Simple Design, and a random overlay is easier to maintain than a tree. Robust (due to inherent redudancy). Negative Aspects: Slightly inefficient (but better than flood). No longer predictable (each message will be disseminated across different links).
76. The Academic View: Data Dissemination Design Alternatives: Embedded Tree All these design alternatives: One-to-All. Spanning Tree. Flood. Gossip.
77. The Academic View: Data Dissemination Design Alternatives: Embedded Tree All these design alternatives: Spanning Tree. Flood. Gossip.
78. The Academic View: Data Dissemination Design Alternatives: Embedded Tree All these design alternatives: Spanning Tree [Efficiency]. Flood. Gossip.
79. The Academic View: Data Dissemination Design Alternatives: Embedded Tree All these design alternatives: Spanning Tree [Efficiency]. Flood. [Robustness]. Gossip. [Robustness].
80. The Academic View: Data Dissemination Design Alternatives: Embedded Tree At some point you start to see trees everywhere...
81. The Academic View: Data Dissemination Design Alternatives: Embedded Tree
82. The Academic View: Data Dissemination Design Alternatives: Embedded Tree
83. The Academic View: Data Dissemination Design Alternatives: Embedded Tree
84. The Academic View: Data Dissemination Design Alternatives: Embedded Tree
85. The Academic View: Data Dissemination Design Alternatives: Embedded Tree
86. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Gossip"fanout"="2"
87. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Gossip"fanout"="2"
88. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Gossip"fanout"="2"
89. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Gossip"fanout"="2"
90. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Gossip"fanout"="2"
91. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Observation: The closure of the paths the lead to the first delivery of a message to each node forms a tree. This observation: Lead us to design the Plumtree protocol.
92. The Academic View: Data Dissemination Design Alternatives: Embedded Tree Observation: The closure of the paths the lead to the first delivery of a message to each node forms a tree. This observation: Lead us to design the Plumtree protocol.
93. Embedded Trees: Plumtree Protocol Epidemic Broadcast Trees. Jo˜ ao Leit˜ ao, Jos´ e Pereira and Lu´ ıs Rodrigues Proceedings of the 26th IEEE International Symposium on Reliable Distributed Systems, Beijing, China, October, 2007.
94. Embedded Trees: Plumtree Protocol Gossip modes of operation There are 1001 ways to Gossip... Eager push Nodes send the message payload to random selected peers as soon as they receive a message for the first time. Lazy push When a node receive a message for the first time, they only send that message identifier to random selected peers. If those peers have not received the message, they make an explicit request.
95. Embedded Trees: Plumtree Protocol Gossip modes of operation There are 1001 ways to Gossip... Eager push Nodes send the message payload to random selected peers as soon as they receive a message for the first time. Lazy push When a node receive a message for the first time, they only send that message identifier to random selected peers. If those peers have not received the message, they make an explicit request.
96. Embedded Trees: Plumtree Protocol Gossip modes of operation There are 1001 ways to Gossip... Eager push Nodes send the message payload to random selected peers as soon as they receive a message for the first time. Lazy push When a node receive a message for the first time, they only send that message identifier to random selected peers. If those peers have not received the message, they make an explicit request.
97. Embedded Trees: Plumtree Protocol Overview Plumtree: push-lazy-push multicast tree. It operates as any gossip protocol, each node gossips with f neighbors on top of a random overlay. Decentralized. Creates (and maintain) an embedded tree on a random overlay. It combines two distinct gossip strategies: Eager Push: In the random overlay links that belong to the embedded spanning tree. Lazy Push: In the remaining random overlay links. TCP is used between nodes as it offers a better reliability and an additional fault detection mechanism.
98. Embedded Trees: Plumtree Protocol Overview Plumtree: push-lazy-push multicast tree. It operates as any gossip protocol, each node gossips with f neighbors on top of a random overlay. Decentralized. Creates (and maintain) an embedded tree on a random overlay. It combines two distinct gossip strategies: Eager Push: In the random overlay links that belong to the embedded spanning tree. Lazy Push: In the remaining random overlay links. TCP is used between nodes as it offers a better reliability and an additional fault detection mechanism.
99. Embedded Trees: Plumtree Protocol Overview Plumtree: push-lazy-push multicast tree. It operates as any gossip protocol, each node gossips with f neighbors on top of a random overlay. Decentralized. Creates (and maintain) an embedded tree on a random overlay. It combines two distinct gossip strategies: Eager Push: In the random overlay links that belong to the embedded spanning tree. Lazy Push: In the remaining random overlay links. TCP is used between nodes as it offers a better reliability and an additional fault detection mechanism.
100. Embedded Trees: Plumtree Protocol Building the Tree Intuition: Use the links in the overlay that generated a message delivery. After initialization all links in the overlay are candidates do belong to the spanning tree. When a message is broadcasted, all links used to disseminate a redundant message are pruned from the tree. If the random overlay is connected, after the dissemination of a message the tree will cover all nodes.
101. Embedded Trees: Plumtree Protocol Building the Tree Intuition: Use the links in the overlay that generated a message delivery. After initialization all links in the overlay are candidates do belong to the spanning tree. When a message is broadcasted, all links used to disseminate a redundant message are pruned from the tree. If the random overlay is connected, after the dissemination of a message the tree will cover all nodes.
102. Embedded Trees: Plumtree Protocol Building the Tree
103. Embedded Trees: Plumtree Protocol Building the Tree
104. Embedded Trees: Plumtree Protocol Building the Tree
105. Embedded Trees: Plumtree Protocol Building the Tree
106. Embedded Trees: Plumtree Protocol Building the Tree
107. Embedded Trees: Plumtree Protocol Building the Tree
108. Embedded Trees: Plumtree Protocol Building the Tree
109. Embedded Trees: Plumtree Protocol Building the Tree
110. Embedded Trees: Plumtree Protocol Building the Tree
111. Embedded Trees: Plumtree Protocol Building the Tree
112. Embedded Trees: Plumtree Protocol Recovering the Tree If a node fails some nodes might become disconnected from the embedded tree. When a node receives a announcement for a message it did not received it starts a timer. If the timer expires before receiving the message the node explicitly requests it to the node that sent the announcement. Reintroduces the link to that node into the embedded tree. The node that had sent the announcement sends the message when requested. It also reintroduces that link into the embedded tree. At the end of this process the embedded tree is repaired.
113. Embedded Trees: Plumtree Protocol Recovering the Tree If a node fails some nodes might become disconnected from the embedded tree. When a node receives a announcement for a message it did not received it starts a timer. If the timer expires before receiving the message the node explicitly requests it to the node that sent the announcement. Reintroduces the link to that node into the embedded tree. The node that had sent the announcement sends the message when requested. It also reintroduces that link into the embedded tree. At the end of this process the embedded tree is repaired.
114. Embedded Trees: Plumtree Protocol Recovering the Tree If a node fails some nodes might become disconnected from the embedded tree. When a node receives a announcement for a message it did not received it starts a timer. If the timer expires before receiving the message the node explicitly requests it to the node that sent the announcement. Reintroduces the link to that node into the embedded tree. The node that had sent the announcement sends the message when requested. It also reintroduces that link into the embedded tree. At the end of this process the embedded tree is repaired.
115. Embedded Trees: Plumtree Protocol Recovering the Tree If a node fails some nodes might become disconnected from the embedded tree. When a node receives a announcement for a message it did not received it starts a timer. If the timer expires before receiving the message the node explicitly requests it to the node that sent the announcement. Reintroduces the link to that node into the embedded tree. The node that had sent the announcement sends the message when requested. It also reintroduces that link into the embedded tree. At the end of this process the embedded tree is repaired.
116. Embedded Trees: Plumtree Protocol Experimental Evaluation Evaluation was conducted with the Peersim simulator. 10.000 nodes system. Hundreds and hundreds of Simulations. Compare the performance between: A standard (push) Gossip Protocol named Eager. Several configurations of Plumtree.
117. Embedded Trees: Plumtree Protocol Experimental Evaluation Evaluation was conducted with the Peersim simulator. 10.000 nodes system. Hundreds and hundreds of Simulations. Compare the performance between: A standard (push) Gossip Protocol named Eager. Several configurations of Plumtree.
118. Embedded Trees: Plumtree Protocol Experimental Evaluation
119. Embedded Trees: Plumtree Protocol Experimental Evaluation
120. Embedded Trees: Plumtree Protocol Experimental Evaluation
121. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
122. The Industry View: Cluster Metadata Management Global Information, Needed Locally Operational Information Mebership, Ownership, Transfers, Capabilities Configuration Replica Counts, Backends, Is Feature X Enabled? Monitoring Data Transfer Statistics, Background Work Tracking, Self-Monitoring
123. The Industry View: Cluster Metadata Management Requirements Failure Tolerance Disable malicious clients despite network partitions Add new nodes despite others being down Quick Convergence Become aware of new nodes ASAP Keep monitoring data as fresh as possible Minimal Resource Usage Goal is to speed up, not slow down, foreground work Network is needed for the data! AP typically, CP rarely necessary
124. The Industry View: Cluster Metadata Management Requirements Failure Tolerance Disable malicious clients despite network partitions Add new nodes despite others being down Quick Convergence Become aware of new nodes ASAP Keep monitoring data as fresh as possible Minimal Resource Usage Goal is to speed up, not slow down, foreground work Network is needed for the data! AP typically, CP rarely necessary
125. The Industry View: Cluster Metadata Management Requirements Failure Tolerance Disable malicious clients despite network partitions Add new nodes despite others being down Quick Convergence Become aware of new nodes ASAP Keep monitoring data as fresh as possible Minimal Resource Usage Goal is to speed up, not slow down, foreground work Network is needed for the data! AP typically, CP rarely necessary
126. The Industry View: Cluster Metadata Management Requirements Failure Tolerance Disable malicious clients despite network partitions Add new nodes despite others being down Quick Convergence Become aware of new nodes ASAP Keep monitoring data as fresh as possible Minimal Resource Usage Goal is to speed up, not slow down, foreground work Network is needed for the data! AP typically, CP rarely necessary
127. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
128. Cluster Metadata Management In Riak Implementations 0.x Peer Service: Classic Anti-Entropy Protocol Data Dissemination: Same 1.x Peer Service: Static Graph Overlay w/ Anti-Entropy Data Dissemination: Same 2.x Peer Service: Static Graph Overlay w/ Anti-Entropy Data Dissemination: Sender-Based Embedded Trees
129. Cluster Metadata Management In Riak Motivations 0.x to 1.0 Anti-Entropy is Reliable but Slow to Converge Improve Convergence Time, Specifically in Failure Scenarios 1.x to 2.0 Static Graph Overlay/Anti-Entropy have several issues: Too Much Network Overhead Improved Convergence Only Needed for Subset of Data Cold Rumor Mongering Improve Resource Usage and Stability
130. Cluster Metadata Management In Riak Plumtree: From Academia to Industry Connectivity “all nodes should have in their partial views...another correct node” is not an invariant we can maintain System Model Must consider dropped, delayed, duplicated* messages in addition to node failures Topology Peer Service is Fully Connected (as is Erlang) Membership Operator-controlled instead of reactive (failure detector is implemented separately from peer service)
131. Cluster Metadata Management In Riak Extending Plumtree What happens if Lazy Messages are Dropped? Add to Outstanding Set. Remove on Graft or Ack Handling Extreme Failures Back protocol by random anti-entropy with peers that are not in eager or lazy sets Shared vs. Sender-Based Trees Sender-Based. Minimize overhead w/ Lazy Construction
132. Cluster Metadata Management In Riak Measurements Reduced convergence time (measured in LDH) by 66% Reduced needless message overhead in stable state 99.9% % Failures before needing to rely on anti-entropy good for small clusters (near 90%) but decreased quickly for larger clusters (under 50%)
133. Cluster Metadata Management In Riak Peer Service Tuning Want to increase percent failures we can tolerate before relying on anti-entropy Fanout of tree given to us by peer service matters For clusters > 10 nodes, fanout = ln(cluster size) + 1 With our use of multiple trees, more peer service tuning may be necessary
134. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
135. The Academic View: When One Tree is not Enough Understanding The Problem We had Plumtree: Efficient. Very Robust. Could even rebalance itself to improve latency (height of the tree). The reality about trees... Trees do not distribute the load evenly among participants.
136. The Academic View: When One Tree is not Enough Understanding The Problem We had Plumtree: Efficient. Very Robust. Could even rebalance itself to improve latency (height of the tree). The reality about trees... Trees do not distribute the load evenly among participants.
137. The Academic View: When One Tree is not Enough Understanding The Problem
138. The Academic View: When One Tree is not Enough Understanding The Problem
139. The Academic View: When One Tree is not Enough Understanding The Problem 4"in"13"nodes"
140. The Academic View: When One Tree is not Enough Understanding The Problem 4"in"13"nodes" 9"in"13"nodes"
141. The Academic View: When One Tree is not Enough Naive Solutions How hard can this be? We started by experimenting with two simple approaches: NUTS: Naive Unstructured spliTStream BOLTS: Basic multiple Overlay-TreeS
142. The Academic View: When One Tree is not Enough Naive Solutions: NUTS Pick t nodes at random and create t Plumtrees; each rooted at a different node.
143. The Academic View: When One Tree is not Enough Naive Solutions: NUTS Pick t nodes at random and create t Plumtrees; each rooted at a different node.
144. The Academic View: When One Tree is not Enough Naive Solutions: NUTS Pick t nodes at random and create t Plumtrees; each rooted at a different node.
145. The Academic View: When One Tree is not Enough Naive Solutions: BOLTS Create t different unstructured overlays and create a Plumtree using each overlay.
146. The Academic View: When One Tree is not Enough Naive Solutions: BOLTS Create t different unstructured overlays and create a Plumtree using each overlay.
147. The Academic View: When One Tree is not Enough Naive Solutions: NUTS & BOLTS
148. The Academic View: When One Tree is not Enough Understanding The Goal From one tree to multiple trees Load balancing: every node should be interior on the same number of trees Use all resources: each node should be interior in at least one tree Fault-tolerance: each node should be interior in at most one tree
149. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
150. The Thicket Protocol Thicket: A Protocol for Building and Maintaining Multiple Trees in a P2P Overlay. M´ ario Ferreira, Jo˜ ao Leit˜ ao, and Lu´ ıs Rodrigues Proceedings of the 29th IEEE Symposium on Reliable Distributed Systems (SRDS), New Delhi, India, 31 October-3 November 2010.
151. The Thicket Protocol Deploying the Trees We use the same principle as in Plumtree to define trees, however: Overlay links are explicitly added to each tree (instead of removed). Trees are deployed by taking into consideration the remaining trees. Finding the minimal coordination necessary to achieve our goal is tricky. Tree coverage is impossible at tree deployment time.
152. The Thicket Protocol Deploying the Trees We use the same principle as in Plumtree to define trees, however: Overlay links are explicitly added to each tree (instead of removed). Trees are deployed by taking into consideration the remaining trees. Finding the minimal coordination necessary to achieve our goal is tricky. Tree coverage is impossible at tree deployment time.
153. The Thicket Protocol Deploying the Trees We use the same principle as in Plumtree to define trees, however: Overlay links are explicitly added to each tree (instead of removed). Trees are deployed by taking into consideration the remaining trees. Finding the minimal coordination necessary to achieve our goal is tricky. Tree coverage is impossible at tree deployment time.
154. The Thicket Protocol Recovering Trees Global coverage of trees is provided by the repair mechanism. All messages exchanged among nodes convey information about the load of the sender. How many messages it sends per time unit. How many child nodes in each tree. Tree repair uses less loaded nodes.
155. The Thicket Protocol Recovering Trees Global coverage of trees is provided by the repair mechanism. All messages exchanged among nodes convey information about the load of the sender. How many messages it sends per time unit. How many child nodes in each tree. Tree repair uses less loaded nodes.
156. The Thicket Protocol Experimental Results Simulated a 10.000 overlay using PeerSim. Comparison with NUTS and BOLTS (and Plumtree). Same underlying unstructured overlay network. Assumes reliable FIFO channels.
157. The Thicket Protocol Experimental Results
158. The Thicket Protocol Experimental Results
159. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
160. The Industry View: Improving Cluster Metadata Measuring Interior Nodes 10 Nodes 4 of 10 nodes are interior in 3 of 10 trees 3 of 10 nodes are interior in 4 of 10 trees 3 of 10 nodes are interior in 5 of 10 trees 20 Nodes 4 nodes are interior only when roots 16 nodes are interior in 5 or more trees 80 Nodes 4 nodes are interior only when roots 36 of 80 nodes are interior in > 25% of trees
161. The Industry View: Improving Cluster Metadata Measuring Interior Nodes 10 Nodes 4 of 10 nodes are interior in 3 of 10 trees 3 of 10 nodes are interior in 4 of 10 trees 3 of 10 nodes are interior in 5 of 10 trees 20 Nodes 4 nodes are interior only when roots 16 nodes are interior in 5 or more trees 80 Nodes 4 nodes are interior only when roots 36 of 80 nodes are interior in > 25% of trees
162. The Industry View: Improving Cluster Metadata Improving Interior Node Counts Coupling between fanout, number of trees, and how many times a node must be interior Height of initial trees constructed by peer service
163. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
164. The Future of Cluster Metadata in Riak Expand Usage Exploring Thicket Research Area Migration & New Data Move data from 1.x subsystem to 2.x subsystem Store data we couldn’t before WAN Replication Integrate with Riak’s Multi-Site Replication Abstract Messaging Layer
165. The Future of Cluster Metadata in Riak Expand Usage Exploring Thicket Research Area Migration & New Data Move data from 1.x subsystem to 2.x subsystem Store data we couldn’t before WAN Replication Integrate with Riak’s Multi-Site Replication Abstract Messaging Layer
166. The Future of Cluster Metadata in Riak Expand Usage Exploring Thicket Research Area Migration & New Data Move data from 1.x subsystem to 2.x subsystem Store data we couldn’t before WAN Replication Integrate with Riak’s Multi-Site Replication Abstract Messaging Layer
167. Outline 1 Motivation 2 The Academic View: Data Dissemination 3 Embedded Tree: Plumtree Protocol 4 The Industry View: Cluster Metadata Management 5 Cluster Metadata Management In Riak 6 The Academic View: When One Tree is not Enough 7 The Thicket Protocol 8 The Industry View: Improving Cluster Metadata 9 The Future of Cluster Metadata in Riak 10 Summary
168. Summary Joining academia and industry is powerful. Riak’s implementation has largely followed academic research, inadvertently... ...even when the research was on sort of a different field... ...and we have now begun to work together with academia.
169. Summary Joining academia and industry is powerful. Riak’s implementation has largely followed academic research, inadvertently... ...even when the research was on sort of a different field... ...and we have now begun to work together with academia.
170. Summary Joining academia and industry is powerful. Riak’s implementation has largely followed academic research, inadvertently... ...even when the research was on sort of a different field... ...and we have now begun to work together with academia.
171. Thanks. Jordan West − @ jrwest Jo˜ ao Leit˜ ao − @jcaleitao