Getting the Word Out: Membership, Dissemination, and Population Protocols

We are building an instrumentation platform that runs across dozens of datacenters to provide operational visibility for internal systems and applications. This platform must remain up as much as possible and allow support and operations staff to understand and diagnose problems quickly. They must be able to ask questions like "what machines and applications are publishing metrics?", "what systems appear to be offline?", "what order did these errors occur in?", all without consulting every datacenter. Furthermore, they must be able to change configuration quickly, with confidence that every affected system will receive and act upon it.

To help with these problems, we are implementing several recently developed protocols for cluster membership, epidemic broadcast, and monotonic time. Respectively, these protocols allow us to know what nodes are peers, to disseminate configuration and status information, and to agree on roughly relative orders of events. Best of all, they are all synchronization-free, meaning we can achieve our goals while remaining highly available. In this talk, we'll discuss the protocols we chose, challenges to implementing them, and some preliminary results from deploying the protocols across our infrastructure.

Transcript

1. M E M B E R S H I P, D I S S E M I N AT I O N & P O P U L AT I O N P R OTO CO LS G E T T I N G T H E W O R D O U T: SEAN CRIBBS SENIOR PRINCIPAL ENGINEER All photos are my own unless attributed.
2. W H Y B U I L D P E E R -TO - P E E R SYST E M S ?
4. Work distribution
5. Work distribution parallelism
6. Work distribution parallelism concurrency
7. Work distribution parallelism concurrency independence
8. Work distribution parallelism concurrency independence Fault Tolerance
9. Work distribution parallelism concurrency independence Fault Tolerance detection
10. Work distribution parallelism concurrency independence Fault Tolerance detection recovery
11. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy
12. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy Locality
13. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy Locality regional presence
14. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy Locality regional presence code-to-data
15. W H Y N OT P E E R -TO - P E E R SYST E M S ?
16. W H Y N OT P E E R -TO - P E E R SYST E M S ? N o w y o u ’v e g o t N p ro b l e m s 

17. W H Y N OT P E E R -TO - P E E R SYST E M S ? N o w y o u ’v e g o t N p ro b l e m s 
 N = 8 p ro b a b l y
18. W H AT A R E W E B U I L D I N G ?
19. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S Emoji provided free by Emoji One
20. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T Emoji provided free by Emoji One
21. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T Emoji provided free by Emoji One
22. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
23. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
24. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
25. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
26. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
27. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
28. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
29. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
30. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
31. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
32. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance ➡ Locality E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
33. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance ➡ Locality E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S ✓ Peer to Peer! Emoji provided free by Emoji One
34. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
35. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other?
36. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other? Distribute code and configuration?
37. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
38. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One Where do agents send data? How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
39. E XT E R N A L S E R V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One Where do agents send data? How to get fault-tolerance without spam? How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
40. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T
41. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery
42. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery ➡ Code and configuration dissemination
43. A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery ➡ Code and configuration dissemination ➡ Relative and convergent time
44. M E M B E R S H I P P R OTO C O L S
45. W H Y N OT Z O O K E E P E R / C O N S U L / E TC D ? J U ST R U B S O M E C O N S E N S U S O N I T
46. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S
47. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness
48. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance
49. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length
50. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering
51. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering ➡ Scalability
52. M E M B E R S H I P : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering ➡ Scalability ➡ Accuracy
53. M E M B E R S H I P : “ V I E W ” F L AV O R S Full Partial
54. Full
55. Full ✓ Connectedness
56. Full ✓ Connectedness ✓ Short path-length
57. Full ✓ Connectedness ✓ Short path-length - Accuracy
58. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance
59. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance ๏ High Clustering
60. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance ๏ High Clustering ๏ Low Scalability
61. Partial
62. Partial ✓ Low Clustering
63. Partial ✓ Low Clustering ✓ High scalability
64. Partial ✓ Low Clustering ✓ High scalability - Connectedness
65. Partial ✓ Low Clustering ✓ High scalability - Connectedness - Balance
66. Partial ✓ Low Clustering ✓ High scalability - Connectedness - Balance - Path-length
67. Partial ✓ Low Clustering ✓ High scalability - Connectedness - Balance - Path-length - Accuracy
68. S W I M - 2 0 0 2
69. S W I M - 2 0 0 2 Emoji provided free by Emoji One
70. S W I M - 2 0 0 2 Emoji provided free by Emoji One Heartbeat protocols
71. S W I M - 2 0 0 2 ๏ Quadratic load Emoji provided free by Emoji One Heartbeat protocols
72. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection Emoji provided free by Emoji One Heartbeat protocols
73. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times Emoji provided free by Emoji One Heartbeat protocols
74. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times ๏ False positives Emoji provided free by Emoji One Heartbeat protocols
75. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times ๏ False positives Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
76. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
77. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection ➡ Randomized probing Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
78. S W I M - 2 0 0 2 ๏ Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection ➡ Randomized probing ➡ Piggyback membership on probes Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
79. S W I M - 2 0 0 2
80. S CA M P - 2 0 0 3
81. S CA M P - 2 0 0 3
82. S CA M P - 2 0 0 3 ๏ Full views limit scalability
83. S CA M P - 2 0 0 3 ๏ Full views limit scalability ➡ Flexible partial-view size, asymmetric
84. S CA M P - 2 0 0 3 ๏ Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management
85. S CA M P - 2 0 0 3 ๏ Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management ➡ Join (“subscribe”) via random walk
86. S CA M P - 2 0 0 3 ๏ Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management ➡ Join (“subscribe”) via random walk ➡ Automatic balancing via indirection and leases
87. S CA M P - 2 0 0 3
88. S CA M P - 2 0 0 3
89. CYC LO N - 2 0 0 5
90. CYC LO N - 2 0 0 5
91. CYC LO N - 2 0 0 5 ๏ Random shuffling doesn’t create good balance
92. CYC LO N - 2 0 0 5 ๏ Random shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric
93. CYC LO N - 2 0 0 5 ๏ Random shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric ➡ Cyclic view management
94. CYC LO N - 2 0 0 5 ๏ Random shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric ➡ Cyclic view management ➡ Join via random walk
95. CYC LO N - 2 0 0 5
96. CYC LO N - 2 0 0 5
97. P R O B L E M S W I T H S CA M P & CYC LO N • No failure detectors • SCAMP: asymmetric views ⟹ disconnection • SCAMP: unbounded view size ⟹ imbalance
98. H Y PA R V I E W - 2 0 0 7
99. H Y PA R V I E W - 2 0 0 7
100. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability
101. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality
102. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality
103. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector
104. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”)
105. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”) ➡ Larger cyclic view (“passive”)
106. H Y PA R V I E W - 2 0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”) ➡ Larger cyclic view (“passive”) ➡ Join and shuffle via random walk
107. H Y PA R V I E W - 2 0 0 7
108. H Y PA R V I E W - 2 0 0 7 A B C D Passive view 
 maintenance
109. H Y PA R V I E W - 2 0 0 7
110. W E C H O S E H Y PA R V I E W • Only active view maintenance • Passive view maintains full membership (unbounded) • Later: switch to complete passive maintenance
111. D I S S E M I N AT I O N P R OTO C O L S
112. D I S S E M I N AT I O N : D E S I RA B L E P R O P E RT I E S
113. D I S S E M I N AT I O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability
114. D I S S E M I N AT I O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability ➡ Scalability
115. D I S S E M I N AT I O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability ➡ Scalability ➡ Efficiency
116. E P I D E M I C B R OA D CAST ( G O S S I P ) !
117. E P I D E M I C B R OA D CAST ( G O S S I P ) ➡ Send to random peers !
118. E P I D E M I C B R OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients !
119. E P I D E M I C B R OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients !
120. E P I D E M I C B R OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients ๏ High redundancy !
121. E P I D E M I C B R OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients ๏ High redundancy ๏ Low scalability !
122. I N C R E AS E D E F F I C I E N CY W I T H O U T R E D U C I N G D E L I V E RY G UA R A N T E E S , W E N E E D
124. P L U M T R E E - 2 0 0 9 ! !
125. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N ! A B
126. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set ! A B
127. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push ! A B
128. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) ! A B
129. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) ! A B
130. P L U M T R E E - 2 0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) • Regular broadcasts proceed with new “eager” sets ! A B
131. P L U M T R E E - 2 0 0 9 R E PA I R ! A B
132. P L U M T R E E - 2 0 0 9 R E PA I R • Lazy-push sends “I Have” messages ! A B
133. P L U M T R E E - 2 0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) ! A B
134. P L U M T R E E - 2 0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) ! A B
135. P L U M T R E E - 2 0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) • Lazy-push batched to reduce overhead ! A B
136. W E C H O S E P L U M T R E E
137. W E C H O S E P L U M T R E E • Good tradeoff between reliability and redundancy
138. W E C H O S E P L U M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths
139. W E C H O S E P L U M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths • Existing open-source implementations
140. W E C H O S E P L U M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths • Existing open-source implementations • Excellent fit with HyParView
141. P O P U L AT I O N P R OTO C O L S
142. R A N D O M I Z E D I N T E R AC T I O N S P O P U L AT I O N P R OTO C O L S U S E
144. D I ST R I B U T E D M O N OTO N I C C LO C K S J O N M O O R E Vidcap from StrangeLoop 2015: https://youtu.be/YqNGbvFHoKM
145. D M C P R O B L E M S ๏ “Wacky clock mode” ๏ Hierarchy imbalances load ๏ Long-lived partitions ๏ No convergence proof
147. A P P LY I N G D M C • Use existing dissemination with DMC • Transmit clocks along with other messages • Use monotonic clocks as a drift-detection mechanism
148. L E S S O N S L E A R N E D
149. Photo by Chris Meiklejohn Vidcap from RICON West: https://youtu.be/s4cCUTPU8GI Photo by Comcast
150. T H A N K YO U ! @ S E A N C R I B B S