Save 37% off PRO during our Black Friday Sale! »

Getting the Word Out: Membership, Dissemination, and Population Protocols

Getting the Word Out: Membership, Dissemination, and Population Protocols

We are building an instrumentation platform that runs across dozens of datacenters to provide operational visibility for internal systems and applications. This platform must remain up as much as possible and allow support and operations staff to understand and diagnose problems quickly. They must be able to ask questions like "what machines and applications are publishing metrics?", "what systems appear to be offline?", "what order did these errors occur in?", all without consulting every datacenter. Furthermore, they must be able to change configuration quickly, with confidence that every affected system will receive and act upon it.

To help with these problems, we are implementing several recently developed protocols for cluster membership, epidemic broadcast, and monotonic time. Respectively, these protocols allow us to know what nodes are peers, to disseminate configuration and status information, and to agree on roughly relative orders of events. Best of all, they are all synchronization-free, meaning we can achieve our goals while remaining highly available. In this talk, we'll discuss the protocols we chose, challenges to implementing them, and some preliminary results from deploying the protocols across our infrastructure.

Ce461b57b6a1f65ce5b6cc1c124673e3?s=128

Sean Cribbs

June 14, 2016
Tweet

Transcript

  1. M E M B E R S H I P,

    D I S S E M I N AT I O N & P O P U L AT I O N P R OTO CO LS G E T T I N G T H E W O R D O U T: SEAN CRIBBS SENIOR PRINCIPAL ENGINEER All photos are my own unless attributed.
  2. W H Y B U I L D P E

    E R -TO - P E E R SYST E M S ?
  3. None
  4. Work distribution

  5. Work distribution parallelism

  6. Work distribution parallelism concurrency

  7. Work distribution parallelism concurrency independence

  8. Work distribution parallelism concurrency independence Fault Tolerance

  9. Work distribution parallelism concurrency independence Fault Tolerance detection

  10. Work distribution parallelism concurrency independence Fault Tolerance detection recovery

  11. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy

  12. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy

    Locality
  13. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy

    Locality regional presence
  14. Work distribution parallelism concurrency independence Fault Tolerance detection recovery redundancy

    Locality regional presence code-to-data
  15. W H Y N OT P E E R -TO

    - P E E R SYST E M S ?
  16. W H Y N OT P E E R -TO

    - P E E R SYST E M S ? N o w y o u ’v e g o t N p ro b l e m s 

  17. W H Y N OT P E E R -TO

    - P E E R SYST E M S ? N o w y o u ’v e g o t N p ro b l e m s 
 N = 8 p ro b a b l y
  18. W H AT A R E W E B U

    I L D I N G ?
  19. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S Emoji provided free by Emoji One
  20. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T Emoji provided free by Emoji One
  21. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T Emoji provided free by Emoji One
  22. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  23. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  24. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  25. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  26. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  27. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  28. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  29. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  30. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  31. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  32. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance ➡ Locality E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  33. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T SYST E M H E A LT H SYST E M H E A LT H A P P L I CAT I O N M E T R I C S A P P L I CAT I O N M E T R I C S AG E N T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G ” “ P L AT F O R M ” ➡ Work distribution ➡ Fault-tolerance ➡ Locality E XT E R N A L S E R V I C E S E XT E R N A L S E R V I C E S ✓ Peer to Peer! Emoji provided free by Emoji One
  34. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One
  35. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other?
  36. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other? Distribute code and configuration?
  37. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
  38. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One Where do agents send data? How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
  39. E XT E R N A L S E R

    V I C E S AG E N T A R G U S O P E RAT I O N A L V I S I B I L I T Y P R OJ EC T AG E N T “ M U LT I -T E N A N T ” “ M U LT I - R E G I O N ” “ H I G H LY- AVA I L A B L E ” “ R E A L-T I M E ” “ ST R E A M I N G” “ P L AT F O R M ” E XT E R N A L S E R V I C E S Emoji provided free by Emoji One Where do agents send data? How to get fault-tolerance without spam? How do cluster nodes find each other? Distribute code and configuration? Know what happened when?
  40. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T
  41. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery
  42. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery ➡ Code and configuration dissemination
  43. A R G U S O P E RAT I

    O N A L V I S I B I L I T Y P R OJ EC T ➡ Cluster membership and discovery ➡ Code and configuration dissemination ➡ Relative and convergent time
  44. M E M B E R S H I P

    P R OTO C O L S
  45. W H Y N OT Z O O K E

    E P E R / C O N S U L / E TC D ? J U ST R U B S O M E C O N S E N S U S O N I T
  46. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S
  47. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness
  48. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance
  49. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length
  50. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering
  51. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering ➡ Scalability
  52. M E M B E R S H I P

    : D E S I RA B L E P R O P E RT I E S ➡ Connectedness ➡ Balance ➡ Short path-length ➡ Low clustering ➡ Scalability ➡ Accuracy
  53. M E M B E R S H I P

    : “ V I E W ” F L AV O R S Full Partial
  54. Full

  55. Full ✓ Connectedness

  56. Full ✓ Connectedness ✓ Short path-length

  57. Full ✓ Connectedness ✓ Short path-length - Accuracy

  58. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance

  59. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance

    ๏ High Clustering
  60. Full ✓ Connectedness ✓ Short path-length - Accuracy - Balance

    ๏ High Clustering ๏ Low Scalability
  61. Partial

  62. Partial ✓ Low Clustering

  63. Partial ✓ Low Clustering ✓ High scalability

  64. Partial ✓ Low Clustering ✓ High scalability - Connectedness

  65. Partial ✓ Low Clustering ✓ High scalability - Connectedness -

    Balance
  66. Partial ✓ Low Clustering ✓ High scalability - Connectedness -

    Balance - Path-length
  67. Partial ✓ Low Clustering ✓ High scalability - Connectedness -

    Balance - Path-length - Accuracy
  68. S W I M - 2 0 0 2

  69. S W I M - 2 0 0 2 Emoji

    provided free by Emoji One
  70. S W I M - 2 0 0 2 Emoji

    provided free by Emoji One Heartbeat protocols
  71. S W I M - 2 0 0 2 ๏

    Quadratic load Emoji provided free by Emoji One Heartbeat protocols
  72. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection Emoji provided free by Emoji One Heartbeat protocols
  73. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times Emoji provided free by Emoji One Heartbeat protocols
  74. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times ๏ False positives Emoji provided free by Emoji One Heartbeat protocols
  75. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times ๏ False positives Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
  76. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
  77. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection ➡ Randomized probing Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
  78. S W I M - 2 0 0 2 ๏

    Quadratic load ๏ Failure detection ๏ Response times ๏ False positives ➡ Separate membership and failure detection ➡ Randomized probing ➡ Piggyback membership on probes Emoji provided free by Emoji One Heartbeat protocols SWIM solutions
  79. S W I M - 2 0 0 2

  80. S CA M P - 2 0 0 3

  81. S CA M P - 2 0 0 3

  82. S CA M P - 2 0 0 3 ๏

    Full views limit scalability
  83. S CA M P - 2 0 0 3 ๏

    Full views limit scalability ➡ Flexible partial-view size, asymmetric
  84. S CA M P - 2 0 0 3 ๏

    Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management
  85. S CA M P - 2 0 0 3 ๏

    Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management ➡ Join (“subscribe”) via random walk
  86. S CA M P - 2 0 0 3 ๏

    Full views limit scalability ➡ Flexible partial-view size, asymmetric ➡ Reactive view management ➡ Join (“subscribe”) via random walk ➡ Automatic balancing via indirection and leases
  87. S CA M P - 2 0 0 3

  88. S CA M P - 2 0 0 3

  89. CYC LO N - 2 0 0 5

  90. CYC LO N - 2 0 0 5

  91. CYC LO N - 2 0 0 5 ๏ Random

    shuffling doesn’t create good balance
  92. CYC LO N - 2 0 0 5 ๏ Random

    shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric
  93. CYC LO N - 2 0 0 5 ๏ Random

    shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric ➡ Cyclic view management
  94. CYC LO N - 2 0 0 5 ๏ Random

    shuffling doesn’t create good balance ➡ Fixed partial-view size, symmetric ➡ Cyclic view management ➡ Join via random walk
  95. CYC LO N - 2 0 0 5

  96. CYC LO N - 2 0 0 5

  97. P R O B L E M S W I

    T H S CA M P & CYC LO N • No failure detectors • SCAMP: asymmetric views ⟹ disconnection • SCAMP: unbounded view size ⟹ imbalance
  98. H Y PA R V I E W - 2

    0 0 7
  99. H Y PA R V I E W - 2

    0 0 7
  100. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability
  101. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality
  102. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality
  103. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector
  104. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”)
  105. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”) ➡ Larger cyclic view (“passive”)
  106. H Y PA R V I E W - 2

    0 0 7 ๏ Fanout is related to reliability ๏ High failure rates decrease quality ➡ TCP for transport and failure detector ➡ Small reactive view (“active”) ➡ Larger cyclic view (“passive”) ➡ Join and shuffle via random walk
  107. H Y PA R V I E W - 2

    0 0 7
  108. H Y PA R V I E W - 2

    0 0 7 A B C D Passive view 
 maintenance
  109. H Y PA R V I E W - 2

    0 0 7
  110. W E C H O S E H Y PA

    R V I E W • Only active view maintenance • Passive view maintains full membership (unbounded) • Later: switch to complete passive maintenance
  111. D I S S E M I N AT I

    O N P R OTO C O L S
  112. D I S S E M I N AT I

    O N : D E S I RA B L E P R O P E RT I E S
  113. D I S S E M I N AT I

    O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability
  114. D I S S E M I N AT I

    O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability ➡ Scalability
  115. D I S S E M I N AT I

    O N : D E S I RA B L E P R O P E RT I E S ➡ Reliability ➡ Scalability ➡ Efficiency
  116. E P I D E M I C B R

    OA D CAST ( G O S S I P ) !
  117. E P I D E M I C B R

    OA D CAST ( G O S S I P ) ➡ Send to random peers !
  118. E P I D E M I C B R

    OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients !
  119. E P I D E M I C B R

    OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients !
  120. E P I D E M I C B R

    OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients ๏ High redundancy !
  121. E P I D E M I C B R

    OA D CAST ( G O S S I P ) ➡ Send to random peers ➡ Messages rebroadcast by recipients ๏ High redundancy ๏ Low scalability !
  122. I N C R E AS E D E F

    F I C I E N CY W I T H O U T R E D U C I N G D E L I V E RY G UA R A N T E E S , W E N E E D
  123. None
  124. P L U M T R E E - 2

    0 0 9 ! !
  125. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N ! A B
  126. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set ! A B
  127. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push ! A B
  128. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) ! A B
  129. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) ! A B
  130. P L U M T R E E - 2

    0 0 9 CO N ST R U C T I O N • All nodes start with full “eager” set • Broadcast triggers eager-push • Duplicate messages cause “pruning” (move to “lazy”) • Regular broadcasts proceed with new “eager” sets ! A B
  131. P L U M T R E E - 2

    0 0 9 R E PA I R ! A B
  132. P L U M T R E E - 2

    0 0 9 R E PA I R • Lazy-push sends “I Have” messages ! A B
  133. P L U M T R E E - 2

    0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) ! A B
  134. P L U M T R E E - 2

    0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) ! A B
  135. P L U M T R E E - 2

    0 0 9 R E PA I R • Lazy-push sends “I Have” messages • Timeout triggers “grafting” (move to “eager”) • Lazy-push batched to reduce overhead ! A B
  136. W E C H O S E P L U

    M T R E E
  137. W E C H O S E P L U

    M T R E E • Good tradeoff between reliability and redundancy
  138. W E C H O S E P L U

    M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths
  139. W E C H O S E P L U

    M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths • Existing open-source implementations
  140. W E C H O S E P L U

    M T R E E • Good tradeoff between reliability and redundancy • Optimizes for lowest-latency paths • Existing open-source implementations • Excellent fit with HyParView
  141. P O P U L AT I O N P

    R OTO C O L S
  142. R A N D O M I Z E D

    I N T E R AC T I O N S P O P U L AT I O N P R OTO C O L S U S E
  143. None
  144. D I ST R I B U T E D

    M O N OTO N I C C LO C K S J O N M O O R E Vidcap from StrangeLoop 2015: https://youtu.be/YqNGbvFHoKM
  145. D M C P R O B L E M

    S ๏ “Wacky clock mode” ๏ Hierarchy imbalances load ๏ Long-lived partitions ๏ No convergence proof
  146. None
  147. A P P LY I N G D M C

    • Use existing dissemination with DMC • Transmit clocks along with other messages • Use monotonic clocks as a drift-detection mechanism
  148. L E S S O N S L E A

    R N E D
  149. Photo by Chris Meiklejohn Vidcap from RICON West: https://youtu.be/s4cCUTPU8GI Photo

    by Comcast
  150. T H A N K YO U ! @ S

    E A N C R I B B S