PWLSF#3 => Bruce Spang On Bimodal Multicast

Bruce Spang from Fastly will covered the Bimodal Multicast paper by Kenneth P. Birman, Mark Hayden, Oznur Ozkasap, Zhen Xiao,. Mihai Budiu, and Yaron Minsky. Discussions available in https://github.com/papers-we-love/papers-we-love/issues/110

Bruce's Bio:
Bruce is a Software Engineer at Fastly, where he works primarily on large-scale distributed systems. He's also a student at UMass Amherst, where he hopes to one day be a Real Computer Scientist. He enjoys good books, rainy days, and dependable gossip.
He tweets at https://twitter.com/brucespang

Transcript

1. Bimodal Multicast And Cache Invalidation
2. Who/What/Where • Bruce Spang • Software Engineer/Student • Fastly
3. `
4. Powderhorn
5. Outline • Frame the problem • The papers we looked at • Bimodal Multicast • What we built
6. Content Delivery Networks
14. Cache Invalidation We would like to be able to update a piece of content globally
15. Cache Invalidation
16. Cache Invalidation
17. Cache Invalidation
18. Approach Notify all servers to remove a piece of content
19. Naïve Approach Central Service.
20. Central Service Message
21. Central Service Ack
22. Central Service
23. Central Service Message
24. Central Service Ack
25. Central Service Works
26. Problems • Unreliable • High Latency
27. Another Idea Cache servers can send purges themselves
28. Fixes • More Reliable • Lower Latency
29. Another Idea Message
30. Another Idea Ack
31. Problem Every server sends an ack to the sender
32. Conclusion This is hard.
33. Reliable Broadcast
34. Basic Problem Send a message to a set of servers
35. Applications • Stock Exchanges • Control Systems • Configuration • Cache Invalidation
36. Atomic Broadcast • Paxos • ZooKeeper Atomic Broadcast • etc…
37. Strong Guarantees • Guaranteed Delivery • Total Order
38. Too Many Guarantees • Don’t need Ordering • Don’t need Atomicity
39. “Best-Effort” Broadcast “Try very hard” to deliver a message
40. Algorithms • Scalable Reliable Multicast • Bimodal Multicast • Plumtree • Sprinkler • etc…
41. ` `
42. Goals • “Best-Effort” Delivery • Predictable Performance
43. Algorithm • Dissemination • Anti-Entropy (Gossip)
44. Dissemination • Unreliably broadcast a message to all other servers • IP multicast, UDP in a for loop, etc…
45. Anti-Entropy • Each server sends a random server a digest of the messages it knows about • If the other server hasn’t received some of those messages, it requests that they be retransmitted
46. Example
47. Example
48. Example
50. Gossip
51. Convergence
52. Goals • “Best-Effort” Delivery • Predictable Performance
53. Stable Throughput Message
54. Stable Throughput Ack
55. Stable Throughput Message
56. Stable Throughput Resend
57. Independence A server that’s behind will recover from many servers in the cluster.
58. Retransmission Limits Don’t DDoS servers that are trying to recover.
59. Soft Failure Detection Ignore servers that are running behind.
61. “Best effort” is ok! • Messages are delivered • Predicable Performance
62. Powderhorn Bimodal Multicast in the Wild
63. Development All the logic is in failure handling.
64. Jepsen • Five nodes • Simulated partitions, packet loss • http://github.com/aphyr/jepsen
65. “Scale” Testing • EC2 • 200 m1.mediums
66. Avalanche • Random, repeatable network fault injection • https://github.com/fastly/avalanche
67. Garbage Collection “I have messages {1,2,3,4,5,6,7,8,9,…}”
68. Garbage Collection
69. Computers are Horrible We see high packet loss and network partitions all the time.
70. Convergence < 40%
71. The Digest List of Message IDs
72. The Digest Doesn’t Have to be a List
73. The Digest • send ranges of ids of known messages • “messages 1 to 1,000,000 from server 1" • can represent large numbers of messages
74. The Digest
75. Behavior
76. End-to-End Latency 42ms 74ms 83ms 133ms New York London San Jose Tokyo 0.00 0.05 0.10 0.00 0.05 0.10 0.00 0.05 0.10 0.00 0.05 0.10 0 50 100 150 Latency (ms) Density Density plot and 95th percentile of purge latency by server location