Developing a Globally Distributed Purging System

(Surge 2014) How do you build a distributed cache invalidation system that can invalidate content in 150 milliseconds across a global network of servers? Fastly CTO Tyler McMullen and engineer Bruce Spang will discuss the process of constructing a production-ready distributed system built on solid theoretical foundations. This talk will cover using research to design systems, the bimodal multicast algorithm, and the behavior of this system in production.

Transcript

1. Prototypes, Papers, and Production Developing a globally distributed purging system
2. Bruce Spang @brucespang
3. Tyler McMullen @tbmcmullen
5. What is a CDN?
13. Cache Invalidation
14. Step One Make it
15. rsyslog
16. E D F C A B Z
17. E D F C A B Z PURGE
18. E D F C A B Z PURGE
19. E D F C A B Z PURGE
20. Already deployed
21. Minimal code
22. Easy to reason about
23. Why does it fail?
24. High latency
25. Partition intolerant
26. Wrong consistency model
27. A B 200ms
28. A B 11 10 9 8 7 6 5 4 3 2 1
29. A B 11 10 9 8 7 6 5 4 3 2
30. A B 22 21 20 19 18 17 16 15 14 SACK 13 12
31. A B SACK 1
32. New Customer Use
33. Step Two Make it Interesting
34. Atomic Broadcast
37. Strong Guarantees Too Strong
38. Thought Real Hard
39. E D F C A B Graph of Responsibility
40. E D F C A B Graph of Responsibility
41. E D F C A B Graph of Responsibility
42. E D F C A B PURGE
43. E D F C A B PURGE
44. E D F C A B PURGE
45. E D F C A B PURGE
46. E D F C A B PURGE
47. This also worked.
48. Arbitrary Partitions
49. Unbounded Queues
50. Failure Dependence
51. New Customer Use
52. Step Three Make it Scale
53. Understood our needs Not the problem
54. I am Lazy Inventing distributed algorithms are hard
55. Read Papers
56. Reliable Broadcast
59. Gossip Protocols
62. “Designed for Scale”
63. Probabilistic Guarantees
65. Bimodal Multicast • Quickly broadcast message to all servers • Gossip to recover lost messages
69. Questions?
72. One Problem Computers have limited space
73. Throw away messages
75. Computers are Terrible We see high packet loss all the time
77. The Digest “I have 1, 2, 3, …”
78. The Digest Doesn’t Have to be a List
79. The Digest Send ranges of ids of known messages! “messages 1 to 3 and 5 to 1,000,000"
81. “with high probability” is fine
82. Real World
83. End-to-End Latency 74ms 83ms 133ms London San Jose Tokyo 0.00 0.00 0.05 0.10 0.00 0.05 0.10 0.00 0.05 0.10 0 50 100 150 Latency (ms) Density
84. End-to-End Latency 42ms 74ms 83ms 133ms New York London San Jose Tokyo 0.00 0.05 0.10 0.00 0.05 0.10 0.00 0.05 0.10 0.00 0.05 0.10 0 50 100 150 Latency (ms) Density Density plot and 95th percentile of purge latency by server location
85. Firewall Partition
86. APAC Packet Loss
87. DDoS • `
88. So what?
89. Good systems are boring
90. What did we learn?
91. Know Your Core Competencies
92. One weird trick…
93. Read More.
94. Thanks!
95. Questions? Come to our booth!