The Economics of Scale: Promises and Perils of Going Distributed

What does it take to scale a system? We'll learn how going distributed can pay dividends in areas like availability and fault tolerance by examining a real-world case study. However, we will also look at the inherent pitfalls. When it comes to distributed systems, for every promise there is a peril.

Transcript

1. The Economics of Scale Tyler Treat Workiva Promises and Perils of Going Distributed September 19, 2015
2. About The Speaker • Backend engineer at Workiva • Messaging platform tech lead • Distributed systems • bravenewgeek.com @tyler_treat tyler.treat@workiva.com
3. About The Talk • Why distributed systems? • Case study • Advantages/Disadvantages • Strategies for scaling and resilience patterns • Scaling Workiva
4. What does it mean to “scale” a system?
5. Scale Up vs. Scale Out ❖ Add resources to a node ❖ Increases node capacity, load is unaffected ❖ System complexity unaffected Vertical Scaling ❖ Add nodes to a cluster ❖ Decreases load, capacity is unaffected ❖ Availability and throughput w/ increased complexity Horizontal Scaling
6. Okay, cool—but what does this actually mean?
7. Let’s look at a real-world example…
8. How does Twitter work?
9. Just write tweets to a big database table.
11. Getting a timeline is a simple join query.
13. But…this is crazy expensive.
14. Joins are sloooooooooow.
15. Prior to 5.5, MySQL used table-level locking. Now it uses row-level locking. Either way,
 lock contention everywhere.
16. As the table grows larger, lock contention on indexes becomes worse too.
17. And UPDATES get higher priority than SELECTS.
19. So now what?
20. Distribute!
21. Specifically, shard.
22. Partition tweets into different databases using some consistent hash scheme
 (put a hash ring on it).
24. This alleviates lock contention and improves throughput…
 
 but fetching timelines is still extremely costly (now scatter-gather query across multiple DBs).
25. Observation: Twitter is a consumption mechanism more than an ingestion one… i.e. cheap reads > cheap writes
26. Move tweet processing to the write path
 rather than read path.
27. Ingestion/Fan-Out Process 1. Tweet comes in 2. Query the social graph service for followers 3. Iterate through each follower and insert tweet ID into their timeline (stored in Redis) 4. Store tweet on disk (MySQL)
29. Ingestion/Fan-Out Process • Lots of processing on ingest, no computation on reads • Redis stores timelines in memory—very fast • Fetching timeline involves no queries—get timeline from Redis cache and rehydrate with multi-get on IDs • If timeline falls out of cache, reconstitute from disk • O(n) on writes, O(1) on reads • http://www.infoq.com/presentations/Twitter-Timeline-Scalability
30. Key Takeaway: think about your access patterns and design accordingly. 
 Optimize for the critical path.
31. Let’s Recap… • Advantages of single database system: • Simple! • Data and invariants are consistent (ACID transactions) • Disadvantages of single database system: • Slow • Doesn’t scale • Single point of failure
32. Going distributed solved the problem,
 but at what cost? (hint: your sanity)
33. Distributed systems are all about trade-offs.
34. By choosing availability, we give up consistency.
35. This problem happens all the time on Twitter.
 
 For example, you tweet, someone else replies, and I see the reply before the original tweet.
36. Can we solve this problem?
37. Sure, just coordinate things before proceeding… “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.” “Have you seen this tweet? Okay, good.”
38. Sooo what do you do when Justin Bieber tweets to his 67 million followers?
40. Coordinating for consistency is expensive
 when data is distributed
 because processes
 can’t make progress independently.
43. Source: Peter Bailis, 2015 https://speakerdeck.com/pbailis/silence-is-golden-coordination-avoiding-systems-design
44. Key Takeaway: strong consistency is slow and distributed coordination is expensive (in terms of latency and throughput).
45. Sharing mutable data at large scale is difficult.
46. If we don’t distribute, we risk scale problems.
47. Let’s say we want to count the number of times a tweet is retweeted.
48. “Get, add 1, and put” transaction will not
 scale.
49. If we do distribute, we risk consistency problems.
51. What do we do when our system is partitioned?
53. If we allow writes on both sides of the partition, how do we resolve conflicts when the partition heals?
54. Distributed systems are hard!
55. But lots of good research going on to solve these problems… CRDTs Lasp SyncFree RAMP transactions etc.
56. Twitter has 316 million monthly active users. Facebook has 1.49 billion monthly active users. Netflix has 62.3 million streaming subscribers.
57. How do you build resilient systems at this scale?
58. Embrace failure.
59. Provide partial availability.
60. If an overloaded service is not essential to
 core business, fail fast to prevent availability or latency problems upstream.
63. It’s better to fail predictably than fail in unexpected ways.
64. Use backpressure to reduce load.
67. Flow-Control Mechanisms • Rate limit • Bound queues/buffers • Backpressure - drop messages on the floor • Increment stat counters for monitoring/alerting • Exponential back-off • Use application-level acks for critical transactions
68. Bounding resource utilization and failing fast helps maintain predictable performance and impedes cascading failures.
69. Going distributed means more wire time.
 How do you improve performance?
71. Cache everything.
73. “There are only two hard things in computer science: cache invalidation and naming things.”
75. Embrace asynchrony.
78. Distributed systems are not just about workload scale, they’re about organizational scale.
79. In 2010, Workiva released a product to streamline financial reporting.
80. A specific solution to solve a very specific problem, originally built by a few dozen engineers.
81. Fast forward to today: a couple hundred engineers, more users, more markets, more solutions.
82. How do you ramp up new products quickly?
83. You stop thinking in terms of products and start thinking in terms of platform.
84. From Product to Platform
85. At this point, going distributed is all but necessary.
86. Service-Oriented Architecture allows us to independently build, deploy, and scale discrete parts of the platform.
87. Loosely coupled services let us tolerate failure. And things fail constantly.
88. Shit happens — network partitions, hardware failure, GC pauses, latency, dropped packets…
89. Build resilient systems.
90. –Ken Arnold “You have to design distributed systems with the expectation of failure.”
91. Design for failure.
92. Consider the trade-off between consistency and availability.
93. Most important?
94. Don’t distribute until you have a reason to!
95. Scale up until you have to
 scale out.
96. –Paul Barham “You can have a second computer once you’ve shown you know how to use the first one.”
97. And when you do distribute,
 don’t go overboard. Walk before you run.
98. Remember, when it comes to distributed systems…
 for every promise there’s a peril.
99. Thanks! @tyler_treat github.com/tylertreat
 bravenewgeek.com