Riak, Latency and Distributed Systems

Everything is distributed. Any engineer building a system that's running in production needs be aware that she is relying on a series of components, services, and processes that need to work together to deliver data to the customers. This is not easy, and the tools you pick to deploy will impact greatly your ability win users and increase revenue.

Riak is an open source, distributed database being used by companies like Comcast, AT&T, and GitHub in production. Like any database, it comes with a specfic set of tradeoff that make it suitable for a certain set of applications. In this talk, Tom will discuss Riak and the types of applications people are building on it with special attention paid to dealing with latencies in such a distributed system.

Transcript

1. riak latency distributed systems and @tsantero Wednesday, June 19, 13
2. What we talk about when we talk about Availability Wednesday, June 19, 13
3. Liveness good things eventually happen Wednesday, June 19, 13
4. User Experience Wednesday, June 19, 13
5. $$$$$$$$$$ Wednesday, June 19, 13
6. Are you Good with computers? Wednesday, June 19, 13
7. Why isn’t this website loading? Wednesday, June 19, 13
8. Things are getting Easier* Wednesday, June 19, 13
9. Users expect it to JUST WORK Wednesday, June 19, 13
10. Frontend Middleware Backend Wednesday, June 19, 13
11. Frontend Middleware Backend Frontend Frontend Middleware Middleware Wednesday, June 19, 13
12. Frontend Middleware Master Frontend Frontend Middleware Middleware S S S Wednesday, June 19, 13
13. Frontend Middleware Master Frontend Frontend Middleware Middleware S S S Wednesday, June 19, 13
14. Frontend Middleware Frontend Frontend Middleware Middleware Riak Wednesday, June 19, 13
15. everything is distributed Wednesday, June 19, 13
16. What we talk about when we talk about Distribution Wednesday, June 19, 13
17. Global Availability Wednesday, June 19, 13
18. Frontend Middleware Master Frontend Frontend Middleware Middleware Riak CDN Wednesday, June 19, 13
19. Frontend Middleware Master Frontend Frontend Middleware Middleware Riak CDN Wednesday, June 19, 13
20. 3.33ms / km (+/- 5µs) Wednesday, June 19, 13
21. San Jose New York Amsterdam Frankfurt Seoul IXP Wednesday, June 19, 13
22. San Jose New York = ~ 74ms Wednesday, June 19, 13
23. San Jose Amsterdam = ~ 200ms Wednesday, June 19, 13
24. Riak Internals Wednesday, June 19, 13
25. node node node node node distributed masterless key/value Store Wednesday, June 19, 13
26. REPLICATION Wednesday, June 19, 13
27. Quorum requests N R W ead rite Wednesday, June 19, 13
28. Tail Latency Misbehaving Nodes & Wednesday, June 19, 13
29. “...widespread underestimation of the specific difficulties of size seems one of the major underlying causes of the current software failure.” --EW Dijkstra Notes on Structured Programming 1969 Wednesday, June 19, 13
30. Google’s BigTable 95th 99.9th 24ms 994ms : : Wednesday, June 19, 13
31. Reduce Tail Latencies with Redundancy Wednesday, June 19, 13
32. IF response > 10ms Send 2nd Request = ~ 99.9 : 50ms (5% increase total reqs.) Wednesday, June 19, 13
33. Overall Latency determined by latency of slowest machine Wednesday, June 19, 13
34. Redundancy in Riak? Wednesday, June 19, 13
35. Setup: basho bench Riak Riak Riak Riak Riak Wednesday, June 19, 13
36. Demo: sudo  tc  qdisc  add  dev  eth0  root  netem  delay  150ms Network: for  x  in  /sys/devices/system/cpu/*;  do (while  true  ;  do  openssl  speed;  done)  & done wait CPU: Wednesday, June 19, 13
37. demo Wednesday, June 19, 13
38. client Riak Riak Riak riak node L B Wednesday, June 19, 13
39. client Riak Riak Riak riak node L B Wednesday, June 19, 13
40. client Riak Riak Riak riak node L B Wednesday, June 19, 13
41. client Riak Riak Riak riak node L B Wednesday, June 19, 13
42. Tunables Wednesday, June 19, 13
43. I <3 Immutability Wednesday, June 19, 13
44. Write Once Read Often N W R = = = 5 N 1 notfound_ok: False Wednesday, June 19, 13
45. Write Once Read Hardly N W R = = = 3 Q Q Wednesday, June 19, 13
46. Write Once N DW R = = = 3 2 1 CRITICAL Wednesday, June 19, 13
47. RSEW? Read Someone Else’s Writes lol Wednesday, June 19, 13
48. Lots of work Remains Wednesday, June 19, 13
49. Thanks! Wednesday, June 19, 13