Building Healthy Distributed Systems

Distributed systems aren't limited to the world of computing. Companies and communities, for example, are also distributed networks of resources. In this talk, we'll take a high-level look at computers, companies, and communities as distributed systems and examine why maintaining their health is crucial if they are to scale.

Transcript

1. Thank you to Jonathan, Duncan, Jonathan, Marije, Jennifer, Lessfuss, and Cape Town. ROCKS! Friday, January 27, 12
2. basho @pharkmillups themarkphillips.com mark@basho.com Mark Phillips Friday, January 27, 12
3. Building Healthy Distributed Systems ScaleConf January 27, 2012 basho Friday, January 27, 12
4. “A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal.” [1] What is a distributed system? basho Friday, January 27, 12
5. Distributed, Scalable, Fault Tolerant No central coordinator; Easy to setup and operate basho Friday, January 27, 12
6. Distributed, Scalable, Fault Tolerant Horizontally Scalable; Add commodity hardware to get more [throughput | processing | storage]. basho Friday, January 27, 12
7. Distributed, Scalable, Fault Tolerant Always Available No Single Point of Failure Self-healing basho Friday, January 27, 12
8. basho Friday, January 27, 12
9. basho { • Founded in 2007 • Collapsed in 2008 • “Pivoted” in 2009 • Commercial Sponsors of Riak, an Open Source, NoSQL Database • Sells Closed Source Extensions to Riak in the form licenses Friday, January 27, 12
10. 2009 2010 2011 14 60 25 Year on Year Growth basho Friday, January 27, 12
11. basho Office Locations Friday, January 27, 12
12. Actual Employee Distribution basho Friday, January 27, 12
13. “A distributed [company] consists of multiple autonomous [team members] that communicate [and collaborate] through various [channels]. The [team members] interact with each other in order to achieve a common goal.” What is a distributed [company]? basho Friday, January 27, 12
14. Hiring where the talent is means we don’t sacrifice great hires for location, but it also presents various hurdles when attempting to build culture and community. basho basho Friday, January 27, 12
15. 1. Make Basho into a Powerhouse 2. Professional Development 3. Employee Happiness 4. Deliver Exceptional Product Common Goals for Basho basho Friday, January 27, 12
16. Internal Communication and Collaboration • Real-time Chat (Jabber, Camp Fire) • Skype (or some for of video chat) • Yammer • GitHub • AgileZen • Email (sort of) • Documentation basho Friday, January 27, 12
17. Good Meetings basho • Quarterly In-person “Summits” • Bi-Monthly, Non-Mandatory Company All Hands • Stands up, Scrum Friday, January 27, 12
18. Make Documentation Part of Your Culture basho • Inside Jokes • Internal Talks • Design Documents • Product Ideas • Product Feedback • New Hire Processes • Everything Else Friday, January 27, 12
19. Open Source Your Code. And Use GitHub. basho • Contributes Directly to Developer Happiness • Makes Your Company’s Product Better • Great Marketing • Use a Permissive License (http://bit.ly/clJyDO) (http://bit.ly/v3OMEf) “Open Source Almost Everything” “Why Your Company Should Have a Permissive Open Source Policy” Friday, January 27, 12
20. Friday, January 27, 12
21. basho basho Hiring Should Not Happen In A Vacuum Friday, January 27, 12
22. Poor Culture Rots a Company from within and Lessens its Resiliency basho Friday, January 27, 12
23. basho Company Fault Tolerance • New CEO + Massive Growth = New Challenges • Our System is Constantly Improving Friday, January 27, 12
24. 2012 1** Planned Growth basho Friday, January 27, 12
25. basho DS2: The Riak Community Friday, January 27, 12
26. “A distributed [community] consists of multiple autonomous [members] that communicate [and collaborate] through various [channels]. The [members] interact with each other in order to achieve a common goal.” What is a distributed [community]? basho Friday, January 27, 12
27. Community Friday, January 27, 12
28. basho Why Build A Community? Friday, January 27, 12
29. Grassroots Marketing, Branding, Awareness: basho Friday, January 27, 12
30. Code Contributions and Bug Fixes : basho 176 names in our THANKS file 1600 hours contributed from Oct 2010 - Sept 2011 Friday, January 27, 12
31. Support: basho Friday, January 27, 12
32. Revenue: basho 75% of new customers in 2011 came from the Open Source Community Friday, January 27, 12
33. Importance of Community for Community Members basho •Working, Quality Code •Recognition and Praise •Desire to Contribute •Jobs (whether they like it or not) •Skills Acquisition Friday, January 27, 12
34. Communication and Collaboration in a Distributed [Community] basho •IRC •Mailing List •Twitter •Riak Recap •Meetups •Q & A Sites •Blogs •Books •Conferences •Actual Meetings •GitHub •Drinking Friday, January 27, 12
35. Riak Recap basho Friday, January 27, 12
36. Books basho http://riakhandbook.com/ Friday, January 27, 12
37. Meetups and Drinking basho Friday, January 27, 12
38. GitHub basho Friday, January 27, 12
39. Give Things Away basho Friday, January 27, 12
40. Build Communities Regardless basho Friday, January 27, 12
41. basho Community Fault Tolerance Friday, January 27, 12
42. DS3: Riak-based Distributed System basho Friday, January 27, 12
43. “A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal.” What is a distributed system? basho Friday, January 27, 12
44. • a database • a key/value store • distributed • fault-tolerant • scalable • Dynamo-inspired • used by startups • used by FORTUNE 100 companies • written (primarily) in Erlang • pronounced “REE-awk” • not the right fit for every project and app basho { Friday, January 27, 12
45. 1000s of Deployments Friday, January 27, 12
46. basho Friday, January 27, 12
47. basho Common Goals for Voxer’s System 1. Serve and Receive App Traffic 2. Perform Queries When Needed 3. Don’t Go Down 4. Scale Out to Meet Demand 5. Low, Consistent Response Times Friday, January 27, 12
48. Voxer’s Initial Riak Cluster Stats (Oct 2011) •11 Riak Nodes •Modest Data Set Size (100s of Gs) •~20,000 Peak Concurrent Users •~4,000,000 Daily Total Requests Then something happened... basho Friday, January 27, 12
49. Friday, January 27, 12
50. basho Friday, January 27, 12
51. Voxer’s Current Riak Cluster Stats • 41 Node Cluster for User Data • 37 Node Cluster to serve app traffic • ~350G/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month • At one point adding three nodes/day basho Friday, January 27, 12
52. basho Voxer’s Fault Tolerance • Have lost a lot of nodes in production • TCP Incast Problem [2] • LevelDB merge issues • Lots of other shit went wrong but it’s still running :) Friday, January 27, 12
53. “Scalability is the ability of a system, network, or process, to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.”[3] basho Friday, January 27, 12
54. Present System Health Dictates Future Ability to Scale basho Friday, January 27, 12
55. credit: http://blogs.ajc.com/jeff-schultz-blog/files/2009/06/closedsign.png basho Distributed [ Companies | Communities | Systems ] are all susceptible to downtime. Friday, January 27, 12
56. Capacity Plan or Perish basho Friday, January 27, 12
57. Everything Is Distributed Now basho Friday, January 27, 12
58. basho @pharkmillups themarkphillips.com mark@basho.com Mark Phillips Questions? Friday, January 27, 12
59. basho References 1. http://en.wikipedia.org/wiki/Distributed_computing 2. http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-is-it/ 3. http://en.wikipedia.org/wiki/Scalability Friday, January 27, 12