Building Healthy Distributed Systems

Slides from my "Building Healthy Distributed Systems" talk delivered at Erlang Factor in March 2012 in San Francisco.

Video here - https://vimeo.com/43027087

Transcript

1. Building Healthy Distributed Systems Erlang Factory SF March 29, 2012 basho Thursday, March 29, 12
2. basho @pharkmillups themarkphillips.com mark@basho.com Mark Phillips Thursday, March 29, 12
3. “A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal.” [1] What is a distributed system? basho Thursday, March 29, 12
4. Distributed, Scalable, Fault Tolerant No central coordinator; Easy to setup and operate basho Thursday, March 29, 12
5. Distributed, Scalable, Fault Tolerant Horizontally Scalable; Add commodity hardware to get more [throughput | processing | storage]. basho Thursday, March 29, 12
6. Distributed, Scalable, Fault Tolerant Always Available No Single Point of Failure Self-healing basho Thursday, March 29, 12
7. basho Thursday, March 29, 12
8. basho { • Founded in 2007 • Collapsed in 2008 • “Pivoted” in 2009 • Commercial Sponsors of Riak, an Open Source, NoSQL Database • Sells Closed Source Extensions to Riak in the form licenses Thursday, March 29, 12
9. Year on Year Growth basho Thursday, March 29, 12
10. 2009 Year on Year Growth basho Thursday, March 29, 12
11. 2009 14 Year on Year Growth basho Thursday, March 29, 12
12. 2009 2010 14 Year on Year Growth basho Thursday, March 29, 12
13. 2009 2010 14 25 Year on Year Growth basho Thursday, March 29, 12
14. 2009 2010 2011 14 25 Year on Year Growth basho Thursday, March 29, 12
15. 2009 2010 2011 14 60 25 Year on Year Growth basho Thursday, March 29, 12
16. basho Office Locations Thursday, March 29, 12
17. basho Office Locations Thursday, March 29, 12
18. basho Office Locations Thursday, March 29, 12
19. basho Office Locations Thursday, March 29, 12
20. Actual Employee Distribution basho Thursday, March 29, 12
21. “A distributed [company] consists of multiple autonomous [team members] that communicate [and collaborate] through various [channels]. The [team members] interact with each other in order to achieve a common goal.” What is a distributed [company]? basho Thursday, March 29, 12
22. Hiring where the talent is means we don’t sacrifice great hires for location, but it also presents various hurdles when attempting to build culture and community. basho basho Thursday, March 29, 12
23. 1. Make Basho into a Powerhouse 2. Professional Development 3. Employee Happiness 4. Deliver Exceptional Product Common Goals for Basho basho Thursday, March 29, 12
24. Internal Communication and Collaboration • Real-time Chat (Jabber, Camp Fire) • Skype (or some for of video chat) • Yammer • GitHub • AgileZen • Email (sort of) • Documentation basho Thursday, March 29, 12
25. Culture: Be Corny and Childish basho Thursday, March 29, 12
26. basho Thursday, March 29, 12
27. basho Thursday, March 29, 12
28. basho Thursday, March 29, 12
29. basho Thursday, March 29, 12
30. basho Thursday, March 29, 12
31. basho Thursday, March 29, 12
32. Good Meetings basho • Quarterly In-person “Summits” • Bi-Monthly, Non-Mandatory Company All Hands • Stands up, Scrum Thursday, March 29, 12
33. Make Documentation Part of Your Culture basho • Inside Jokes • Internal Talks • Design Documents • Product Ideas • Product Feedback • New Hire Processes • Everything Else Thursday, March 29, 12
34. Open Source Your Code. And Use GitHub. basho • Contributes Directly to Developer Happiness • Makes Your Company’s Product Better • Great Marketing • Use a Permissive License (http://bit.ly/clJyDO) (http://bit.ly/v3OMEf) “Open Source Almost Everything” “Why Your Company Should Have a Permissive Open Source Policy” Thursday, March 29, 12
35. Thursday, March 29, 12
36. basho basho Hiring Should Not Happen In A Vacuum Thursday, March 29, 12
37. Poor Culture Rots a Company from within and Lessens its Resiliency basho Thursday, March 29, 12
38. basho Company Fault Tolerance • New CEO + Massive Growth = New Challenges • Our System is Constantly Improving Thursday, March 29, 12
39. 2012 1** Planned Growth basho Thursday, March 29, 12
40. Community Thursday, March 29, 12
41. “A distributed [community] consists of multiple autonomous [members] that communicate [and collaborate] through various [channels]. The [members] interact with each other in order to achieve a common goal.” What is a distributed [community]? basho Thursday, March 29, 12
42. basho Why Build A Community? Thursday, March 29, 12
43. Grassroots Marketing, Branding, Awareness: basho Thursday, March 29, 12
44. Code Contributions and Bug Fixes : basho 180 names in our THANKS file 1600 hours contributed from Oct 2010 - Sept 2011 Thursday, March 29, 12
45. Support: basho Thursday, March 29, 12
46. Revenue: basho 75% of new customers in 2011 came from the Open Source Community Thursday, March 29, 12
47. Importance of Community for Community Members basho •Working, Quality Code •Recognition and Praise •Desire to Contribute •Jobs (whether they like it or not) •Skills Acquisition Thursday, March 29, 12
48. Communication and Collaboration in a Distributed [Community] basho •IRC •Mailing List •Twitter •Riak Recap •Meetups •Q & A Sites •Blogs •Books •Conferences •Actual Meetings •GitHub •Drinking Thursday, March 29, 12
49. Riak Recap basho Thursday, March 29, 12
50. Books basho http://riakhandbook.com/ Thursday, March 29, 12
51. Meetups and Drinking basho Thursday, March 29, 12
52. GitHub basho Thursday, March 29, 12
53. Give Things Away basho Thursday, March 29, 12
54. Build Communities Regardless basho Thursday, March 29, 12
55. basho Community Fault Tolerance Thursday, March 29, 12
56. basho Thursday, March 29, 12
57. “A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal.” What is a distributed system? basho Thursday, March 29, 12
58. • a database • a key/value store • distributed • fault-tolerant • scalable • Dynamo-inspired • used by startups • used by FORTUNE 100 companies • written (primarily) in Erlang • pronounced “REE-awk” • not the right fit for every project and app basho { Thursday, March 29, 12
59. 1000s of Deployments Thursday, March 29, 12
60. basho Thursday, March 29, 12
61. basho Common Goals for Voxer’s System 1. Serve and Receive App Traffic 2. Perform Queries When Needed 3. Don’t Go Down 4. Scale Out to Meet Demand 5. Low, Consistent Response Times Thursday, March 29, 12
62. Voxer’s Initial Riak Cluster Stats (Oct 2011) •11 Riak Nodes •Modest Data Set Size (100s of Gs) •~20,000 Peak Concurrent Users •~4,000,000 Daily Total Requests Then something happened... basho Thursday, March 29, 12
63. Thursday, March 29, 12
64. basho Thursday, March 29, 12
65. Voxer’s Current Riak Cluster Stats basho Thursday, March 29, 12
66. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data basho Thursday, March 29, 12
67. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data • >40 Node Cluster to serve app traffic basho Thursday, March 29, 12
68. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily basho Thursday, March 29, 12
69. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak basho Thursday, March 29, 12
70. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month basho Thursday, March 29, 12
71. Voxer’s Current Riak Cluster Stats • >40 Node Cluster for User Data • >40 Node Cluster to serve app traffic • ~1TB/day of user data being added daily • 100,000s of concurrent users at peak • Went from 11 to about 80 nodes in a month • At one point adding three nodes/day basho Thursday, March 29, 12
72. basho Voxer’s Fault Tolerance • Have lost a lot of nodes in production • TCP Incast Problem [2] • LevelDB merge issues • Lots of other shit went wrong but it’s still running :) Thursday, March 29, 12
73. “Scalability is the ability of a system, network, or process, to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.”[3] basho Thursday, March 29, 12
74. Present System Health Dictates Future Ability to Scale basho Thursday, March 29, 12
75. credit: http://blogs.ajc.com/jeff-schultz-blog/files/2009/06/closedsign.png basho Distributed [ Companies | Communities | Systems ] are all susceptible to downtime. Thursday, March 29, 12
76. Capacity Plan or Perish basho Thursday, March 29, 12
77. Everything Is Distributed Now basho Thursday, March 29, 12
78. basho @pharkmillups themarkphillips.com mark@basho.com Mark Phillips Questions? Thursday, March 29, 12
79. basho References 1. http://en.wikipedia.org/wiki/Distributed_computing 2. http://www.snookles.com/slf-blog/2012/01/05/tcp-incast-what-is-it/ 3. http://en.wikipedia.org/wiki/Scalability Thursday, March 29, 12