OSCON: Scaling a distributed engineering team from 50-250

The Elastic engineering team is made up of ~250 employees, across more than ten teams, in over 30 countries — it’s been distributed since day one. It’s also been open source since day one, as the company behind the OSS projects Elasticsearch, Logstash, Kibana, and Beats.

This talk will cover how the engineering team has evolved as we've grown, how we work and communicate effectively across all timezones, the tools we use, and how we make decisions. We’ll discuss the core values we’ve defined and re-defined at different stages of growth, and how we identified and communicated those values. Additionally, we’ll talk about what team structures work well, how to hire, and balancing open source and commercial development.

Transcript

1. ‹#› Michael Basnight Suyog Rao From 50-250: Scaling a distributed engineering team
2. Who are we?
3. Who are we? Kibana Elasticsearch Beats Logstash Security Alerting Monitoring Reporting X-Pack Graph Elastic Cloud
4. Who are we?
5. Who are we? 485 elasticians
6. Who are we? 485 elasticians 32 countries
7. Who are we? 485 elasticians 250 engineers 32 countries
8. Who are we? 485 elasticians 250 engineers 32 countries 50+ million downloads
9. 5 At the beginning
10. The founders: 2012 Shay Uri Simon Steven
11. Next few hires:
12. Next few hires:CFO ??
13. 7 Distributed By Design
14. ‹#› A distributed system is a software system in which components located on network computers communicate and coordinate their actions by passing messages
15. Elasticsearch the software, was distributed from day 1
16. What does a distributed software and distributed, open source company have in common?
17. Lots
18. ‹#› Distributed systems: communication 12
19. Asynchronous Comms
20. Email: Filters included Timezone friendly
21. Email: Filters included How we reached a decision
22. Email transparency
23. 16 Some email become history
24. 17 Reply-all is fun!
25. 18 Emails become t-shirts
26. GitHub all the things
27. 20 @mention someone directly on GH issue instead of discussing in Slack
28. ‹#›
29. Synchronous Comms
30. 23 Join any room Technical discussions in public Public communication will educate everyone
32. 25 Meetings
33. 25 Dev Standups 25 Meetings
34. 25 25 Dev Standups Collapses 25 Meetings
35. 25 25 50 Demo Day Dev Standups Collapses 25 Meetings
36. 25 25 Table Group 50 70 Demo Day Dev Standups Collapses 25 Meetings
37. 25 25 Table Group 50 70 Demo Day Dev Standups Collapses Collapses 70 25 Meetings
38. 25 25 Table Group Black Belt Session 50 70 140 Demo Day Dev Standups Collapses Collapses 70 25 Meetings
39. 25 25 Table Group Black Belt Session Cross Product/
 Per Product Sync 50 70 140 200 Demo Day Dev Standups Collapses Collapses 70 25 Meetings
40. 26 Meetings are open: dev cal
41. 27 Persistent zoom session Always On Zoom: hop on, hop off
42. Zoom release party
43. Engineering all-hands: 2x a year
44. Engineering all-hands: 2x a year Vision, long term plans Talks from CEO and investors 2 days team time 1 day cross product
45. Engineering all-hands:meetings with a view!
46. Engineering all-hands: fun day!
47. 33 dev process Agile, Waterfall
48. 34 Consensus based
49. ‹#› GitHub to track things
50. 36 Pull Requests: Peer Review
51. 37 LGTMs == ACKs
52. Releases
53. 39 Before: Independent releases
54. Time based releases
55. 41 Release Process: Time based x.y.z —> major.minor.patches
56. Release: Train analogy
57. Release Process: Time based Merge when ready and scalable Otherwise catch the next train Important: No pressure on devs for pushing features
58. 44 Team dynamic
60. How’s the org structure Flat org Self organising teams Highly available
62. Hire the best talent from anywhere Globally distributed
63. A node has many roles
64. Engineers are nodes
65. Engineers are evangelists
66. Engineers write blogs
67. Engineers do support
68. Engineers write documentation
69. Engineers have autonomy
70. 51 node discovery node looking for other nodes in elastic
72. Hiring “how many golf balls fit in an airplane ?”
73. Hiring “how many golf balls fit in an airplane ?”
74. Hiring No puzzle questions Each team has different interview styles Technical and team fit for distributed environment
75. Nodes are independent
76. 56 0 10 20 30 40 50 0-3 3-6 9-12 12-15 15-18 18+ Average experience
77. On Boarding: Mentoring Day 1 High touch mentoring Mentor != Manager
78. On Boarding: x-school Week-long orientation in Mountain View, CA Meet team leads, product team Big Picture: Understand business functions
79. 59 Takeaways
80. Key takeaways Don’t manage by outliers
81. Key takeaways Progress over perfection
82. In closing… They say it won’t work at 50, 100, 500..
83. Questions?
84. More resources https://www.elastic.co/elasticon/conf/2016/sf/inside-elastic-engineering https://www.elastic.co/elasticon/conf/2017/sf/on-distributed-systems-and-distributed-teams https://zachholman.com/posts/how-github-works/ Fonts and slide design were inspired by Zach Holman: https://zachholman.com/posts/slide-design-for-developers/