OSCON: Scaling a distributed engineering team from 50-250

OSCON: Scaling a distributed engineering team from 50-250

The Elastic engineering team is made up of ~250 employees, across more than ten teams, in over 30 countries — it’s been distributed since day one. It’s also been open source since day one, as the company behind the OSS projects Elasticsearch, Logstash, Kibana, and Beats.

This talk will cover how the engineering team has evolved as we've grown, how we work and communicate effectively across all timezones, the tools we use, and how we make decisions. We’ll discuss the core values we’ve defined and re-defined at different stages of growth, and how we identified and communicated those values. Additionally, we’ll talk about what team structures work well, how to hire, and balancing open source and commercial development.

098332e9d988080a9057816f84d668f7?s=128

Elasticsearch Inc

May 23, 2017
Tweet

Transcript

  1. ‹#› Michael Basnight Suyog Rao From 50-250: Scaling a distributed

    engineering team
  2. Who are we?

  3. Who are we? Kibana Elasticsearch Beats Logstash Security Alerting Monitoring

    Reporting X-Pack Graph Elastic Cloud
  4. Who are we?

  5. Who are we? 485 elasticians

  6. Who are we? 485 elasticians 32 countries

  7. Who are we? 485 elasticians 250 engineers 32 countries

  8. Who are we? 485 elasticians 250 engineers 32 countries 50+

    million downloads
  9. 5 At the beginning

  10. The founders: 2012 Shay Uri Simon Steven

  11. Next few hires:

  12. Next few hires:CFO ??

  13. 7 Distributed By Design

  14. ‹#› A distributed system is a software system in which

    components located on network computers communicate and coordinate their actions by passing messages
  15. Elasticsearch the software, was distributed from day 1

  16. What does a distributed software and distributed, open source company

    have in common?
  17. Lots

  18. ‹#› Distributed systems: communication 12

  19. Asynchronous Comms

  20. Email: Filters included Timezone friendly

  21. Email: Filters included How we reached a decision

  22. Email transparency

  23. 16 Some email become history

  24. 17 Reply-all is fun!

  25. 18 Emails become t-shirts

  26. GitHub all the things

  27. 20 @mention someone directly on GH issue instead of discussing

    in Slack
  28. ‹#›

  29. Synchronous Comms

  30. 23 Join any room Technical discussions in public Public communication

    will educate everyone
  31. 24

  32. 25 Meetings

  33. 25 Dev Standups 25 Meetings

  34. 25 25 Dev Standups Collapses 25 Meetings

  35. 25 25 50 Demo Day Dev Standups Collapses 25 Meetings

  36. 25 25 Table Group 50 70 Demo Day Dev Standups

    Collapses 25 Meetings
  37. 25 25 Table Group 50 70 Demo Day Dev Standups

    Collapses Collapses 70 25 Meetings
  38. 25 25 Table Group Black Belt Session 50 70 140

    Demo Day Dev Standups Collapses Collapses 70 25 Meetings
  39. 25 25 Table Group Black Belt Session Cross Product/
 Per

    Product Sync 50 70 140 200 Demo Day Dev Standups Collapses Collapses 70 25 Meetings
  40. 26 Meetings are open: dev cal

  41. 27 Persistent zoom session Always On Zoom: hop on, hop

    off
  42. Zoom release party

  43. Engineering all-hands: 2x a year

  44. Engineering all-hands: 2x a year Vision, long term plans Talks

    from CEO and investors 2 days team time 1 day cross product
  45. Engineering all-hands:meetings with a view!

  46. Engineering all-hands: fun day!

  47. 33 dev process Agile, Waterfall

  48. 34 Consensus based

  49. ‹#› GitHub to track things

  50. 36 Pull Requests: Peer Review

  51. 37 LGTMs == ACKs

  52. Releases

  53. 39 Before: Independent releases

  54. Time based releases

  55. 41 Release Process: Time based x.y.z —> major.minor.patches

  56. Release: Train analogy

  57. Release Process: Time based Merge when ready and scalable Otherwise

    catch the next train Important: No pressure on devs for pushing features
  58. 44 Team dynamic

  59. None
  60. How’s the org structure Flat org Self organising teams Highly

    available
  61. 47

  62. Hire the best talent from anywhere Globally distributed

  63. A node has many roles

  64. Engineers are nodes

  65. Engineers are evangelists

  66. Engineers write blogs

  67. Engineers do support

  68. Engineers write documentation

  69. Engineers have autonomy

  70. 51 node discovery node looking for other nodes in elastic

  71. 52

  72. Hiring “how many golf balls fit in an airplane ?”

  73. Hiring “how many golf balls fit in an airplane ?”

  74. Hiring No puzzle questions Each team has different interview styles

    Technical and team fit for distributed environment
  75. Nodes are independent

  76. 56 0 10 20 30 40 50 0-3 3-6 9-12

    12-15 15-18 18+ Average experience
  77. On Boarding: Mentoring Day 1 High touch mentoring Mentor !=

    Manager
  78. On Boarding: x-school Week-long orientation in Mountain View, CA Meet

    team leads, product team Big Picture: Understand business functions
  79. 59 Takeaways

  80. Key takeaways Don’t manage by outliers

  81. Key takeaways Progress over perfection

  82. In closing… They say it won’t work at 50, 100,

    500..
  83. Questions?

  84. More resources https://www.elastic.co/elasticon/conf/2016/sf/inside-elastic-engineering https://www.elastic.co/elasticon/conf/2017/sf/on-distributed-systems-and-distributed-teams https://zachholman.com/posts/how-github-works/ Fonts and slide design were

    inspired by Zach Holman: https://zachholman.com/posts/slide-design-for-developers/