Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OSCON: Scaling a distributed engineering team from 50-250

OSCON: Scaling a distributed engineering team from 50-250

The Elastic engineering team is made up of ~250 employees, across more than ten teams, in over 30 countries — it’s been distributed since day one. It’s also been open source since day one, as the company behind the OSS projects Elasticsearch, Logstash, Kibana, and Beats.

This talk will cover how the engineering team has evolved as we've grown, how we work and communicate effectively across all timezones, the tools we use, and how we make decisions. We’ll discuss the core values we’ve defined and re-defined at different stages of growth, and how we identified and communicated those values. Additionally, we’ll talk about what team structures work well, how to hire, and balancing open source and commercial development.

Elasticsearch Inc

May 23, 2017
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. ‹#›
    Michael Basnight
    Suyog Rao
    From 50-250: Scaling a distributed
    engineering team

    View Slide

  2. Who are we?

    View Slide

  3. Who are we?
    Kibana
    Elasticsearch
    Beats Logstash
    Security
    Alerting
    Monitoring
    Reporting
    X-Pack
    Graph
    Elastic Cloud

    View Slide

  4. Who are we?

    View Slide

  5. Who are we?
    485 elasticians

    View Slide

  6. Who are we?
    485 elasticians
    32 countries

    View Slide

  7. Who are we?
    485 elasticians
    250 engineers
    32 countries

    View Slide

  8. Who are we?
    485 elasticians
    250 engineers
    32 countries
    50+ million downloads

    View Slide

  9. 5
    At the beginning

    View Slide

  10. The founders: 2012
    Shay Uri Simon Steven

    View Slide

  11. Next few hires:

    View Slide

  12. Next few hires:CFO ??

    View Slide

  13. 7
    Distributed By
    Design

    View Slide

  14. ‹#›
    A distributed system is a software system in
    which components located on network
    computers communicate and coordinate their
    actions by passing messages

    View Slide

  15. Elasticsearch the software,
    was distributed from day 1

    View Slide

  16. What does a distributed software
    and distributed, open source
    company have in common?

    View Slide

  17. Lots

    View Slide

  18. ‹#›
    Distributed systems:
    communication
    12

    View Slide

  19. Asynchronous Comms

    View Slide

  20. Email: Filters included
    Timezone friendly

    View Slide

  21. Email: Filters included
    How we reached a decision

    View Slide

  22. Email transparency

    View Slide

  23. 16
    Some email become history

    View Slide

  24. 17
    Reply-all is fun!

    View Slide

  25. 18
    Emails become t-shirts

    View Slide

  26. GitHub all the things

    View Slide

  27. 20
    @mention someone
    directly on
    GH issue instead of
    discussing
    in Slack

    View Slide

  28. ‹#›

    View Slide

  29. Synchronous Comms

    View Slide

  30. 23
    Join any room
    Technical discussions in public
    Public communication will educate
    everyone

    View Slide

  31. 24

    View Slide

  32. 25
    Meetings

    View Slide

  33. 25
    Dev Standups
    25
    Meetings

    View Slide

  34. 25
    25
    Dev Standups
    Collapses
    25
    Meetings

    View Slide

  35. 25
    25 50
    Demo Day
    Dev Standups
    Collapses
    25
    Meetings

    View Slide

  36. 25
    25
    Table Group
    50 70
    Demo Day
    Dev Standups
    Collapses
    25
    Meetings

    View Slide

  37. 25
    25
    Table Group
    50 70
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View Slide

  38. 25
    25
    Table Group Black Belt
    Session
    50 70 140
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View Slide

  39. 25
    25
    Table Group Black Belt
    Session
    Cross Product/

    Per Product
    Sync
    50 70 140 200
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View Slide

  40. 26
    Meetings are open: dev cal

    View Slide

  41. 27
    Persistent zoom session
    Always On Zoom: hop on, hop off

    View Slide

  42. Zoom release party

    View Slide

  43. Engineering all-hands: 2x a year

    View Slide

  44. Engineering all-hands: 2x a year
    Vision, long term plans
    Talks from CEO and investors
    2 days team time
    1 day cross product

    View Slide

  45. Engineering all-hands:meetings with a view!

    View Slide

  46. Engineering all-hands: fun day!

    View Slide

  47. 33
    dev process
    Agile, Waterfall

    View Slide

  48. 34
    Consensus based

    View Slide

  49. ‹#›
    GitHub to track things

    View Slide

  50. 36
    Pull Requests: Peer Review

    View Slide

  51. 37
    LGTMs == ACKs

    View Slide

  52. Releases

    View Slide

  53. 39
    Before: Independent releases

    View Slide

  54. Time based releases

    View Slide

  55. 41
    Release Process: Time based
    x.y.z —> major.minor.patches

    View Slide

  56. Release: Train analogy

    View Slide

  57. Release Process: Time based
    Merge when ready and scalable
    Otherwise catch the next train
    Important: No pressure on devs for pushing
    features

    View Slide

  58. 44
    Team dynamic

    View Slide

  59. View Slide

  60. How’s the org structure
    Flat org
    Self organising teams
    Highly available

    View Slide

  61. 47

    View Slide

  62. Hire the best talent from anywhere
    Globally distributed

    View Slide

  63. A node has many roles

    View Slide

  64. Engineers are nodes

    View Slide

  65. Engineers are evangelists

    View Slide

  66. Engineers write blogs

    View Slide

  67. Engineers do support

    View Slide

  68. Engineers write documentation

    View Slide

  69. Engineers have autonomy

    View Slide

  70. 51
    node
    discovery node
    looking for other
    nodes in elastic

    View Slide

  71. 52

    View Slide

  72. Hiring
    “how many golf balls fit in an airplane ?”

    View Slide

  73. Hiring
    “how many golf balls fit in an airplane ?”

    View Slide

  74. Hiring
    No puzzle questions
    Each team has different interview styles
    Technical and team fit for distributed
    environment

    View Slide

  75. Nodes are independent

    View Slide

  76. 56
    0
    10
    20
    30
    40
    50
    0-3 3-6 9-12 12-15 15-18 18+
    Average experience

    View Slide

  77. On Boarding: Mentoring
    Day 1
    High touch mentoring
    Mentor != Manager

    View Slide

  78. On Boarding: x-school
    Week-long orientation in Mountain View, CA
    Meet team leads, product team
    Big Picture: Understand business functions

    View Slide

  79. 59
    Takeaways

    View Slide

  80. Key takeaways
    Don’t manage by outliers

    View Slide

  81. Key takeaways
    Progress over perfection

    View Slide

  82. In closing…
    They say it won’t work at 50, 100, 500..

    View Slide

  83. Questions?

    View Slide

  84. More resources
    https://www.elastic.co/elasticon/conf/2016/sf/inside-elastic-engineering
    https://www.elastic.co/elasticon/conf/2017/sf/on-distributed-systems-and-distributed-teams
    https://zachholman.com/posts/how-github-works/
    Fonts and slide design were inspired by Zach Holman: https://zachholman.com/posts/slide-design-for-developers/

    View Slide