Upgrade to Pro — share decks privately, control downloads, hide ads and more …

OSCON: Scaling a distributed engineering team from 50-250

OSCON: Scaling a distributed engineering team from 50-250

The Elastic engineering team is made up of ~250 employees, across more than ten teams, in over 30 countries — it’s been distributed since day one. It’s also been open source since day one, as the company behind the OSS projects Elasticsearch, Logstash, Kibana, and Beats.

This talk will cover how the engineering team has evolved as we've grown, how we work and communicate effectively across all timezones, the tools we use, and how we make decisions. We’ll discuss the core values we’ve defined and re-defined at different stages of growth, and how we identified and communicated those values. Additionally, we’ll talk about what team structures work well, how to hire, and balancing open source and commercial development.

Elasticsearch Inc

May 23, 2017
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. ‹#›
    Michael Basnight
    Suyog Rao
    From 50-250: Scaling a distributed
    engineering team

    View full-size slide

  2. Who are we?
    Kibana
    Elasticsearch
    Beats Logstash
    Security
    Alerting
    Monitoring
    Reporting
    X-Pack
    Graph
    Elastic Cloud

    View full-size slide

  3. Who are we?
    485 elasticians

    View full-size slide

  4. Who are we?
    485 elasticians
    32 countries

    View full-size slide

  5. Who are we?
    485 elasticians
    250 engineers
    32 countries

    View full-size slide

  6. Who are we?
    485 elasticians
    250 engineers
    32 countries
    50+ million downloads

    View full-size slide

  7. 5
    At the beginning

    View full-size slide

  8. The founders: 2012
    Shay Uri Simon Steven

    View full-size slide

  9. Next few hires:

    View full-size slide

  10. Next few hires:CFO ??

    View full-size slide

  11. 7
    Distributed By
    Design

    View full-size slide

  12. ‹#›
    A distributed system is a software system in
    which components located on network
    computers communicate and coordinate their
    actions by passing messages

    View full-size slide

  13. Elasticsearch the software,
    was distributed from day 1

    View full-size slide

  14. What does a distributed software
    and distributed, open source
    company have in common?

    View full-size slide

  15. ‹#›
    Distributed systems:
    communication
    12

    View full-size slide

  16. Asynchronous Comms

    View full-size slide

  17. Email: Filters included
    Timezone friendly

    View full-size slide

  18. Email: Filters included
    How we reached a decision

    View full-size slide

  19. Email transparency

    View full-size slide

  20. 16
    Some email become history

    View full-size slide

  21. 17
    Reply-all is fun!

    View full-size slide

  22. 18
    Emails become t-shirts

    View full-size slide

  23. GitHub all the things

    View full-size slide

  24. 20
    @mention someone
    directly on
    GH issue instead of
    discussing
    in Slack

    View full-size slide

  25. Synchronous Comms

    View full-size slide

  26. 23
    Join any room
    Technical discussions in public
    Public communication will educate
    everyone

    View full-size slide

  27. 25
    Dev Standups
    25
    Meetings

    View full-size slide

  28. 25
    25
    Dev Standups
    Collapses
    25
    Meetings

    View full-size slide

  29. 25
    25 50
    Demo Day
    Dev Standups
    Collapses
    25
    Meetings

    View full-size slide

  30. 25
    25
    Table Group
    50 70
    Demo Day
    Dev Standups
    Collapses
    25
    Meetings

    View full-size slide

  31. 25
    25
    Table Group
    50 70
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View full-size slide

  32. 25
    25
    Table Group Black Belt
    Session
    50 70 140
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View full-size slide

  33. 25
    25
    Table Group Black Belt
    Session
    Cross Product/

    Per Product
    Sync
    50 70 140 200
    Demo Day
    Dev Standups
    Collapses Collapses
    70
    25
    Meetings

    View full-size slide

  34. 26
    Meetings are open: dev cal

    View full-size slide

  35. 27
    Persistent zoom session
    Always On Zoom: hop on, hop off

    View full-size slide

  36. Zoom release party

    View full-size slide

  37. Engineering all-hands: 2x a year

    View full-size slide

  38. Engineering all-hands: 2x a year
    Vision, long term plans
    Talks from CEO and investors
    2 days team time
    1 day cross product

    View full-size slide

  39. Engineering all-hands:meetings with a view!

    View full-size slide

  40. Engineering all-hands: fun day!

    View full-size slide

  41. 33
    dev process
    Agile, Waterfall

    View full-size slide

  42. 34
    Consensus based

    View full-size slide

  43. ‹#›
    GitHub to track things

    View full-size slide

  44. 36
    Pull Requests: Peer Review

    View full-size slide

  45. 37
    LGTMs == ACKs

    View full-size slide

  46. 39
    Before: Independent releases

    View full-size slide

  47. Time based releases

    View full-size slide

  48. 41
    Release Process: Time based
    x.y.z —> major.minor.patches

    View full-size slide

  49. Release: Train analogy

    View full-size slide

  50. Release Process: Time based
    Merge when ready and scalable
    Otherwise catch the next train
    Important: No pressure on devs for pushing
    features

    View full-size slide

  51. 44
    Team dynamic

    View full-size slide

  52. How’s the org structure
    Flat org
    Self organising teams
    Highly available

    View full-size slide

  53. Hire the best talent from anywhere
    Globally distributed

    View full-size slide

  54. A node has many roles

    View full-size slide

  55. Engineers are nodes

    View full-size slide

  56. Engineers are evangelists

    View full-size slide

  57. Engineers write blogs

    View full-size slide

  58. Engineers do support

    View full-size slide

  59. Engineers write documentation

    View full-size slide

  60. Engineers have autonomy

    View full-size slide

  61. 51
    node
    discovery node
    looking for other
    nodes in elastic

    View full-size slide

  62. Hiring
    “how many golf balls fit in an airplane ?”

    View full-size slide

  63. Hiring
    “how many golf balls fit in an airplane ?”

    View full-size slide

  64. Hiring
    No puzzle questions
    Each team has different interview styles
    Technical and team fit for distributed
    environment

    View full-size slide

  65. Nodes are independent

    View full-size slide

  66. 56
    0
    10
    20
    30
    40
    50
    0-3 3-6 9-12 12-15 15-18 18+
    Average experience

    View full-size slide

  67. On Boarding: Mentoring
    Day 1
    High touch mentoring
    Mentor != Manager

    View full-size slide

  68. On Boarding: x-school
    Week-long orientation in Mountain View, CA
    Meet team leads, product team
    Big Picture: Understand business functions

    View full-size slide

  69. Key takeaways
    Don’t manage by outliers

    View full-size slide

  70. Key takeaways
    Progress over perfection

    View full-size slide

  71. In closing…
    They say it won’t work at 50, 100, 500..

    View full-size slide

  72. More resources
    https://www.elastic.co/elasticon/conf/2016/sf/inside-elastic-engineering
    https://www.elastic.co/elasticon/conf/2017/sf/on-distributed-systems-and-distributed-teams
    https://zachholman.com/posts/how-github-works/
    Fonts and slide design were inspired by Zach Holman: https://zachholman.com/posts/slide-design-for-developers/

    View full-size slide