‹#›
Michael Basnight
Suyog Rao
From 50-250: Scaling a distributed
engineering team
Slide 2
Slide 2 text
Who are we?
Slide 3
Slide 3 text
Who are we?
Kibana
Elasticsearch
Beats Logstash
Security
Alerting
Monitoring
Reporting
X-Pack
Graph
Elastic Cloud
Slide 4
Slide 4 text
Who are we?
Slide 5
Slide 5 text
Who are we?
485 elasticians
Slide 6
Slide 6 text
Who are we?
485 elasticians
32 countries
Slide 7
Slide 7 text
Who are we?
485 elasticians
250 engineers
32 countries
Slide 8
Slide 8 text
Who are we?
485 elasticians
250 engineers
32 countries
50+ million downloads
Slide 9
Slide 9 text
5
At the beginning
Slide 10
Slide 10 text
The founders: 2012
Shay Uri Simon Steven
Slide 11
Slide 11 text
Next few hires:
Slide 12
Slide 12 text
Next few hires:CFO ??
Slide 13
Slide 13 text
7
Distributed By
Design
Slide 14
Slide 14 text
‹#›
A distributed system is a software system in
which components located on network
computers communicate and coordinate their
actions by passing messages
Slide 15
Slide 15 text
Elasticsearch the software,
was distributed from day 1
Slide 16
Slide 16 text
What does a distributed software
and distributed, open source
company have in common?
Slide 17
Slide 17 text
Lots
Slide 18
Slide 18 text
‹#›
Distributed systems:
communication
12
Slide 19
Slide 19 text
Asynchronous Comms
Slide 20
Slide 20 text
Email: Filters included
Timezone friendly
Slide 21
Slide 21 text
Email: Filters included
How we reached a decision
Slide 22
Slide 22 text
Email transparency
Slide 23
Slide 23 text
16
Some email become history
Slide 24
Slide 24 text
17
Reply-all is fun!
Slide 25
Slide 25 text
18
Emails become t-shirts
Slide 26
Slide 26 text
GitHub all the things
Slide 27
Slide 27 text
20
@mention someone
directly on
GH issue instead of
discussing
in Slack
Slide 28
Slide 28 text
‹#›
Slide 29
Slide 29 text
Synchronous Comms
Slide 30
Slide 30 text
23
Join any room
Technical discussions in public
Public communication will educate
everyone
Slide 31
Slide 31 text
24
Slide 32
Slide 32 text
25
Meetings
Slide 33
Slide 33 text
25
Dev Standups
25
Meetings
Slide 34
Slide 34 text
25
25
Dev Standups
Collapses
25
Meetings
Slide 35
Slide 35 text
25
25 50
Demo Day
Dev Standups
Collapses
25
Meetings
Slide 36
Slide 36 text
25
25
Table Group
50 70
Demo Day
Dev Standups
Collapses
25
Meetings
Slide 37
Slide 37 text
25
25
Table Group
50 70
Demo Day
Dev Standups
Collapses Collapses
70
25
Meetings
Slide 38
Slide 38 text
25
25
Table Group Black Belt
Session
50 70 140
Demo Day
Dev Standups
Collapses Collapses
70
25
Meetings
Slide 39
Slide 39 text
25
25
Table Group Black Belt
Session
Cross Product/
Per Product
Sync
50 70 140 200
Demo Day
Dev Standups
Collapses Collapses
70
25
Meetings
Slide 40
Slide 40 text
26
Meetings are open: dev cal
Slide 41
Slide 41 text
27
Persistent zoom session
Always On Zoom: hop on, hop off
Slide 42
Slide 42 text
Zoom release party
Slide 43
Slide 43 text
Engineering all-hands: 2x a year
Slide 44
Slide 44 text
Engineering all-hands: 2x a year
Vision, long term plans
Talks from CEO and investors
2 days team time
1 day cross product
Slide 45
Slide 45 text
Engineering all-hands:meetings with a view!
Slide 46
Slide 46 text
Engineering all-hands: fun day!
Slide 47
Slide 47 text
33
dev process
Agile, Waterfall
Slide 48
Slide 48 text
34
Consensus based
Slide 49
Slide 49 text
‹#›
GitHub to track things
Slide 50
Slide 50 text
36
Pull Requests: Peer Review
Slide 51
Slide 51 text
37
LGTMs == ACKs
Slide 52
Slide 52 text
Releases
Slide 53
Slide 53 text
39
Before: Independent releases
Slide 54
Slide 54 text
Time based releases
Slide 55
Slide 55 text
41
Release Process: Time based
x.y.z —> major.minor.patches
Slide 56
Slide 56 text
Release: Train analogy
Slide 57
Slide 57 text
Release Process: Time based
Merge when ready and scalable
Otherwise catch the next train
Important: No pressure on devs for pushing
features
Slide 58
Slide 58 text
44
Team dynamic
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
How’s the org structure
Flat org
Self organising teams
Highly available
Slide 61
Slide 61 text
47
Slide 62
Slide 62 text
Hire the best talent from anywhere
Globally distributed
Slide 63
Slide 63 text
A node has many roles
Slide 64
Slide 64 text
Engineers are nodes
Slide 65
Slide 65 text
Engineers are evangelists
Slide 66
Slide 66 text
Engineers write blogs
Slide 67
Slide 67 text
Engineers do support
Slide 68
Slide 68 text
Engineers write documentation
Slide 69
Slide 69 text
Engineers have autonomy
Slide 70
Slide 70 text
51
node
discovery node
looking for other
nodes in elastic
Slide 71
Slide 71 text
52
Slide 72
Slide 72 text
Hiring
“how many golf balls fit in an airplane ?”
Slide 73
Slide 73 text
Hiring
“how many golf balls fit in an airplane ?”
Slide 74
Slide 74 text
Hiring
No puzzle questions
Each team has different interview styles
Technical and team fit for distributed
environment
On Boarding: Mentoring
Day 1
High touch mentoring
Mentor != Manager
Slide 78
Slide 78 text
On Boarding: x-school
Week-long orientation in Mountain View, CA
Meet team leads, product team
Big Picture: Understand business functions
Slide 79
Slide 79 text
59
Takeaways
Slide 80
Slide 80 text
Key takeaways
Don’t manage by outliers
Slide 81
Slide 81 text
Key takeaways
Progress over perfection
Slide 82
Slide 82 text
In closing…
They say it won’t work at 50, 100, 500..
Slide 83
Slide 83 text
Questions?
Slide 84
Slide 84 text
More resources
https://www.elastic.co/elasticon/conf/2016/sf/inside-elastic-engineering
https://www.elastic.co/elasticon/conf/2017/sf/on-distributed-systems-and-distributed-teams
https://zachholman.com/posts/how-github-works/
Fonts and slide design were inspired by Zach Holman: https://zachholman.com/posts/slide-design-for-developers/