This isn’t a “management problem”.
Everyone needs to worry about this.
Slide 14
Slide 14 text
Hiring an employee is the
most thing
you can do to your startup.
T O X I C
Slide 15
Slide 15 text
Hiring an employee is the
most thing
you can do to your startup.
T O X I C
work slower
more bugs less features
worse culture
Slide 16
Slide 16 text
Hiring an employee is the
most thing
you can do to your startup.
EXCITING
work faster
fewer bugs more features
better culture
Slide 17
Slide 17 text
so how can you
score excitement
and avoid the toxic?
Slide 18
Slide 18 text
TOXIC EXCITEMENT
would be a great name for a rock band
yeah, i know...
Slide 19
Slide 19 text
k
S
e
k 2
k
S
S
k
k
k
k
S
S
k
S
UKeep your employees happy.
Really happy.
Slide 20
Slide 20 text
Your servers, offices, and ideas are bullshit.
Worry about your coworkers.
Slide 21
Slide 21 text
EMPLOYEES NEW HIRES
Know your codebase
Know your process
Know your mistakes
Know your mission
Don’t know jack
Know your jokes
Know your priorities
Slide 22
Slide 22 text
Imprison your employees with happiness and
nice things and cuddly work practices.
Slide 23
Slide 23 text
GitHub Jail
work whenever you want
work however you want
work on what you want
health, dental, vision
paid conference trips
retirement plans
solid salaries
a product people love
four beers on tap stock
Slide 24
Slide 24 text
get out of the way
NO MEETINGS
NO PLANNING SESSIONS
NO NEED TO BE IN THE OFFICE
chat, pull requests, email
MORE DIRECT
FASTER
ALWAYS RECORDED
Slide 25
Slide 25 text
This is designed to retain people.
We’re at 56 employees. We haven’t lost one.
This is a huge, massive competitive advantage.
It justifies the extra expense.
Slide 26
Slide 26 text
Communication.
Slide 27
Slide 27 text
Don’t have the server guy who knows everything.
the billing girl
the testing dude
the customer support maven
the performance czar
the software licensing file hoarder
Slide 28
Slide 28 text
Don’t have the person who knows everything.
Slide 29
Slide 29 text
Specialization is great,
but only having one person
is a synchronous bottleneck.
V Every internal GitHub talk
is automatically recorded,
uploaded, and viewable to
every future employee.
Slide 33
Slide 33 text
V ...on a Kinect-powered
Arduino-based motion-
detecting portable video
recording platform.
Slide 34
Slide 34 text
Your new hire is stoked to dive in,
start reading, and start contributing
...so don’t get in their way.
Slide 35
Slide 35 text
Hire well.
Slide 36
Slide 36 text
Hiring poorly is just as bad
as losing people.
Slide 37
Slide 37 text
Aim for really great people.
Slide 38
Slide 38 text
WE SELF-STARTERS
k
less babysitting, more code
Slide 39
Slide 39 text
k
S
e
k 2
k
S
S
k
k
k
k
S
S
k
S
UKeep your employees happy.
Really happy.
(future!)
Slide 40
Slide 40 text
Don’t just market your product;
market your team and company too.
Slide 41
Slide 41 text
Always think
about attracting
good people,
even if you’re
not hiring.
OPEN SOURCE
CONFERENCES
TECHNICAL POSTS
SPONSORSHIPS
MEETUPS
TALKS
Slide 42
Slide 42 text
Technical
robots can be pretty finicky too
Slide 43
Slide 43 text
Automate.
Slide 44
Slide 44 text
hubot deploy github to production
COMPILATION
CoffeeScript
SCSS and SASS
bundles assets
caches Python dependencies
compiles Erlang changes
compiles C changes
builds static pages
APP SETUP
installs gems
symlink directories
14 rolling app server restarts
NOTIFY
Campfire
New Relic
graphite
fs fs fs fs fs fs fs fs fs fs
fs fs fs fs fs fs fs fs fs fs
fe fe fe fe fe fe fe fe fe fe
fe fe fe fe fe fe fe fe fe fe
fe fe fe fe fe fe fe fe fe fe
fs fs fs fs fs fs fs fs fs fs
Slide 45
Slide 45 text
deploys
current process overview
multi-server shell commands
new employee setup
app bootstrap
Slide 46
Slide 46 text
Automating now will save you way
more time down the road.
Slide 47
Slide 47 text
Ship.
Slide 48
Slide 48 text
Ship early, ship often.
5x-30x
deploys per day
Slide 49
Slide 49 text
master = always deployable
always green tests
always a safe rollback
Slide 50
Slide 50 text
Limit your deployments
to staff-only
to beta users only
to one server only
to one app process on one server only
Slide 51
Slide 51 text
@github tweets
exceptions
deploys
deploys
Slide 52
Slide 52 text
Graph.
Slide 53
Slide 53 text
everyone loves fancy graphs
quickly see trends
quickly see problems
historical data as basis for alerts
Slide 54
Slide 54 text
METRICS ARE GREAT
But use them wisely.
Slide 55
Slide 55 text
162ms
average overall response time
Slide 56
Slide 56 text
Valueless metric.
Slide 57
Slide 57 text
59ms
average API response time
with 4x throughput of web
Slide 58
Slide 58 text
23ms
average raw response time
with 2x throughput of web
Slide 59
Slide 59 text
The responsiveness is a lie.
Slide 60
Slide 60 text
199ms
average browser response time
Slide 61
Slide 61 text
16,000
requests in the last week over 4.5s
Slide 62
Slide 62 text
Needed to look at the
right stuff.
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
throttled google
googlebot
2-3x throughput
3-4x CPU usage
had
web requests
compared to
Slide 65
Slide 65 text
Collect a lot of metrics,
but make sure they’re
important metrics.
Slide 66
Slide 66 text
GitHub scale.
Slide 67
Slide 67 text
Everyone has different
growth patterns.
Slide 68
Slide 68 text
GitHub has had three.
Slide 69
Slide 69 text
Launch
2008
Bare metal servers
2009
net-shard
2010
major github
infrastructure milestones
Slide 70
Slide 70 text
Launch
2008
Hosted on Engine Yard
10 VMs
54GB RAM
shared GFS mount
one metric shit-ton of caching
Slide 71
Slide 71 text
Bare metal servers
2009
Hosted on Rackspace
16 bare metal servers
288GB of RAM
redundant disk storage
Slide 72
Slide 72 text
net-shard
2010
networks share a common repository
rails/rails
holman/rails github/rails
+1 commit +30 commits
classic net-shard
rails network repo
...multiplied 2,600 times
holman/rails rails/rails github/rails
fat network, skeleton forks
Slide 73
Slide 73 text
net-shard
2010
networks share a common repository
they also share the same fs and partition
halves storage requirements
improves hit rate of kernel disk cache
speeds up backups
allows fast forks, merge button, network GC
Slide 74
Slide 74 text
For GitHub, scaling involved a lot of
predictions of future trends, then
acting appropriately.
Slide 75
Slide 75 text
Side Projects.
Slide 76
Slide 76 text
A THOUGHT EXPERIMENT:
Imagine I told you to build...
Slide 77
Slide 77 text
No content
Slide 78
Slide 78 text
This grew organically, over dozens of
projects, written by dozens of employees,
when they felt like it.
Slide 79
Slide 79 text
Figure out how to let this happen. It’s hard.
Slide 80
Slide 80 text
Small hack days can result in
real, imma-make-us-money impact.
Slide 81
Slide 81 text
Small hack days can also keep your
developers insanely happy.
Slide 82
Slide 82 text
Small hack days can also lead to
learning new techniques.
Slide 83
Slide 83 text
Projects and Posts.
Slide 84
Slide 84 text
JENKINS + CAMPFIRE
github.com/github/janky
CHAT ROOM ROBOT
github.com/github/hubot
OFFICE MUSIC DJ
github.com/holman/play
Slide 85
Slide 85 text
BLOG: GITHUB IS MOVING TO RACKSPACE
git.io/jByrlQ
BLOG: HOW WE MADE GITHUB FAST
git.io/p5v2Ag
BLOG: UNICORN
git.io/77Onfg
Slide 86
Slide 86 text
+
Technical
Organizational
Slide 87
Slide 87 text
Continually refine your
process + workflow.
Slide 88
Slide 88 text
Worry about your
computers, and worry
about your humans.