Slide 1

Slide 1 text

Scaling GitHub Scaling GitHub Scaling GitHub ling GitHub Scaling GitHub Scaling GitHub Scaling GitHub Scaling GitHub Scaling GitHub Scaling github SCALING GITHUB scalin’ githubs Scaling GitHub Scaling GitHub Scaling GitHub githubs and shit Scaling GitHub Scaling GitHub Scaling Startups B=======D~~~~ Scaling GitHub

Slide 2

Slide 2 text

“Scale”

Slide 3

Slide 3 text

Two problems.

Slide 4

Slide 4 text

SyntaxError: compile error. I’m too hungover to work. ORGANIZATIONAL TECHNICAL

Slide 5

Slide 5 text

Scaling is people + technology

Slide 6

Slide 6 text

@holman

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Organizational jeez humans are so finicky

Slide 9

Slide 9 text

Happiness.

Slide 10

Slide 10 text

0 250,000 500,000 750,000 1,000,000 Happiness vs Productivity ☹ ☺ $ $$$

Slide 11

Slide 11 text

happy employees are productive employees

Slide 12

Slide 12 text

productive employees are happy employees

Slide 13

Slide 13 text

This isn’t a “management problem”. Everyone needs to worry about this.

Slide 14

Slide 14 text

Hiring an employee is the most thing you can do to your startup. T O X I C

Slide 15

Slide 15 text

Hiring an employee is the most thing you can do to your startup. T O X I C work slower more bugs less features worse culture

Slide 16

Slide 16 text

Hiring an employee is the most thing you can do to your startup. EXCITING work faster fewer bugs more features better culture

Slide 17

Slide 17 text

so how can you score excitement and avoid the toxic?

Slide 18

Slide 18 text

TOXIC EXCITEMENT would be a great name for a rock band yeah, i know...

Slide 19

Slide 19 text

k S e k 2 k S S k k k k S S k S UKeep your employees happy. Really happy.

Slide 20

Slide 20 text

Your servers, offices, and ideas are bullshit. Worry about your coworkers.

Slide 21

Slide 21 text

EMPLOYEES NEW HIRES Know your codebase Know your process Know your mistakes Know your mission Don’t know jack Know your jokes Know your priorities

Slide 22

Slide 22 text

Imprison your employees with happiness and nice things and cuddly work practices.

Slide 23

Slide 23 text

GitHub Jail work whenever you want work however you want work on what you want health, dental, vision paid conference trips retirement plans solid salaries a product people love four beers on tap stock

Slide 24

Slide 24 text

get out of the way NO MEETINGS NO PLANNING SESSIONS NO NEED TO BE IN THE OFFICE chat, pull requests, email MORE DIRECT FASTER ALWAYS RECORDED

Slide 25

Slide 25 text

This is designed to retain people. We’re at 56 employees. We haven’t lost one. This is a huge, massive competitive advantage. It justifies the extra expense.

Slide 26

Slide 26 text

Communication.

Slide 27

Slide 27 text

Don’t have the server guy who knows everything. the billing girl the testing dude the customer support maven the performance czar the software licensing file hoarder

Slide 28

Slide 28 text

Don’t have the person who knows everything.

Slide 29

Slide 29 text

Specialization is great, but only having one person is a synchronous bottleneck.

Slide 30

Slide 30 text

Reduce institutional knowledge.

Slide 31

Slide 31 text

Reduce institutional knowledge. wikis issues chat logs pull requests {

Slide 32

Slide 32 text

V Every internal GitHub talk is automatically recorded, uploaded, and viewable to every future employee.

Slide 33

Slide 33 text

V ...on a Kinect-powered Arduino-based motion- detecting portable video recording platform.

Slide 34

Slide 34 text

Your new hire is stoked to dive in, start reading, and start contributing ...so don’t get in their way.

Slide 35

Slide 35 text

Hire well.

Slide 36

Slide 36 text

Hiring poorly is just as bad as losing people.

Slide 37

Slide 37 text

Aim for really great people.

Slide 38

Slide 38 text

WE SELF-STARTERS k less babysitting, more code

Slide 39

Slide 39 text

k S e k 2 k S S k k k k S S k S UKeep your employees happy. Really happy. (future!)

Slide 40

Slide 40 text

Don’t just market your product; market your team and company too.

Slide 41

Slide 41 text

Always think about attracting good people, even if you’re not hiring. OPEN SOURCE CONFERENCES TECHNICAL POSTS SPONSORSHIPS MEETUPS TALKS

Slide 42

Slide 42 text

Technical robots can be pretty finicky too

Slide 43

Slide 43 text

Automate.

Slide 44

Slide 44 text

hubot deploy github to production COMPILATION CoffeeScript SCSS and SASS bundles assets caches Python dependencies compiles Erlang changes compiles C changes builds static pages APP SETUP installs gems symlink directories 14 rolling app server restarts NOTIFY Campfire New Relic graphite fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fs fs fs fs fs fs fs fs fs fs

Slide 45

Slide 45 text

deploys current process overview multi-server shell commands new employee setup app bootstrap

Slide 46

Slide 46 text

Automating now will save you way more time down the road.

Slide 47

Slide 47 text

Ship.

Slide 48

Slide 48 text

Ship early, ship often. 5x-30x deploys per day

Slide 49

Slide 49 text

master = always deployable always green tests always a safe rollback

Slide 50

Slide 50 text

Limit your deployments to staff-only to beta users only to one server only to one app process on one server only

Slide 51

Slide 51 text

@github tweets exceptions deploys deploys

Slide 52

Slide 52 text

Graph.

Slide 53

Slide 53 text

everyone loves fancy graphs quickly see trends quickly see problems historical data as basis for alerts

Slide 54

Slide 54 text

METRICS ARE GREAT But use them wisely.

Slide 55

Slide 55 text

162ms average overall response time

Slide 56

Slide 56 text

Valueless metric.

Slide 57

Slide 57 text

59ms average API response time with 4x throughput of web

Slide 58

Slide 58 text

23ms average raw response time with 2x throughput of web

Slide 59

Slide 59 text

The responsiveness is a lie.

Slide 60

Slide 60 text

199ms average browser response time

Slide 61

Slide 61 text

16,000 requests in the last week over 4.5s

Slide 62

Slide 62 text

Needed to look at the right stuff.

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

throttled google googlebot 2-3x throughput 3-4x CPU usage had web requests compared to

Slide 65

Slide 65 text

Collect a lot of metrics, but make sure they’re important metrics.

Slide 66

Slide 66 text

GitHub scale.

Slide 67

Slide 67 text

Everyone has different growth patterns.

Slide 68

Slide 68 text

GitHub has had three.

Slide 69

Slide 69 text

Launch 2008 Bare metal servers 2009 net-shard 2010 major github infrastructure milestones

Slide 70

Slide 70 text

Launch 2008 Hosted on Engine Yard 10 VMs 54GB RAM shared GFS mount one metric shit-ton of caching

Slide 71

Slide 71 text

Bare metal servers 2009 Hosted on Rackspace 16 bare metal servers 288GB of RAM redundant disk storage

Slide 72

Slide 72 text

net-shard 2010 networks share a common repository rails/rails holman/rails github/rails +1 commit +30 commits classic net-shard rails network repo ...multiplied 2,600 times holman/rails rails/rails github/rails fat network, skeleton forks

Slide 73

Slide 73 text

net-shard 2010 networks share a common repository they also share the same fs and partition halves storage requirements improves hit rate of kernel disk cache speeds up backups allows fast forks, merge button, network GC

Slide 74

Slide 74 text

For GitHub, scaling involved a lot of predictions of future trends, then acting appropriately.

Slide 75

Slide 75 text

Side Projects.

Slide 76

Slide 76 text

A THOUGHT EXPERIMENT: Imagine I told you to build...

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

This grew organically, over dozens of projects, written by dozens of employees, when they felt like it.

Slide 79

Slide 79 text

Figure out how to let this happen. It’s hard.

Slide 80

Slide 80 text

Small hack days can result in real, imma-make-us-money impact.

Slide 81

Slide 81 text

Small hack days can also keep your developers insanely happy.

Slide 82

Slide 82 text

Small hack days can also lead to learning new techniques.

Slide 83

Slide 83 text

Projects and Posts.

Slide 84

Slide 84 text

JENKINS + CAMPFIRE github.com/github/janky CHAT ROOM ROBOT github.com/github/hubot OFFICE MUSIC DJ github.com/holman/play

Slide 85

Slide 85 text

BLOG: GITHUB IS MOVING TO RACKSPACE git.io/jByrlQ BLOG: HOW WE MADE GITHUB FAST git.io/p5v2Ag BLOG: UNICORN git.io/77Onfg

Slide 86

Slide 86 text

+ Technical Organizational

Slide 87

Slide 87 text

Continually refine your process + workflow.

Slide 88

Slide 88 text

Worry about your computers, and worry about your humans.

Slide 89

Slide 89 text

Thanks.

Slide 90

Slide 90 text

ZACH HOLMAN zachholman.com/talks @holman twitter+github: