Scaling GitHub

Scaling GitHub

A month after launching, GitHub hosted one thousand repositories. Three years later, we host over three million. In the same time we've gone from one thousand users to over a million.

This type of scaling presents some interesting technical challenges. I'll dig into our development workflow and how we address concepts like scaling, deployment, code review, and testing.

It also presents some interesting business challenges, too. How you grow your company from three employees, how you work in teams, and how you split your app up into services all help ensure that you'll be able to react to your product's growth.

http://zachholman.com/talk/scaling-github

78b475797a14c84799063c7cd073962f?s=128

Zach Holman

January 26, 2012
Tweet

Transcript

  1. 1.

    Scaling GitHub Scaling GitHub Scaling GitHub ling GitHub Scaling GitHub

    Scaling GitHub Scaling GitHub Scaling GitHub Scaling GitHub Scaling github SCALING GITHUB scalin’ githubs Scaling GitHub Scaling GitHub Scaling GitHub githubs and shit Scaling GitHub Scaling GitHub Scaling Startups B=======D~~~~ Scaling GitHub
  2. 6.
  3. 7.
  4. 14.
  5. 15.

    Hiring an employee is the most thing you can do

    to your startup. T O X I C work slower more bugs less features worse culture
  6. 16.

    Hiring an employee is the most thing you can do

    to your startup. EXCITING work faster fewer bugs more features better culture
  7. 19.

    k S e k 2 k S S k k

    k k S S k S UKeep your employees happy. Really happy.
  8. 21.

    EMPLOYEES NEW HIRES Know your codebase Know your process Know

    your mistakes Know your mission Don’t know jack Know your jokes Know your priorities
  9. 23.

    GitHub Jail work whenever you want work however you want

    work on what you want health, dental, vision paid conference trips retirement plans solid salaries a product people love four beers on tap stock
  10. 24.

    get out of the way NO MEETINGS NO PLANNING SESSIONS

    NO NEED TO BE IN THE OFFICE chat, pull requests, email MORE DIRECT FASTER ALWAYS RECORDED
  11. 25.

    This is designed to retain people. We’re at 56 employees.

    We haven’t lost one. This is a huge, massive competitive advantage. It justifies the extra expense.
  12. 27.

    Don’t have the server guy who knows everything. the billing

    girl the testing dude the customer support maven the performance czar the software licensing file hoarder
  13. 34.

    Your new hire is stoked to dive in, start reading,

    and start contributing ...so don’t get in their way.
  14. 39.

    k S e k 2 k S S k k

    k k S S k S UKeep your employees happy. Really happy. (future!)
  15. 41.

    Always think about attracting good people, even if you’re not

    hiring. OPEN SOURCE CONFERENCES TECHNICAL POSTS SPONSORSHIPS MEETUPS TALKS
  16. 43.
  17. 44.

    hubot deploy github to production COMPILATION CoffeeScript SCSS and SASS

    bundles assets caches Python dependencies compiles Erlang changes compiles C changes builds static pages APP SETUP installs gems symlink directories 14 rolling app server restarts NOTIFY Campfire New Relic graphite fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fs fs fs fs fs fs fs fs fs fs
  18. 47.
  19. 50.

    Limit your deployments to staff-only to beta users only to

    one server only to one app process on one server only
  20. 52.
  21. 63.
  22. 70.

    Launch 2008 Hosted on Engine Yard 10 VMs 54GB RAM

    shared GFS mount one metric shit-ton of caching
  23. 71.

    Bare metal servers 2009 Hosted on Rackspace 16 bare metal

    servers 288GB of RAM redundant disk storage
  24. 72.

    net-shard 2010 networks share a common repository rails/rails holman/rails github/rails

    +1 commit +30 commits classic net-shard rails network repo ...multiplied 2,600 times holman/rails rails/rails github/rails fat network, skeleton forks
  25. 73.

    net-shard 2010 networks share a common repository they also share

    the same fs and partition halves storage requirements improves hit rate of kernel disk cache speeds up backups allows fast forks, merge button, network GC
  26. 74.
  27. 77.
  28. 78.
  29. 85.

    BLOG: GITHUB IS MOVING TO RACKSPACE git.io/jByrlQ BLOG: HOW WE

    MADE GITHUB FAST git.io/p5v2Ag BLOG: UNICORN git.io/77Onfg
  30. 89.