Scaling GitHub

Scaling GitHub

A month after launching, GitHub hosted one thousand repositories. Three years later, we host over three million. In the same time we've gone from one thousand users to over a million.

This type of scaling presents some interesting technical challenges. I'll dig into our development workflow and how we address concepts like scaling, deployment, code review, and testing.

It also presents some interesting business challenges, too. How you grow your company from three employees, how you work in teams, and how you split your app up into services all help ensure that you'll be able to react to your product's growth.

http://zachholman.com/talk/scaling-github

78b475797a14c84799063c7cd073962f?s=128

Zach Holman

January 26, 2012
Tweet

Transcript

  1. Scaling GitHub Scaling GitHub Scaling GitHub ling GitHub Scaling GitHub

    Scaling GitHub Scaling GitHub Scaling GitHub Scaling GitHub Scaling github SCALING GITHUB scalin’ githubs Scaling GitHub Scaling GitHub Scaling GitHub githubs and shit Scaling GitHub Scaling GitHub Scaling Startups B=======D~~~~ Scaling GitHub
  2. “Scale”

  3. Two problems.

  4. SyntaxError: compile error. I’m too hungover to work. ORGANIZATIONAL TECHNICAL

  5. Scaling is people + technology

  6. @holman

  7. None
  8. Organizational jeez humans are so finicky

  9. Happiness.

  10. 0 250,000 500,000 750,000 1,000,000 Happiness vs Productivity ☹ ☺

    $ $$$
  11. happy employees are productive employees

  12. productive employees are happy employees

  13. This isn’t a “management problem”. Everyone needs to worry about

    this.
  14. Hiring an employee is the most thing you can do

    to your startup. T O X I C
  15. Hiring an employee is the most thing you can do

    to your startup. T O X I C work slower more bugs less features worse culture
  16. Hiring an employee is the most thing you can do

    to your startup. EXCITING work faster fewer bugs more features better culture
  17. so how can you score excitement and avoid the toxic?

  18. TOXIC EXCITEMENT would be a great name for a rock

    band yeah, i know...
  19. k S e k 2 k S S k k

    k k S S k S UKeep your employees happy. Really happy.
  20. Your servers, offices, and ideas are bullshit. Worry about your

    coworkers.
  21. EMPLOYEES NEW HIRES Know your codebase Know your process Know

    your mistakes Know your mission Don’t know jack Know your jokes Know your priorities
  22. Imprison your employees with happiness and nice things and cuddly

    work practices.
  23. GitHub Jail work whenever you want work however you want

    work on what you want health, dental, vision paid conference trips retirement plans solid salaries a product people love four beers on tap stock
  24. get out of the way NO MEETINGS NO PLANNING SESSIONS

    NO NEED TO BE IN THE OFFICE chat, pull requests, email MORE DIRECT FASTER ALWAYS RECORDED
  25. This is designed to retain people. We’re at 56 employees.

    We haven’t lost one. This is a huge, massive competitive advantage. It justifies the extra expense.
  26. Communication.

  27. Don’t have the server guy who knows everything. the billing

    girl the testing dude the customer support maven the performance czar the software licensing file hoarder
  28. Don’t have the person who knows everything.

  29. Specialization is great, but only having one person is a

    synchronous bottleneck.
  30. Reduce institutional knowledge.

  31. Reduce institutional knowledge. wikis issues chat logs pull requests {

  32. V Every internal GitHub talk is automatically recorded, uploaded, and

    viewable to every future employee.
  33. V ...on a Kinect-powered Arduino-based motion- detecting portable video recording

    platform.
  34. Your new hire is stoked to dive in, start reading,

    and start contributing ...so don’t get in their way.
  35. Hire well.

  36. Hiring poorly is just as bad as losing people.

  37. Aim for really great people.

  38. WE SELF-STARTERS k less babysitting, more code

  39. k S e k 2 k S S k k

    k k S S k S UKeep your employees happy. Really happy. (future!)
  40. Don’t just market your product; market your team and company

    too.
  41. Always think about attracting good people, even if you’re not

    hiring. OPEN SOURCE CONFERENCES TECHNICAL POSTS SPONSORSHIPS MEETUPS TALKS
  42. Technical robots can be pretty finicky too

  43. Automate.

  44. hubot deploy github to production COMPILATION CoffeeScript SCSS and SASS

    bundles assets caches Python dependencies compiles Erlang changes compiles C changes builds static pages APP SETUP installs gems symlink directories 14 rolling app server restarts NOTIFY Campfire New Relic graphite fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fs fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fe fs fs fs fs fs fs fs fs fs fs
  45. deploys current process overview multi-server shell commands new employee setup

    app bootstrap
  46. Automating now will save you way more time down the

    road.
  47. Ship.

  48. Ship early, ship often. 5x-30x deploys per day

  49. master = always deployable always green tests always a safe

    rollback
  50. Limit your deployments to staff-only to beta users only to

    one server only to one app process on one server only
  51. @github tweets exceptions deploys deploys

  52. Graph.

  53. everyone loves fancy graphs quickly see trends quickly see problems

    historical data as basis for alerts
  54. METRICS ARE GREAT But use them wisely.

  55. 162ms average overall response time

  56. Valueless metric.

  57. 59ms average API response time with 4x throughput of web

  58. 23ms average raw response time with 2x throughput of web

  59. The responsiveness is a lie.

  60. 199ms average browser response time

  61. 16,000 requests in the last week over 4.5s

  62. Needed to look at the right stuff.

  63. None
  64. throttled google googlebot 2-3x throughput 3-4x CPU usage had web

    requests compared to
  65. Collect a lot of metrics, but make sure they’re important

    metrics.
  66. GitHub scale.

  67. Everyone has different growth patterns.

  68. GitHub has had three.

  69. Launch 2008 Bare metal servers 2009 net-shard 2010 major github

    infrastructure milestones
  70. Launch 2008 Hosted on Engine Yard 10 VMs 54GB RAM

    shared GFS mount one metric shit-ton of caching
  71. Bare metal servers 2009 Hosted on Rackspace 16 bare metal

    servers 288GB of RAM redundant disk storage
  72. net-shard 2010 networks share a common repository rails/rails holman/rails github/rails

    +1 commit +30 commits classic net-shard rails network repo ...multiplied 2,600 times holman/rails rails/rails github/rails fat network, skeleton forks
  73. net-shard 2010 networks share a common repository they also share

    the same fs and partition halves storage requirements improves hit rate of kernel disk cache speeds up backups allows fast forks, merge button, network GC
  74. For GitHub, scaling involved a lot of predictions of future

    trends, then acting appropriately.
  75. Side Projects.

  76. A THOUGHT EXPERIMENT: Imagine I told you to build...

  77. None
  78. This grew organically, over dozens of projects, written by dozens

    of employees, when they felt like it.
  79. Figure out how to let this happen. It’s hard.

  80. Small hack days can result in real, imma-make-us-money impact.

  81. Small hack days can also keep your developers insanely happy.

  82. Small hack days can also lead to learning new techniques.

  83. Projects and Posts.

  84. JENKINS + CAMPFIRE github.com/github/janky CHAT ROOM ROBOT github.com/github/hubot OFFICE MUSIC

    DJ github.com/holman/play
  85. BLOG: GITHUB IS MOVING TO RACKSPACE git.io/jByrlQ BLOG: HOW WE

    MADE GITHUB FAST git.io/p5v2Ag BLOG: UNICORN git.io/77Onfg
  86. + Technical Organizational

  87. Continually refine your process + workflow.

  88. Worry about your computers, and worry about your humans.

  89. Thanks.

  90. ZACH HOLMAN zachholman.com/talks @holman twitter+github: