How to Build a GitHub

How to Build a GitHub

Learn about the growth patterns and the architecture behind github.com.

78b475797a14c84799063c7cd073962f?s=128

Zach Holman

August 05, 2012
Tweet

Transcript

  1.  githu H O W t B U I L

    D GITHUB a 
  2. githu

  3. 6.5MM REPOSITORIES LARGEST GIT HOST 1.9MM USERS SINCE 2008

  4. 6.5MM REPOSITORIES LARGEST GIT HOST 1.9MM USERS SINCE 2008 SVN

    HOST
  5. gh  gh  gh  gh  gh 

    gh 
  6. gh  gh  gh  gh  gh 

    gh  SHOW YOU OUR CARDS going t
  7. MAGIC BULLET there i n

  8. FOUR STAGES OF GROWTH happiness the EVERYTHING automate

  9. NO FORKING HOLMAN @ LOST YO QUIT READING THIS SHIT

  10. ho DID WE GIT HERE

  11. 1809: PERL INVENTED

  12. 1814: COMPUTERS INVENTED

  13. 1814-2004: ANARCHY AND CHAOS AND ZOMG EVERYONE’S DYING

  14. 2005: VERSION CONTROL INVENTED git

  15. 2007: githu GLOBAL PEACE AND HAPPINESS ACHIEVED

  16. ...or something like that

  17. PRESTON-WERNER TOM GRIT O C TOBER 9, 2 0 07

    git via ruby
  18. GRIT git via ruby github’s interface to git object-oriented, read/write

    open source
  19. repo = Grit::Repo.new('/tmp/repository') grit repo.commits

  20. grit shelling out to git is expensive grit reimplements portions

    of git in ruby native packfile and git object support 2x-100x speedup on low-level operations
  21. grit slowly reimplement grit for speed allows for incremental improvements

  22. LED TO GITHUB grit O C TOBER 19, 2 0

    07
  23. TODAY ADDING 2TB A MONTH 22 FILESERVER PAIRS 23TB OF

    REPO DATA
  24. GITHUB GROWTH THE FOUR STAGES of

  25. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:

  26. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB: 2008

    2009 2010 2012
  27. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:

  28. JAN 2008 DEC 2008 FOUR STAGES OF GROWTH GITHUB: 42,000

    USERS 
  29. JAN 2008 DEC 2008 FOUR STAGES OF GROWTH GITHUB: 80,000

    REPOSITORIES 
  30. LOCAL MULTI-VM SHARED GFS MOUNT

  31. LOCAL MULTI-VM WEB FRONTENDS BACKGROUND WORKERS

  32. LOCAL MULTI-VM SIMPLE ARCHITECTURE HORIZONTALLY SCALABLE-ish

  33. LOCAL SHARED GFS MOUNT SHARED MOUNT ON EACH VM SIMILAR

    PRODUCTION + DEVELOPMENT ACCESS ALLOWED LOCAL ACCESS VIA GRIT
  34. SIMPLE APPROACH, COMMON GIT INTERFACE, QUICK TO BUILD AND SHIP

    LOCAL
  35. LOCAL NETWORKED FOUR STAGES OF GROWTH GITHUB: NET-SHARD GITRPC

  36. 2008 2009 2010 FOUR STAGES OF GROWTH GITHUB: 166,000 USERS

  37. 2008 2009 2010 FOUR STAGES OF GROWTH GITHUB: 484,000 REPOSITORIES

  38. the problem: is slow GFS performance degraded as repos added

  39. the problem: i/o-bound we’re read/write to disk needs to be

    fast
  40. THE PLAN NETWORKED HARDWARE MOVE DATACENTERS

  41. NETWORKED HARDWARE bare metal servers 16 machines 6x RAM machine

    roles solid datacenter got dat cloud
  42. NETWORKED FRONTENDS FILESERVERS AUX DB LAUNCH: SERVER PAIRS

  43. NETWORKED GRIT IS LOCAL NEEDS TO BE NETWORKED

  44. NETWORKED smoke service is run on each fs; facilitates disk

    access chimney routes the smoke, stores routing table in redis stub local grit calls, retain API usage, but send over network
  45. NETWORKED server pairs offer failover via DRBD real servers, real

    big RAM allocations
  46. NETWORKED LATENCY networked routing adds 2-10ms per request optimize for

    the roundtrip smoke contains smarter server-side logic
  47. NETWORKED LATENCY smoke has custom git extension commands git-distinct-commits returns

    commits only contained on a given branch calls to git-show-refs and git-rev-list run all calls server-side in one roundtrip
  48. NETWORKED HORIZONTALLY-SCALABLE, LATENCY- CONSIDERATE, API-COMPATIBLE WITH GRIT

  49. LOCAL FOUR STAGES OF GROWTH GITHUB: NET-SHARD GITRPC NETWORKED

  50. 2008 2009 2010 2011 FOUR STAGES OF GROWTH GITHUB: 510,000

    USERS 
  51. 2008 2009 2010 2011 FOUR STAGES OF GROWTH GITHUB: 1.3MM

    REPOSITORIES 
  52. the problem: duplication data each fork is a full project

    history 
  53. duplication data  i create a repo you fork my

    repo fs5:/data/repositories/6/nw/6b/de/92/1/1.git fs7:/data/repositories/4/na/3b/dr/72/2/2.git
  54. duplication data  1,000 commits 1,001 commits 10MB 10MB 20MB

    total disk }
  55. duplication data  1,000 commits 1 commit 1KB 10MB 10MB

    total disk }GOAL:
  56. duplication data  75 MB repo 3.5k forks x ~250

    GB x 2 fs pairs + offsite backups
  57. NET-SHARD shard by repository network (“forks”)

  58. NET-SHARD network.git 1.git 2.git 3.git 4.git CONTAINS DELTA }CONTAINS ALL

    REFS ›
  59. NET-SHARD network.git GIT ALTERNATES store git object data externally to

    repository we fetch refs into your fork, transparently
  60. NET-SHARD network.git PRIVACY potential leaking of refs cross-network net-shard enabled

    on all-public and all-private repository networks only
  61. NET-SHARD network.git DISK halves disk usage increase disk and kernel

    cache hits
  62. NET-SHARD network.git MIGRATION gradually transitioned repos to network.git effectively feature-flagged

    by repo
  63. NET-SHARD SAVE DISK, IMPROVE PERFORMANCE

  64. LOCAL FOUR STAGES OF GROWTH GITHUB: GITRPC NETWORKED NET-SHARD

  65. 2008 2009 2010 2011 2012 FOUR STAGES OF GROWTH GITHUB:

    1.2MM USERS 
  66. 2008 2009 2010 2011 2012 AUGUST FOUR STAGES OF GROWTH

    GITHUB: 1.9MM USERS 
  67. 2008 2009 2010 2011 2012 FOUR STAGES OF GROWTH GITHUB:

    3.4MM REPOSITORIES 
  68. 2008 2009 2010 2011 2012 AUGUST FOUR STAGES OF GROWTH

    GITHUB: 6.5MM REPOSITORIES 
  69. the problem: GRIT git via ruby

  70. the problem: local, ruby-based grit ended up in a high-traffic

    distributed system
  71. the problem: inelegant code spread out everywhere

  72. GITRPC network-oriented library for git access GitRPC

  73. GITRPC open source fastest git implementation (C) github-sponsored project bindings

    for all major languages used in our mac, windows clients
  74. GITRPC rugged (RUBY) libgit2 (C) gitrpc (RUBY)

  75. GITRPC like smoke, gitrpc aims to reduce latency by reducing

    roundtrips LATENCY
  76. GITRPC operations cached on library level CACHING yank out tons

    of app-level cache logic
  77. GITRPC the move to gitrpc started this summer and will

    take months MIGRATION gradually replace smoke and grit; avoids a risky deploy
  78. FAST AND STABLE NETWORKED GIT ACCESS GITRPC

  79. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:

  80. identify WHAT’S BROKEN

  81. sma CHANGES, FAST DEVELOPMENT

  82. realCODE BEATS IMAGINARY CODE

  83. EVERYTHING automate automate automate automate automate AUTOMATE automate automate automate

    automate automate automate
  84.     m . manage LOL DEVELOPERS SOFTWARE

    DEVELOPMENT
  85.    m . manage DEADLINES MEETINGS PRIORITIES ESTIMATES

  86.    m . manage DEADLINES MEETINGS PRIORITIES ESTIMATES

  87.  EVERYONE i A MANAGER

  88. AUTOMATE AWAY PAIN DEPLOYMENT RECOVERY DEVELOPMENT

  89. DEVELOPMENT automate

  90. DEVELOPMENT > ./do-work RUN THIS IN EACH PROJECT: ...AND YOU’RE

    DONE! loljk
  91. DEVELOPMENT YOU CAN AUTOMATE THE PAIN OF DEVELOPMENT

  92. SETUP DEVELOPMENT the

  93. SETUP DEVELOPMENT the ONE-LINER INSTALLS ALL GITHUB DEVELOPMENT DEPENDENCIES

  94.  30 min SETUP DEVELOPMENT the CLEAN MACHINE TO FULL

    DEVELOPMENT ENVIRONMENT
  95. SETUP DEVELOPMENT the NEW EMPLOYEES SHIP THEIR FIRST WEEK 

  96. SETUP DEVELOPMENT the PUPPET HANDLES ALL DEPENDENCIES

  97. DEPLOYMENT automate

  98. DEPLOYMENT REAL BROGRAMMERS DEPLOY WITH NO FEAR SO FUCK THAT

  99. DEPLOYMENT DEPLOYS SHOULD BE CAUTIOUS, COMMONPLACE, AND AUTOMATED

  100. DEPLOYMENT GITHUB DEPLOYS 20-40 TIMES A DAY

  101. DEPLOYMENT PUSH BRANCH DEPLOY BRANCH EVERYWHERE · MACHINE CLASS ·

    SPECIFIC SERVERS HUBOT RUNS TESTS IN ABOUT 200 SECONDS USUALLY OPEN A PULL REQUEST
  102. DEPLOYMENT DEPLOY LOCKING CAN’T DEPLOY IF A BRANCH IS DEPLOYED

    AUTODEPLOYS PUSHED TO MASTER WITH GREEN TESTS? DEPLOY.
  103. DEPLOYMENT STAFF-ONLY FEATURE FLAGS LIMITS EXPOSURE · REAL-WORLD · AVOIDS

    MERGES
  104. RECOVERY automate

  105. RECOVERY SOMETHING WILL ALWAYS BREAK

  106. RECOVERY HUBOT IS A SYSADMIN

  107. RECOVERY HUBOT LOAD HUBOT QUERIES HUBOT CONNS SERVER LOAD RUNNING

    DB QUERIES ALL OPEN CONNECTIONS
  108. RECOVERY HUBOT RESTORE <REPO> HUBOT PUSH-LOG <REPO> HUBOT GH-EACH <HOST>

    <COMMAND> RESTORE A REPO FROM BACKUPS SEE RECENT PUSH LOGS TO A REPO RUN COMMAND ON SPECIFIC HOSTS
  109. HIGH-LEVEL OVERVIEW IN MINUTES SPEND MORE TIME FIXING AND LESS

    TIME INVESTIGATING RECOVERY
  110. — happiness the — — — —

  111. EMPLOYEES HAVE QUIT YEARS 5 EMPLOYEES 108 ZERO

  112. 1-2 MONTHS HIRE 1-3 MONTHS RAMP-UP 2 WEEKS LEAVE

  113. LOSING AN EMPLOYEE CAN SET YOU BACK HALF A YEAR

  114. remove ANY REASON TO LEAVE — — — — —

    — — — — — — — — — — — —
  115. TDD✓ PAIR PROGRAMMING ✓ BDD ✓ TEST-FIRST ✓ DESIGN-FIRST ✓

    (just kidding) EMACS x NONE OF THESE ✓
  116. WE CARE ABOUT THE WORK YOU DO, NOT ABOUT HOW

    YOU DO IT
  117. LOCATION  HOURS  DIRECTION 

  118. LOCATION  HOURS  DIRECTION  GITHUB EMPLOYEES WORK REMOTELY

  119. LOCATION  HOURS  DIRECTION  FAMILY RELOCATION, TRAVEL FREEDOM

  120. LOCATION  HOURS  DIRECTION  CHOOSE YOUR SCHEDULE CHOOSE

    YOUR VACATIONS FRESH, CREATIVE EMPLOYEES 
  121. LOCATION  HOURS  DIRECTION  YOU HACK ON THINGS

    THAT INTEREST YOU REDUCES BURNOUT 
  122. flexible LOCATION  HOURS  DIRECTION  BE TOWARDS WORK/LIFE

  123. githu

  124. basica y, MOVE FAST = SMALL CHANGES

  125. basica y, BE STABLE = DEPLOY CONSTANTLY

  126. basica y, HAPPY COMPANY = HAPPY EMPLOYEES

  127. thank

  128. NO FORKING HOLMAN @ LOST YO QUIT READING THIS SHIT

    ZACHHOLMAN.COM/TALKS