Unsucking Your Team's Development Environment

Success can bring many glamorous changes to your company: hiring more employees, getting free coffee, and giving everyone a private jet filled with cash and endangered African predatory cats.

Success can lead to less-glamorous problems, though. As you grow, your team's development environment becomes really important. How long does it take to clone, set up, and boot your apps? Can your employees still be productive on an aging codebase? How can you automate CI, hooks, and other setup for new projects? Is any of this fun anymore?

GitHub ran into these problems as we expanded our team tremendously over the last two years. Let's look at some of the ways we've improved our employees' development environments.

Transcript

1. unsucking
   YOUR TEAM’S
   ENVIRONMENT
   DEVELOPMENT
   

   View Slide

2. NO
   FORKING
   HOLMAN
   @
   LOST
   YO QUIT READING THIS SHIT
   

   View Slide

3. githu
   

   View Slide

4. WE STARTED WITH
   one QUEUE
   

   View Slide

5. THIS QUEUE WORKED SORTA
   OBJECTS IN,
   TASK OCCURS
   

   View Slide

6. OH SHIT
   BLOCKING
   OPERATIONS
   

   View Slide

7. THEN SPLIT INTO 2 QUEUES
   SEGREGATE BY
   CATEGORY
   

   View Slide

8. 2 QUEUES, 3x THROUGHPUT
   EXPONENTIAL
   GROWTH
   

   View Slide

9. SATURATED AVAILABILITY
   EXPERIENCING
   BOTTLENECKS
   

   View Slide

10. THIS IS GITHUB’S GODDAMN
    TOILET SITUATION
    

    View Slide

11. TWO GENDERED BATHROOMS
    ZOMG WHY
    AM I WAITING
    

    View Slide

12. It’s because of this:
    

    View Slide

13. GITHUB EMPLOYEES
    5 8
    2009
    2008
    

    View Slide

14. GITHUB EMPLOYEES
    2009 2010
    5 8 14
    2008
    

    View Slide

15. GITHUB EMPLOYEES
    2009 2010 2011
    5 8
    14 54
    2008
    

    View Slide

16. GITHUB EMPLOYEES
    2008 2009 2010 2011 2012
    5 8
    14 54
    114
    

    View Slide

17. WE’VE THOUGHT A LOT
    ABOUT ONBOARDING
    

    View Slide

18. IT’S NOT JUST
    TOILET LATENCY
    ABOUT
    

    View Slide

19. 2.5 YEARS AGO:
    1 WEEK
    TODAY:
    20 MINUTES
    DURATION FROM HIRE TO BOOT
    (a )
    (every a )
    

    View Slide

20. THINK THROUGH MAKING
    YOUR ENVIRONMENT
    SUCK LESS
    

    View Slide

21. PAIN IS MULTIPLIED
    ACROSS YOUR
    WHOLE COMPANY
    

    View Slide

22. THE APP
    THE STACK
    THE DEPLOY
    T H E C O M P A N Y
    

    View Slide

23. THE APP
    THE STACK
    THE DEPLOY
    T H E C O M P A N Y
    

    View Slide

24. I S
    Y O U R
    F O U N D A T I O N
    THE APP
    

    View Slide

25. remembe :
    EVERY APP SHOULD HAVE A
    ONE-LINER INSTALL
    REALLY.
    

    View Slide

26. EVEN LARGE PROJECTS CAN HAVE A
    TUTORIAL-STYLE SCRIPT
    remembe :
    

    View Slide

27. THINGS
    SUCK
    LESS HOMEBREW
    FOREMAN
    RBENV
    BUNDLER
    

    View Slide

28. AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    

    View Slide

29. APP BOOTSTRAP
    

    View Slide

30. BOOTSTRAP
    REPLICATE-REPO
    CIBUILD
    SCRIPT/
    

    View Slide

31. BOOTSTRAP
    SCRIPT/
    ALL-IN-ONE SCRIPT
    THAT SETS UP
    EVERYTHING
    

    View Slide

32. BOOTSTRAP
    SCRIPT/
    DEPENDENCY CHECKS + MIGRATIONS
    +
    BUNDLER ASSET COMPILATION
    

    View Slide

33. BOOTSTRAP
    SCRIPT/
    S P E E D
    MD5 CHECKSUM:
    GEMFILE + ASSETS
    

    View Slide

34. BOOTSTRAP
    SCRIPT/
    everywhere SCRIPT/SERVER
    SCRIPT/CONSOLE
    CI
    

    View Slide

35. BOOTSTRAP
    SCRIPT/
    DESIGNERS ❤ THIS
    

    View Slide

36. BOOTSTRAP
    SCRIPT/
    NON-TECHNICALS ❤ THIS
    

    View Slide

37. BOOTSTRAP
    SCRIPT/
    DEVELOPERS ❤ THIS
    

    View Slide

38. REPLICATE-REPO
    SCRIPT/
    A SUPPORT SCRIPT
    THAT LOADS
    PRODUCTION DATA
    LOCALLY
    

    View Slide

39. REPLICATE-REPO
    SCRIPT/
    FLY YOUR
    PRODUCTION AR
    ASSOCIATIONS TO
    YOU IN DEVELOPMENT
    an rail
    dhh
    

    View Slide

40. REPLICATE-REPO
    SCRIPT/
    an rail
    

    View Slide

41. REPLICATE-REPO
    SCRIPT/
    FOR US, IT’S REPLICATE-REPO
    REPLICATE-#{OBJECT}
    BUT IT COULD BE YOUR
    

    View Slide

42. REPLICATE-REPO
    SCRIPT/
    /RTOMAYKO/REPLICATE
    HTTP://
    

    View Slide

43. REPLICATE-REPO
    SCRIPT/
    DUMPS BELONGS_TO &
    HAS_ONE ASSOCIATIONS
    

    View Slide

44. REPLICATE-REPO
    SCRIPT/
    $ script/replicate-repo holman/spark
    CLONES ISSUES, PULL REQUESTS,
    USERS, GIT DATA, ETC.
    

    View Slide

45. REPLICATE-REPO
    SCRIPT/
    AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    AUTOMATE
    BORING
    SUPPORT TASKS
    

    View Slide

46. CIBUILD
    SCRIPT/
    OUR
    CONTINUOUS INTEGRATION
    ENVIRONMENT
    PREPARATION SCRIPT
    

    View Slide

47. CIBUILD
    SCRIPT/
    SET ENV VARS + LOAD GC CONFIG
    +
    CLEAN WORK DIR BOOTSTRAP
    

    View Slide

48. CIBUILD
    SCRIPT/
    VERSION CONTROL
    you TEST ENVIRONMENT
    

    View Slide

49. USE
    COMPUTERS
    TO DO
    THINGS
    

    View Slide

50. USE
    HUMANS
    TO DRINK
    WITH
    

    View Slide

51. USE
    COMPUTERS
    TO DO
    THINGS
    

    View Slide

52. THE STACK
    THE APP
    THE DEPLOY
    T H E C O M P A N Y
    

    View Slide

53. { GROUP PARTICIPATION TIME }
    

    View Slide

54. EVERYONE LIFT YOUR
    
    

    View Slide

55. OKAY NOW
    SMASH THEM
    
    

    View Slide

56. NO?
    

    View Slide

57. OKAY, MAYBE THEY’RE
    EXPENSIVE, BUT THEY’RE
    ALSO A PAIN TO SET UP
    

    View Slide

58. MACHINES SHOULD BE
    expendable
    

    View Slide

59. WE’VE DONE THAT
    

    View Slide

60. I COULD
    SMASH
    THIS
    LAPTOP
    

    View Slide

61. JUST KIDDING
    OMG I’D GET MURDERED
    

    View Slide

62. AUTOMATED
    MACHINE
    SETUP
    
    20 min
    

    View Slide

63. setu
    THE
    

    View Slide

64. FIRST DAY
    GET A LAPTOP
    RUN A ONE-LINER
    HAVE A BEER
    HACK
    

    View Slide

65. $ gh-setup project
    < >
    

    View Slide

66. DEPENDENCIES PROCESSES
    +
    fo 50 SERVICES
    & APPS
    

    View Slide

67. APPS
    DROPBOX HOMEBREW
    + +
    CHROME + 1PASSWORD + ETC.
    

    View Slide

68. DEPENDENCIES
    MYSQL NODE
    + + REDIS +
    POSTGRES
    RIAK
    + + RBENV + ETC.
    

    View Slide

69. +
    CONFIG
    FULL-DISK ENCRYPTION
    SCREEN LOCKS
    VPN
    + ETC.
    

    View Slide

70. PUPPET
    

    View Slide

71. AUTO-REPORTING
    OF ANY FAILURE TO GITHUB ISSUES
    

    View Slide

72. USE
    COMPUTERS
    TO DO
    THINGS
    

    View Slide

73. THE DEPLOY
    THE APP
    THE STACK
    T H E C O M P A N Y
    

    View Slide

74. remembe :
    DON’T HANDLE
    YOUR OWN DEPLOYS
    

    View Slide

75. HUBOT
    HEAVEN
    +
    

    View Slide

76. HUBOT
    NODE.JS CHAT BOT
    HUNDREDS OF PLUGINS
    

    View Slide

77. HUBOT DEPLOY X TO Y
    

    View Slide

78. HUBOT DEPLOY X TO Y
    DEPLOY GITHUB TO PRODUCTION
    DEPLOY GITHUB TO PRODUCTION/FE
    DEPLOY SMOKE TO FS12,FE1
    

    View Slide

79. HUBOT DEPLOY X TO Y
    DEPLOY GITHUB TO PRODUCTION
    DEPLOY GITHUB TO PRODUCTION/FE
    DEPLOY SMOKE/HOTFIX TO FS12,FE1
    

    View Slide

80. HUBOT
    HEAVEN
    +
    

    View Slide

81. HEAVEN
    CAPISTRANO OR HEROKU
    SINATRA APP
    API TO HUBOT
    RUNS DEPLOYS ON OUR BOXES
    

    View Slide

82. HEAVEN
    DEPLOY LOCKING
    1. YOU PUSH A BRANCH
    2. CI STARTS A BUILD
    3. BUILD PASSES
    4. YOU DEPLOY BRANCH
    5. HEAVEN PREVENTS DEPLOY CONFLICTS
    6.YOU MERGE BRANCH
    7. HEAVEN UNLOCKS DEPLOYS
    

    View Slide

83. HUBOT
    HEAVEN
    +
    FLEXIBLE
    SAFE
    EASY
    

    View Slide

84. USE
    COMPUTERS
    TO DO
    THINGS
    

    View Slide

85. T H E C O M P A N Y
    THE APP
    THE STACK
    THE DEPLOY
    

    View Slide

86. remembe :
    TAKE AWAY
    THE BULLSHIT
    

    View Slide

87. JUST BEING AN EMPLOYEE
    ADDS A LOT OF OVERHEAD
    LOGINS
    SITES
    TERMS
    REPORTS PAYROLL
    NAMES COMMUNICATION
    

    View Slide

88. REDUCE IT
    OAUTH EVERYTHING
    50+ INTERNAL APPS UNDER OAUTH
    

    View Slide

89. REDUCE IT
    BUILD HELPERS
    TEAM APP HELPS US COMMUNICATE
    

    View Slide

90. CLARIFY IT
    WRITE A GUIDE
    FIRST-DAY GUIDE TO THE COMPANY
    

    View Slide

91. CLARIFY IT
    MAKE IT ACCESSIBLE
    MONITORS APP AND OFFICE TVS
    

    View Slide

92. USE
    COMPUTERS
    TO BUILD
    THINGS THAT CAN HELP YOUR
    ORGANIZATION SCALE TO THE EXTENT
    THAT NEW HIRES ACCLIMATE
    WITHOUT A LOT OF STRESS
    

    View Slide

93. THE APP
    THE STACK
    THE DEPLOY
    T H E C O M P A N Y
    

    View Slide

94. AUTOMATE EVERYTHING
    

    View Slide

95. WE HAVE ZERO MANAGERS
    BECAUSE WE AUTOMATE
    

    View Slide

96. LONG-TERM BENEFIT
    WILL ALWAYS OUTWEIGH
    SHORT-TERM COST
    

    View Slide

97. THANKS
    

    View Slide

98. NO
    FORKING
    HOLMAN
    @
    LOST
    YO QUIT READING THIS SHIT
    ZACHHOLMAN.COM/TALKS
    

    View Slide