The Setup: Managing an Army of Laptops with Puppet

As a rapidly growing company, GitHub's faced some challenges in how to make sure that everyone can get up and running on projects or continue to work on one of the dozens of existing projects if they've never set it up before. What a nightmare! That's what prompted @jbarnette and @wfarr to develop The Setup. The Setup aims to solve all the problems that used to plague on-boarding GitHubbers onto projects with a lot of automation and a case of the "It Just Works". This presentation talks about the history of The Setup, how it works, and some of the lessons learned along the way.

Transcript

1. Setup
   the
   

   View Slide

2. managing an
   army of laptops
   ...with puppet
   

   View Slide

3. hi, my name is will farrington
   but @wfarr is way shorter
   

   View Slide

4. i work at a “little-known startup” called
   github
   

   View Slide

5. I work on System operations
   System operations
   internal apps
   the setup
   

   View Slide

6. why
   

   View Slide

7. once
   upon a
   time
   

   View Slide

8. there
   was a
   developer
   

   View Slide

9. there
   was a
   designer
   

   View Slide

10. oh man, this new macbook pro is sweet! let me run
    a bunch of stuff by hand so I can get some work
    done in about 6 to 8 hours, maybe.
    - some perfectly normal people
    

    View Slide

11. once
    upon a
    time
    

    View Slide

12. there
    was a
    crazy
    person
    

    View Slide

13. oh man, this new macbook pro is sweet! let me run
    a bunch of stuff by hand so I can get some work
    done in about 6 to 8 hours, maybe.
    ~ crazy people
    

    View Slide

14. View Slide

15. we made a
    HUGE
    mistake
    

    View Slide

16. yep, it’s
    pretty
    bad
    

    View Slide

17. i thought we cared about testing
    

    View Slide

18. i thought we cared about repeatability
    

    View Slide

19. i thought we cared about automation
    

    View Slide

20. we say those things matter for software
    and they totally do
    

    View Slide

21. we do not treat our laptops like they matter
    

    View Slide

22. wtf is wrong with us
    

    View Slide

23. wtf is wrong with us
    i mean srsly guise
    

    View Slide

24. spock
    approves
    this
    message
    

    View Slide

25. what
    

    View Slide

26. a living, breathing software project
    that tests and automates every little
    bit of your machine with love
    

    View Slide

27. @jbarnette
    

    View Slide

28. about 6000
    about 3600
    1186
    26
    lines of Ruby (not incl. tests)
    lines of Puppet (not incl. tests)
    commits
    contributors
    some numbers
    

    View Slide

29. chrome, colloquy, dropbox, elasticsearch, emacs,
    erlang, git, homebrew, hub, iterm2, java, macvim,
    memcache, mongodb, mysql, nginx, nodejs, nvm,
    1passwd, personalization, postgresql, projects,
    python, qt, rbenv, rdio, redis, ruby, solr, sparrow,
    textmate, virtualbox, viscosity, wget, xcode,
    xquartz, zeromq, zsh, and more daily.
    

    View Slide

30. less than 6 months
    

    View Slide

31. a githubber’s first day
    in

    View Slide

32. View Slide

33. hours of frustration getting
    github.com and other projects
    even working locally
    

    View Slide

34. View Slide

35. a githubber’s first day
    now
    

    View Slide

36. View Slide

37. View Slide

38. drink coffee for 20 minutes
    

    View Slide

39. Text
    

    View Slide

40. View Slide

41. The Command Line Interface
    

    View Slide

42. » gh-setup --projects
    * github
    

    View Slide

43. github » script/bootstrap
    github » script/server
    github » open github.dev
    # Ready to ship things!
    

    View Slide

44. » gh-setup octostatus
    # sets it all up. boom.
    

    View Slide

45. » gh-setup --stealth
    # development mode
    

    View Slide

46. View Slide

47. The Puppet DSL
    

    View Slide

48. The Puppet DSL is pretty easy actually
    

    View Slide

49. Resources
    

    View Slide

50. Resources are the basic unit of puppet
    

    View Slide

51. package { ‘redis-server’: }
    

    View Slide

52. file { ‘/etc/motd’:
    content => ‘Hello, world!’
    }
    

    View Slide

53. service { ‘riak’:
    ensure => running,
    subscribe => File[‘/etc/riak/vm.args’]
    }
    resource type
    

    View Slide

54. service { ‘riak’:
    ensure => running,
    subscribe => File[‘/etc/riak/vm.args’]
    }
    resource name
    

    View Slide

55. service { ‘riak’:
    ensure => running,
    subscribe => File[‘/etc/riak/vm.args’]
    }
    resource parameters
    

    View Slide

56. service { ‘riak’:
    ensure => running,
    subscribe => File[‘/etc/riak/vm.args’]
    }
    metaparameter
    

    View Slide

57. metaparameters work on every resource
    

    View Slide

58. metaparameters describe relationships
    

    View Slide

59. before
    require
    subscribe
    notify
    this resource must run before
    this resource must run after
    this resource runs after and watches
    this resource runs before and tells
    

    View Slide

60. resources can have custom types
    

    View Slide

61. define git::config::local($key, $value) {
    $safekey = shellquote($key)
    $safevalue = shellquote($value)
    $command = "git config --local ${safekey} ${safevalue}"
    exec { "${command} in ${name}":
    command => $command,
    cwd => $name,
    onlyif => 'test -d .git',
    unless => "git config --local ${safekey}",
    require => Package['git']
    }
    }
    

    View Slide

62. git::config::local { ‘~/github/hubot’:
    key => ‘heroku.account’,
    value => ‘work’
    }
    

    View Slide

63. Resources can be grouped with classes
    

    View Slide

64. class redis {
    package { ‘redis-server’: }
    service { ‘redis’:
    require => Package[‘redis-server’]
    }
    }
    

    View Slide

65. classes can mixin other classes
    

    View Slide

66. classes can require other classes
    

    View Slide

67. this matters a whole lot everybody
    

    View Slide

68. Puppet has its own dsl for a couple reasons
    

    View Slide

69. 1. puppet was designed for sysadmin-types
    

    View Slide

70. 2. puppet has semantics about execution that
    would be confusing and unintuitive in ruby or
    python or most other languages
    

    View Slide

71. puppet uses a DAG to apply resources
    

    View Slide

72. a DAG is a directed, acyclic graph
    

    View Slide

73. a DAG means relationships matter
    

    View Slide

74. a DAG also means order of execution does not matter
    

    View Slide

75. View Slide

76. The Setup as a standard library for puppet
    but designed for personal laptops
    

    View Slide

77. class github::projects::octostatus {
    require redis
    ...
    }
    

    View Slide

78. class github::projects::octostatus {
    ...
    $dir = "${github::config::srcdir}/octostatus"
    ...
    }
    

    View Slide

79. class github::projects::octostatus {
    ...
    git::repo { $dir:
    source => 'github/octostatus'
    }
    ...
    }
    

    View Slide

80. class github::projects::octostatus {
    ...
    ruby::local { $dir:
    version => '1.9.3-p194',
    require => Git::Repo[$dir]
    }
    }
    

    View Slide

81. I like doing things my way
    

    View Slide

82. class github::people::hubot {
    include github::projects::all
    }
    

    View Slide

83. class github::people::keavy {
    require gitx
    require onepassword
    require sizeup
    require sparrow
    require textmate
    }
    

    View Slide

84. class github::people::yossef {
    package {
    'Alfred':
    provider => 'appdmg',
    source => $alfred_dmg_url;
    }
    }
    

    View Slide

85. class github::people::yossef {
    setup::tarballed_app {
    'Mailplane':
    source => $mailplane_url;
    }
    }
    

    View Slide

86. class github::people::wfarr {
    git::config::global {
    'alias.st':
    value => 'status';
    'alias.ci':
    value => 'commit';
    }
    }
    

    View Slide

87. class github::people::wfarr {
    file { '/Users/wfarr/.gitignore':
    ensure => present,
    content => '.DS_STORE'
    }
    }
    

    View Slide

88. class github::people::wfarr {
    package { 'emacs': }
    git::repo { '/Users/wfarr/.emacs.d':
    source => 'wfarr/.emacs.d',
    submodules => true
    }
    }
    

    View Slide

89. class github::people::wfarr {
    setup::osx_defaults { 'require password always':
    ensure => present,
    domain => 'com.apple.screensaver',
    key => 'askForPassword',
    value => 1,
    user => 'wfarr'
    }
    }
    

    View Slide

90. lessons
    

    View Slide

91. puppeteer
    supportocat
    

    View Slide

92. people will do crazy shit to it
    

    View Slide

93. if it breaks it is your job to fix it
    

    View Slide

94. if it breaks it is your job to fix it
    no matter what
    

    View Slide

95. devs are really good at breaking things
    

    View Slide

96. people report about 1% of issues they have
    

    View Slide

97. people report about 1% of issues they have
    on a really good day
    

    View Slide

98. View Slide

99. rapid iteration on bugs makes people happy
    

    View Slide

100. avoiding more bugs makes us happy
     

     View Slide

101. reduce potential for bugs by vendoring
     

     View Slide

102. EVERYTHING
     

     View Slide

103. View Slide

104. » du -sh /opt/github
     3.8G /opt/github
     

     View Slide

105. there’s

     View Slide

106. View Slide

107. =(
     

     View Slide

108. » env | grep GH_ | wc -l
     24
     

     View Slide

109. » env | grep GH_REDIS_PORT
     GH_REDIS_PORT=16379
     

     View Slide

110. » env | grep GH_MYSQL_PORT
     GH_MYSQL_PORT=13306
     

     View Slide

111. View Slide

112. people just will never remember to update
     

     View Slide

113. tell people when they should update
     

     View Slide

114. View Slide

115. omg sneak
     peek time
     

     View Slide

116. => Installing qt...
     

     View Slide

117. View Slide

118. View Slide

119. => Fetching snapshot...
     => Installing qt...
     

     View Slide

120. => Fetching snapshot...
     => Installing ruby-1.9.3...
     

     View Slide

121. => Fetching snapshot...
     => Installing node-0.6.8...
     

     View Slide

122. we don't have the ability to
     mandate environments
     

     View Slide

123. we have to provide something
     dramatically better than just
     doing it yourself
     

     View Slide

124. prim
     donn
     

     View Slide

125. the setup is not a tool for it administrators
     

     View Slide

126. the setup is a tool for people to get stuff done
     

     View Slide

127. the setup is extensible by users
     

     View Slide

128. the setup is a stable framework for config
     

     View Slide

129. the setup is about enabling good patterns
     

     View Slide

130. the setup is not about dictating machine policy
     

     View Slide

131. how do i get it
     

     View Slide

132. you don’t =(
     

     View Slide

133. ... yet
     

     View Slide

134. we are working
     hard to open
     source it
     

     View Slide

135. Soon!
     

     View Slide

136. Setup
     the
     

     View Slide

137. questions
     

     View Slide

138. love ops? come work with us
     puppet, erlang, ruby, shell, c
     and all the graphs you can shake a stick at
     

     View Slide

139. View Slide

140. thanks
     

     View Slide