Puppet at GitHub: PuppetConf 2013

Transcript

1. Puppet at GitHub

2. @wfarr GitHub Operations Known Aliases: King of Kebabs The Chairman

   Mr. Caremad !

3. The State of Puppet at GitHub "

4. " The State of Puppet at GitHub github/boxen

5. " The State of Puppet at GitHub ~1.5 years old

6. " The State of Puppet at GitHub open-sourced ~7 months

   ago

7. " The State of Puppet at GitHub ~240 open-source puppet

   modules

8. " The State of Puppet at GitHub puppet 3.latest

9. " The State of Puppet at GitHub supports ruby 1.8.7,

   1.9.3, 2.0.0

10. " The State of Puppet at GitHub hiera

11. " The State of Puppet at GitHub OS X

12. " The State of Puppet at GitHub Linux support in-progress

13. " The State of Puppet at GitHub # tomorrow @

    2:20pm

14. " The State of Puppet at GitHub github/puppet

15. " The State of Puppet at GitHub $

16. " The State of Puppet at GitHub ~5 years old

17. " The State of Puppet at GitHub 0.24.x — 2.7.x

18. " The State of Puppet at GitHub 121 modules

19. " The State of Puppet at GitHub ~280k lines of

    code

20. " The State of Puppet at GitHub every single employee

    has commit access

21. " The State of Puppet at GitHub 15.5k commits to

    master past 12 months

22. " The State of Puppet at GitHub by 86 contributors

23. " The State of Puppet at GitHub % % %

    % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %

24. " The State of Puppet at GitHub % % %

    % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % 42.5%

25. " The State of Puppet at GitHub % % %

    % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % now with ops taken out

26. " The State of Puppet at GitHub 0 100 200

    300 400 2012-08-25 2012-10-20 2012-12-15 2013-02-09 2013-04-06 2013-06-01 2013-07-27 commits on master per week, last 12 months

27. " The State of Puppet at GitHub 0 1000 2000

    3000 4000 total commits by author past year

28. " The State of Puppet at GitHub 0 175 350

    525 700 total commits by author past year, except ops

29. " The State of Puppet at GitHub 0 175 350

    525 700 total commits by authors with >10 commits, past year, except ops

30. " The State of Puppet at GitHub single puppetmaster

31. " The State of Puppet at GitHub rubygems

32. " The State of Puppet at GitHub ruby 1.8.7

33. " The State of Puppet at GitHub unicorn

34. " The State of Puppet at GitHub puppet 2.7.latest

35. " The State of Puppet at GitHub ~600 nodes

36. " The State of Puppet at GitHub run hourly via

    crond

37. " The State of Puppet at GitHub puppetdb

38. " The State of Puppet at GitHub nagiosdb

39. " The State of Puppet at GitHub "yo puppetdb, gimme

    all your nagios::object::* resources so I can update this nagios config"

40. " The State of Puppet at GitHub filtergendb

41. " The State of Puppet at GitHub "yo puppetdb, gimme

    all the filtergen::rule resources I would realize so I can update this filtergen config"

42. " The State of Puppet at GitHub ⚡ puppetdb ⚡

43. " The State of Puppet at GitHub as it turns

    out, an api call is faster than running puppet on a host

44. " The State of Puppet at GitHub gpanel

45. " The State of Puppet at GitHub "Imagine Puppet Dashboard

    meets Razor and went on a weekend trip to the beach with the Heroku API and drank epic amounts of blue drink"

46. " The State of Puppet at GitHub aka we reinvented

    our own, smaller version of Foreman

47. " The State of Puppet at GitHub inventory

48. " The State of Puppet at GitHub

49. " The State of Puppet at GitHub

50. " The State of Puppet at GitHub app configuration

51. " The State of Puppet at GitHub

52. " The State of Puppet at GitHub versioning of configuration

    values

53. " The State of Puppet at GitHub Create nil =>

    1 Update 1 => 2 Delete 2 => nil

54. " The State of Puppet at GitHub let's make credential

    rolling less awful

55. " The State of Puppet at GitHub provisioning

56. " The State of Puppet at GitHub provisioning is typically

    awful

57. " The State of Puppet at GitHub we sprinkled in

    some ChatOps

58. " The State of Puppet at GitHub

59. " The State of Puppet at GitHub a little bit

    later...

60. " The State of Puppet at GitHub <MACHINE>

61. " The State of Puppet at GitHub enc

62. " The State of Puppet at GitHub we have the

    dumbest ENC out there

63. " The State of Puppet at GitHub ⋆

64. " The State of Puppet at GitHub # /usr/local/sbin/fetch_gpanel_enc \

    fe1.rs.github.com --- parameters: gpanel_cabinet: D20-13 gpanel_enabled: true gpanel_monitored: false

65. " The State of Puppet at GitHub we never specify

    classes via the ENC

66. " The State of Puppet at GitHub any variables we

    pass through are prefixed with gpanel_

67. How GitHub writes Puppet (

68. ( How GitHub Writes Puppet rodjek/puppet-lint

69. ( How GitHub Writes Puppet

70. ( How GitHub Writes Puppet if you aren't using puppet-lint

    to audit your puppet codebase, you are doing it wrong

71. ( How GitHub Writes Puppet puppet-lint enforces the Puppet Labs

    style guide

72. ( How GitHub Writes Puppet puppet-lint can even fix a

    ton of linter errors for you

73. ( How GitHub Writes Puppet put it in a pre-commit

    hook

74. ( How GitHub Writes Puppet $ git commit -am "can't

    lint this" modules/github/manifests/role/redis.pp: syntax ok modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118 1 errors found, aborting commit.

75. ( How GitHub Writes Puppet and then buy rodjek a

    beer

76. ( How GitHub Writes Puppet rodjek/rspec-puppet

77. ( How GitHub Writes Puppet

78. ( How GitHub Writes Puppet if you aren't writing tests

    for your puppet code before running it on a server, you are doing it wrong

79. ( How GitHub Writes Puppet use whatever framework/library

80. ( How GitHub Writes Puppet rspec-puppet just happens to be

    a pretty good one

81. ( How GitHub Writes Puppet put it in a pre-commit

    hook

82. ( How GitHub Writes Puppet $ git commit -am "tests

    dont pass but whatever lol" 1) Expected redis::server would include class "more_than_a_single_c_thread" 1 failures encountered, aborting commit.

83. ( How GitHub Writes Puppet and then buy rodjek another

    beer

84. ( How GitHub Writes Puppet node definitions

85. ( How GitHub Writes Puppet we don't use an ENC

    to describe node classes

86. ( How GitHub Writes Puppet node /^github-redis\d+/ { class {

    'github::role::polling_redis': enabled => $::gpanel_enabled, environment => $::gpanel_environment, private_ipv4 => $::ipaddress, } }

87. ( How GitHub Writes Puppet abstractions all the way down

88. ( How GitHub Writes Puppet treat your site classes as

    cascades down to your dist classes

89. ( How GitHub Writes Puppet class redis::server( # params )

    { class { 'redis::config': ... } -> class { 'redis::package': ... } ~> class { 'redis::service': ... } }

90. ( How GitHub Writes Puppet class github::redis( # params )

    inherits github::defaults { $memory = $environment ? { 'stg' => '2G', default => $half_memory_gb } class { 'redis::server': ... } }

91. ( How GitHub Writes Puppet class github::role::polling_redis( # params )

    { class { 'github::redis': # overrides based on specific node } }

92. ( How GitHub Writes Puppet augeas

93. ( How GitHub Writes Puppet you know what's not awesome?

94. ( How GitHub Writes Puppet an erb template that requires

    your class to take 52876423 parameters so you can configure every possible value in my.cnf

95. ( How GitHub Writes Puppet augeas { 'my.cnf/performance': context =>

    '/files/etc/mysql/my.cnf/mysqld', changes => [ # automatic dump/restore 'set innodb_auto_lru_dump 18000', # innodb "set innodb_buffer_pool_size ${innodb_buffer_pool_size}", 'set innodb_log_file_size 256M', 'set innodb_log_buffer_size 8M', 'set innodb_lazy_drop_table 1', ], require => Percona::Server[$::fqdn] }

96. ( How GitHub Writes Puppet it can seem complex and

    scary

97. ( How GitHub Writes Puppet it is an amazing tool

98. How GitHub ships Puppet )

99. ) How GitHub Ships Puppet continuous integration

100. ) How GitHub Ships Puppet commit gets pushed

101. ) How GitHub Ships Puppet jenkins runs the test suite

102. ) How GitHub Ships Puppet status gets posted back to

     GitHub.com

103. ) How GitHub Ships Puppet tmm1/test-queue

104. ) How GitHub Ships Puppet

105. ) How GitHub Ships Puppet remembers how to better parallelize

     tests

106. ) How GitHub Ships Puppet continuous deployment

107. ) How GitHub Ships Puppet tests passed for commit, Hubot

     auto-deploys

108. ) How GitHub Ships Puppet branch deploy everything

109. ) How GitHub Ships Puppet a lot of shops have

     a few environments

110. ) How GitHub Ships Puppet testing staging production

111. ) How GitHub Ships Puppet currently we have 181 environments

112. ) How GitHub Ships Puppet that's not the same as

     181 nodes not running production

113. ) How GitHub Ships Puppet we only have 5 of

     those

114. ) How GitHub Ships Puppet Hubot automatically merges the master

     branch before allowing any branch to deploy

115. ) How GitHub Ships Puppet ChatOps

116. ) How GitHub Ships Puppet 202 employees 100% access to

     Puppet 100% trust

117. ) How GitHub Ships Puppet the list of roles you'd

     never think would touch production puppet webkit developer core git developers core ruby developers svn developers search developer frontend designers windows developers mac developers support enterprise sales

118. ) How GitHub Ships Puppet every puppet run happens in

     chat

119. ) How GitHub Ships Puppet all puppet output goes to

     chat

120. ) How GitHub Ships Puppet everyone can see everything

121. ) How GitHub Ships Puppet everyone can do anything

122. ) How GitHub Ships Puppet hands-on learning, by accident

123. ) How GitHub Ships Puppet eliminate disruptive questions

124. The Future of Puppet at GitHub *

125. * The Future of Puppet at GitHub puppet 3.x

126. * The Future of Puppet at GitHub upgrading a large,

     old codebase from 2.7 to 3 is really painful

127. * The Future of Puppet at GitHub we've been talking

     about it for a year

128. * The Future of Puppet at GitHub we're still trying

     to push forward, but...

129. * The Future of Puppet at GitHub helping maintain security

     fixes for 2.7

130. * The Future of Puppet at GitHub goal is to

     be on 3.2 by end of year

131. * The Future of Puppet at GitHub ruby 2.0

132. * The Future of Puppet at GitHub moar faster rubby

133. * The Future of Puppet at GitHub we get this

     for free when we move to 3

134. * The Future of Puppet at GitHub mcollective

135. * The Future of Puppet at GitHub aka the story

     of github/shell and the wonders of ssh in a for-loop

136. * The Future of Puppet at GitHub there are a

     lot of cool things about mcollective

137. * The Future of Puppet at GitHub I think it

     will be a part of our stack in the future

138. * The Future of Puppet at GitHub but you can

     still go pretty damn far with bash and ssh

139. * The Future of Puppet at GitHub even more puppetdb

     tooling

140. * The Future of Puppet at GitHub waiting for multiple

     runs to converge exported resources is painful

141. * The Future of Puppet at GitHub use the puppetdb

     API to skip all that

142. * The Future of Puppet at GitHub even more gpanel

     integration

143. * The Future of Puppet at GitHub "databags"

144. * The Future of Puppet at GitHub "node search"

145. * The Future of Puppet at GitHub steal good ideas

     from other tools

146. closing thoughts

147. "all software is terrible" — anyone who's worked with software

     long enough

148. "no software is better than no software" — rtomayko

149. the plumbing doesn't matter when all you care about is

     the porcelain

150. write some damned good porcelain

151. None

152. THIS IS RODJEK

153. HE'S NORMALLY NOT THIS BLURRY

154. HE IS ALSO NOT MY BROTHER, BUT YOU REALLY SHOULD

     BUY HIM A BEER

155. YOU SHOULD BUY THIS MAN A LOT OF BEER

156. THANKS https://speakerdeck.com/wfarr/ puppet-at-github-puppetconf-2013 !