$30 off During Our Annual Pro Sale. View Details »

Puppet at GitHub: PuppetConf 2013

Puppet at GitHub: PuppetConf 2013

Will Farrington

August 22, 2013
Tweet

More Decks by Will Farrington

Other Decks in Technology

Transcript

  1. Puppet at GitHub

    View Slide

  2. @wfarr
    GitHub
    Operations
    Known Aliases:
    King of Kebabs
    The Chairman
    Mr. Caremad
    !

    View Slide

  3. The State of Puppet at GitHub
    "

    View Slide

  4. " The State of Puppet at GitHub
    github/boxen

    View Slide

  5. " The State of Puppet at GitHub
    ~1.5 years old

    View Slide

  6. " The State of Puppet at GitHub
    open-sourced ~7 months ago

    View Slide

  7. " The State of Puppet at GitHub
    ~240 open-source puppet modules

    View Slide

  8. " The State of Puppet at GitHub
    puppet 3.latest

    View Slide

  9. " The State of Puppet at GitHub
    supports ruby 1.8.7, 1.9.3, 2.0.0

    View Slide

  10. " The State of Puppet at GitHub
    hiera

    View Slide

  11. " The State of Puppet at GitHub
    OS X

    View Slide

  12. " The State of Puppet at GitHub
    Linux support in-progress

    View Slide

  13. " The State of Puppet at GitHub
    #
    tomorrow @ 2:20pm

    View Slide

  14. " The State of Puppet at GitHub
    github/puppet

    View Slide

  15. " The State of Puppet at GitHub
    $

    View Slide

  16. " The State of Puppet at GitHub
    ~5 years old

    View Slide

  17. " The State of Puppet at GitHub
    0.24.x — 2.7.x

    View Slide

  18. " The State of Puppet at GitHub
    121 modules

    View Slide

  19. " The State of Puppet at GitHub
    ~280k lines of code

    View Slide

  20. " The State of Puppet at GitHub
    every single employee has commit access

    View Slide

  21. " The State of Puppet at GitHub
    15.5k commits to master past 12 months

    View Slide

  22. " The State of Puppet at GitHub
    by 86 contributors

    View Slide

  23. " The State of Puppet at GitHub
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % %

    View Slide

  24. " The State of Puppet at GitHub
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % %
    42.5%

    View Slide

  25. " The State of Puppet at GitHub
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % % % % % % % % % % % % % % % % % % % %
    % %
    now with ops taken out

    View Slide

  26. " The State of Puppet at GitHub
    0
    100
    200
    300
    400
    2012-08-25
    2012-10-20
    2012-12-15
    2013-02-09
    2013-04-06
    2013-06-01
    2013-07-27
    commits on master per week, last 12 months

    View Slide

  27. " The State of Puppet at GitHub
    0
    1000
    2000
    3000
    4000
    total commits by author past year

    View Slide

  28. " The State of Puppet at GitHub
    0
    175
    350
    525
    700
    total commits by author past year, except ops

    View Slide

  29. " The State of Puppet at GitHub
    0
    175
    350
    525
    700
    total commits by authors with >10 commits, past year, except ops

    View Slide

  30. " The State of Puppet at GitHub
    single puppetmaster

    View Slide

  31. " The State of Puppet at GitHub
    rubygems

    View Slide

  32. " The State of Puppet at GitHub
    ruby 1.8.7

    View Slide

  33. " The State of Puppet at GitHub
    unicorn

    View Slide

  34. " The State of Puppet at GitHub
    puppet 2.7.latest

    View Slide

  35. " The State of Puppet at GitHub
    ~600 nodes

    View Slide

  36. " The State of Puppet at GitHub
    run hourly via crond

    View Slide

  37. " The State of Puppet at GitHub
    puppetdb

    View Slide

  38. " The State of Puppet at GitHub
    nagiosdb

    View Slide

  39. " The State of Puppet at GitHub
    "yo puppetdb, gimme all your
    nagios::object::* resources so I can
    update this nagios config"

    View Slide

  40. " The State of Puppet at GitHub
    filtergendb

    View Slide

  41. " The State of Puppet at GitHub
    "yo puppetdb, gimme all the
    filtergen::rule resources I would realize
    so I can update this filtergen config"

    View Slide

  42. " The State of Puppet at GitHub
    ⚡ puppetdb ⚡

    View Slide

  43. " The State of Puppet at GitHub
    as it turns out, an api call is faster
    than running puppet on a host

    View Slide

  44. " The State of Puppet at GitHub
    gpanel

    View Slide

  45. " The State of Puppet at GitHub
    "Imagine Puppet Dashboard meets
    Razor and went on a weekend trip to the
    beach with the Heroku API and drank
    epic amounts of blue drink"

    View Slide

  46. " The State of Puppet at GitHub
    aka we reinvented our own, smaller
    version of Foreman

    View Slide

  47. " The State of Puppet at GitHub
    inventory

    View Slide

  48. " The State of Puppet at GitHub

    View Slide

  49. " The State of Puppet at GitHub

    View Slide

  50. " The State of Puppet at GitHub
    app configuration

    View Slide

  51. " The State of Puppet at GitHub

    View Slide

  52. " The State of Puppet at GitHub
    versioning of configuration values

    View Slide

  53. " The State of Puppet at GitHub
    Create nil => 1
    Update 1 => 2
    Delete 2 => nil

    View Slide

  54. " The State of Puppet at GitHub
    let's make credential rolling less awful

    View Slide

  55. " The State of Puppet at GitHub
    provisioning

    View Slide

  56. " The State of Puppet at GitHub
    provisioning is typically awful

    View Slide

  57. " The State of Puppet at GitHub
    we sprinkled in some ChatOps

    View Slide

  58. " The State of Puppet at GitHub

    View Slide

  59. " The State of Puppet at GitHub
    a little bit later...

    View Slide

  60. " The State of Puppet at GitHub

    View Slide

  61. " The State of Puppet at GitHub
    enc

    View Slide

  62. " The State of Puppet at GitHub
    we have the dumbest ENC out there

    View Slide

  63. " The State of Puppet at GitHub

    View Slide

  64. " The State of Puppet at GitHub
    # /usr/local/sbin/fetch_gpanel_enc \
    fe1.rs.github.com
    ---
    parameters:
    gpanel_cabinet: D20-13
    gpanel_enabled: true
    gpanel_monitored: false

    View Slide

  65. " The State of Puppet at GitHub
    we never specify classes via the ENC

    View Slide

  66. " The State of Puppet at GitHub
    any variables we pass through
    are prefixed with gpanel_

    View Slide

  67. How GitHub writes Puppet
    (

    View Slide

  68. ( How GitHub Writes Puppet
    rodjek/puppet-lint

    View Slide

  69. ( How GitHub Writes Puppet

    View Slide

  70. ( How GitHub Writes Puppet
    if you aren't using puppet-lint
    to audit your puppet codebase,
    you are doing it wrong

    View Slide

  71. ( How GitHub Writes Puppet
    puppet-lint enforces the
    Puppet Labs style guide

    View Slide

  72. ( How GitHub Writes Puppet
    puppet-lint can even fix a ton
    of linter errors for you

    View Slide

  73. ( How GitHub Writes Puppet
    put it in a pre-commit hook

    View Slide

  74. ( How GitHub Writes Puppet
    $ git commit -am "can't lint this"
    modules/github/manifests/role/redis.pp: syntax ok
    modules/github/manifests/role/redis.pp - WARNING: => is not
    properly aligned on line 118
    1 errors found, aborting commit.

    View Slide

  75. ( How GitHub Writes Puppet
    and then buy rodjek a beer

    View Slide

  76. ( How GitHub Writes Puppet
    rodjek/rspec-puppet

    View Slide

  77. ( How GitHub Writes Puppet

    View Slide

  78. ( How GitHub Writes Puppet
    if you aren't writing tests for
    your puppet code before
    running it on a server,
    you are doing it wrong

    View Slide

  79. ( How GitHub Writes Puppet
    use whatever framework/library

    View Slide

  80. ( How GitHub Writes Puppet
    rspec-puppet just happens to be
    a pretty good one

    View Slide

  81. ( How GitHub Writes Puppet
    put it in a pre-commit hook

    View Slide

  82. ( How GitHub Writes Puppet
    $ git commit -am "tests dont pass but whatever lol"
    1) Expected redis::server would include class
    "more_than_a_single_c_thread"
    1 failures encountered, aborting commit.

    View Slide

  83. ( How GitHub Writes Puppet
    and then buy rodjek another beer

    View Slide

  84. ( How GitHub Writes Puppet
    node definitions

    View Slide

  85. ( How GitHub Writes Puppet
    we don't use an ENC to describe
    node classes

    View Slide

  86. ( How GitHub Writes Puppet
    node /^github-redis\d+/ {
    class { 'github::role::polling_redis':
    enabled => $::gpanel_enabled,
    environment => $::gpanel_environment,
    private_ipv4 => $::ipaddress,
    }
    }

    View Slide

  87. ( How GitHub Writes Puppet
    abstractions all the way down

    View Slide

  88. ( How GitHub Writes Puppet
    treat your site classes as cascades
    down to your dist classes

    View Slide

  89. ( How GitHub Writes Puppet
    class redis::server(
    # params
    ) {
    class { 'redis::config': ... }
    -> class { 'redis::package': ... }
    ~> class { 'redis::service': ... }
    }

    View Slide

  90. ( How GitHub Writes Puppet
    class github::redis(
    # params
    ) inherits github::defaults {
    $memory = $environment ? {
    'stg' => '2G',
    default => $half_memory_gb
    }
    class { 'redis::server': ... }
    }

    View Slide

  91. ( How GitHub Writes Puppet
    class github::role::polling_redis(
    # params
    ) {
    class { 'github::redis':
    # overrides based on specific node
    }
    }

    View Slide

  92. ( How GitHub Writes Puppet
    augeas

    View Slide

  93. ( How GitHub Writes Puppet
    you know what's not awesome?

    View Slide

  94. ( How GitHub Writes Puppet
    an erb template that requires your
    class to take 52876423 parameters
    so you can configure every possible
    value in my.cnf

    View Slide

  95. ( How GitHub Writes Puppet
    augeas { 'my.cnf/performance':
    context => '/files/etc/mysql/my.cnf/mysqld',
    changes => [
    # automatic dump/restore
    'set innodb_auto_lru_dump 18000',
    # innodb
    "set innodb_buffer_pool_size ${innodb_buffer_pool_size}",
    'set innodb_log_file_size 256M',
    'set innodb_log_buffer_size 8M',
    'set innodb_lazy_drop_table 1',
    ],
    require => Percona::Server[$::fqdn]
    }

    View Slide

  96. ( How GitHub Writes Puppet
    it can seem complex and scary

    View Slide

  97. ( How GitHub Writes Puppet
    it is an amazing tool

    View Slide

  98. How GitHub ships Puppet
    )

    View Slide

  99. ) How GitHub Ships Puppet
    continuous integration

    View Slide

  100. ) How GitHub Ships Puppet
    commit gets pushed

    View Slide

  101. ) How GitHub Ships Puppet
    jenkins runs the test suite

    View Slide

  102. ) How GitHub Ships Puppet
    status gets posted back to GitHub.com

    View Slide

  103. ) How GitHub Ships Puppet
    tmm1/test-queue

    View Slide

  104. ) How GitHub Ships Puppet

    View Slide

  105. ) How GitHub Ships Puppet
    remembers how to better parallelize tests

    View Slide

  106. ) How GitHub Ships Puppet
    continuous deployment

    View Slide

  107. ) How GitHub Ships Puppet
    tests passed for commit, Hubot auto-deploys

    View Slide

  108. ) How GitHub Ships Puppet
    branch deploy everything

    View Slide

  109. ) How GitHub Ships Puppet
    a lot of shops have a few environments

    View Slide

  110. ) How GitHub Ships Puppet
    testing
    staging
    production

    View Slide

  111. ) How GitHub Ships Puppet
    currently we have 181 environments

    View Slide

  112. ) How GitHub Ships Puppet
    that's not the same as 181 nodes
    not running production

    View Slide

  113. ) How GitHub Ships Puppet
    we only have 5 of those

    View Slide

  114. ) How GitHub Ships Puppet
    Hubot automatically merges the
    master branch before allowing any
    branch to deploy

    View Slide

  115. ) How GitHub Ships Puppet
    ChatOps

    View Slide

  116. ) How GitHub Ships Puppet
    202 employees
    100% access to Puppet
    100% trust

    View Slide

  117. ) How GitHub Ships Puppet
    the list of roles you'd never think would touch production puppet
    webkit developer
    core git developers
    core ruby developers
    svn developers
    search developer
    frontend designers
    windows developers
    mac developers
    support
    enterprise sales

    View Slide

  118. ) How GitHub Ships Puppet
    every puppet run happens in chat

    View Slide

  119. ) How GitHub Ships Puppet
    all puppet output goes to chat

    View Slide

  120. ) How GitHub Ships Puppet
    everyone can see everything

    View Slide

  121. ) How GitHub Ships Puppet
    everyone can do anything

    View Slide

  122. ) How GitHub Ships Puppet
    hands-on learning, by accident

    View Slide

  123. ) How GitHub Ships Puppet
    eliminate disruptive questions

    View Slide

  124. The Future of Puppet at GitHub
    *

    View Slide

  125. * The Future of Puppet at GitHub
    puppet 3.x

    View Slide

  126. * The Future of Puppet at GitHub
    upgrading a large, old codebase
    from 2.7 to 3 is really painful

    View Slide

  127. * The Future of Puppet at GitHub
    we've been talking about it for a year

    View Slide

  128. * The Future of Puppet at GitHub
    we're still trying to push forward, but...

    View Slide

  129. * The Future of Puppet at GitHub
    helping maintain security fixes for 2.7

    View Slide

  130. * The Future of Puppet at GitHub
    goal is to be on 3.2 by end of year

    View Slide

  131. * The Future of Puppet at GitHub
    ruby 2.0

    View Slide

  132. * The Future of Puppet at GitHub
    moar faster rubby

    View Slide

  133. * The Future of Puppet at GitHub
    we get this for free when we move to 3

    View Slide

  134. * The Future of Puppet at GitHub
    mcollective

    View Slide

  135. * The Future of Puppet at GitHub
    aka the story of github/shell and the
    wonders of ssh in a for-loop

    View Slide

  136. * The Future of Puppet at GitHub
    there are a lot of cool things about
    mcollective

    View Slide

  137. * The Future of Puppet at GitHub
    I think it will be a part of our stack in
    the future

    View Slide

  138. * The Future of Puppet at GitHub
    but you can still go pretty damn far
    with bash and ssh

    View Slide

  139. * The Future of Puppet at GitHub
    even more puppetdb tooling

    View Slide

  140. * The Future of Puppet at GitHub
    waiting for multiple runs to converge
    exported resources is painful

    View Slide

  141. * The Future of Puppet at GitHub
    use the puppetdb API to skip all that

    View Slide

  142. * The Future of Puppet at GitHub
    even more gpanel integration

    View Slide

  143. * The Future of Puppet at GitHub
    "databags"

    View Slide

  144. * The Future of Puppet at GitHub
    "node search"

    View Slide

  145. * The Future of Puppet at GitHub
    steal good ideas from other tools

    View Slide

  146. closing thoughts

    View Slide

  147. "all software is terrible"
    — anyone who's worked with software long enough

    View Slide

  148. "no software is better than no software"
    — rtomayko

    View Slide

  149. the plumbing doesn't matter when all you
    care about is the porcelain

    View Slide

  150. write some damned good porcelain

    View Slide

  151. View Slide

  152. THIS IS RODJEK

    View Slide

  153. HE'S NORMALLY
    NOT THIS BLURRY

    View Slide

  154. HE IS ALSO NOT MY BROTHER,
    BUT YOU REALLY SHOULD BUY
    HIM A BEER

    View Slide

  155. YOU SHOULD BUY THIS
    MAN A LOT OF BEER

    View Slide

  156. THANKS
    https://speakerdeck.com/wfarr/
    puppet-at-github-puppetconf-2013
    !

    View Slide