Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet at GitHub (PuppetCamp Raleigh 2013)

Puppet at GitHub (PuppetCamp Raleigh 2013)

Will Farrington

May 16, 2013
Tweet

More Decks by Will Farrington

Other Decks in Technology

Transcript

  1. and other adventures in ChatOps
    Puppet at GitHub

    View full-size slide

  2. @wfarr
    GitHub Operations
    Known Aliases: King of Kebabs
    The Computer Guy
    Mr. RealTalk
    Probably a delinquent

    View full-size slide

  3. The State of Puppet at GitHub

    View full-size slide

  4.  The State of Puppet at GitHub
    boxen/puppet-*

    View full-size slide

  5.  The State of Puppet at GitHub
    ~1 YEAR OLD

    View full-size slide

  6.  The State of Puppet at GitHub
    OPEN SOURCED 3 MONTHS AGO

    View full-size slide

  7.  The State of Puppet at GitHub
    ~180 PUBLIC MODULES

    View full-size slide

  8.  The State of Puppet at GitHub
    2 EXTERNAL MAINTAINERS
    @fromonesrc @JHaals

    View full-size slide

  9. The State of Puppet at GitHub
    THE STACK

    View full-size slide

  10. The State of Puppet at GitHub
    PUPPET 3.2

    View full-size slide

  11. The State of Puppet at GitHub
    MASTER-LESS

    View full-size slide

  12. The State of Puppet at GitHub
    RUBY 1.8.7

    View full-size slide

  13. The State of Puppet at GitHub
    RUN MANUALLY

    View full-size slide

  14.  The State of Puppet at GitHub
    github/puppet

    View full-size slide

  15.  The State of Puppet at GitHub
    ~5 YEARS OLD

    View full-size slide

  16. 2010
    2011
    2012
    2013

    View full-size slide

  17. CODE COMMITS
     The State of Puppet at GitHub

    View full-size slide

  18. CODE ADDITIONS
     The State of Puppet at GitHub

    View full-size slide

  19. CODE DELETIONS
     The State of Puppet at GitHub

    View full-size slide

  20. CONTRIBUTIONS (PAST YEAR)
    ~2k commits
    ~150 commits
     The State of Puppet at GitHub

    View full-size slide

  21. CONTRIBUTIONS (PAST YEAR)
    ~23% of all commits
     The State of Puppet at GitHub

    View full-size slide

  22. The State of Puppet at GitHub
    AN AVERAGE WEEK
    About 50 Pull Requests and 25 Issues comprising
    300 commits across 24 authors

    View full-size slide

  23. The State of Puppet at GitHub
    THE STACK

    View full-size slide

  24. The State of Puppet at GitHub
    PUPPET 2.7.GITHUB

    View full-size slide

  25. The State of Puppet at GitHub
    SINGLE PUPPETMASTER

    View full-size slide

  26. The State of Puppet at GitHub
    RUBY 1.8.7

    View full-size slide

  27. The State of Puppet at GitHub
    RUN VIA CRON JOB

    View full-size slide

  28. The State of Puppet at GitHub
    PUPPETDB

    View full-size slide

  29. The State of Puppet at GitHub
    CUSTOM NODE DEFINITIONS

    View full-size slide

  30. How GitHub writes Puppet

    View full-size slide

  31.  How GitHub Writes Puppet
    PUPPETLABS/STDLIB

    View full-size slide

  32.  How GitHub Writes Puppet
    PARAMETER VALIDATION

    View full-size slide

  33.  How GitHub Writes Puppet
    class redis::server(
    $data_dir = '/var/lib/redis',
    $manage_service = false,
    $package = 'redis-server'
    ) {
    validate_bool($manage_service)
    validate_absolute_path($data_dir)
    validate_re($package, '^redis2?-server$',
    "Redis::Server[${name}]: package must be either redis-server or redis2-server: $
    {package}")
    }

    View full-size slide

  34.  How GitHub Writes Puppet
    DATA MUNGING

    View full-size slide

  35.  How GitHub Writes Puppet
    define ruby::version(
    $env = {}
    ) {
    $default_environment = {
    'CC' => 'clang',
    'CFLAGS' => '-O2'
    }
    $ruby_build_environment = merge($default_environment, $env)
    exec { "install ruby version ${name}":
    environment => join_keys_to_values($ruby_build_environment, '=')
    }
    }

    View full-size slide

  36.  How GitHub Writes Puppet
    RESOURCE HANDLING

    View full-size slide

  37.  How GitHub Writes Puppet
    class redis::server(
    $data_dir = '/var/lib/redis'
    ) {
    if ! defined_with_params(User[redis], { ensure => 'present' }) {
    user { 'redis': ensure => 'present', group => 'redis' }
    }
    }

    View full-size slide

  38.  How GitHub Writes Puppet
    $latest_tcs_version = "${::ruby::root}/versions/1.9.3-p231-tcs-github-1.0.30"
    $tcs_alias = "${::ruby::root}/versions/1.9.3-p231-tcs-github"
    $desired_params = {
    'ensure' => 'link',
    'target' => $latest_tcs_version,
    'force' => true
    }
    File <| title == $tcs_version |> {
    ensure => link,
    target => $latest_tcs_version,
    force => true
    }
    ensure_resource('file', $tcs_alias, $desired_params)

    View full-size slide

  39.  How GitHub Writes Puppet
    GITHUB::ROLE::*

    View full-size slide

  40.  How GitHub Writes Puppet
    NODE CONFIGURATION

    View full-size slide

  41. node /^github-redis\d+[a-z]?-rs1-prd.iad.github.net$/ {
    class { 'github::role::redis':
    env => 'production',
    private_ipv4 => $::ipaddress,
    }
    }
     How GitHub Writes Puppet

    View full-size slide

  42.  How GitHub Writes Puppet
    ROLE CONFIGURATION

    View full-size slide

  43. class github::role::redis($env, $private_ipv4) {
    validate_re($env, '^(vagrant|staging|production)$')
    validate_re($private_ipv4, '^\d+\.\d+\.\d+\.\d+$')
    $monitor = $env ? { 'production' => true, default => false }
    class {
    'github::core':
    monitor => $monitor,
    private_address => $private_ipv4 ;
    'redis::server':
    bind_address => $private_ipv4,
    monitor => $monitor ;
    }
    }
     How GitHub Writes Puppet

    View full-size slide


  44. ABSTRACTION
    How GitHub Writes Puppet

    View full-size slide

  45. class github::core($monitor) {
    include github::common_packages
    include github::staff
    class {
    'github::ssh':
    monitor => $monitor ;
    'github::ipv6':
    ensure => absent ;
    }
    }
     How GitHub Writes Puppet

    View full-size slide

  46. class github::ipv6($ensure = present) {
    if $::lsbdistcodename != 'squeeze' {
    file { '/etc/modprobe.d/ipv6':
    ensure => $ensure,
    mode => '0444',
    source => 'puppet:///modules/github//etc/modprobe.d/ipv6',
    }
    } else {
    $value = $ensure ? { present => 0, default => 1 }
    sysctl { 'net.ipv6.conf.all.disable_ipv6':
    value => $value,
    }
    }
    }
     How GitHub Writes Puppet

    View full-size slide


  47. AUGEAS
    How GitHub Writes Puppet

    View full-size slide

  48. class redis::server($bind_address, $data_dir, $monitor, $port) {
    redis::config {
    'dir':
    value => $data_dir,
    require => File[$data_dir];
    'bind':
    value => $bind_address;
    'port':
    value => $port;
    'daemonize':
    value => 'yes';
    }
    }
     How GitHub Writes Puppet

    View full-size slide

  49. define redis::config($value, $ensure = present) {
    validate_re($ensure, '^(present|absent)$')
    $changes = $ensure ? { present => "set ${name} ${value}", default => "rm ${name}" }
    augeas { "Set Redis config '${name}' to '${value}'":
    changes => $changes,
    context => '/files/etc/redis/redis.conf',
    lens => 'Redis.lns',
    incl => '/etc/redis/redis.conf',
    require => File['/etc/redis/redis.conf']
    }
    }
     How GitHub Writes Puppet

    View full-size slide

  50.  How GitHub Writes Puppet
    CODE SHARE

    View full-size slide


  51. LIBRARIAN-PUPPET
    How GitHub Writes Puppet

    View full-size slide


  52. HENSON
    How GitHub Writes Puppet

    View full-size slide

  53. How GitHub deploys Puppet

    View full-size slide

  54. KEEP IT CLEAN
     How GitHub Deploys Puppet

    View full-size slide

  55. rodjek/puppet-lint

     How GitHub Deploys Puppet

    View full-size slide

  56. KEEP IT GREEN
     How GitHub Deploys Puppet

    View full-size slide

  57. rodjek/rspec-puppet

     How GitHub Deploys Puppet

    View full-size slide

  58. tmm1/test-queue

     How GitHub Deploys Puppet

    View full-size slide

  59. KEEP IT LEAN
     How GitHub Deploys Puppet

    View full-size slide

  60. $ git commit -am "can't lint this"
    modules/github/manifests/role/redis.pp: syntax ok
    modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118
    1 errors found, aborting commit.
     How GitHub Deploys Puppet

    View full-size slide

  61.  How GitHub Deploys Puppet

    View full-size slide

  62.  How GitHub Deploys Puppet

    View full-size slide

  63. CHATOPS
     How GitHub Deploys Puppet

    View full-size slide

  64. /puppet env worker
    #=>
    worker1.rs.github.com: production
    worker2.rs.github.com: production
    worker3.rs.github.com: production
    worker4.rs.github.com: production
    worker5.rs.github.com: production
    worker6.rs.github.com: production
     How GitHub Deploys Puppet

    View full-size slide

  65. /puppet run worker2
    #=> Running puppet on worker2.rs.github.com/production
     How GitHub Deploys Puppet

    View full-size slide

  66. /puppet noop feature_branch worker2
    #=> Running puppet on worker2.rs.github.com/feature_branch --noop
     How GitHub Deploys Puppet

    View full-size slide

  67. /puppet force feature_branch worker2
    #=> Running puppet on worker2.rs.github.com/feature_branch
     How GitHub Deploys Puppet

    View full-size slide

  68. /puppet disable worker2
    #=> Disabling puppet on worker2.rs.github.com
     How GitHub Deploys Puppet

    View full-size slide

  69. /puppet enable worker2
    #=> Disabling puppet on worker2.rs.github.com
     How GitHub Deploys Puppet

    View full-size slide

  70. /puppet last_run worker2
     How GitHub Deploys Puppet

    View full-size slide

  71. /puppet certs
    #=>
    "wills-macbook-pro.local" (A4:5B:AC:B9:E1:85:8B:2B:0E:8B:62:F9:03:32:C9:03)
     How GitHub Deploys Puppet

    View full-size slide

  72. The Future of Puppet at GitHub

    View full-size slide

  73. boxen/puppet-*

    The Future of Puppet at GitHub

    View full-size slide

  74. SHELL "REMEMBER TO RUN" SUPPORT
    The Future of Puppet at GitHub

    View full-size slide

  75. You haven't run Boxen in over a week. =(
    We really recommend running Boxen regularly. It's way better that way!
    Do you want to run boxen now? (y/N) y
    # boxen
    Updating Boxen.
    ...
    The Future of Puppet at GitHub

    View full-size slide

  76. OPT-OUT
    The Future of Puppet at GitHub

    View full-size slide

  77. touch $HOME/.boxen-never-prompt-for-updates
    The Future of Puppet at GitHub

    View full-size slide

  78. PUPPETMASTER SUPPORT
    The Future of Puppet at GitHub

    View full-size slide

  79. OPT-IN ONLY
    The Future of Puppet at GitHub

    View full-size slide

  80. HIERA SUPPORT
    The Future of Puppet at GitHub

    View full-size slide

  81. "uncomfortunities"
    The Future of Puppet at GitHub

    View full-size slide

  82. UBUNTU SUPPORT
    The Future of Puppet at GitHub

    View full-size slide

  83. UBUNTU PRECISE
    The Future of Puppet at GitHub

    View full-size slide

  84. github/puppet

    The Future of Puppet at GitHub

    View full-size slide

  85. The Future of Puppet at GitHub
    RUBY 1.9.3

    View full-size slide

  86. The Future of Puppet at GitHub
    PUPPET 3.X

    View full-size slide

  87. The Future of Puppet at GitHub
    GPANEL: ENC

    View full-size slide

  88. The Future of Puppet at GitHub

    lol censored

    View full-size slide

  89. The Future of Puppet at GitHub
    MCOLLECTIVE

    View full-size slide

  90. The Future of Puppet at GitHub
    REPLACING PARTS OF GITHUB/SHELL

    View full-size slide

  91. The Future of Puppet at GitHub
    STRUCTURED DATA > SED/AWK

    View full-size slide

  92. The Future of Puppet at GitHub
    HIERA

    View full-size slide

  93. THANKS
    speakerdeck.com/wfarr/
    puppet-at-github-
    puppetcamp-raleigh-2013

    View full-size slide