Puppet at GitHub (PuppetCamp Raleigh 2013)

Transcript

1. and other adventures in ChatOps
   Puppet at GitHub
   

   View Slide

2. @wfarr
   GitHub Operations
   Known Aliases: King of Kebabs
   The Computer Guy
   Mr. RealTalk
   Probably a delinquent
   

   View Slide

3. The State of Puppet at GitHub
   
   

   View Slide

4.  The State of Puppet at GitHub
   boxen/puppet-*
   
   

   View Slide

5.  The State of Puppet at GitHub
   ~1 YEAR OLD
   

   View Slide

6.  The State of Puppet at GitHub
   OPEN SOURCED 3 MONTHS AGO
   

   View Slide

7.  The State of Puppet at GitHub
   ~180 PUBLIC MODULES
   

   View Slide

8.  The State of Puppet at GitHub
   2 EXTERNAL MAINTAINERS
   @fromonesrc @JHaals
   

   View Slide

9. The State of Puppet at GitHub
   THE STACK
   
   

   View Slide

10. The State of Puppet at GitHub
    PUPPET 3.2
    
    

    View Slide

11. The State of Puppet at GitHub
    MASTER-LESS
    
    

    View Slide

12. The State of Puppet at GitHub
    RUBY 1.8.7
    
    

    View Slide

13. The State of Puppet at GitHub
    RUN MANUALLY
    
    

    View Slide

14.  The State of Puppet at GitHub
    github/puppet
    
    

    View Slide

15.  The State of Puppet at GitHub
    ~5 YEARS OLD
    

    View Slide

16. View Slide

17. CODE COMMITS
     The State of Puppet at GitHub
    

    View Slide

18. CODE ADDITIONS
     The State of Puppet at GitHub
    

    View Slide

19. CODE DELETIONS
     The State of Puppet at GitHub
    

    View Slide

20. CONTRIBUTIONS (PAST YEAR)
    ~2k commits
    ~150 commits
     The State of Puppet at GitHub
    

    View Slide

21. CONTRIBUTIONS (PAST YEAR)
    ~23% of all commits
     The State of Puppet at GitHub
    

    View Slide

22. The State of Puppet at GitHub
    AN AVERAGE WEEK
    About 50 Pull Requests and 25 Issues comprising
    300 commits across 24 authors
    
    

    View Slide

23. The State of Puppet at GitHub
    THE STACK
    
    

    View Slide

24. The State of Puppet at GitHub
    PUPPET 2.7.GITHUB
    
    

    View Slide

25. The State of Puppet at GitHub
    SINGLE PUPPETMASTER
    
    

    View Slide

26. The State of Puppet at GitHub
    RUBY 1.8.7
    
    

    View Slide

27. The State of Puppet at GitHub
    RUN VIA CRON JOB
    
    

    View Slide

28. The State of Puppet at GitHub
    PUPPETDB
    
    

    View Slide

29. The State of Puppet at GitHub
    CUSTOM NODE DEFINITIONS
    
    

    View Slide

30. How GitHub writes Puppet
    
    

    View Slide

31.  How GitHub Writes Puppet
    PUPPETLABS/STDLIB
    

    View Slide

32.  How GitHub Writes Puppet
    PARAMETER VALIDATION
    

    View Slide

33.  How GitHub Writes Puppet
    class redis::server(
    $data_dir = '/var/lib/redis',
    $manage_service = false,
    $package = 'redis-server'
    ) {
    validate_bool($manage_service)
    validate_absolute_path($data_dir)
    validate_re($package, '^redis2?-server$',
    "Redis::Server[${name}]: package must be either redis-server or redis2-server: $
    {package}")
    }
    

    View Slide

34.  How GitHub Writes Puppet
    DATA MUNGING
    

    View Slide

35.  How GitHub Writes Puppet
    define ruby::version(
    $env = {}
    ) {
    $default_environment = {
    'CC' => 'clang',
    'CFLAGS' => '-O2'
    }
    $ruby_build_environment = merge($default_environment, $env)
    exec { "install ruby version ${name}":
    environment => join_keys_to_values($ruby_build_environment, '=')
    }
    }
    

    View Slide

36.  How GitHub Writes Puppet
    RESOURCE HANDLING
    

    View Slide

37.  How GitHub Writes Puppet
    class redis::server(
    $data_dir = '/var/lib/redis'
    ) {
    if ! defined_with_params(User[redis], { ensure => 'present' }) {
    user { 'redis': ensure => 'present', group => 'redis' }
    }
    }
    

    View Slide

38.  How GitHub Writes Puppet
    $latest_tcs_version = "${::ruby::root}/versions/1.9.3-p231-tcs-github-1.0.30"
    $tcs_alias = "${::ruby::root}/versions/1.9.3-p231-tcs-github"
    $desired_params = {
    'ensure' => 'link',
    'target' => $latest_tcs_version,
    'force' => true
    }
    File {
    ensure => link,
    target => $latest_tcs_version,
    force => true
    }
    ensure_resource('file', $tcs_alias, $desired_params)
    

    View Slide

39.  How GitHub Writes Puppet
    GITHUB::ROLE::*
    

    View Slide

40.  How GitHub Writes Puppet
    NODE CONFIGURATION
    

    View Slide

41. node /^github-redis\d+[a-z]?-rs1-prd.iad.github.net$/ {
    class { 'github::role::redis':
    env => 'production',
    private_ipv4 => $::ipaddress,
    }
    }
     How GitHub Writes Puppet
    

    View Slide

42.  How GitHub Writes Puppet
    ROLE CONFIGURATION
    

    View Slide

43. class github::role::redis($env, $private_ipv4) {
    validate_re($env, '^(vagrant|staging|production)$')
    validate_re($private_ipv4, '^\d+\.\d+\.\d+\.\d+$')
    $monitor = $env ? { 'production' => true, default => false }
    class {
    'github::core':
    monitor => $monitor,
    private_address => $private_ipv4 ;
    'redis::server':
    bind_address => $private_ipv4,
    monitor => $monitor ;
    }
    }
     How GitHub Writes Puppet
    

    View Slide

44. 
    ABSTRACTION
    How GitHub Writes Puppet
    

    View Slide

45. class github::core($monitor) {
    include github::common_packages
    include github::staff
    class {
    'github::ssh':
    monitor => $monitor ;
    'github::ipv6':
    ensure => absent ;
    }
    }
     How GitHub Writes Puppet
    

    View Slide

46. class github::ipv6($ensure = present) {
    if $::lsbdistcodename != 'squeeze' {
    file { '/etc/modprobe.d/ipv6':
    ensure => $ensure,
    mode => '0444',
    source => 'puppet:///modules/github//etc/modprobe.d/ipv6',
    }
    } else {
    $value = $ensure ? { present => 0, default => 1 }
    sysctl { 'net.ipv6.conf.all.disable_ipv6':
    value => $value,
    }
    }
    }
     How GitHub Writes Puppet
    

    View Slide

47. 
    AUGEAS
    How GitHub Writes Puppet
    

    View Slide

48. class redis::server($bind_address, $data_dir, $monitor, $port) {
    redis::config {
    'dir':
    value => $data_dir,
    require => File[$data_dir];
    'bind':
    value => $bind_address;
    'port':
    value => $port;
    'daemonize':
    value => 'yes';
    }
    }
     How GitHub Writes Puppet
    

    View Slide

49. define redis::config($value, $ensure = present) {
    validate_re($ensure, '^(present|absent)$')
    $changes = $ensure ? { present => "set ${name} ${value}", default => "rm ${name}" }
    augeas { "Set Redis config '${name}' to '${value}'":
    changes => $changes,
    context => '/files/etc/redis/redis.conf',
    lens => 'Redis.lns',
    incl => '/etc/redis/redis.conf',
    require => File['/etc/redis/redis.conf']
    }
    }
     How GitHub Writes Puppet
    

    View Slide

50.  How GitHub Writes Puppet
    CODE SHARE
    

    View Slide

51. 
    LIBRARIAN-PUPPET
    How GitHub Writes Puppet
    

    View Slide

52. 
    HENSON
    How GitHub Writes Puppet
    

    View Slide

53. How GitHub deploys Puppet
    
    

    View Slide

54. KEEP IT CLEAN
     How GitHub Deploys Puppet
    

    View Slide

55. rodjek/puppet-lint
    
     How GitHub Deploys Puppet
    

    View Slide

56. KEEP IT GREEN
     How GitHub Deploys Puppet
    

    View Slide

57. rodjek/rspec-puppet
    
     How GitHub Deploys Puppet
    

    View Slide

58. tmm1/test-queue
    
     How GitHub Deploys Puppet
    

    View Slide

59. KEEP IT LEAN
     How GitHub Deploys Puppet
    

    View Slide

60. $ git commit -am "can't lint this"
    modules/github/manifests/role/redis.pp: syntax ok
    modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118
    1 errors found, aborting commit.
     How GitHub Deploys Puppet
    

    View Slide

61.  How GitHub Deploys Puppet
    

    View Slide

62.  How GitHub Deploys Puppet
    

    View Slide

63. CHATOPS
     How GitHub Deploys Puppet
    

    View Slide

64. /puppet env worker
    #=>
    worker1.rs.github.com: production
    worker2.rs.github.com: production
    worker3.rs.github.com: production
    worker4.rs.github.com: production
    worker5.rs.github.com: production
    worker6.rs.github.com: production
     How GitHub Deploys Puppet
    

    View Slide

65. /puppet run worker2
    #=> Running puppet on worker2.rs.github.com/production
     How GitHub Deploys Puppet
    

    View Slide

66. /puppet noop feature_branch worker2
    #=> Running puppet on worker2.rs.github.com/feature_branch --noop
     How GitHub Deploys Puppet
    

    View Slide

67. /puppet force feature_branch worker2
    #=> Running puppet on worker2.rs.github.com/feature_branch
     How GitHub Deploys Puppet
    

    View Slide

68. /puppet disable worker2
    #=> Disabling puppet on worker2.rs.github.com
     How GitHub Deploys Puppet
    

    View Slide

69. /puppet enable worker2
    #=> Disabling puppet on worker2.rs.github.com
     How GitHub Deploys Puppet
    

    View Slide

70. /puppet last_run worker2
     How GitHub Deploys Puppet
    

    View Slide

71. /puppet certs
    #=>
    "wills-macbook-pro.local" (A4:5B:AC:B9:E1:85:8B:2B:0E:8B:62:F9:03:32:C9:03)
     How GitHub Deploys Puppet
    

    View Slide

72. The Future of Puppet at GitHub
    
    

    View Slide

73. boxen/puppet-*
    
    The Future of Puppet at GitHub
    
    

    View Slide

74. SHELL "REMEMBER TO RUN" SUPPORT
    The Future of Puppet at GitHub
    
    

    View Slide

75. You haven't run Boxen in over a week. =(
    We really recommend running Boxen regularly. It's way better that way!
    Do you want to run boxen now? (y/N) y
    # boxen
    Updating Boxen.
    ...
    The Future of Puppet at GitHub
    
    

    View Slide

76. OPT-OUT
    The Future of Puppet at GitHub
    
    

    View Slide

77. touch $HOME/.boxen-never-prompt-for-updates
    The Future of Puppet at GitHub
    
    

    View Slide

78. PUPPETMASTER SUPPORT
    The Future of Puppet at GitHub
    
    

    View Slide

79. OPT-IN ONLY
    The Future of Puppet at GitHub
    
    

    View Slide

80. HIERA SUPPORT
    The Future of Puppet at GitHub
    
    

    View Slide

81. "uncomfortunities"
    The Future of Puppet at GitHub
    
    

    View Slide

82. UBUNTU SUPPORT
    The Future of Puppet at GitHub
    
    

    View Slide

83. UBUNTU PRECISE
    The Future of Puppet at GitHub
    
    

    View Slide

84. github/puppet
    
    The Future of Puppet at GitHub
    
    

    View Slide

85. The Future of Puppet at GitHub
    RUBY 1.9.3
    
    

    View Slide

86. The Future of Puppet at GitHub
    PUPPET 3.X
    
    

    View Slide

87. The Future of Puppet at GitHub
    GPANEL: ENC
    
    

    View Slide

88. The Future of Puppet at GitHub
    
    lol censored
    

    View Slide

89. The Future of Puppet at GitHub
    MCOLLECTIVE
    
    

    View Slide

90. The Future of Puppet at GitHub
    REPLACING PARTS OF GITHUB/SHELL
    
    

    View Slide

91. The Future of Puppet at GitHub
    STRUCTURED DATA > SED/AWK
    
    

    View Slide

92. The Future of Puppet at GitHub
    HIERA
    
    

    View Slide

93. THANKS
    speakerdeck.com/wfarr/
    puppet-at-github-
    puppetcamp-raleigh-2013
    

    View Slide