Puppet at GitHub (PuppetCamp Raleigh 2013)

Slide 1

Slide 1 text

and other adventures in ChatOps Puppet at GitHub

Slide 2

Slide 2 text

@wfarr GitHub Operations Known Aliases: King of Kebabs The Computer Guy Mr. RealTalk Probably a delinquent

Slide 3

Slide 3 text

The State of Puppet at GitHub 

Slide 4

Slide 4 text

 The State of Puppet at GitHub boxen/puppet-* 

Slide 5

Slide 5 text

 The State of Puppet at GitHub ~1 YEAR OLD

Slide 6

Slide 6 text

 The State of Puppet at GitHub OPEN SOURCED 3 MONTHS AGO

Slide 7

Slide 7 text

 The State of Puppet at GitHub ~180 PUBLIC MODULES

Slide 8

Slide 8 text

 The State of Puppet at GitHub 2 EXTERNAL MAINTAINERS @fromonesrc @JHaals

Slide 9

Slide 9 text

The State of Puppet at GitHub THE STACK 

Slide 10

Slide 10 text

The State of Puppet at GitHub PUPPET 3.2 

Slide 11

Slide 11 text

The State of Puppet at GitHub MASTER-LESS 

Slide 12

Slide 12 text

The State of Puppet at GitHub RUBY 1.8.7 

Slide 13

Slide 13 text

The State of Puppet at GitHub RUN MANUALLY 

Slide 14

Slide 14 text

 The State of Puppet at GitHub github/puppet 

Slide 15

Slide 15 text

 The State of Puppet at GitHub ~5 YEARS OLD

Slide 16

Slide 16 text

2010 2011 2012 2013

Slide 17

Slide 17 text

CODE COMMITS  The State of Puppet at GitHub

Slide 18

Slide 18 text

CODE ADDITIONS  The State of Puppet at GitHub

Slide 19

Slide 19 text

CODE DELETIONS  The State of Puppet at GitHub

Slide 20

Slide 20 text

CONTRIBUTIONS (PAST YEAR) ~2k commits ~150 commits  The State of Puppet at GitHub

Slide 21

Slide 21 text

CONTRIBUTIONS (PAST YEAR) ~23% of all commits  The State of Puppet at GitHub

Slide 22

Slide 22 text

The State of Puppet at GitHub AN AVERAGE WEEK About 50 Pull Requests and 25 Issues comprising 300 commits across 24 authors 

Slide 23

Slide 23 text

The State of Puppet at GitHub THE STACK 

Slide 24

Slide 24 text

The State of Puppet at GitHub PUPPET 2.7.GITHUB 

Slide 25

Slide 25 text

The State of Puppet at GitHub SINGLE PUPPETMASTER 

Slide 26

Slide 26 text

The State of Puppet at GitHub RUBY 1.8.7 

Slide 27

Slide 27 text

The State of Puppet at GitHub RUN VIA CRON JOB 

Slide 28

Slide 28 text

The State of Puppet at GitHub PUPPETDB 

Slide 29

Slide 29 text

The State of Puppet at GitHub CUSTOM NODE DEFINITIONS 

Slide 30

Slide 30 text

How GitHub writes Puppet 

Slide 31

Slide 31 text

 How GitHub Writes Puppet PUPPETLABS/STDLIB

Slide 32

Slide 32 text

 How GitHub Writes Puppet PARAMETER VALIDATION

Slide 33

Slide 33 text

 How GitHub Writes Puppet class redis::server( $data_dir = '/var/lib/redis', $manage_service = false, $package = 'redis-server' ) { validate_bool($manage_service) validate_absolute_path($data_dir) validate_re($package, '^redis2?-server$', "Redis::Server[${name}]: package must be either redis-server or redis2-server: $ {package}") }

Slide 34

Slide 34 text

 How GitHub Writes Puppet DATA MUNGING

Slide 35

Slide 35 text

 How GitHub Writes Puppet define ruby::version( $env = {} ) { $default_environment = { 'CC' => 'clang', 'CFLAGS' => '-O2' } $ruby_build_environment = merge($default_environment, $env) exec { "install ruby version ${name}": environment => join_keys_to_values($ruby_build_environment, '=') } }

Slide 36

Slide 36 text

 How GitHub Writes Puppet RESOURCE HANDLING

Slide 37

Slide 37 text

 How GitHub Writes Puppet class redis::server( $data_dir = '/var/lib/redis' ) { if ! defined_with_params(User[redis], { ensure => 'present' }) { user { 'redis': ensure => 'present', group => 'redis' } } }

Slide 38

Slide 38 text

 How GitHub Writes Puppet $latest_tcs_version = "${::ruby::root}/versions/1.9.3-p231-tcs-github-1.0.30" $tcs_alias = "${::ruby::root}/versions/1.9.3-p231-tcs-github" $desired_params = { 'ensure' => 'link', 'target' => $latest_tcs_version, 'force' => true } File <| title == $tcs_version |> { ensure => link, target => $latest_tcs_version, force => true } ensure_resource('file', $tcs_alias, $desired_params)

Slide 39

Slide 39 text

 How GitHub Writes Puppet GITHUB::ROLE::*

Slide 40

Slide 40 text

 How GitHub Writes Puppet NODE CONFIGURATION

Slide 41

Slide 41 text

node /^github-redis\d+[a-z]?-rs1-prd.iad.github.net$/ { class { 'github::role::redis': env => 'production', private_ipv4 => $::ipaddress, } }  How GitHub Writes Puppet

Slide 42

Slide 42 text

 How GitHub Writes Puppet ROLE CONFIGURATION

Slide 43

Slide 43 text

class github::role::redis($env, $private_ipv4) { validate_re($env, '^(vagrant|staging|production)$') validate_re($private_ipv4, '^\d+\.\d+\.\d+\.\d+$') $monitor = $env ? { 'production' => true, default => false } class { 'github::core': monitor => $monitor, private_address => $private_ipv4 ; 'redis::server': bind_address => $private_ipv4, monitor => $monitor ; } }  How GitHub Writes Puppet

Slide 44

Slide 44 text

 ABSTRACTION How GitHub Writes Puppet

Slide 45

Slide 45 text

class github::core($monitor) { include github::common_packages include github::staff class { 'github::ssh': monitor => $monitor ; 'github::ipv6': ensure => absent ; } }  How GitHub Writes Puppet

Slide 46

Slide 46 text

class github::ipv6($ensure = present) { if $::lsbdistcodename != 'squeeze' { file { '/etc/modprobe.d/ipv6': ensure => $ensure, mode => '0444', source => 'puppet:///modules/github//etc/modprobe.d/ipv6', } } else { $value = $ensure ? { present => 0, default => 1 } sysctl { 'net.ipv6.conf.all.disable_ipv6': value => $value, } } }  How GitHub Writes Puppet

Slide 47

Slide 47 text

 AUGEAS How GitHub Writes Puppet

Slide 48

Slide 48 text

class redis::server($bind_address, $data_dir, $monitor, $port) { redis::config { 'dir': value => $data_dir, require => File[$data_dir]; 'bind': value => $bind_address; 'port': value => $port; 'daemonize': value => 'yes'; } }  How GitHub Writes Puppet

Slide 49

Slide 49 text

define redis::config($value, $ensure = present) { validate_re($ensure, '^(present|absent)$') $changes = $ensure ? { present => "set ${name} ${value}", default => "rm ${name}" } augeas { "Set Redis config '${name}' to '${value}'": changes => $changes, context => '/files/etc/redis/redis.conf', lens => 'Redis.lns', incl => '/etc/redis/redis.conf', require => File['/etc/redis/redis.conf'] } }  How GitHub Writes Puppet

Slide 50

Slide 50 text

 How GitHub Writes Puppet CODE SHARE

Slide 51

Slide 51 text

 LIBRARIAN-PUPPET How GitHub Writes Puppet

Slide 52

Slide 52 text

 HENSON How GitHub Writes Puppet

Slide 53

Slide 53 text

How GitHub deploys Puppet 

Slide 54

Slide 54 text

KEEP IT CLEAN  How GitHub Deploys Puppet

Slide 55

Slide 55 text

rodjek/puppet-lint   How GitHub Deploys Puppet

Slide 56

Slide 56 text

KEEP IT GREEN  How GitHub Deploys Puppet

Slide 57

Slide 57 text

rodjek/rspec-puppet   How GitHub Deploys Puppet

Slide 58

Slide 58 text

tmm1/test-queue   How GitHub Deploys Puppet

Slide 59

Slide 59 text

KEEP IT LEAN  How GitHub Deploys Puppet

Slide 60

Slide 60 text

$ git commit -am "can't lint this" modules/github/manifests/role/redis.pp: syntax ok modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118 1 errors found, aborting commit.  How GitHub Deploys Puppet

Slide 61

Slide 61 text

 How GitHub Deploys Puppet

Slide 62

Slide 62 text

 How GitHub Deploys Puppet

Slide 63

Slide 63 text

CHATOPS  How GitHub Deploys Puppet

Slide 64

Slide 64 text

/puppet env worker #=> worker1.rs.github.com: production worker2.rs.github.com: production worker3.rs.github.com: production worker4.rs.github.com: production worker5.rs.github.com: production worker6.rs.github.com: production  How GitHub Deploys Puppet

Slide 65

Slide 65 text

/puppet run worker2 #=> Running puppet on worker2.rs.github.com/production  How GitHub Deploys Puppet

Slide 66

Slide 66 text

/puppet noop feature_branch worker2 #=> Running puppet on worker2.rs.github.com/feature_branch --noop  How GitHub Deploys Puppet

Slide 67

Slide 67 text

/puppet force feature_branch worker2 #=> Running puppet on worker2.rs.github.com/feature_branch  How GitHub Deploys Puppet

Slide 68

Slide 68 text

/puppet disable worker2 #=> Disabling puppet on worker2.rs.github.com  How GitHub Deploys Puppet

Slide 69

Slide 69 text

/puppet enable worker2 #=> Disabling puppet on worker2.rs.github.com  How GitHub Deploys Puppet

Slide 70

Slide 70 text

/puppet last_run worker2  How GitHub Deploys Puppet

Slide 71

Slide 71 text

/puppet certs #=> "wills-macbook-pro.local" (A4:5B:AC:B9:E1:85:8B:2B:0E:8B:62:F9:03:32:C9:03)  How GitHub Deploys Puppet

Slide 72

Slide 72 text

The Future of Puppet at GitHub 

Slide 73

Slide 73 text

boxen/puppet-*  The Future of Puppet at GitHub 

Slide 74

Slide 74 text

SHELL "REMEMBER TO RUN" SUPPORT The Future of Puppet at GitHub 

Slide 75

Slide 75 text

You haven't run Boxen in over a week. =( We really recommend running Boxen regularly. It's way better that way! Do you want to run boxen now? (y/N) y # boxen Updating Boxen. ... The Future of Puppet at GitHub 

Slide 76

Slide 76 text

OPT-OUT The Future of Puppet at GitHub 

Slide 77

Slide 77 text

touch $HOME/.boxen-never-prompt-for-updates The Future of Puppet at GitHub 

Slide 78

Slide 78 text

PUPPETMASTER SUPPORT The Future of Puppet at GitHub 

Slide 79

Slide 79 text

OPT-IN ONLY The Future of Puppet at GitHub 

Slide 80

Slide 80 text

HIERA SUPPORT The Future of Puppet at GitHub 

Slide 81

Slide 81 text

"uncomfortunities" The Future of Puppet at GitHub 

Slide 82

Slide 82 text

UBUNTU SUPPORT The Future of Puppet at GitHub 

Slide 83

Slide 83 text

UBUNTU PRECISE The Future of Puppet at GitHub 

Slide 84

Slide 84 text

github/puppet  The Future of Puppet at GitHub 

Slide 85

Slide 85 text

The Future of Puppet at GitHub RUBY 1.9.3 

Slide 86

Slide 86 text

The Future of Puppet at GitHub PUPPET 3.X 

Slide 87

Slide 87 text

The Future of Puppet at GitHub GPANEL: ENC 

Slide 88

Slide 88 text

The Future of Puppet at GitHub  lol censored

Slide 89

Slide 89 text

The Future of Puppet at GitHub MCOLLECTIVE 

Slide 90

Slide 90 text

The Future of Puppet at GitHub REPLACING PARTS OF GITHUB/SHELL 

Slide 91

Slide 91 text

The Future of Puppet at GitHub STRUCTURED DATA > SED/AWK 

Slide 92

Slide 92 text

The Future of Puppet at GitHub HIERA 

Slide 93

Slide 93 text

THANKS speakerdeck.com/wfarr/ puppet-at-github- puppetcamp-raleigh-2013