Puppet at GitHub: PuppetConf 2013

Slide 1

Slide 1 text

Puppet at GitHub

Slide 2

Slide 2 text

@wfarr GitHub Operations Known Aliases: King of Kebabs The Chairman Mr. Caremad !

Slide 3

Slide 3 text

The State of Puppet at GitHub "

Slide 4

Slide 4 text

" The State of Puppet at GitHub github/boxen

Slide 5

Slide 5 text

" The State of Puppet at GitHub ~1.5 years old

Slide 6

Slide 6 text

" The State of Puppet at GitHub open-sourced ~7 months ago

Slide 7

Slide 7 text

" The State of Puppet at GitHub ~240 open-source puppet modules

Slide 8

Slide 8 text

" The State of Puppet at GitHub puppet 3.latest

Slide 9

Slide 9 text

" The State of Puppet at GitHub supports ruby 1.8.7, 1.9.3, 2.0.0

Slide 10

Slide 10 text

" The State of Puppet at GitHub hiera

Slide 11

Slide 11 text

" The State of Puppet at GitHub OS X

Slide 12

Slide 12 text

" The State of Puppet at GitHub Linux support in-progress

Slide 13

Slide 13 text

" The State of Puppet at GitHub # tomorrow @ 2:20pm

Slide 14

Slide 14 text

" The State of Puppet at GitHub github/puppet

Slide 15

Slide 15 text

" The State of Puppet at GitHub $

Slide 16

Slide 16 text

" The State of Puppet at GitHub ~5 years old

Slide 17

Slide 17 text

" The State of Puppet at GitHub 0.24.x — 2.7.x

Slide 18

Slide 18 text

" The State of Puppet at GitHub 121 modules

Slide 19

Slide 19 text

" The State of Puppet at GitHub ~280k lines of code

Slide 20

Slide 20 text

" The State of Puppet at GitHub every single employee has commit access

Slide 21

Slide 21 text

" The State of Puppet at GitHub 15.5k commits to master past 12 months

Slide 22

Slide 22 text

" The State of Puppet at GitHub by 86 contributors

Slide 23

Slide 23 text

" The State of Puppet at GitHub % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

" The State of Puppet at GitHub 0 100 200 300 400 2012-08-25 2012-10-20 2012-12-15 2013-02-09 2013-04-06 2013-06-01 2013-07-27 commits on master per week, last 12 months

Slide 27

Slide 27 text

" The State of Puppet at GitHub 0 1000 2000 3000 4000 total commits by author past year

Slide 28

Slide 28 text

" The State of Puppet at GitHub 0 175 350 525 700 total commits by author past year, except ops

Slide 29

Slide 29 text

" The State of Puppet at GitHub 0 175 350 525 700 total commits by authors with >10 commits, past year, except ops

Slide 30

Slide 30 text

" The State of Puppet at GitHub single puppetmaster

Slide 31

Slide 31 text

" The State of Puppet at GitHub rubygems

Slide 32

Slide 32 text

" The State of Puppet at GitHub ruby 1.8.7

Slide 33

Slide 33 text

" The State of Puppet at GitHub unicorn

Slide 34

Slide 34 text

" The State of Puppet at GitHub puppet 2.7.latest

Slide 35

Slide 35 text

" The State of Puppet at GitHub ~600 nodes

Slide 36

Slide 36 text

" The State of Puppet at GitHub run hourly via crond

Slide 37

Slide 37 text

" The State of Puppet at GitHub puppetdb

Slide 38

Slide 38 text

" The State of Puppet at GitHub nagiosdb

Slide 39

Slide 39 text

" The State of Puppet at GitHub "yo puppetdb, gimme all your nagios::object::* resources so I can update this nagios conﬁg"

Slide 40

Slide 40 text

" The State of Puppet at GitHub ﬁltergendb

Slide 41

Slide 41 text

" The State of Puppet at GitHub "yo puppetdb, gimme all the filtergen::rule resources I would realize so I can update this filtergen config"

Slide 42

Slide 42 text

" The State of Puppet at GitHub ⚡ puppetdb ⚡

Slide 43

Slide 43 text

" The State of Puppet at GitHub as it turns out, an api call is faster than running puppet on a host

Slide 44

Slide 44 text

" The State of Puppet at GitHub gpanel

Slide 45

Slide 45 text

" The State of Puppet at GitHub "Imagine Puppet Dashboard meets Razor and went on a weekend trip to the beach with the Heroku API and drank epic amounts of blue drink"

Slide 46

Slide 46 text

" The State of Puppet at GitHub aka we reinvented our own, smaller version of Foreman

Slide 47

Slide 47 text

" The State of Puppet at GitHub inventory

Slide 48

Slide 48 text

" The State of Puppet at GitHub

Slide 49

Slide 49 text

" The State of Puppet at GitHub

Slide 50

Slide 50 text

" The State of Puppet at GitHub app conﬁguration

Slide 51

Slide 51 text

" The State of Puppet at GitHub

Slide 52

Slide 52 text

" The State of Puppet at GitHub versioning of conﬁguration values

Slide 53

Slide 53 text

" The State of Puppet at GitHub Create nil => 1 Update 1 => 2 Delete 2 => nil

Slide 54

Slide 54 text

" The State of Puppet at GitHub let's make credential rolling less awful

Slide 55

Slide 55 text

" The State of Puppet at GitHub provisioning

Slide 56

Slide 56 text

" The State of Puppet at GitHub provisioning is typically awful

Slide 57

Slide 57 text

" The State of Puppet at GitHub we sprinkled in some ChatOps

Slide 58

Slide 58 text

" The State of Puppet at GitHub

Slide 59

Slide 59 text

" The State of Puppet at GitHub a little bit later...

Slide 60

Slide 60 text

" The State of Puppet at GitHub

Slide 61

Slide 61 text

" The State of Puppet at GitHub enc

Slide 62

Slide 62 text

" The State of Puppet at GitHub we have the dumbest ENC out there

Slide 63

Slide 63 text

" The State of Puppet at GitHub ⋆

Slide 64

Slide 64 text

" The State of Puppet at GitHub # /usr/local/sbin/fetch_gpanel_enc \ fe1.rs.github.com --- parameters: gpanel_cabinet: D20-13 gpanel_enabled: true gpanel_monitored: false

Slide 65

Slide 65 text

" The State of Puppet at GitHub we never specify classes via the ENC

Slide 66

Slide 66 text

" The State of Puppet at GitHub any variables we pass through are preﬁxed with gpanel_

Slide 67

Slide 67 text

How GitHub writes Puppet (

Slide 68

Slide 68 text

( How GitHub Writes Puppet rodjek/puppet-lint

Slide 69

Slide 69 text

( How GitHub Writes Puppet

Slide 70

Slide 70 text

( How GitHub Writes Puppet if you aren't using puppet-lint to audit your puppet codebase, you are doing it wrong

Slide 71

Slide 71 text

( How GitHub Writes Puppet puppet-lint enforces the Puppet Labs style guide

Slide 72

Slide 72 text

( How GitHub Writes Puppet puppet-lint can even ﬁx a ton of linter errors for you

Slide 73

Slide 73 text

( How GitHub Writes Puppet put it in a pre-commit hook

Slide 74

Slide 74 text

( How GitHub Writes Puppet $ git commit -am "can't lint this" modules/github/manifests/role/redis.pp: syntax ok modules/github/manifests/role/redis.pp - WARNING: => is not properly aligned on line 118 1 errors found, aborting commit.

Slide 75

Slide 75 text

( How GitHub Writes Puppet and then buy rodjek a beer

Slide 76

Slide 76 text

( How GitHub Writes Puppet rodjek/rspec-puppet

Slide 77

Slide 77 text

( How GitHub Writes Puppet

Slide 78

Slide 78 text

( How GitHub Writes Puppet if you aren't writing tests for your puppet code before running it on a server, you are doing it wrong

Slide 79

Slide 79 text

( How GitHub Writes Puppet use whatever framework/library

Slide 80

Slide 80 text

( How GitHub Writes Puppet rspec-puppet just happens to be a pretty good one

Slide 81

Slide 81 text

( How GitHub Writes Puppet put it in a pre-commit hook

Slide 82

Slide 82 text

( How GitHub Writes Puppet $ git commit -am "tests dont pass but whatever lol" 1) Expected redis::server would include class "more_than_a_single_c_thread" 1 failures encountered, aborting commit.

Slide 83

Slide 83 text

( How GitHub Writes Puppet and then buy rodjek another beer

Slide 84

Slide 84 text

( How GitHub Writes Puppet node deﬁnitions

Slide 85

Slide 85 text

( How GitHub Writes Puppet we don't use an ENC to describe node classes

Slide 86

Slide 86 text

( How GitHub Writes Puppet node /^github-redis\d+/ { class { 'github::role::polling_redis': enabled => $::gpanel_enabled, environment => $::gpanel_environment, private_ipv4 => $::ipaddress, } }

Slide 87

Slide 87 text

( How GitHub Writes Puppet abstractions all the way down

Slide 88

Slide 88 text

( How GitHub Writes Puppet treat your site classes as cascades down to your dist classes

Slide 89

Slide 89 text

( How GitHub Writes Puppet class redis::server( # params ) { class { 'redis::config': ... } -> class { 'redis::package': ... } ~> class { 'redis::service': ... } }

Slide 90

Slide 90 text

( How GitHub Writes Puppet class github::redis( # params ) inherits github::defaults { $memory = $environment ? { 'stg' => '2G', default => $half_memory_gb } class { 'redis::server': ... } }

Slide 91

Slide 91 text

( How GitHub Writes Puppet class github::role::polling_redis( # params ) { class { 'github::redis': # overrides based on specific node } }

Slide 92

Slide 92 text

( How GitHub Writes Puppet augeas

Slide 93

Slide 93 text

( How GitHub Writes Puppet you know what's not awesome?

Slide 94

Slide 94 text

( How GitHub Writes Puppet an erb template that requires your class to take 52876423 parameters so you can conﬁgure every possible value in my.cnf

Slide 95

Slide 95 text

( How GitHub Writes Puppet augeas { 'my.cnf/performance': context => '/files/etc/mysql/my.cnf/mysqld', changes => [ # automatic dump/restore 'set innodb_auto_lru_dump 18000', # innodb "set innodb_buffer_pool_size ${innodb_buffer_pool_size}", 'set innodb_log_file_size 256M', 'set innodb_log_buffer_size 8M', 'set innodb_lazy_drop_table 1', ], require => Percona::Server[$::fqdn] }

Slide 96

Slide 96 text

( How GitHub Writes Puppet it can seem complex and scary

Slide 97

Slide 97 text

( How GitHub Writes Puppet it is an amazing tool

Slide 98

Slide 98 text

How GitHub ships Puppet )

Slide 99

Slide 99 text

) How GitHub Ships Puppet continuous integration

Slide 100

Slide 100 text

) How GitHub Ships Puppet commit gets pushed

Slide 101

Slide 101 text

) How GitHub Ships Puppet jenkins runs the test suite

Slide 102

Slide 102 text

) How GitHub Ships Puppet status gets posted back to GitHub.com

Slide 103

Slide 103 text

) How GitHub Ships Puppet tmm1/test-queue

Slide 104

Slide 104 text

) How GitHub Ships Puppet

Slide 105

Slide 105 text

) How GitHub Ships Puppet remembers how to better parallelize tests

Slide 106

Slide 106 text

) How GitHub Ships Puppet continuous deployment

Slide 107

Slide 107 text

) How GitHub Ships Puppet tests passed for commit, Hubot auto-deploys

Slide 108

Slide 108 text

) How GitHub Ships Puppet branch deploy everything

Slide 109

Slide 109 text

) How GitHub Ships Puppet a lot of shops have a few environments

Slide 110

Slide 110 text

) How GitHub Ships Puppet testing staging production

Slide 111

Slide 111 text

) How GitHub Ships Puppet currently we have 181 environments

Slide 112

Slide 112 text

) How GitHub Ships Puppet that's not the same as 181 nodes not running production

Slide 113

Slide 113 text

) How GitHub Ships Puppet we only have 5 of those

Slide 114

Slide 114 text

) How GitHub Ships Puppet Hubot automatically merges the master branch before allowing any branch to deploy

Slide 115

Slide 115 text

) How GitHub Ships Puppet ChatOps

Slide 116

Slide 116 text

) How GitHub Ships Puppet 202 employees 100% access to Puppet 100% trust

Slide 117

Slide 117 text

) How GitHub Ships Puppet the list of roles you'd never think would touch production puppet webkit developer core git developers core ruby developers svn developers search developer frontend designers windows developers mac developers support enterprise sales

Slide 118

Slide 118 text

) How GitHub Ships Puppet every puppet run happens in chat

Slide 119

Slide 119 text

) How GitHub Ships Puppet all puppet output goes to chat

Slide 120

Slide 120 text

) How GitHub Ships Puppet everyone can see everything

Slide 121

Slide 121 text

) How GitHub Ships Puppet everyone can do anything

Slide 122

Slide 122 text

) How GitHub Ships Puppet hands-on learning, by accident

Slide 123

Slide 123 text

) How GitHub Ships Puppet eliminate disruptive questions

Slide 124

Slide 124 text

The Future of Puppet at GitHub *

Slide 125

Slide 125 text

* The Future of Puppet at GitHub puppet 3.x

Slide 126

Slide 126 text

* The Future of Puppet at GitHub upgrading a large, old codebase from 2.7 to 3 is really painful

Slide 127

Slide 127 text

* The Future of Puppet at GitHub we've been talking about it for a year

Slide 128

Slide 128 text

* The Future of Puppet at GitHub we're still trying to push forward, but...

Slide 129

Slide 129 text

* The Future of Puppet at GitHub helping maintain security ﬁxes for 2.7

Slide 130

Slide 130 text

* The Future of Puppet at GitHub goal is to be on 3.2 by end of year

Slide 131

Slide 131 text

* The Future of Puppet at GitHub ruby 2.0

Slide 132

Slide 132 text

* The Future of Puppet at GitHub moar faster rubby

Slide 133

Slide 133 text

* The Future of Puppet at GitHub we get this for free when we move to 3

Slide 134

Slide 134 text

* The Future of Puppet at GitHub mcollective

Slide 135

Slide 135 text

* The Future of Puppet at GitHub aka the story of github/shell and the wonders of ssh in a for-loop

Slide 136

Slide 136 text

* The Future of Puppet at GitHub there are a lot of cool things about mcollective

Slide 137

Slide 137 text

* The Future of Puppet at GitHub I think it will be a part of our stack in the future

Slide 138

Slide 138 text

* The Future of Puppet at GitHub but you can still go pretty damn far with bash and ssh

Slide 139

Slide 139 text

* The Future of Puppet at GitHub even more puppetdb tooling

Slide 140

Slide 140 text

* The Future of Puppet at GitHub waiting for multiple runs to converge exported resources is painful