Puppet at GitHub: PuppetConf 2013

Puppet at GitHub

View Slide

@wfarr
GitHub
Operations
Known Aliases:
King of Kebabs
The Chairman
Mr. Caremad
!

View Slide

The State of Puppet at GitHub
"

View Slide

" The State of Puppet at GitHub
github/boxen

View Slide

~1.5 years old

View Slide

open-sourced ~7 months ago

View Slide

~240 open-source puppet modules

View Slide

puppet 3.latest

View Slide

supports ruby 1.8.7, 1.9.3, 2.0.0

View Slide

hiera

View Slide

OS X

View Slide

Linux support in-progress

View Slide

#
tomorrow @ 2:20pm

View Slide

github/puppet

View Slide

$

View Slide

~5 years old

View Slide

0.24.x — 2.7.x

View Slide

121 modules

View Slide

~280k lines of code

View Slide

every single employee has commit access

View Slide

15.5k commits to master past 12 months

View Slide

by 86 contributors

View Slide

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %

View Slide

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %
42.5%

View Slide

% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% % % % % % % % % % % % % % % % % % % %
% %
now with ops taken out

View Slide

0
100
200
300
400
2012-08-25
2012-10-20
2012-12-15
2013-02-09
2013-04-06
2013-06-01
2013-07-27
commits on master per week, last 12 months

View Slide

0
1000
2000
3000
4000
total commits by author past year

View Slide

0
175
350
525
700
total commits by author past year, except ops

View Slide

0
175
350
525
700
total commits by authors with >10 commits, past year, except ops

View Slide

single puppetmaster

View Slide

rubygems

View Slide

ruby 1.8.7

View Slide

unicorn

View Slide

puppet 2.7.latest

View Slide

~600 nodes

View Slide

run hourly via crond

View Slide

puppetdb

View Slide

nagiosdb

View Slide

"yo puppetdb, gimme all your
nagios::object::* resources so I can
update this nagios conﬁg"

View Slide

ﬁltergendb

View Slide

"yo puppetdb, gimme all the
filtergen::rule resources I would realize
so I can update this filtergen config"

View Slide

⚡ puppetdb ⚡

View Slide

as it turns out, an api call is faster
than running puppet on a host

View Slide

gpanel

View Slide

"Imagine Puppet Dashboard meets
Razor and went on a weekend trip to the
beach with the Heroku API and drank
epic amounts of blue drink"

View Slide

aka we reinvented our own, smaller
version of Foreman

View Slide

inventory

View Slide


View Slide

app conﬁguration

View Slide


View Slide

versioning of conﬁguration values

View Slide

Create nil => 1
Update 1 => 2
Delete 2 => nil

View Slide

let's make credential rolling less awful

View Slide

provisioning

View Slide

provisioning is typically awful

View Slide

we sprinkled in some ChatOps

View Slide


View Slide

a little bit later...

View Slide


View Slide

enc

View Slide

we have the dumbest ENC out there

View Slide

⋆

View Slide

# /usr/local/sbin/fetch_gpanel_enc \
fe1.rs.github.com
---
parameters:
gpanel_cabinet: D20-13
gpanel_enabled: true
gpanel_monitored: false

View Slide

we never specify classes via the ENC

View Slide

any variables we pass through
are preﬁxed with gpanel_

View Slide

How GitHub writes Puppet
(

View Slide

( How GitHub Writes Puppet
rodjek/puppet-lint

View Slide


View Slide

if you aren't using puppet-lint
to audit your puppet codebase,
you are doing it wrong

View Slide

puppet-lint enforces the
Puppet Labs style guide

View Slide

puppet-lint can even ﬁx a ton
of linter errors for you

View Slide

put it in a pre-commit hook

View Slide

$ git commit -am "can't lint this"
modules/github/manifests/role/redis.pp: syntax ok
modules/github/manifests/role/redis.pp - WARNING: => is not
properly aligned on line 118
1 errors found, aborting commit.

View Slide

and then buy rodjek a beer

View Slide

rodjek/rspec-puppet

View Slide


View Slide

if you aren't writing tests for
your puppet code before
running it on a server,
you are doing it wrong

View Slide

use whatever framework/library

View Slide

rspec-puppet just happens to be
a pretty good one

View Slide

put it in a pre-commit hook

View Slide

$ git commit -am "tests dont pass but whatever lol"
1) Expected redis::server would include class
"more_than_a_single_c_thread"
1 failures encountered, aborting commit.

View Slide

and then buy rodjek another beer

View Slide

node deﬁnitions

View Slide

we don't use an ENC to describe
node classes

View Slide

node /^github-redis\d+/ {
class { 'github::role::polling_redis':
enabled => $::gpanel_enabled,
environment => $::gpanel_environment,
private_ipv4 => $::ipaddress,
}
}

View Slide

abstractions all the way down

View Slide

treat your site classes as cascades
down to your dist classes

View Slide

class redis::server(
# params
) {
class { 'redis::config': ... }
-> class { 'redis::package': ... }
~> class { 'redis::service': ... }
}

View Slide

class github::redis(
# params
) inherits github::defaults {
$memory = $environment ? {
'stg' => '2G',
default => $half_memory_gb
}
class { 'redis::server': ... }
}

View Slide

class github::role::polling_redis(
# params
) {
class { 'github::redis':
# overrides based on specific node
}
}

View Slide

augeas

View Slide

you know what's not awesome?

View Slide

an erb template that requires your
class to take 52876423 parameters
so you can conﬁgure every possible
value in my.cnf

View Slide

augeas { 'my.cnf/performance':
context => '/files/etc/mysql/my.cnf/mysqld',
changes => [
# automatic dump/restore
'set innodb_auto_lru_dump 18000',
# innodb
"set innodb_buffer_pool_size ${innodb_buffer_pool_size}",
'set innodb_log_file_size 256M',
'set innodb_log_buffer_size 8M',
'set innodb_lazy_drop_table 1',
],
require => Percona::Server[$::fqdn]
}

View Slide

it can seem complex and scary

View Slide

it is an amazing tool

View Slide

How GitHub ships Puppet
)

View Slide

) How GitHub Ships Puppet
continuous integration

View Slide

commit gets pushed

View Slide

jenkins runs the test suite

View Slide

status gets posted back to GitHub.com

View Slide

tmm1/test-queue

View Slide


View Slide

remembers how to better parallelize tests

View Slide

continuous deployment

View Slide

tests passed for commit, Hubot auto-deploys

View Slide

branch deploy everything

View Slide

a lot of shops have a few environments

View Slide

testing
staging
production

View Slide

currently we have 181 environments

View Slide

that's not the same as 181 nodes
not running production

View Slide

we only have 5 of those

View Slide

Hubot automatically merges the
master branch before allowing any
branch to deploy

View Slide

ChatOps

View Slide

202 employees
100% access to Puppet
100% trust

View Slide

the list of roles you'd never think would touch production puppet
webkit developer
core git developers
core ruby developers
svn developers
search developer
frontend designers
windows developers
mac developers
support
enterprise sales

View Slide

every puppet run happens in chat

View Slide

all puppet output goes to chat

View Slide

everyone can see everything

View Slide

everyone can do anything

View Slide

hands-on learning, by accident

View Slide

eliminate disruptive questions

View Slide

The Future of Puppet at GitHub
*

View Slide

* The Future of Puppet at GitHub
puppet 3.x

View Slide

upgrading a large, old codebase
from 2.7 to 3 is really painful

View Slide

we've been talking about it for a year

View Slide

we're still trying to push forward, but...

View Slide

helping maintain security ﬁxes for 2.7

View Slide

goal is to be on 3.2 by end of year

View Slide

ruby 2.0

View Slide

moar faster rubby

View Slide

we get this for free when we move to 3

View Slide

mcollective

View Slide

aka the story of github/shell and the
wonders of ssh in a for-loop

View Slide

there are a lot of cool things about
mcollective

View Slide

I think it will be a part of our stack in
the future

View Slide

but you can still go pretty damn far
with bash and ssh

View Slide

even more puppetdb tooling

View Slide

waiting for multiple runs to converge
exported resources is painful

View Slide

use the puppetdb API to skip all that

View Slide

even more gpanel integration

View Slide

"databags"

View Slide

"node search"

View Slide

steal good ideas from other tools

View Slide

closing thoughts

View Slide

"all software is terrible"
— anyone who's worked with software long enough

View Slide

"no software is better than no software"
— rtomayko

View Slide

the plumbing doesn't matter when all you
care about is the porcelain

View Slide

write some damned good porcelain

View Slide

View Slide

THIS IS RODJEK

View Slide

HE'S NORMALLY
NOT THIS BLURRY

View Slide

HE IS ALSO NOT MY BROTHER,
BUT YOU REALLY SHOULD BUY
HIM A BEER

View Slide

YOU SHOULD BUY THIS
MAN A LOT OF BEER

View Slide

THANKS
https://speakerdeck.com/wfarr/
puppet-at-github-puppetconf-2013
!

View Slide

Puppet at GitHub: PuppetConf 2013

Puppet at GitHub: PuppetConf 2013

Will Farrington

More Decks by Will Farrington

Other Decks in Technology

Featured

Transcript