Puppet in the Trenches: Guerilla DevOps at Braintree

This talk will be a straight-dope report of our real-world experiences using Puppet to manage our infrastructure over the last 3 years. We run Puppet in masterless mode to manage the environments that support our payments processing platform [1].

Our domain provides especially stringent requirements when it comes to security and availability, and I'll be speaking about our practices around Puppet that support these requirements.

When you're in the trenches, it's not always pretty. I'll spend some time talking about the spectacular ways we've screwed up with Puppet over the past few years. I'll link each of those war stories with lessons we've learned about good Puppet practices. I'll provide a few examples of where we feel like applying these lessons have yielded some cool modules (which will hopefully be open sourced and pushed to the Forge by PuppetConf).

There will be no stunning conclusions. As I get up to the more recent lessons we've learned, I'll talk about a few of our unsolved problems, where we run up against the edges of what Puppet itself is meant for as a tool. I'll talk about unanswered questions around where truth should be managed, and outline the ideas we have about where we're headed. In the end, I'll be plugging for the Puppet community as a place for us to collaboratively solve the Hard Problems of managing infrastructure.

[1] https://www.braintreepayments.com/

Transcript

1. puppet in the trenches ™ guerrilla devops @
2. ™ mission objective
3. — $ 1. 2.
4. $ dead simple
5. ™
6. You ( )
7. Secure Storage Bank Integration Clean APIs
8. ™ modern tech killer support rock-solid platform
9. why puppet?
10. a small startup...
11. ...with a big job
12. ...with a big job
13. Use what you've got <3
14. Use what you've got
15. automated consistency version controlled layer(s) of abstraction puppet
16. why puppet? ^ masterless
17. master puppet puppet master puppet agent puppet agent puppet agent who am i? a proxy server! a db box! an app node!
18. master puppet less you two: mongodb cluster! APP node ASAP!
19. master puppet less fine-grained control parallelization distributed (no S.P.O.F.)
20. supply_drop pitluga/supply_drop
21. supply_drop $ mkdir infrastructure && cd infrastructure [infrastructure] $ gem install supply_drop Successfully installed capistrano-2.13.4 Successfully installed supply_drop-0.11.1 [infrastructure] $ capify . [done] capified! [infrastructure] $ vim config/deploy.rb let's get
22. # config/deploy.rb require 'rubygems' require 'supply_drop' def tasks_for_datacenter(datacenter, servers) task(datacenter) do role :server, *servers end servers.each do |server| task(server) { role :server, server } end end supply_drop let's use
23. # ...config/deploy.rb tasks_for_datacenter :qa, %w( app01.qa db01.qa ) tasks_for_datacenter :production, %w( app01.prod db01.prod ) supply_drop let's use
24. # site.pp node 'app01.qa' { package { 'python': ensure => installed } } supply_drop let's use
25. supply_drop let's use [infrastructure] $ cap app01.qa puppet:noop ... notice: /Stage[main]/Package[python]/ensure: current_value absent, should be present (noop) [infrastructure] $ cap app01.qa puppet:apply ... notice: /Stage[main]/Package[python]/ensure: ensure changed ‘purged’ to ‘present’
26. cap tasks parallel rsync collision detection supply_drop
27. fine grained control $ cap db01.qa puppet:noop # single boxes $ cap app{01..05}.qa puppet:noop # classes of boxes $ cap qa puppet:noop # full environment $ git branch --list * master # maps to QA environments * staging # maps to pre-prod environments * production # maps to production environments we always read the diff
28. parallelization anybody using qa? working on a nagios check i'm tweaking the postgres configs, but we should be fine $ cap monitor01.qa puppet:noop $ cap monitor01.qa puppet:apply $ git pull --rebase $ git commit $ cap db01.qa puppet:noop $ cap db01.qa puppet:apply $ git pull --rebase $ git commit
29. parallelization Squad! Prepping a merge to staging! $ git co master && git pull $ git co staging && git pull $ git merge master $ git log origin/staging..HEAD $ git diff origin/staging..HEAD # review git log and diff $ cap staging puppet:noop # review puppet noop output $ cap db01.stg puppet:apply # selectively apply high-risk changes $ cap staging puppet:apply # flush out environment $ cap staging puppet:noop # double check "clean noop" $ git push roger! ten-four!
30. distributed by default well... thanks git!
31. putting it together
32. . ├── config # capistrano tasks & supply_drop config ├── datacenters # variable farms inherited by all nodes │ ├── qa.pp │ ├── staging.pp │ └── production.pp ├── manifests # legacy manifests; here be dragons ├── modules # new-style modules ├── nodes # node definitions │ ├── qa │ │ ├── app.pp # divided into per-class files │ │ ├── db.pp │ │ └── proxy.pp │ ├── staging │ └── production ├── puppet.pp # top-level puppet file ├── templates # legacy templates; here too be dragons └── vendor # vendored puppet pushed to machines ├── facter-1.6.4 ├── puppet-2.6.13 └── supply_drop-0.10.2
33. lessons learned and unanswered questions
34. puppet weapon is a dangerous
35. <VirtualHost *:443> # apache2.conf.erb <% if rails_env == 'prod' -%> HostName www.example.com <% else -%> Hostname qa.example.com <% end -%> </VirtualHost> deceptively simple node "app1.qa" { $rails_env = "qa" include users include apache include rails } node "app1.prod" { $rails_env = "prod" include users include apache include rails }
36. mind your abstractions node "app1.qa" { include apache apache::site { 'example': hostname => 'qa.example.com' } } node "app1.prod" { include apache apache::site { 'example': hostname => 'www.example.com' } }
37. mind your abstractions 0..1 or 0..n? ☑ generic or unique? ☑ what changes? ☑ required variables ☑ optional variables ☑ defaults ☑
38. mind your abstractions modules. use them.
39. mind your scope watch your six! location node class/type template/file
40. truth
41. does not manage truth puppet maps truth to complexity puppet
42. how should we manage truth?
43. puppet in the trenches ™ guerrilla devops @