Bootstrap the world, homey At first it was beautiful

Then the first development change “leaked” to production

The setup

The setup • 5 Staging environments (1 per team, 1 pre-prod)

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud”

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud” • Hosted chef

$ /bin/knife kill me --now

The AGONY • Managing multiple versions of a cookbook for different developers each environment • Wanting to make and test small changes on different environments • Lack of visibility when deploying changes • Impossible to correlate CHANGES (git) with STATE (chef-server)

We want to work like everyone else works

$ git checkout -b new_feature # do some work $ git commit -am “Did dat” $ git push -u # pushed to quirkey/new_feature # Pull request/code review $ pp rails deploy earth quirkey/new_feature # Deploys to staging # Sends alert via email + campfire # runs deploy in viewable Jenkins environment

But it’s how we work Not just cause it’s pretty

Make it work. Start with the API

$ pp chef deploy earth johnnydtan/graphite_cluster # git merging # cookbook testing # cookbook uploading # environment munging

No content

checkout the staging branch (earth)

checkout the staging branch (earth) git merge the deploying branch(es)

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file sends notifications to campfire/email

No content

The Minor Discomforts

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata)

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well • Chef server consistency is problematic

To improve • More code review as part of the process • Simple roles that map to recipes so that roles can be managed/tested at the env level • Using secondary store to keep track of versions (Zookeeper) • Chef versions could be tied to SHA instead of SemVer # (??)

github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS!