Slide 1

Slide 1 text

Bootstrap the world, homey At first it was beautiful

Slide 2

Slide 2 text

Then the first development change “leaked” to production

Slide 3

Slide 3 text

The setup

Slide 4

Slide 4 text

The setup • 5 Staging environments (1 per team, 1 pre-prod)

Slide 5

Slide 5 text

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members

Slide 6

Slide 6 text

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes

Slide 7

Slide 7 text

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef

Slide 8

Slide 8 text

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud”

Slide 9

Slide 9 text

The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud” • Hosted chef

Slide 10

Slide 10 text

$ /bin/knife kill me --now

Slide 11

Slide 11 text

The AGONY • Managing multiple versions of a cookbook for different developers each environment • Wanting to make and test small changes on different environments • Lack of visibility when deploying changes • Impossible to correlate CHANGES (git) with STATE (chef-server)

Slide 12

Slide 12 text

We want to work like everyone else works

Slide 13

Slide 13 text

$ git checkout -b new_feature # do some work $ git commit -am “Did dat” $ git push -u # pushed to quirkey/new_feature # Pull request/code review $ pp rails deploy earth quirkey/new_feature # Deploys to staging # Sends alert via email + campfire # runs deploy in viewable Jenkins environment

Slide 14

Slide 14 text

But it’s how we work Not just cause it’s pretty

Slide 15

Slide 15 text

Make it work. Start with the API

Slide 16

Slide 16 text

$ pp chef deploy earth johnnydtan/graphite_cluster # git merging # cookbook testing # cookbook uploading # environment munging

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

checkout the staging branch (earth)

Slide 19

Slide 19 text

checkout the staging branch (earth) git merge the deploying branch(es)

Slide 20

Slide 20 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging

Slide 21

Slide 21 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server

Slide 22

Slide 22 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json

Slide 23

Slide 23 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env

Slide 24

Slide 24 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins

Slide 25

Slide 25 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file

Slide 26

Slide 26 text

checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file sends notifications to campfire/email

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

The Minor Discomforts

Slide 29

Slide 29 text

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata)

Slide 30

Slide 30 text

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless

Slide 31

Slide 31 text

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well

Slide 32

Slide 32 text

The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well • Chef server consistency is problematic

Slide 33

Slide 33 text

To improve • More code review as part of the process • Simple roles that map to recipes so that roles can be managed/tested at the env level • Using secondary store to keep track of versions (Zookeeper) • Chef versions could be tied to SHA instead of SemVer # (??)

Slide 34

Slide 34 text

github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS!