Paperless Ops Chef Workflow

F04bfa14141dca6713f0d9caa763e26b?s=47 Aaron Quint
January 29, 2013
150

Paperless Ops Chef Workflow

A short talk given at the January 2013 NYC Chef meetup at Paperless Post HQ. This is a brief overview of how the Paperless Post ops team has formulated it's chef worfklow and the good and bad parts of it.

F04bfa14141dca6713f0d9caa763e26b?s=128

Aaron Quint

January 29, 2013
Tweet

Transcript

  1. Bootstrap the world, homey At first it was beautiful

  2. Then the first development change “leaked” to production

  3. The setup

  4. The setup • 5 Staging environments (1 per team, 1

    pre-prod)
  5. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members
  6. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes
  7. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef
  8. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud”
  9. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud” • Hosted chef
  10. $ /bin/knife kill me --now

  11. The AGONY • Managing multiple versions of a cookbook for

    different developers each environment • Wanting to make and test small changes on different environments • Lack of visibility when deploying changes • Impossible to correlate CHANGES (git) with STATE (chef-server)
  12. We want to work like everyone else works

  13. $ git checkout -b new_feature # do some work $

    git commit -am “Did dat” $ git push -u # pushed to quirkey/new_feature # Pull request/code review $ pp rails deploy earth quirkey/new_feature # Deploys to staging # Sends alert via email + campfire # runs deploy in viewable Jenkins environment
  14. But it’s how we work Not just cause it’s pretty

  15. Make it work. Start with the API

  16. $ pp chef deploy earth johnnydtan/graphite_cluster # git merging #

    cookbook testing # cookbook uploading # environment munging
  17. None
  18. checkout the staging branch (earth)

  19. checkout the staging branch (earth) git merge the deploying branch(es)

  20. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging
  21. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server
  22. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json
  23. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env
  24. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins
  25. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file
  26. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file sends notifications to campfire/email
  27. None
  28. The Minor Discomforts

  29. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata)
  30. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless
  31. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well
  32. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well • Chef server consistency is problematic
  33. To improve • More code review as part of the

    process • Simple roles that map to recipes so that roles can be managed/tested at the env level • Using secondary store to keep track of versions (Zookeeper) • Chef versions could be tied to SHA instead of SemVer # (??)
  34. github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS!