Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paperless Ops Chef Workflow

Aaron Quint
January 29, 2013
200

Paperless Ops Chef Workflow

A short talk given at the January 2013 NYC Chef meetup at Paperless Post HQ. This is a brief overview of how the Paperless Post ops team has formulated it's chef worfklow and the good and bad parts of it.

Aaron Quint

January 29, 2013
Tweet

Transcript

  1. Bootstrap the world, homey
    At first it was
    beautiful

    View full-size slide

  2. Then the first
    development change
    “leaked” to
    production

    View full-size slide

  3. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)

    View full-size slide

  4. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)
    • 4 Full time Ops team members

    View full-size slide

  5. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)
    • 4 Full time Ops team members
    • > 10 Developers who should be able to make
    changes

    View full-size slide

  6. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)
    • 4 Full time Ops team members
    • > 10 Developers who should be able to make
    changes
    • Many different applications and technologies
    being managed by chef

    View full-size slide

  7. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)
    • 4 Full time Ops team members
    • > 10 Developers who should be able to make
    changes
    • Many different applications and technologies
    being managed by chef
    • Private VMWare/VSphere backed “cloud”

    View full-size slide

  8. The setup
    • 5 Staging environments (1 per team, 1 pre-prod)
    • 4 Full time Ops team members
    • > 10 Developers who should be able to make
    changes
    • Many different applications and technologies
    being managed by chef
    • Private VMWare/VSphere backed “cloud”
    • Hosted chef

    View full-size slide

  9. $ /bin/knife kill me --now

    View full-size slide

  10. The AGONY
    • Managing multiple versions of a cookbook for
    different developers each environment
    • Wanting to make and test small changes on
    different environments
    • Lack of visibility when deploying changes
    • Impossible to correlate CHANGES (git) with STATE
    (chef-server)

    View full-size slide

  11. We want to work like
    everyone else works

    View full-size slide

  12. $ git checkout -b new_feature
    # do some work
    $ git commit -am “Did dat”
    $ git push -u
    # pushed to quirkey/new_feature
    # Pull request/code review
    $ pp rails deploy earth quirkey/new_feature
    # Deploys to staging
    # Sends alert via email + campfire
    # runs deploy in viewable Jenkins environment

    View full-size slide

  13. But it’s how we work
    Not just cause
    it’s pretty

    View full-size slide

  14. Make it work.
    Start with the API

    View full-size slide

  15. $ pp chef deploy earth johnnydtan/graphite_cluster
    # git merging
    # cookbook testing
    # cookbook uploading
    # environment munging

    View full-size slide

  16. checkout the staging branch (earth)

    View full-size slide

  17. checkout the staging branch (earth)
    git merge the deploying branch(es)

    View full-size slide

  18. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging

    View full-size slide

  19. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server

    View full-size slide

  20. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json

    View full-size slide

  21. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env

    View full-size slide

  22. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins

    View full-size slide

  23. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins
    jenkins uploads cookbooks and environment file

    View full-size slide

  24. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins
    jenkins uploads cookbooks and environment file
    sends notifications to campfire/email

    View full-size slide

  25. The Minor Discomforts

    View full-size slide

  26. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)

    View full-size slide

  27. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless

    View full-size slide

  28. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless
    • Doesn’t handle changes to roles (you cant scope a
    role to an environment) or new cookbooks very
    well

    View full-size slide

  29. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless
    • Doesn’t handle changes to roles (you cant scope a
    role to an environment) or new cookbooks very
    well
    • Chef server consistency is problematic

    View full-size slide

  30. To improve
    • More code review as part of the process
    • Simple roles that map to recipes so that
    roles can be managed/tested at the env
    level
    • Using secondary store to keep track of
    versions (Zookeeper)
    • Chef versions could be tied to SHA
    instead of SemVer # (??)

    View full-size slide

  31. github.com/quirkey
    twitter.com/aq
    quirkey.com
    paperlesspost.com
    THANKS!

    View full-size slide