Paperless Ops Chef Workflow

A short talk given at the January 2013 NYC Chef meetup at Paperless Post HQ. This is a brief overview of how the Paperless Post ops team has formulated it's chef worfklow and the good and bad parts of it.

Transcript

1. Bootstrap the world, homey
   At first it was
   beautiful
   

   View Slide

2. Then the first
   development change
   “leaked” to
   production
   

   View Slide

3. The setup
   

   View Slide

4. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   

   View Slide

5. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   • 4 Full time Ops team members
   

   View Slide

6. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   • 4 Full time Ops team members
   • > 10 Developers who should be able to make
   changes
   

   View Slide

7. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   • 4 Full time Ops team members
   • > 10 Developers who should be able to make
   changes
   • Many different applications and technologies
   being managed by chef
   

   View Slide

8. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   • 4 Full time Ops team members
   • > 10 Developers who should be able to make
   changes
   • Many different applications and technologies
   being managed by chef
   • Private VMWare/VSphere backed “cloud”
   

   View Slide

9. The setup
   • 5 Staging environments (1 per team, 1 pre-prod)
   • 4 Full time Ops team members
   • > 10 Developers who should be able to make
   changes
   • Many different applications and technologies
   being managed by chef
   • Private VMWare/VSphere backed “cloud”
   • Hosted chef
   

   View Slide

10. $ /bin/knife kill me --now
    

    View Slide

11. The AGONY
    • Managing multiple versions of a cookbook for
    different developers each environment
    • Wanting to make and test small changes on
    different environments
    • Lack of visibility when deploying changes
    • Impossible to correlate CHANGES (git) with STATE
    (chef-server)
    

    View Slide

12. We want to work like
    everyone else works
    

    View Slide

13. $ git checkout -b new_feature
    # do some work
    $ git commit -am “Did dat”
    $ git push -u
    # pushed to quirkey/new_feature
    # Pull request/code review
    $ pp rails deploy earth quirkey/new_feature
    # Deploys to staging
    # Sends alert via email + campfire
    # runs deploy in viewable Jenkins environment
    

    View Slide

14. But it’s how we work
    Not just cause
    it’s pretty
    

    View Slide

15. Make it work.
    Start with the API
    

    View Slide

16. $ pp chef deploy earth johnnydtan/graphite_cluster
    # git merging
    # cookbook testing
    # cookbook uploading
    # environment munging
    

    View Slide

17. View Slide

18. checkout the staging branch (earth)
    

    View Slide

19. checkout the staging branch (earth)
    git merge the deploying branch(es)
    

    View Slide

20. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    

    View Slide

21. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    

    View Slide

22. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    

    View Slide

23. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    

    View Slide

24. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins
    

    View Slide

25. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins
    jenkins uploads cookbooks and environment file
    

    View Slide

26. checkout the staging branch (earth)
    git merge the deploying branch(es)
    check for cookbooks changed from the last deploy to staging
    check latest versions of cookbooks in chef server
    bump version in metadata and environment.json
    commit and git push to env
    send deploy to jenkins
    jenkins uploads cookbooks and environment file
    sends notifications to campfire/email
    

    View Slide

27. View Slide

28. The Minor Discomforts
    

    View Slide

29. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    

    View Slide

30. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless
    

    View Slide

31. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless
    • Doesn’t handle changes to roles (you cant scope a
    role to an environment) or new cookbooks very
    well
    

    View Slide

32. The Minor Discomforts
    • Dealing with conflicts is hard (especially in
    metadata)
    • MANY MANY VERSIONS (0.3.192) aka version
    numbers are meaningless
    • Doesn’t handle changes to roles (you cant scope a
    role to an environment) or new cookbooks very
    well
    • Chef server consistency is problematic
    

    View Slide

33. To improve
    • More code review as part of the process
    • Simple roles that map to recipes so that
    roles can be managed/tested at the env
    level
    • Using secondary store to keep track of
    versions (Zookeeper)
    • Chef versions could be tied to SHA
    instead of SemVer # (??)
    

    View Slide

34. github.com/quirkey
    twitter.com/aq
    quirkey.com
    paperlesspost.com
    THANKS!
    

    View Slide