Pipeline Cookbooks Meetup Talk Chef Seattle

Transcript

1. View Slide

2. Pipeline
   Automated delivery of cookbooks using Berkshelf
   and Jenkins
   

   View Slide

3. Stephen Lauck
   lauck@getchef.com
   @stephenlauck
   https://github.com/stephenlauck
   

   View Slide

4. Live Demo?
   

   View Slide

5. Why?
   Control the flow of cookbooks through your
   environments
   * Automate testing and delivery of cookbooks
   * Track all cookbooks for an org or enterprise
   * Uniform space to do integration testing
   * Manage dependency and versions of cookbooks for many environments using one codebase
   * Teams collaborate/integrate in source code only ie git
   * Developers work locally, no direct access to chef servers/orgs via knife
   

   View Slide

6. Automation of your workflow is a good thing.
   !
   • one way moving of code towards production
   • one place for integration
   • local development before integration
   

   View Slide

7. Wrapper Cookbook
   Wrap the pipeline cookbook with your settings
   

   View Slide

8. How it works
   !
   • 1. create berksfile with all the cookbooks
   • 2. pipeline job for the berksfile
   • 4. CD/CI job per cookbook uploads to artifact chef server
   • 3. promote jobs move cookbooks across environments
   

   View Slide

9. Berksfile site :opscode
   !
   group :community do
   # opscode community
   cookbook 'nginx', '= 2.0.8'
   cookbook 'runit', '= 1.4.0'
   cookbook 'yum', '= 2.4.4'
   cookbook 'git', '= 2.8.4'
   cookbook 'pipeline', git: 'git@github.com:stephenlauck/pipeline.git'
   end
   !
   group :sml do
   cookbook 'thresholder_pipeline', git: 'git@github.com:thresholderio/thresholder_pipeline.git'
   cookbook 'quasar_app', git: 'git@github.com:thresholderio/quasar_app.git'
   cookbook 'appliation_ruby', git: 'git@github.com:stephenlauck/application_ruby', branch:
   'fix_integration'
   end
   Example berksfile
   

   View Slide

10. Continuous Delivery
    reads a master berksfile
    creates jenkins job per cookbook
    watches for changes on cookbooks
    runs tests
    uploads cookbook to chef server/org artifact?
    !
    creates spiceweasel job
    watches for changes on yml for chef artifacts (data_bags, environments and roles)
    uploads artifacts
    

    View Slide

11. Berksfile Job
    sudo chef-client
    reads a master berksfile
    creates jenkins job per cookbook
    watches for changes on cookbooks
    runs tests
    uploads cookbook to chef server/org artifact?
    !
    creates spiceweasel job
    watches for changes on yml for chef artifacts (data_bags, environments and roles)
    uploads artifacts
    

    View Slide

12. Cookbook Job
    foodcritic -f correctness .
    if [ -f Berksfile.lock ];
    then
    berks update -c /var/lib/jenkins/.berkshelf/config.json
    else
    berks install -c /var/lib/jenkins/.berkshelf/config.json
    fi
    berks upload -c /var/lib/jenkins/.berkshelf/config.json
    reads a master berksfile
    creates jenkins job per cookbook
    watches for changes on cookbooks
    runs tests
    uploads cookbook to chef server/org artifact?
    !
    creates spiceweasel job
    watches for changes on yml for chef artifacts (data_bags, environments and roles)
    uploads artifacts
    

    View Slide

13. Deployment
    export PATH="/opt/chef/embedded/bin:$PATH"
    /opt/chef/embedded/bin/spiceweasel -e -c /var/lib/jenkins/.chef/%>.rb --novalidation
    if [ -f Berksfile.lock ];
    then
    /opt/chef/embedded/bin/berks update -c /var/lib/jenkins/.berkshelf/-
    config.json
    else
    /opt/chef/embedded/bin/berks install -c /var/lib/jenkins/.berkshelf/-
    config.json
    fi
    /opt/chef/embedded/bin/berks upload -c /var/lib/jenkins/.berkshelf/-config.json
    /opt/chef/embedded/bin/berks apply -c /var/lib/jenkins/.berkshelf/-
    config.json
    create promote jobs per environment AND/OR chef server/org
    uploads cookbooks/artifacts from one chef server/org to another
    pin versions in environments
    

    View Slide

14. Deployment
    spiceweasel upload
    berks install from artifact chef server
    berks upload to PROD chef server
    berks apply PROD
    create promote jobs per environment AND/OR chef server/org
    uploads cookbooks/artifacts from one chef server/org to another
    pin versions in environments
    

    View Slide

15. * file[/var/lib/jenkins/.chef/chef-zero-validator.pem] action create [2014-01-21T22:15:48+00:00] INFO: Processing file[/var/lib/
    jenkins/.chef/chef-zero-validator.pem] action create (pipeline::knife line 36)
    [2014-01-21T22:15:48+00:00] INFO: file[/var/lib/jenkins/.chef/chef-zero-validator.pem] created file /var/lib/jenkins/.chef/chef-zero-
    validator.pem
    - create new file /var/lib/jenkins/.chef/chef-zero-validator.pem
    [2014-01-21T22:15:48+00:00] INFO: file[/var/lib/jenkins/.chef/chef-zero-validator.pem] updated file contents /var/lib/jenkins/.chef/chef-
    zero-validator.pem
    - update content in file /var/lib/jenkins/.chef/chef-zero-validator.pem from none to 6fe8f1
    --- /var/lib/jenkins/.chef/chef-zero-validator.pem 2014-01-21 22:15:48.136061549 +0000
    +++ /tmp/.chef-zero-validator.pem20140121-1306-1s3u300 2014-01-21 22:15:48.136061549 +0000
    @@ -1 +1,28 @@
    +-----BEGIN RSA PRIVATE KEY-----
    +MIIEogIBAAKCAQEAyd4VInXBK+XlGFSkFv80Kgifr8/5LOi/HBF3VhLRxM/GkgnN
    +fds/lhzNq3NH3ZjDhMbS5vOisrVW00DgvlyzsFJAjVGKNEW+S9BqTiYAQzmpRp6Y
    +QRYAm5sEuhZgY5m7WWuFrW/zayR2wllanirBdIlfY9TnF+w1NecKAqbW4QH5XIWk
    +dri2aq/wW9to7X0mdotAhZxfZ5sRPoyN9Kno0qjqJ6+zzmRUOY76Rq3CD4FDYEMJ
    +v3hMT2yOzqavLq2rUaApENLrnKd9SJXFrnuhfhKGZ21hTVn4kkiD5BAHst6k8+m9
    +j3KcJXmHXYkQjNME8JZ3iwpBtN+xKyoknYGRNwIDAQABAoIBACFbz8ZIC0oDzZ39
    +rrgWKDqh/jGBfr3LIHm08TGKHpwVcc0ETa70okdeLyacAE5ARl8UtBlyPXqmuNhk
    +Kj+K9i63CO/Rf7Mvq0jAAjEz2mtBhhWjc6mdxy/vqBJQTFFpQCqAuDB3BZS5C98G
    +ARGOIzXs1ZSbxCyR3iEwMtlJVM0NyQj7XOEMvbPXll3ODYuhLIYCFzRYE9Doa2QX
    +XDwTTQ6e2NtEUI2l9PgV1l27oQxuKFQ9EiZdCj74BgsNuoUcB45ZvSsFWbvXJdhD
    +lCoO8oTG7cR/hrf/LhdyFck/AaX0zqIfkfN0YFhUmXkbR9pDgibNsRIoSrAcAE1c
    +f08/umECgYEA+DTqCdfAQzwGtFbXSeqGimVZwEKCvjPYnJLcJJAVNUzBCXW0TOF9
    ++DynHEU4o/FwRPfm2H9UC/ebzJnUAvsBMT4R86mlk2rmogeuXw1X5w6iIzLR/nPs
    Fail
    !
    keys, auth, credentials
    users deploy/jenkins/github
    cookbook dependency and berkshelf
    upstream cookbooks
    

    View Slide

16. Improve
    better ssh key management
    partials for job commands
    LWRP pipeline
    git tagging / metadata bump
    berkshelf DSL
    test before merge
    data bags for chef orgs
    documentation (derp)
    better ssh key management
    partials
    all providers
    tagging back
    berkshelf DSL
    test before merge
    data in data bags
    how to test the pipeline cookbook
    

    View Slide

17. Patterns
    wrapper cookbook
    berkshelf / spiceweasel manifest
    vagrant / test-kitchen
    cookbook / wrapper cookbook
    berkshelf / spiceweasel manifest
    vagrant / test-kitchen
    

    View Slide

18. Actual Code
    https://github.com/stephenlauck/pipeline
    https://github.com/stephenlauck/pipeline_chef
    https://github.com/stephenlauck/wrapper_pipeline
    

    View Slide

19. Questions?
    cookbook / wrapper cookbook
    berkshelf / spiceweasel manifest
    vagrant / test-kitchen
    

    View Slide