ChefConf 2016 - Build Cookbooks and the Delivery Service Canvas

So you have Chef Automate, now what? This talk is a practical introduction to build cookbooks and how to figure out what should run in each stage. Have you ever found yourself wondering: "What is a build cookbook?" "Where should I put performance tests?" "How does this work with Docker?" If you have, this talk is for you. Come join us as we plan out a few projects in Delivery. We will start by using the Service Delivery Canvas, a tool for thinking about where things go in the pipeline, to layout our project. From there, we will take what we learned from the canvas and walk through building out a few applications, including examples from Chef's own infrastructure. We will also demonstrate multiple language runtimes and methodologies. Come be part of delivering all the things!

Transcript

2. BUILD COOKBOOKS AND THE SERVICE DELIVERY CANVAS Christopher Webber Engineering Manager – CIA, Chef 13 Jul 16
3. WHO AM I? • The person that was behind the scenes when we Delivered Delivery with Delivery on stage at ChefConf 2015 • Author of 20+ build-cookbooks • Engineering Manager at Chef for the CIA Team
4. WHAT WE ARE GOING TO COVER • Prerequisites and Assumptions • Concepts and Vocabulary • Mechanics • The Service Delivery Canvas • Examples Using the Service Delivery Canvas • Implemented Code Samples
5. PREREQUISITES AND ASSUMPTIONS
6. DOING THE DEVOPS • App/Dev teams and Ops teams are able to use Git • Each group is willing to see the work the other is doing • Each group cares about the application in production • The source code for the application, the building of the application, and the deployment of the application can all live in harmony in the same repo
7. API • Infrastructure as code is possible • You can do most, if not all infrastructure things via an API • Ideal minimum API access: • Provision new VMs/Instances/Compute, including associated network connections • Make load balancer changes • Make DNS changes
8. CONCEPTS AND VOCABULARY
9. DELIVERING SERVICES • What is a service? • Artifact vs Code • An artifact that can be promoted and delivered
10. CONTINUOUS INTEGRATION/DELIVERY • Merge to master • Remember, we care about services, not code • Every Pull Request/Change Request should be shippable
11. USEFUL TOOLS • delivery-truck • Cookbook for use with cookbook and application workflows • Handles aspects of workflow • delivery-sugar • Useful helpers for use in build cookbooks
12. MECHANICS
13. PHASES AND STAGES APPROVE DELIVER Lint Syntax Unit Security Quality Publish Lint Syntax Unit Provision Deploy Smoke Functional Provision Deploy Smoke Functional Provision Deploy Smoke Functional Provision Deploy Smoke Functional
14. PHASES AND STAGES • Each stage is made up of phases • You can skip phases, but not stages • Each phase is a collection of two independent Chef runs • The first, run as root, executes the default.rb recipe • The second, run as the dbuild user, executes the phase_name.rb
15. VERSIONS AND MISC • Version promotion • Create data bag item with version and artifact information needed for deploy • Use delivery-truck to move versions through the environments • Secrets Management • Still needs some work… • I use encrypted data bag items
16. BREAKING APART THE APP • Are all the pieces related to the app? • Is the database/load balancer/caching layer dedicated or shared? • Understanding dependencies • Is it build time or run time? • Can I test the dependency?
17. MECHANICS SUMMARY • Merge after Verify • Build from Master • Move Artifacts, not code, through Acceptance, Union, Rehearsal, and Delivered
18. THE SERVICE DELIVERY CANVAS
19. BASICS • Start with the artifact • It is a canvas, which means it is non-linear… fill it in, in the order that makes sense • Identify the Operational Concerns as you go • Identify the Dependencies as you go • Remember, anything you declare as a dependency, you are responsible for testing
20. SHOW THE CANVAS HERE Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish Unit Lint Syntax Functional Smoke Deploy Provision Dependencies
21. FOR REALS https://cwebber.net/delivery_canvas.pdf
22. SAMPLE RUBY APP • Sinatra • No Database • Runit, Unicorn, Nginx
23. DISCUSS PUBLISH Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish Unit Lint Syntax Functional Smoke Deploy Provision Dependencies
24. DISCUSS PROVISION Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit Lint Syntax Functional Smoke Deploy Provision Dependencies
25. DISCUSS DEPLOY Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit Lint Syntax Functional Smoke Deploy Provision • EC2 Instances • Elastic Load Balancer Dependencies
26. DISCUSS SMOKE Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit Lint Syntax Functional Smoke Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
27. DISCUSS FUNCTIONAL Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit Lint Syntax Functional Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
28. DISCUSS UNIT Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit Lint Syntax Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
29. DISCUSS LINT Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit • minitest • chefspec Lint Syntax Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
30. DISCUSS SYNTAX Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit • minitest • chefspec Lint • rubocop • foodcritic Syntax Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
31. DISCUSS SECURITY Operational Concerns Pre-Artifact Post-Artifact Security Quality Publish • Cookbook • Tar Ball Unit • minitest • chefspec Lint • rubocop • foodcritic Syntax • ruby -c Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
32. DISCUSS QUALITY Operational Concerns Pre-Artifact Post-Artifact Security • bundler-audit Quality Publish • Cookbook • Tar Ball Unit • minitest • chefspec Lint • rubocop • foodcritic Syntax • ruby -c Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies
33. MORE COMPLEX EXAMPLE • Ruby on Rails App • Dedicated Sidekiq Instance • Redis • Postgres Database • CDN via Fastly
34. RAILS APP Operational Concerns Pre-Artifact Post-Artifact Security • bundler-audit • brakeman Quality • simplecov Publish • Cookbook • Tar Ball Unit • rspec • chefspec Lint • rubocop • foodcritic Syntax • ruby -c Functional • Selenium Smoke • Check for 200 from / Deploy • Run Migrations • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer • Elasticache Redis • RDS Postgres • CDN Setup Dependencies
35. HARDWARE • Simple PHP App • MySQL • Memcache • F5 for load balancer • VMware for compute • Shared storage via EMC Isilon
36. HARDWARE PHP APP Operational Concerns • F5 Cluster • VMware Cluster • Infoblox Cluster • Zones, etc Pre-Artifact Post-Artifact Security Quality • phpunit coverage tool Publish • Cookbook • Tar Ball Unit • phpunit • chefspec Lint • rubocop • foodcritic Syntax • php -l • ruby -c Functional • Selenium Smoke • Check for 200 from / Deploy • Chef Run MySQL • Chef Run Memcache • Chef Run PHP instances • DNS Provision • VMware Instances • F5 VIP Dependencies
37. IMPLEMENTED CODE SAMPLES
38. CHEF-WEB-PURCHASE Sinatra app that handles purchases of one-off services
40. CHEF-WEB-PURCHASE CANVAS Operational Concerns Pre-Artifact Post-Artifact Security • bundler-audit Quality • Skipped Publish • Cookbook • Tar Ball Unit • minitest • chefspec Lint • rubocop • foodcritic Syntax • ruby -c Functional • Skipped Smoke • Check for 200 from /architect_days/ Deploy • Chef Run Nodes to upgrade app • Setup DNS Provision • EC2 Instances • Elastic Load Balancer Dependencies • Salesforce • Avalara • Stripe
41. DEFAULT.RB include_recipe 'delivery-truck::default’ include_recipe 'cia_infra::aws_prereq’ include_recipe 'cia_infra::ruby’
42. UNIT.RB include_recipe 'delivery-truck::unit’ include_recipe 'cia_infra::bundler_install_deps’ link File.join(node['delivery']['workspace']['repo'], '.env') do to File.join(node['delivery']['workspace']['repo'], '.env-example') end cia_delivery_bundle_exec 'rake test’
43. LINT.RB include_recipe 'delivery-truck::lint’ include_recipe 'cia_infra::bundler_install_deps’ cia_delivery_bundle_exec 'rubocop -D’
44. SYNTAX.RB include_recipe 'delivery-truck::syntax’ ruby_files = Dir["#{node['delivery']['workspace']['repo']}/**/*.rb"] \ - Dir["#{node['delivery']['workspace']['repo']}/vendor/**/*.rb"] ruby_files.each do |file| execute "ruby -c #{file}" do environment( 'PATH' => '/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games’ ) end end
45. SECURITY.RB include_recipe 'cia_infra::bundler_install_deps’ cia_delivery_bundle_exec 'bundle-audit update’ cia_delivery_bundle_exec 'bundle-audit check’
46. PROVISION.RB include_recipe 'delivery-truck::provision’ cia_infra_cluster node['delivery']['change']['project'] do host_name 'purchase’ size 2 end
47. DEPLOY.RB cia_infra_cluster node['delivery']['change']['project'] do action :deploy host_name 'purchase’ size 2 end
48. SMOKE.RB cia_delivery_http_status 'purchase' do path '/architect_days/’ end
49. USEFUL PATHS AND FILES TO KNOW ABOUT • cookbooks/ • Any cookbooks in this directory will be run through the delivery-truck process • .delivery/config.json • The file that we actually use to configure our project • Allows us to skip phases • Allows us to specify the search that we use to pick build nodes • .delivery/build-cookbook • The “standard” location for the build-cookbook
50. OTHER THOUGHTS • Take advantage of custom resources • Helper methods are awesome • Can be placed in libraries directory • Put shared things in shared cookbooks