So Continuous. Much Delivery. Very Chef. Wow.

An anonymous case-study of how I got to help an Org get started with Continuous Delivery using Chef

Transcript

1. So Continuous. Much Delivery. Very Chef. Wow. A Case Study on using Chef to start building a Continuous Delivery Pipeline
2. • George Miranda • Sr Consultant at Chef Software, Inc. • Unix guy (15+ years) About Me
6. Minimum Viable Pipeline
7. • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ What we know
8. • Must utilize existing tools within the company • Git/Github Enterprise for SCM • Jenkins approved for use • Working in a static VM environment • Just migrated to single cookbook repos • Starting with infrastructure cookbooks • Want a manual go-to-production button (ugh!) Case Study: Requirements
9. • Git PR model: branch from master for any new feature • 4-person team, only 3 active at any time • Code review done manually and informally • Simple communication/reqs (makes it easy!) Case Study: Code Review Model
10. • How are developers expected to work locally? • When do they push to remote? How do we verify their work? • Code Review criteria: what does it mean to be ready to merge? • How do we go from merged code to artifact? • How do we get that artifact all the way to Production? Figuring out new workflow
11. • New branch for every feature • Create a failing test • Write a resource to pass the test • Local commits • Test-Kitchen + guard • Once local tests passed, push to remote Local Development Work
12. • Open a Pull Request (new branch to master) • Triggers a build via Jenkins GHPRB plugin Push to remote
13. The Verify Build Job • Verify syntax (knife cookbook check) • Foodcritic Rules • Rubocop • Test-Kitchen w/ BATS busser Push to remote
14. @test "My directory is created" {! test -d /foo/bar! }! ! @test "A basharific test" {! if [ foo != bar ]; then! skip "foo isn't bar"! fi! ! run foo! [ "$status" -eq 0 ]! }! ! • https://github.com/sstephenson/bats • Super low learning curve (but also very limited) BATS: Simple Unit Tests
15. • If failed, notify • Another commit to the same branch triggers another Verify Build Job • Super easy to track, comment, and approve • If passed, let’s go to Human Code Review Push to remote
16. • Only one feature branch per one cookbook at one time • Must have test for feature that changed • One for one: resource unit tests • Consider the smoke test Human Code Review Rules
17. • Unit tests: small, fast, check one single concern • In this context: checking Chef resources • Smoke tests: test multiple things in the course of one concern • In this context: check the intent of a recipe • Note: that was testing for this use case Unit Test vs Smoke Test
18. • Only 3 active team members at any given time • Submitter cannot approve • Merge approval requires 2 approvals • Code review can happen at any time, but only merge when you’re ready to fix it. When are we ready to merge?
19. • Freeze your cookbooks! • Semantic versioning: Major.Minor.Patch • You own Major.Minor • The Pipeline owns .Patch • No one gets to knife upload No one.! Ever.! • "git merge" is the new "knife upload" Merged code to artifact
21. • Bumps Cookbook version • Re-commits to master • Upload frozen cookbook (via berks) • Pin that new cookbook to the Integration environment • Converge all nodes that use that cookbook The Integration Job
22. • First sign that things may be broken • These nodes also run smoke tests • serverspec, minitest, etc The Integration Job
23. • We survived! Trigger the next job(s) • The Jenkins Build Pipelines Plugin allows upstream/downstream definitions to string together jobs • From here out, it’s all the same Promote Job* • After the Integration job, we just run X number of Promote Jobs The Integration Job * (mostly)
27. • Pin cookbook to new Chef Environment • Converge all nodes using this cookbook • Run Tests Promote Jobs
28. #!/opt/chef/embedded/bin/ruby ! require 'chef/environment' require 'chef' Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb") ! def pin_env(env, cookbook_versions) to = Chef::Environment.load(env) cookbook_versions.each do |cb, version| puts "Pinning #{cb} #{version} in #{env}" to.cookbook_versions[cb] = version end to.save end ! cookbook_data = Array.new ! if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))) metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')) File.read(metadata_file).each_line do |line| if line =~ /^name\s+["'](\w+)["'].*$/ cookbook_data << $1 end if line =~ /^version\s+["'](\d+\.\d+\.\d+)["'].*$/ cookbook_data << "= #{$1}" end end end ! cookbook_versions = Hash[*cookbook_data] ! pin_env(ARGV[0], cookbook_versions) Pin the cookbook to Env
29. $ berks apply <environment> Pin the cookbook to Env
30. $ knife ssh "recipes:mycookbook AND chef_environment:promote-environment” 'sudo chef-client'! … OR … Pushy! Converge Nodes
31. • Most testing frameworks have a Report Handler to automatically run tests • chef-serverspec-handler • minitest-handler • Deploy to your nodes by adding ‘chef_handler’ to their run_list • Many community cookbooks are already packaged with tests Run Tests
32. • In this particular use case: • Build job: BATS (unit tests) • Integration & Promote jobs: serverspec (smoke tests) • UAT: also ran Cucumber tests (acceptance) Run Tests
34. • Can string together N number of promotions • UAT • Production A • Production B • etc Promoting to more environments
37. • In production monitoring is the test • Could not queue up changes reliably anyway • There is no spoon Push to Production
38. • Small incremental deployments led to greater confidence • TDD was pushed to the forefront of priorities • Commitment from Dev group to write application deployment cookbooks • But the biggest lesson learned… Results
39. • Continuous Delivery is a practice, not a tool • Small incremental changes in code • Small incremental changes in workflow • Small incremental changes in tooling • You will constantly improve your code, your workflow, your tools, your team, and your skills. Let’s Go Devop with a CD tool
40. RECAP
41. • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ What We Wanted
42. • (Pre-req) Test Driven Development • 2A. Establish development workflow before submitting changes * • 2B. Auto verification of submission before humans look at it • 2C. Humans Apply Code Review Criteria * • 2D. Don’t merge unless you mean it * • 2E. Merge kicks off an Integration Job • 2F. Followed by a series of Promotion Jobs • 2G. There is no spoon * Wait… what was Step 2?
43. • Step 1: Develop a new change • Step 2: ! ! • Step 3: Production! • Step 4: Level Up. This is great! • Step 5: MOAR THINGS! Wait. This is hard! • Go to Step 1 What We Got (Pre-req) Test Driven Development 2A. Establish development workflow before submitting changes * 2B. Auto verification of submission before humans look at it 2C. Humans Apply Code Review Criteria * 2D. Don’t merge unless you mean it * 2E. Merge kicks off an Integration Job 2F. Followed by a series of Promotion Jobs 2G. There is no spoon *
44. • Test Kitchen — http://kitchen.ci/ • Guard Plugin for Test Kitchen — https://github.com/test-kitchen/guard-kitchen • Foodcritic — http://acrmp.github.io/foodcritic/ • Berkshelf — http://berkshelf.com/ Key Chef Ecosystem Tools
45. • git • github • build-pipeline-plugin • ghprb • warnings • mailer Helpful Jenkins Plugins
46. I want to hear from you! ! @gmiranda23 gmiranda@getchef.com