Policyfiles: Why, What, and How

https://www.chef.io/blog/event/webinar-chef-policies-why-what-and-how/

Transcript

1. Policyfiles: Why, What, How 1
2. Why • Roles, Environments, dependencies • A tale of two workflows   Monorepo   The Berkshelf Way • Workflow RFC 2
3. Implicit Dependencies Are Confusing • In the beginning we had no run lists • Nodes had recipes • Recipes included other recipes • No way to depend on another cookbook 3
4. Everyone loves a depsolver • Multiple solutions came out of the community • No cohesive workflow • Different tools • Different *files in cookbooks • Lack of a consistent supported interface 4
5. Roles Are Global Named State • Make a change across every node • Even nodes you didn't want to change • Not just run lists, but attributes too • "Just version roles" 5
6. Environments Are A Crutch • Version pinning is important • Environments can still get inconsistently applied • They weren't a complete solution for everyone 6
7. We won't mention the attributes hierarchy • Roles and Environments brought unprecedented complexity to attributes 7
8. Workflow support • Local users with a Chef Server • Local-mode (neé, Solo) users • CI/CD Pipelines 8
9. Monorepo vs Separate repos • Everyone started with a monorepo • The Berkshelf Way introduced using separate repos, per cookbook git  clone  git://github.com/chef/chef-­‐repo.git   9
10. Many workflow solutions isn’t awesome • The Berkshelf Way was polarizing • Having an opinion is important • Having a client-side dependency solver is super important 10
11. We Knew We Needed Tools • The workflow RFC • ChefDK is the place where the workflow tools are built • ChefDK isn't the only place where we need to have support, though   (chef, chef-zero, chef-server, chef-provisioning, test-kitchen, third party tools like ChefSpec) 11
12. What • What are policyfiles? • What are Policies? • What does it mean to have a policy? 12
13. Policyfiles • Policyfiles are where users define node(s) policy • Policyfiles are a Ruby DSL • Generate a lock file and cache cookbooks with install or update • Upload the policy and the cookbooks with push • Export the policy and cookbooks to a directory 13
14. Policy • Nodes apply a policy • Named run list • Set of attributes 14
15. Fulfilling a node's run list: The Eternal Struggle™ • Get the cookbooks required • Get the correct version of the cookbooks required • Get the cookbooks to a place the node can download them   Account for chef-solo and local-mode   Account for chef-server 15
16. Caching cookbooks • Each cookbook is a unique artifact • Every change should be tracked in version control • Local development testing can be a hassle • We cache every change to a cookbook • The lock file specifies the shasum to use for upload • This gaurantees consistency in policies being applied • Cookbooks missing from the cache are re-retrieved 16
17. Policies have a "lock file" • Checksum (SHA1) based on cookbook content • Any cookbook change == the cookbook and policy revisions change • This prevents dangerous behavior, like uploading the same version of a cookbook • Deploy across SDLC with confidence 17
18. Example - cookbook-dir.rb name  'cookbook-­‐dir'   default_source  :chef_repo,  File.dirname(__FILE__)     run_list  'example'   18
19. Example - chef install %  chef  install  cookbook-­‐dir.rb   Installing  cookbooks  from  lock   Installing  example  1.0.0   19
20. Example - cookbook-dir.lock.json {    "revision_id":   "5228e045bfc3d258a0f589e2796dc33a06a47db8120a050e4191573e521e43a5",    "name":  "cookbook-­‐dir",    "run_list":  [        "recipe[example::default]"    ],    "cookbook_locks":  {        "example":  {            "version":  "1.0.0",            "identifier":  "0061b3e1897fb689d7b45c1c56d0a7243bace1ad",            "dotted_decimal_identifier":   "107425210924982.38799241448806096.183774061846957",   ....  SNIPPED   20
21. Example - change default recipe diff  -­‐-­‐git  a/cookbooks/example/recipes/default.rb  b/cookbooks/example/recipes/default.rb   index  61bfd5d..56702da  100644   -­‐-­‐-­‐  a/cookbooks/example/recipes/default.rb   +++  b/cookbooks/example/recipes/default.rb   @@  -­‐1,8  +1,3  @@   -­‐#  This  is  a  Chef  recipe  file.  It  can  be  used  to  specify  resources  which  will   -­‐#  apply  configuration  to  a  server.   -­‐    log  "Welcome  to  Chef,  #{node["example"]["name"]}!"  do        level  :info    end   -­‐   -­‐#  For  more  information,  see  the  documentation:  http://docs.getchef.com/ essentials_cookbook_recipes.html   21
22. Example - chef update %  chef  update  cookbook-­‐dir.rb   Building  policy  cookbook-­‐dir   Expanded  run  list:  recipe[example]   Caching  Cookbooks...   Installing  example  1.0.0     Lockfile  written  to  /Users/jtimberman/ Development/sandbox/policyfile-­‐demo/cookbook-­‐ dir.lock.json   22
23. Example - cookbook-dir.lock.json diff diff  -­‐-­‐git  a/cookbook-­‐dir.lock.json  b/cookbook-­‐dir.lock.json   index  fb7187e..44e8029  100644   -­‐-­‐-­‐  a/cookbook-­‐dir.lock.json   +++  b/cookbook-­‐dir.lock.json   @@  -­‐1,5  +1,5  @@    {   -­‐    "revision_id":  "5228e045bfc3d258a0f589e2796dc33a06a47db8120a050e4191573e521e43a5",   +    "revision_id":  "22e57345ff686babc582b1a1956424729b22fe7f4a9c24b8375d009b0cc818ac",        "name":  "cookbook-­‐dir",        "run_list":  [            "recipe[example::default]"   @@  -­‐7,15  +7,15  @@        "cookbook_locks":  {            "example":  {                "version":  "1.0.0",   -­‐            "identifier":  "0061b3e1897fb689d7b45c1c56d0a7243bace1ad",   -­‐            "dotted_decimal_identifier":  "107425210924982.38799241448806096.183774061846957",   +            "identifier":  "7c44db3777a0ed08267df19a97c5f00aaf5f5150",   +            "dotted_decimal_identifier":  "34978605431234797.2294122179893189.263928682598736",   ...  SNIPPED   23
24. Or, chef diff! %  chef  diff  -­‐-­‐head  cookbook-­‐dir.rb     Policy  lock  'cookbook-­‐dir'  differs  between  'git:HEAD'  and   'local:cookbook-­‐dir.lock.json':     REVISION  ID  CHANGED   ===================     @@  -­‐1,2  +1,2  @@   -­‐5228e045bfc3d258a0f589e2796dc33a06a47db8120a050e4191573e521e43a5   +22e57345ff686babc582b1a1956424729b22fe7f4a9c24b8375d009b0cc818ac     ...  SNIPPED   24
25. The lock file... • Changed based on the content change in the cookbook • This makes it a "new version" • This doesn't absolve users from managing versions in metadata.rb! • Still follow best practices for releasing artifacts - policies aren't applicable to Supermarket or SCM repos 25
26. Policies Are a Single Configuration Unit • Store the run list • Store the attributes used in that run list • Store the specific version of all cookbook content • Bound to the node • The node is still the authority about itself 26
27. How • Create a Policyfile.rb • Add the run list • Define the default source (e.g., supermarket) • Modify any attributes (at the same priority as "role" attributes) • Specify any source options for cookbooks (local path or git sources, specific versions) 27
28. Chef Requirements • ChefDK 0.8.0+ (workstation) • Chef 12.4.3+ (nodes) • Chef Server 12.2.2+   (or Hosted Chef) 28
29. Create a repository %  chef  generate  repo  policyfile-­‐demo   %  cd  policyfile-­‐demo   %  git  init   %  git  add  .   %  git  commit  -­‐m  'Gotta  start  somewhere.'   29
30. Create a policyfile.rb %  chef  generate  policyfile   %  cat  Policyfile.rb   name  "example_application"     default_source  :community     run_list  "cookbook::recipe",  "other_cookbook::recipe"   30
31. Change the name and run list name  'policyfile-­‐demo'     default_source  :supermarket     run_list  'vim',  'zsh'   31
32. Install the policy %  chef  install   Building  policy  policyfile-­‐demo   Expanded  run  list:  recipe[vim],  recipe[zsh]   Caching  Cookbooks...   Using            vim  1.1.4   Using            zsh  1.0.3     Lockfile  written  to  /Users/jtimberman/Development/ sandbox/policyfile-­‐demo/Policyfile.lock.json   32
33. Install the policy • Generates Policyfile.lock.json • Caches the cookbooks locally in ~/.chefdk/cache/cookbooks 33
34. Push the Policy %  chef  push  demo   Uploading  policy  to  policy  group  demo   Uploaded  vim  1.1.4  (1c7a1cf5)   Uploaded  zsh  1.0.3  (cb687099)   34
35. What does that do? %  knife  raw  /policies/policyfile-­‐demo   {      "revisions":  {          "54e091dd430d3f369108ad8bb91941a88e0b8843590ad983f0458dabdb5c707d":  {            }      }   }     %  grep  54e091dd430d3f369108ad8bb91941a88e0b8843590ad983f0458dabdb5c707d   Policyfile.lock.json      "revision_id":  "54e091dd430d3f369108ad8bb91941a88e0b8843590ad983f0458dabdb5c707d",   35
36. What does that do? %  knife  raw  /cookbook_artifacts/zsh   {      "zsh":  {          "url":  "https://api.opscode.com/organizations/joshtest/cookbook_artifacts/zsh",          "versions":  [              {                  "url":  "https://api.opscode.com/organizations/joshtest/cookbook_artifacts/zsh/ b512ef33af29b8d34ad7e4e9b6ad38d42dea4945",                  "identifier":  "b512ef33af29b8d34ad7e4e9b6ad38d42dea4945"              },              {                  "url":  "https://api.opscode.com/organizations/joshtest/cookbook_artifacts/zsh/ cb687099809a18f819439d358fea9a6bf2ceb129",                  "identifier":  "cb687099809a18f819439d358fea9a6bf2ceb129"              }          ]      }   }   36
37. From the Policyfile.lock.json...        "zsh":  {              "version":  "1.0.3",              "identifier":  "cb687099809a18f819439d358fea9a6bf2ceb129",              "dotted_decimal_identifier":   "57254253093231128.69833572415279082.169788425810217",              "cache_key":  "zsh-­‐1.0.3-­‐supermarket.chef.io",              "origin":  "https://supermarket.chef.io:443/api/v1/cookbooks/zsh/ versions/1.0.3/download",              "source_options":  {                  "artifactserver":  "https://supermarket.chef.io:443/api/v1/cookbooks/ zsh/versions/1.0.3/download",                  "version":  "1.0.3"              }          }   37
38. (Optional) Export the policy • To a directory or as a tarball • Use for chef-zero / chef-client local mode workflows 38
39. %  chef  export  chef-­‐repo   Exported  policy  'policyfile-­‐demo'  to  chef-­‐ repo   39 (Optional) Export the policy
40. What does that do? %  ls  chef-­‐repo   Policyfile.lock.json   cookbooks/   data_bags/   40
41. What does that do? %  ls  chef-­‐repo/cookbooks   vim-­‐8015564136978712.3094730730143422.401418 93797988/   zsh-­‐57254253093231128.69833572415279082.1697 88425810217/   41
42. % cat chef-repo/data_bags/policyfiles/policyfile-demo-local.json {      "id":  "policyfile-­‐demo-­‐local",      "name":  "data_bag_item_policyfiles_policyfile-­‐demo-­‐local",      "data_bag":  "policyfiles",      "raw_data":  {          "revision_id":   "54e091dd430d3f369108ad8bb91941a88e0b8843590ad983f0458dabdb5c707d",          "name":  "policyfile-­‐demo",          "run_list":  [              "recipe[vim::default]",              "recipe[zsh::default]"          ],          "cookbook_locks":  {   ...SNIPPED   42
43. Export to a tarball %  chef  export  -­‐a  chef-­‐repo   Exported  policy  'policyfile-­‐demo'  to  /Users/jtimberman/Development/sandbox/ policyfile-­‐demo/chef-­‐repo/policyfile-­‐ demo-­‐54e091dd430d3f369108ad8bb91941a88e0b8843590ad983f0458dabdb5c707d.tgz     %  tar  -­‐tvf  policyfile-­‐demo-­‐*.tgz   -­‐rw-­‐r-­‐-­‐r-­‐-­‐    0  501        20              1564  Sep  22  13:50  Policyfile.lock.json   drwxr-­‐xr-­‐x    0  501        20                    0  Sep  22  13:50  cookbooks   ...  SNIPPED   drwxr-­‐xr-­‐x    0  501        20                    0  Sep  22  13:50  data_bags   drwxr-­‐xr-­‐x    0  501        20                    0  Sep  22  13:50  data_bags/policyfiles   -­‐rw-­‐r-­‐-­‐r-­‐-­‐    0  501        20              1883  Sep  22  13:50  data_bags/policyfiles/ policyfile-­‐demo-­‐local.json   43
44. Configure the nodes • Configuration is required • With Chef 12.5 and Chef Server 12.3, Node objects can have policy properties   These replace run lists   These replace "environments" 44
45. /etc/chef/client.rb use_policyfile  true   policy_group  "#{policy_group}"   policy_name  "#{policy_name}"     #  For  example:   use_policyfile  true   policy_group  "dev"   policy_name  "policyfile-­‐demo"   45
46. Configuration for compatibility mode use_policyfile  true   policy_document_native_api  false   deployment_group  "#{policy_name}-­‐#{policy_group}"   46
47. Anatomy of a Policyfile • name • run_list • cookbook • default_source 47
48. name • "name" of the policy • Corresponds to "policy_name" in the configuration • Used with commands taking a POLICY_NAME • Combined with the policy_group when applying to nodes 48
49. run_list • Array of run list items • Currently only recipes are allowed - roles are not supported • Just like the run_list you know and love from roles and nodes • E.g., ['zsh'] or ['recipe[zsh]'] 49
50. cookbook • Add version constraints on cookbooks • Optionally add alternative locations such as local paths or git repositories • Specify git options like tags, ref, etc. 50
51. cookbook examples #  cookbook  version  constraint   cookbook  'zsh',  '=  1.0.2'   cookbook  'zsh',  '~>  1.0.0'     #  cookbook  from  a  local  path   cookbook  'zsh',  '/Users/jtimberman/dev/cookbooks/zsh'     #  cookbook  from  a  git  repository   cookbook  'zsh',  git:  'https://github.com/chef-­‐cookbooks/zsh'     #  cookbook  from  github  repository   cookbook  'zsh',  github:  'chef-­‐cookbooks/zsh'     #  cookbook  from  the  same  github  repository  by  a  release  tag   cookbook  'zsh',  github:  'chef-­‐cookbooks/zsh',  tag:  '1.0.2'   51
52. default_source • Default location where to retrieve cookbooks • Often used for Supermarket • Can be used for a Chef Repository (local path) location 52
53. default_source examples #  uses  supermarket.chef.io:   default_source  :supermarket     #  local/private  supermarket  server:   default_source  :supermarket,  'https://supermarket.example.com'     #  the  cookbooks  from  the  current  working  directory:   default_source  :chef_repo,  File.dirname(__FILE__)     #  cookbooks  from  another  filesystem  location:   default_source  :chef_repo,  "/home/src/cookbook-­‐repo"   53
54. Attributes • Policyfiles can specify attributes • These take the place of “role” attributes in the attribute matrix • Use the same syntax as cookbook attributes files 54
55. Attribute examples default['ntp']['servers']  =  ['0.us.pool.ntp.org']   default['webapp']['port']  =  80   55
56. Using Policyfiles with ChefSpec • https://github.com/sethvargo/chefspec/commit/ cd57e28fdbd59fc26962c0dd3b1809b8841312f3 • Unreleased at the time of this writing, otherwise may be in version 4.4.0.   https://github.com/sethvargo/chefspec/pull/634 • Use bundler to get the right version • Require the policyfile library in spec_helper 56
57. ChefSpec – your Gemfile source  'https://rubygems.org'     #  This  is  required  to  have  policyfile  support  for  ChefSpec   gem  'chefspec',      git:  'https://github.com/sethvargo/chefspec',      ref:  'cd57e28fdbd59fc26962c0dd3b1809b8841312f3'     gem  'chef-­‐dk',  ‘=>  0.7.0'   57
58. ChefSpec spec_helper #  in  spec/spec_helper.rb,  e.g.     require  'chefspec/policyfile'     58
59. Using Policyfiles with Test Kitchen • Use the policyfile_zero provisioner • Note that this uses chef-client with local mode (chef-zero) and compatibility mode 59
60. Test kitchen #  in  kitchen.yml:     provisioner:      name:  policyfile_zero   60
61. Using Policyfiles with Provisioning • “chef provision” command • Create a provision cookbook • Use ChefDK::ProvisioningData.context • Set convergence_options from the context in the machine options • Take the machine action from the context 61
62. Provisioning recipe example context  =  ChefDK::ProvisioningData.context     with_driver  'vagrant:~/.vagrant.d/boxes'  do    options  =  {          vagrant_options:  {              'vm.box'  =>  'opscode-­‐centos-­‐7.1',        },          convergence_options:  context.convergence_options      }        machine  context.node_name  do          machine_options  options          converge  true          action  context.action      end   end   62
63. chef provision %  chef  provision  dev  -­‐-­‐sync  -­‐n  demothing   63
64. chef provision populates client.rb %  cat  /etc/chef/client.rb   chef_server_url  "https://api.opscode.com/organizations/joshtest"   node_name  "demothing"   client_key  "/etc/chef/client.pem"   ssl_verify_mode  :verify_peer     #  Policyfile  Settings:   use_policyfile  true   policy_document_native_api  true     policy_group  "dev"   policy_name  "policyfile-­‐demo"   64
65. Migrating to Policyfiles • Use Chef Server 12.2.0   (or Hosted Chef :) ) • Use ChefDK 0.8.0   These versions ensure the latest in API support and tools 65
66. Migrating to Policyfiles • "cookbook" lines in Berksfiles likely copy/paste fine • "wrapper" cookbooks that only set attributes and include recipes are easy • "roles" and "role cookbooks" simply move the attributes and run list • We plan to create a "role to policyfile" conversion tool 66
67. Known Limitations • Chef zero requires compatibility mode • No support for roles • Must have certain versions of Chef, ChefDK, Chef Server • Not implemented for knife bootstrap • Cannot “nest” or “include” other policies • Rapidly working towards "1.0 Feature Complete" status 67
68. Recap: Benefits • Most akin to "versioned" roles, oft-requested feature • Version constraints • Attributes • Everything kept in lockstep throughout the lifecycle • Consistent workflow commands • Integration with chef provisioning in `chef provision` • Easier to reason about than roles, environments, berksfiles, role cookbooks, different “knife” upload commands, etc 68
69. • Chef (12.5.0): node properties as first class citizens, named run lists become a thing • ChefDK: cookbook generate policyfile instead of berksfile, and a policies directory for a repo • ChefSpec (4.4.0): Support for policyfiles • Chef Zero (unknown version): Policy APIs instead of compatibility mode • Chef Analytics, Chef Sync product integration 69 Coming soon - or may have arrived!
70. Questions? https://www.chef.io/blog/2015/08/18/policyfiles-a-guided-tour/ https://github.com/chef/chef-dk/blob/master/POLICYFILE_README.md 70