Chef Vault

Chef Vault

Slides for a talk about Chef Vault

4eb6098fc8de5a5f37199c3668b11590?s=128

Joshua Timberman

June 16, 2014
Tweet

Transcript

  1. Chef Vault Paul Collins https://flic.kr/p/dg1kZW Joshua Timberman Sr. Operations System

    Administrator ! joshua@getchef.com @jtimberman
  2. Managing Secrets Cedward Brice https://flic.kr/p/m6hFPH

  3. Managing Secrets is a Solved Problem? Daniel Fleming https://flic.kr/p/9boLH5

  4. None
  5. None
  6. Types of Secrets •SSL certificates •SSH keys •Passwords

  7. Ways of storing secrets… •Password managers (1password, lastpass, keepass) •GPG/PGP

    encrypted file •Firewall-protected databass •The Configuration Management Database
  8. None
  9. Chef Server An infrastructure publishing system with an API and

    search index
  10. Ways to Store Secrets in Chef Attributes Data Bag Items

    Encrypted Data Bag Items Chef Vault
  11. Secrets in Chef Attributes!

  12. Secrets in Chef: Attributes ! # In a recipe or

    cookbook default attributes: node.default['mysql']['root_password'] = 'This-Is-Secure-Right?' ! # In a role: { "name": "mysql-server", "default_attributes": { "mysql": { "root_password": "This-is-Secure-Right?" } } }
  13. Secrets in Chef: Attributes % knife search node "mysql*password:*" -a

    mysql 1 items found ! chef-vault-examples: mysql: root_password: This-Is-Secure-Right?
  14. Secure? NOPE. Michael Sarver https://flic.kr/p/6xHF1

  15. Problems with attributes •Plain text •Easy to search •Secrets aren't

    usually node specific
  16. Secrets in Chef Data Bag Items!

  17. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
  18. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
  19. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
  20. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
  21. Problems with Data Bags •Still plain text •Still easy to

    search •Can use RBAC... •But /search doesn't have RBAC :( •Using Hosted Chef?
  22. None
  23. None
  24. Chef Server RBAC •Enterprise Chef has RBAC •Remove read permissions

    from certain groups/users •Remove read permissions from certain API clients (nodes)
  25. This helps, but...

  26. Secrets in Chef ENCRYPTED Data Bag Items!

  27. Secrets in Chef: Encrypted Data Bags •Added in Chef 10

    •Data bag items encrypted at rest •Encrypted with a secret (phrase or file) •Hosted Chef is the primary use case
  28. I read your email...

  29. I read your email... W ho has tim e for

    that?!
  30. Encrypted Data Bags: Local files % cat secret-key -----BEGIN RSA

    PRIVATE KEY—— ... some stuff in a random key i made for examples -----END RSA PRIVATE KEY----- ! % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Now-we-re-getting-somewhere!" }
  31. Encrypted Data Bags: Knife % knife data bag from file

    -h knife data bag from file BAG FILE|FOLDER [FILE|FOLDER..] (options) -s, --secret The secret key to use to encrypt data bag item values --secret-file SECRET_FILE A file containing the secret key to use to encrypt data bag item values
  32. Encrypted Data Bags: Upload % knife data bag from file

    secrets \ data_bags/secrets/mysql_root.json --secret-file secret-key ! Updated data_bag_item[secrets::mysql_root]
  33. Encrypted Data Bags: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: {

    "id": "mysql_root", "password": { "encrypted_data": "aCztoQZuBvPQopBgxY32AjAVQ8QXxcaj0VkuIxO/ YZxkVNgClcJv3RpDvLFF\nZT6s\n", "iv": "nYTlXs9tCMOQw2ikgXCwrA==\n", "version": 1, "cipher": "aes-256-cbc" } }
  34. Encrypted Data Bags: Decrypt % knife data bag show secrets

    mysql_root \ --secret-file secret-key id: mysql_root password: Now-we-re-getting-somewhere!
  35. Encrypted Data Bags: Recipe root_pw = Chef::EncryptedDataBagItem.load('secrets', 'mysql_root')

  36. Wait... how do we get the secret? % grep secret

    /etc/chef/client.rb encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"
  37. The secret must be present... •Built into a machine image

    •Copied with SCP/Rsync •Put into place with knife bootstrap...
  38. Problems with Encrypted Items... •They're encrypted, so that's better •But

    now we have a key distribution problem • how do we share the secret key with others? • oops. •There's no centralized point of control •The key lives on the systems that need it...
  39. "That's why storing encryption keys on the same system where

    the protected data resides violates all of the core principles of data protection." ! - Patrick Townsend Townsend Security ! http://web.townsendsecurity.com/bid/23881/PCI-DSS-2-0- and-Encryption-Key-Management
  40. None
  41. Back to Square One? Rix Tox https://flic.kr/p/jhYXGH

  42. Obviously not

  43. Leading the witness! Martin Bowling https://flic.kr/p/aFh3TV

  44. But first, a story...

  45. I used to write/teach Chef Fundamentals

  46. Tiffany Von Arnim - https://flic.kr/p/nzRhZT

  47. Bill Holmes https://flic.kr/p/tujYE

  48. How do you manage secrets with Chef? attributes... data bags...

    encrypted data bags... Dennis Hill https://flic.kr/p/Lzpyo
  49. TL;DR, Chef Vault Teresa https://flic.kr/p/hvAhJ

  50. https://github.com/Nordstrom/chef-vault

  51. How does it work? • Chef Client: Private RSA key

    (/etc/chef/client.pem) • Chef Server: Public RSA key • Chef Vault: Generates a Shared Secret • Chef Vault: Encrypts w/ Public Keys • API Clients (nodes) • Users • Search Query
  52. Chef Vault: Gem and Knife Plugin % gem install chef-vault

    % knife vault ** VAULT COMMANDS ** knife vault create VAULT ITEM VALUES (options) knife vault decrypt VAULT ITEM [VALUES] (options) knife vault delete VAULT ITEM (options) knife vault edit VAULT ITEM (options) knife vault remove VAULT ITEM VALUES (options) knife vault rotate all keys knife vault rotate keys VAULT ITEM (options) knife vault show VAULT ITEM [VALUES] (options)
  53. Chef Vault: Configure Knife % grep vault .chef/knife.rb knife[:vault_mode] =

    'client'
  54. Chef Vault: Create % knife vault create secrets mysql_root \

    -J data_bags/secrets/mysql_root_unencrypted.json \ -A jtimberman -S 'role:mysql-server'
  55. Chef Vault: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id":

    "mysql_root", "password": { "encrypted_data": "naJm6HHz0ti7kzVJxFpwnmvR2/ u38yYinBpfUXUsERJQBwtfLjY2wnrLo0S3\nxO2G\n", "iv": "A0RjHlcOer+K0XT2BSI5KQ==\n", "version": 1, "cipher": "aes-256-cbc" } }
  56. Chef Vault: Show Keys % knife data bag show secrets

    mysql_root_keys admins: jtimberman chef-vault-examples: KqshDPU49J217p/xH0HAIy5Ik5j/cHflM0QYuhAtFOuWqUh4ZkV983148UcT r3hCjhQ9MkE+5NOz3CMRiOv5p1gGJ17HaM3Vtn6UY3AIN8A5biZHJPsO47uk AZyn0gsxzys561lmCAaZA3T/F/TGlv2dppf/iPmjtCX/84barkcO1F8qPrUt 94s4+LrVGconqQs3YmhdNKzfnO1im82CHZyv7DM2D2hmojGl6Re1K5UYbnvz Lo/Circ5+dninO8JXbBqvX1pNwbflWf1p7J+KqJvEnY2ZwortXNNN9FABH9+ 34WAzpZLVfICGvKuMSrofRGDfXa9Vp7Aw44cbxu1tQ== ! clients: chef-vault-examples id: mysql_root_keys jtimberman: dlmahCPrKrMF0Vh35rzOzT57hNAohC2iIU32Iit/TwFjMiGZSB0GGEXi6R7y oXjlJX1Dusqk8R4zoYWdxIvjcgWoyFTtpC1QM2ArOPiV2FMQNur+OB3RmbRM fTxygbF84PGxPHPd17pZP+hBvlxfv4sARUAfxVXr9uQIc9i14uVsZozIZ8KP YE305QuUwPhi/vAIf2Z8GAVN7X2qN4AnWKRq8dowCID67f8NdvXiwCzWQT7N pFjRSa2DdE2BeAq77YsLmdph3wd5Qg+uViQqcMnxMx0KHZOcUvWvkXUZjvCt U5bOdQCZYrI4mmMCftKN4XVKFzbzlbH1FEKmwDfwyw== ! search_query: role:mysql-server
  57. Chef Vault: Show (Decrypt) % knife vault show secrets mysql_root

    id: mysql_root password: Now-we-re-getting-somewhere!
  58. Chef Vault: Who Has Access? % knife search secrets 'id:mysql_root_keys'

    -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples ! % knife search secrets 'id:mysql_root_keys' -a admins 1 items found ! data_bag_item_secrets_mysql_root_keys: admins: jtimberman
  59. Chef Vault: Update is Additive % knife vault update secrets

    mysql_root -S 'role:mysql- server' ! % knife search secrets 'id:mysql_root_keys' -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples second-node
  60. Chef Vault: Edit % knife vault show secrets mysql_root id:

    mysql_root password: Now-we-re-getting-somewhere! ! % knife vault edit secrets mysql_root ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?
  61. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB !
  62. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB ! % knife vault rotate keys secrets mysql_root
  63. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: ZR73Skl9CntebESAPyj8MxLuLb0TXDj/6A/PSUar +eqyDy4wKW8Bvknv8Ckr+EJ7 ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?
  64. Chef Vault: Recipes ! require 'chef-vault' ! root_pw = ChefVault::Item.load('secrets',

    'mysql_root') !
  65. chef-vault Cookbook # in your cookbook's metadata depends 'chef-vault' !

    # in your recipe include_recipe 'chef-vault' ! root_pw = chef_vault_item('secrets', 'mysql_root')
  66. "Developer mode" fallback node.default['dev_mode'] = true ! include_recipe 'chef-vault' !

    root_pw = chef_vault_item('secrets', 'mysql_root')
  67. Saving Secrets in Recipes

  68. Chef Servers Generate Secrets ubuntu@chef-server:~$ sudo chef-server-ctl reconfigure ... *

    execute[/opt/chef-server/embedded/bin/psql --port 5432 -d 'opscode_chef' -c "CREATE USER opscode_chef WITH SUPERUSER ENCRYPTED PASSWORD '0f3ca56ae76af84181'"] action run * template[/var/opt/chef-server/erchef/etc/app.config] action create + %% Database connection parameters + {db_host, "127.0.0.1"}, + {db_port, 5432}, + {db_user, "opscode_chef"}, + {db_pass, "0f3ca56ae76af84181"}, + {db_name, "opscode_chef" }, + {idle_check, 10000},
  69. /etc/chef-server/chef-server-secrets.json ubuntu@chef-server:~$ sudo grep sql_password /etc/chef- server/chef-server-secrets.json "sql_password": "0f3ca56ae76af84181", !

    ubuntu@chef-server:~$ ls /etc/chef-server/ admin.pem chef-server-running.json chef-server- secrets.json chef-validator.pem chef-webui.pem ! ubuntu@chef-server:~$ scp -r /etc/chef-server root@chef- server-replica
  70. But what if we manage Chef Servers with Chef?

  71. We don't want to scp...

  72. So let's upload in a recipe!

  73. chef_vault_secret resource data = JSON.parse(IO.read('/etc/chef-server/chef-server- secrets.json')) ! chef_vault_secret 'store-chef-server-secrets' do

    data_bag 'chef-server-secrets' raw_data 'data' => data admins 'jtimberman' search 'role:chef-server' end
  74. Generate and store a password... require 'secure_random' secure_pw = SecureRandom.hex(48)

    ! chef_vault_secret 'mysql_root' do data_bag 'secrets' raw_data 'password' => secure_pw admins 'jtimberman' search 'role:mysql_server' end
  75. Share Secrets with a Coworker % cat secret_data.json { "id":

    "secret_data", "data": { "file_contents": "---- BEGIN SUPER SECRET FILE...." }, } ! % knife vault create secrets secret_data \ -J secret_data.json -A your_coworker
  76. Best Practices

  77. Best Practices GOOD IDEAS

  78. Good Ideas •Configure knife (knife[:vault_mode] = 'client') •Restrict access as

    needed (by user, node) •Rotate keys when necessary • on a schedule, when admins leave, etc. •Use the chef-vault cookbook!
  79. Chef Vault Benefits •Encrypted data bag item •No "shared secret"

    to copy •Flexible library and tooling •Easily revoke access to items •Easily rotate the shared secret •Chef Software uses this in production
  80. Things to Know •Chef Vault 3 in the works •Citadel:

    an alternative for AWS / S3 • https://github.com/poise/citadel •Cheffish & Chef Metal • https://github.com/opscode/cheffish • https://github.com/opscode/chef-metal
  81. Questions? Joshua Timberman joshua@getchef.com @jtimberman