Chef Vault

Slides for a talk about Chef Vault

Transcript

1. Chef Vault Paul Collins https://flic.kr/p/dg1kZW Joshua Timberman Sr. Operations System Administrator ! joshua@getchef.com @jtimberman
2. Managing Secrets Cedward Brice https://flic.kr/p/m6hFPH
3. Managing Secrets is a Solved Problem? Daniel Fleming https://flic.kr/p/9boLH5
6. Types of Secrets •SSL certificates •SSH keys •Passwords
7. Ways of storing secrets… •Password managers (1password, lastpass, keepass) •GPG/PGP encrypted file •Firewall-protected databass •The Configuration Management Database
9. Chef Server An infrastructure publishing system with an API and search index
10. Ways to Store Secrets in Chef Attributes Data Bag Items Encrypted Data Bag Items Chef Vault
11. Secrets in Chef Attributes!
12. Secrets in Chef: Attributes ! # In a recipe or cookbook default attributes: node.default['mysql']['root_password'] = 'This-Is-Secure-Right?' ! # In a role: { "name": "mysql-server", "default_attributes": { "mysql": { "root_password": "This-is-Secure-Right?" } } }
13. Secrets in Chef: Attributes % knife search node "mysql*password:*" -a mysql 1 items found ! chef-vault-examples: mysql: root_password: This-Is-Secure-Right?
14. Secure? NOPE. Michael Sarver https://flic.kr/p/6xHF1
15. Problems with attributes •Plain text •Easy to search •Secrets aren't usually node specific
16. Secrets in Chef Data Bag Items!
17. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
18. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
19. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
20. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end
21. Problems with Data Bags •Still plain text •Still easy to search •Can use RBAC... •But /search doesn't have RBAC :( •Using Hosted Chef?
24. Chef Server RBAC •Enterprise Chef has RBAC •Remove read permissions from certain groups/users •Remove read permissions from certain API clients (nodes)
25. This helps, but...
26. Secrets in Chef ENCRYPTED Data Bag Items!
27. Secrets in Chef: Encrypted Data Bags •Added in Chef 10 •Data bag items encrypted at rest •Encrypted with a secret (phrase or file) •Hosted Chef is the primary use case
28. I read your email...
29. I read your email... W ho has tim e for that?!
30. Encrypted Data Bags: Local files % cat secret-key -----BEGIN RSA PRIVATE KEY—— ... some stuff in a random key i made for examples -----END RSA PRIVATE KEY----- ! % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Now-we-re-getting-somewhere!" }
31. Encrypted Data Bags: Knife % knife data bag from file -h knife data bag from file BAG FILE|FOLDER [FILE|FOLDER..] (options) -s, --secret The secret key to use to encrypt data bag item values --secret-file SECRET_FILE A file containing the secret key to use to encrypt data bag item values
32. Encrypted Data Bags: Upload % knife data bag from file secrets \ data_bags/secrets/mysql_root.json --secret-file secret-key ! Updated data_bag_item[secrets::mysql_root]
33. Encrypted Data Bags: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": { "encrypted_data": "aCztoQZuBvPQopBgxY32AjAVQ8QXxcaj0VkuIxO/ YZxkVNgClcJv3RpDvLFF\nZT6s\n", "iv": "nYTlXs9tCMOQw2ikgXCwrA==\n", "version": 1, "cipher": "aes-256-cbc" } }
34. Encrypted Data Bags: Decrypt % knife data bag show secrets mysql_root \ --secret-file secret-key id: mysql_root password: Now-we-re-getting-somewhere!
35. Encrypted Data Bags: Recipe root_pw = Chef::EncryptedDataBagItem.load('secrets', 'mysql_root')
36. Wait... how do we get the secret? % grep secret /etc/chef/client.rb encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"
37. The secret must be present... •Built into a machine image •Copied with SCP/Rsync •Put into place with knife bootstrap...
38. Problems with Encrypted Items... •They're encrypted, so that's better •But now we have a key distribution problem • how do we share the secret key with others? • oops. •There's no centralized point of control •The key lives on the systems that need it...
39. "That's why storing encryption keys on the same system where the protected data resides violates all of the core principles of data protection." ! - Patrick Townsend Townsend Security ! http://web.townsendsecurity.com/bid/23881/PCI-DSS-2-0- and-Encryption-Key-Management
41. Back to Square One? Rix Tox https://flic.kr/p/jhYXGH
42. Obviously not
43. Leading the witness! Martin Bowling https://flic.kr/p/aFh3TV
44. But first, a story...
45. I used to write/teach Chef Fundamentals
46. Tiffany Von Arnim - https://flic.kr/p/nzRhZT
47. Bill Holmes https://flic.kr/p/tujYE
48. How do you manage secrets with Chef? attributes... data bags... encrypted data bags... Dennis Hill https://flic.kr/p/Lzpyo
49. TL;DR, Chef Vault Teresa https://flic.kr/p/hvAhJ
50. https://github.com/Nordstrom/chef-vault
51. How does it work? • Chef Client: Private RSA key (/etc/chef/client.pem) • Chef Server: Public RSA key • Chef Vault: Generates a Shared Secret • Chef Vault: Encrypts w/ Public Keys • API Clients (nodes) • Users • Search Query
52. Chef Vault: Gem and Knife Plugin % gem install chef-vault % knife vault ** VAULT COMMANDS ** knife vault create VAULT ITEM VALUES (options) knife vault decrypt VAULT ITEM [VALUES] (options) knife vault delete VAULT ITEM (options) knife vault edit VAULT ITEM (options) knife vault remove VAULT ITEM VALUES (options) knife vault rotate all keys knife vault rotate keys VAULT ITEM (options) knife vault show VAULT ITEM [VALUES] (options)
53. Chef Vault: Configure Knife % grep vault .chef/knife.rb knife[:vault_mode] = 'client'
54. Chef Vault: Create % knife vault create secrets mysql_root \ -J data_bags/secrets/mysql_root_unencrypted.json \ -A jtimberman -S 'role:mysql-server'
55. Chef Vault: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": { "encrypted_data": "naJm6HHz0ti7kzVJxFpwnmvR2/ u38yYinBpfUXUsERJQBwtfLjY2wnrLo0S3\nxO2G\n", "iv": "A0RjHlcOer+K0XT2BSI5KQ==\n", "version": 1, "cipher": "aes-256-cbc" } }
56. Chef Vault: Show Keys % knife data bag show secrets mysql_root_keys admins: jtimberman chef-vault-examples: KqshDPU49J217p/xH0HAIy5Ik5j/cHflM0QYuhAtFOuWqUh4ZkV983148UcT r3hCjhQ9MkE+5NOz3CMRiOv5p1gGJ17HaM3Vtn6UY3AIN8A5biZHJPsO47uk AZyn0gsxzys561lmCAaZA3T/F/TGlv2dppf/iPmjtCX/84barkcO1F8qPrUt 94s4+LrVGconqQs3YmhdNKzfnO1im82CHZyv7DM2D2hmojGl6Re1K5UYbnvz Lo/Circ5+dninO8JXbBqvX1pNwbflWf1p7J+KqJvEnY2ZwortXNNN9FABH9+ 34WAzpZLVfICGvKuMSrofRGDfXa9Vp7Aw44cbxu1tQ== ! clients: chef-vault-examples id: mysql_root_keys jtimberman: dlmahCPrKrMF0Vh35rzOzT57hNAohC2iIU32Iit/TwFjMiGZSB0GGEXi6R7y oXjlJX1Dusqk8R4zoYWdxIvjcgWoyFTtpC1QM2ArOPiV2FMQNur+OB3RmbRM fTxygbF84PGxPHPd17pZP+hBvlxfv4sARUAfxVXr9uQIc9i14uVsZozIZ8KP YE305QuUwPhi/vAIf2Z8GAVN7X2qN4AnWKRq8dowCID67f8NdvXiwCzWQT7N pFjRSa2DdE2BeAq77YsLmdph3wd5Qg+uViQqcMnxMx0KHZOcUvWvkXUZjvCt U5bOdQCZYrI4mmMCftKN4XVKFzbzlbH1FEKmwDfwyw== ! search_query: role:mysql-server
57. Chef Vault: Show (Decrypt) % knife vault show secrets mysql_root id: mysql_root password: Now-we-re-getting-somewhere!
58. Chef Vault: Who Has Access? % knife search secrets 'id:mysql_root_keys' -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples ! % knife search secrets 'id:mysql_root_keys' -a admins 1 items found ! data_bag_item_secrets_mysql_root_keys: admins: jtimberman
59. Chef Vault: Update is Additive % knife vault update secrets mysql_root -S 'role:mysql- server' ! % knife search secrets 'id:mysql_root_keys' -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples second-node
60. Chef Vault: Edit % knife vault show secrets mysql_root id: mysql_root password: Now-we-re-getting-somewhere! ! % knife vault edit secrets mysql_root ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?
61. Chef Vault: Rotate % knife data bag show secrets mysql_root id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB !
62. Chef Vault: Rotate % knife data bag show secrets mysql_root id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB ! % knife vault rotate keys secrets mysql_root
63. Chef Vault: Rotate % knife data bag show secrets mysql_root id: mysql_root password: encrypted_data: ZR73Skl9CntebESAPyj8MxLuLb0TXDj/6A/PSUar +eqyDy4wKW8Bvknv8Ckr+EJ7 ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?
64. Chef Vault: Recipes ! require 'chef-vault' ! root_pw = ChefVault::Item.load('secrets', 'mysql_root') !
65. chef-vault Cookbook # in your cookbook's metadata depends 'chef-vault' ! # in your recipe include_recipe 'chef-vault' ! root_pw = chef_vault_item('secrets', 'mysql_root')
66. "Developer mode" fallback node.default['dev_mode'] = true ! include_recipe 'chef-vault' ! root_pw = chef_vault_item('secrets', 'mysql_root')
67. Saving Secrets in Recipes
68. Chef Servers Generate Secrets ubuntu@chef-server:~$ sudo chef-server-ctl reconfigure ... * execute[/opt/chef-server/embedded/bin/psql --port 5432 -d 'opscode_chef' -c "CREATE USER opscode_chef WITH SUPERUSER ENCRYPTED PASSWORD '0f3ca56ae76af84181'"] action run * template[/var/opt/chef-server/erchef/etc/app.config] action create + %% Database connection parameters + {db_host, "127.0.0.1"}, + {db_port, 5432}, + {db_user, "opscode_chef"}, + {db_pass, "0f3ca56ae76af84181"}, + {db_name, "opscode_chef" }, + {idle_check, 10000},
69. /etc/chef-server/chef-server-secrets.json ubuntu@chef-server:~$ sudo grep sql_password /etc/chef- server/chef-server-secrets.json "sql_password": "0f3ca56ae76af84181", ! ubuntu@chef-server:~$ ls /etc/chef-server/ admin.pem chef-server-running.json chef-server- secrets.json chef-validator.pem chef-webui.pem ! ubuntu@chef-server:~$ scp -r /etc/chef-server root@chef- server-replica
70. But what if we manage Chef Servers with Chef?
71. We don't want to scp...
72. So let's upload in a recipe!
73. chef_vault_secret resource data = JSON.parse(IO.read('/etc/chef-server/chef-server- secrets.json')) ! chef_vault_secret 'store-chef-server-secrets' do data_bag 'chef-server-secrets' raw_data 'data' => data admins 'jtimberman' search 'role:chef-server' end
74. Generate and store a password... require 'secure_random' secure_pw = SecureRandom.hex(48) ! chef_vault_secret 'mysql_root' do data_bag 'secrets' raw_data 'password' => secure_pw admins 'jtimberman' search 'role:mysql_server' end
75. Share Secrets with a Coworker % cat secret_data.json { "id": "secret_data", "data": { "file_contents": "---- BEGIN SUPER SECRET FILE...." }, } ! % knife vault create secrets secret_data \ -J secret_data.json -A your_coworker
76. Best Practices
77. Best Practices GOOD IDEAS
78. Good Ideas •Configure knife (knife[:vault_mode] = 'client') •Restrict access as needed (by user, node) •Rotate keys when necessary • on a schedule, when admins leave, etc. •Use the chef-vault cookbook!
79. Chef Vault Benefits •Encrypted data bag item •No "shared secret" to copy •Flexible library and tooling •Easily revoke access to items •Easily rotate the shared secret •Chef Software uses this in production
80. Things to Know •Chef Vault 3 in the works •Citadel: an alternative for AWS / S3 • https://github.com/poise/citadel •Cheffish & Chef Metal • https://github.com/opscode/cheffish • https://github.com/opscode/chef-metal
81. Questions? Joshua Timberman joshua@getchef.com @jtimberman