Chef Vault

Transcript

1. Chef Vault Paul Collins https://flic.kr/p/dg1kZW Joshua Timberman Sr. Operations System

   Administrator ! joshua@getchef.com @jtimberman

2. Managing Secrets Cedward Brice https://flic.kr/p/m6hFPH

3. Managing Secrets is a Solved Problem? Daniel Fleming https://flic.kr/p/9boLH5

4. None
5. None

6. Types of Secrets •SSL certificates •SSH keys •Passwords

7. Ways of storing secrets… •Password managers (1password, lastpass, keepass) •GPG/PGP

   encrypted file •Firewall-protected databass •The Configuration Management Database
8. None

9. Chef Server An infrastructure publishing system with an API and

   search index

10. Ways to Store Secrets in Chef Attributes Data Bag Items

    Encrypted Data Bag Items Chef Vault

11. Secrets in Chef Attributes!

12. Secrets in Chef: Attributes ! # In a recipe or

    cookbook default attributes: node.default['mysql']['root_password'] = 'This-Is-Secure-Right?' ! # In a role: { "name": "mysql-server", "default_attributes": { "mysql": { "root_password": "This-is-Secure-Right?" } } }

13. Secrets in Chef: Attributes % knife search node "mysql*password:*" -a

    mysql 1 items found ! chef-vault-examples: mysql: root_password: This-Is-Secure-Right?

14. Secure? NOPE. Michael Sarver https://flic.kr/p/6xHF1

15. Problems with attributes •Plain text •Easy to search •Secrets aren't

    usually node specific

16. Secrets in Chef Data Bag Items!

17. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end

18. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end

19. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end

20. Secrets in Chef: Data Bag Items % knife show data_bags/secrets/mysql_root.json

    data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Is-This-More-Secure?" } ! % cat cookbooks/mysql/recipes/grants root_pw = data_bag_item('secrets', 'mysql_root') ! template '/etc/mysql/grants.sql' do variables :root_pw => root_pw end

21. Problems with Data Bags •Still plain text •Still easy to

    search •Can use RBAC... •But /search doesn't have RBAC :( •Using Hosted Chef?
22. None
23. None

24. Chef Server RBAC •Enterprise Chef has RBAC •Remove read permissions

    from certain groups/users •Remove read permissions from certain API clients (nodes)

25. This helps, but...

26. Secrets in Chef ENCRYPTED Data Bag Items!

27. Secrets in Chef: Encrypted Data Bags •Added in Chef 10

    •Data bag items encrypted at rest •Encrypted with a secret (phrase or file) •Hosted Chef is the primary use case

28. I read your email...

29. I read your email... W ho has tim e for

    that?!

30. Encrypted Data Bags: Local files % cat secret-key -----BEGIN RSA

    PRIVATE KEY—— ... some stuff in a random key i made for examples -----END RSA PRIVATE KEY----- ! % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id": "mysql_root", "password": "Now-we-re-getting-somewhere!" }

31. Encrypted Data Bags: Knife % knife data bag from file

    -h knife data bag from file BAG FILE|FOLDER [FILE|FOLDER..] (options) -s, --secret The secret key to use to encrypt data bag item values --secret-file SECRET_FILE A file containing the secret key to use to encrypt data bag item values

32. Encrypted Data Bags: Upload % knife data bag from file

    secrets \ data_bags/secrets/mysql_root.json --secret-file secret-key ! Updated data_bag_item[secrets::mysql_root]

33. Encrypted Data Bags: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: {

    "id": "mysql_root", "password": { "encrypted_data": "aCztoQZuBvPQopBgxY32AjAVQ8QXxcaj0VkuIxO/ YZxkVNgClcJv3RpDvLFF\nZT6s\n", "iv": "nYTlXs9tCMOQw2ikgXCwrA==\n", "version": 1, "cipher": "aes-256-cbc" } }

34. Encrypted Data Bags: Decrypt % knife data bag show secrets

    mysql_root \ --secret-file secret-key id: mysql_root password: Now-we-re-getting-somewhere!

35. Encrypted Data Bags: Recipe root_pw = Chef::EncryptedDataBagItem.load('secrets', 'mysql_root')

36. Wait... how do we get the secret? % grep secret

    /etc/chef/client.rb encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"

37. The secret must be present... •Built into a machine image

    •Copied with SCP/Rsync •Put into place with knife bootstrap...

38. Problems with Encrypted Items... •They're encrypted, so that's better •But

    now we have a key distribution problem • how do we share the secret key with others? • oops. •There's no centralized point of control •The key lives on the systems that need it...

39. "That's why storing encryption keys on the same system where

    the protected data resides violates all of the core principles of data protection." ! - Patrick Townsend Townsend Security ! http://web.townsendsecurity.com/bid/23881/PCI-DSS-2-0- and-Encryption-Key-Management
40. None

41. Back to Square One? Rix Tox https://flic.kr/p/jhYXGH

42. Obviously not

43. Leading the witness! Martin Bowling https://flic.kr/p/aFh3TV

44. But first, a story...

45. I used to write/teach Chef Fundamentals

46. Tiffany Von Arnim - https://flic.kr/p/nzRhZT

47. Bill Holmes https://flic.kr/p/tujYE

48. How do you manage secrets with Chef? attributes... data bags...

    encrypted data bags... Dennis Hill https://flic.kr/p/Lzpyo

49. TL;DR, Chef Vault Teresa https://flic.kr/p/hvAhJ

50. https://github.com/Nordstrom/chef-vault

51. How does it work? • Chef Client: Private RSA key

    (/etc/chef/client.pem) • Chef Server: Public RSA key • Chef Vault: Generates a Shared Secret • Chef Vault: Encrypts w/ Public Keys • API Clients (nodes) • Users • Search Query

52. Chef Vault: Gem and Knife Plugin % gem install chef-vault

    % knife vault ** VAULT COMMANDS ** knife vault create VAULT ITEM VALUES (options) knife vault decrypt VAULT ITEM [VALUES] (options) knife vault delete VAULT ITEM (options) knife vault edit VAULT ITEM (options) knife vault remove VAULT ITEM VALUES (options) knife vault rotate all keys knife vault rotate keys VAULT ITEM (options) knife vault show VAULT ITEM [VALUES] (options)

53. Chef Vault: Configure Knife % grep vault .chef/knife.rb knife[:vault_mode] =

    'client'

54. Chef Vault: Create % knife vault create secrets mysql_root \

    -J data_bags/secrets/mysql_root_unencrypted.json \ -A jtimberman -S 'role:mysql-server'

55. Chef Vault: Show % knife show data_bags/secrets/mysql_root.json data_bags/secrets/mysql_root.json: { "id":

    "mysql_root", "password": { "encrypted_data": "naJm6HHz0ti7kzVJxFpwnmvR2/ u38yYinBpfUXUsERJQBwtfLjY2wnrLo0S3\nxO2G\n", "iv": "A0RjHlcOer+K0XT2BSI5KQ==\n", "version": 1, "cipher": "aes-256-cbc" } }

56. Chef Vault: Show Keys % knife data bag show secrets

    mysql_root_keys admins: jtimberman chef-vault-examples: KqshDPU49J217p/xH0HAIy5Ik5j/cHflM0QYuhAtFOuWqUh4ZkV983148UcT r3hCjhQ9MkE+5NOz3CMRiOv5p1gGJ17HaM3Vtn6UY3AIN8A5biZHJPsO47uk AZyn0gsxzys561lmCAaZA3T/F/TGlv2dppf/iPmjtCX/84barkcO1F8qPrUt 94s4+LrVGconqQs3YmhdNKzfnO1im82CHZyv7DM2D2hmojGl6Re1K5UYbnvz Lo/Circ5+dninO8JXbBqvX1pNwbflWf1p7J+KqJvEnY2ZwortXNNN9FABH9+ 34WAzpZLVfICGvKuMSrofRGDfXa9Vp7Aw44cbxu1tQ== ! clients: chef-vault-examples id: mysql_root_keys jtimberman: dlmahCPrKrMF0Vh35rzOzT57hNAohC2iIU32Iit/TwFjMiGZSB0GGEXi6R7y oXjlJX1Dusqk8R4zoYWdxIvjcgWoyFTtpC1QM2ArOPiV2FMQNur+OB3RmbRM fTxygbF84PGxPHPd17pZP+hBvlxfv4sARUAfxVXr9uQIc9i14uVsZozIZ8KP YE305QuUwPhi/vAIf2Z8GAVN7X2qN4AnWKRq8dowCID67f8NdvXiwCzWQT7N pFjRSa2DdE2BeAq77YsLmdph3wd5Qg+uViQqcMnxMx0KHZOcUvWvkXUZjvCt U5bOdQCZYrI4mmMCftKN4XVKFzbzlbH1FEKmwDfwyw== ! search_query: role:mysql-server

57. Chef Vault: Show (Decrypt) % knife vault show secrets mysql_root

    id: mysql_root password: Now-we-re-getting-somewhere!

58. Chef Vault: Who Has Access? % knife search secrets 'id:mysql_root_keys'

    -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples ! % knife search secrets 'id:mysql_root_keys' -a admins 1 items found ! data_bag_item_secrets_mysql_root_keys: admins: jtimberman

59. Chef Vault: Update is Additive % knife vault update secrets

    mysql_root -S 'role:mysql- server' ! % knife search secrets 'id:mysql_root_keys' -a clients 1 items found ! data_bag_item_secrets_mysql_root_keys: clients: chef-vault-examples second-node

60. Chef Vault: Edit % knife vault show secrets mysql_root id:

    mysql_root password: Now-we-re-getting-somewhere! ! % knife vault edit secrets mysql_root ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?

61. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB !

62. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: B7uY/ zhphnyd0T7OVFRC0n0PNneCBG5Gcx3MbN6KYwS0MiAfQT9wAyuWzG2EKReB ! % knife vault rotate keys secrets mysql_root

63. Chef Vault: Rotate % knife data bag show secrets mysql_root

    id: mysql_root password: encrypted_data: ZR73Skl9CntebESAPyj8MxLuLb0TXDj/6A/PSUar +eqyDy4wKW8Bvknv8Ckr+EJ7 ! % knife vault show secrets mysql_root id: mysql_root password: Time-to-update-passwords?

64. Chef Vault: Recipes ! require 'chef-vault' ! root_pw = ChefVault::Item.load('secrets',

    'mysql_root') !

65. chef-vault Cookbook # in your cookbook's metadata depends 'chef-vault' !

    # in your recipe include_recipe 'chef-vault' ! root_pw = chef_vault_item('secrets', 'mysql_root')

66. "Developer mode" fallback node.default['dev_mode'] = true ! include_recipe 'chef-vault' !

    root_pw = chef_vault_item('secrets', 'mysql_root')

67. Saving Secrets in Recipes

68. Chef Servers Generate Secrets ubuntu@chef-server:~$ sudo chef-server-ctl reconfigure ... *

    execute[/opt/chef-server/embedded/bin/psql --port 5432 -d 'opscode_chef' -c "CREATE USER opscode_chef WITH SUPERUSER ENCRYPTED PASSWORD '0f3ca56ae76af84181'"] action run * template[/var/opt/chef-server/erchef/etc/app.config] action create + %% Database connection parameters + {db_host, "127.0.0.1"}, + {db_port, 5432}, + {db_user, "opscode_chef"}, + {db_pass, "0f3ca56ae76af84181"}, + {db_name, "opscode_chef" }, + {idle_check, 10000},

69. /etc/chef-server/chef-server-secrets.json ubuntu@chef-server:~$ sudo grep sql_password /etc/chef- server/chef-server-secrets.json "sql_password": "0f3ca56ae76af84181", !

    ubuntu@chef-server:~$ ls /etc/chef-server/ admin.pem chef-server-running.json chef-server- secrets.json chef-validator.pem chef-webui.pem ! ubuntu@chef-server:~$ scp -r /etc/chef-server root@chef- server-replica

70. But what if we manage Chef Servers with Chef?

71. We don't want to scp...

72. So let's upload in a recipe!

73. chef_vault_secret resource data = JSON.parse(IO.read('/etc/chef-server/chef-server- secrets.json')) ! chef_vault_secret 'store-chef-server-secrets' do

    data_bag 'chef-server-secrets' raw_data 'data' => data admins 'jtimberman' search 'role:chef-server' end

74. Generate and store a password... require 'secure_random' secure_pw = SecureRandom.hex(48)

    ! chef_vault_secret 'mysql_root' do data_bag 'secrets' raw_data 'password' => secure_pw admins 'jtimberman' search 'role:mysql_server' end

75. Share Secrets with a Coworker % cat secret_data.json { "id":

    "secret_data", "data": { "file_contents": "---- BEGIN SUPER SECRET FILE...." }, } ! % knife vault create secrets secret_data \ -J secret_data.json -A your_coworker

76. Best Practices

77. Best Practices GOOD IDEAS

78. Good Ideas •Configure knife (knife[:vault_mode] = 'client') •Restrict access as

    needed (by user, node) •Rotate keys when necessary • on a schedule, when admins leave, etc. •Use the chef-vault cookbook!

79. Chef Vault Benefits •Encrypted data bag item •No "shared secret"

    to copy •Flexible library and tooling •Easily revoke access to items •Easily rotate the shared secret •Chef Software uses this in production

80. Things to Know •Chef Vault 3 in the works •Citadel:

    an alternative for AWS / S3 • https://github.com/poise/citadel •Cheffish & Chef Metal • https://github.com/opscode/cheffish • https://github.com/opscode/chef-metal

81. Questions? Joshua Timberman joshua@getchef.com @jtimberman