Chef server and know how to secure them • Participants will learn how to utilize industry standards for security and how the Chef analytics platform can be leveraged to streamline auditing and compliance 6 Goals
compared to encrypted data bags • Participants will successfully use Chef Vault to generate, distribute and encrypt/decrypt secrets both via CLI and the Chef client run • Participants will learn strategies for managing and rotating keys 7 Goals
are present or not • @here notifies everyone currently in the room • @username to address a specific person in the room • /code <paste code> allows you to paste code snippets in the room
key fingerprint is d9:95:a3:b9:02:27:e9:cd:74:e4:a2:34:23:f5:a6:8b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'uvo1qrwls0jdgs3blvt.vm.cld.sr,69.195.232.110' (RSA) to the list of known hosts. [email protected]'s password: Last login: Mon Jan 6 16:26:24 2014 from host86-145-117-53.range86-145.btcentralplus.com [chef@CentOS63 ~]$ Lab - Login $ ssh chef@<EXTERNAL_ADDRESS>
(VM) or server that you’ll use for the lab exercises •The IP address or public hostname •An application for establishing an ssh connection •‘sudo’ or ‘root’ permissions on the VM
on your laptop under your home directory called ‘~/workshop’, i.e. •Windows:- C:\Users\you\workshop •Mac/*nix:- /Users/you/workshop Navigate to this working directory
server setup from the morning Analytics workshop •Only do this if you don’t already have the .pem file available on your laptop •Reset and copy/paste your key to <USER.pem>
$ mv ../<your-org>-validator.pem .chef/ $ cp ~/Downloads/knife.rb .chef/ Exercise: Create and populate a .chef dir • knife.rb & .pem files reside in the .chef directory which can be in o <current-directory>/.chef o /etc/.chef o ~/.chef
your trusted_cert directory (/Users/larryeichenbaum/workshop/chef-fundamentals-repo-master/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. Adding certificate for ec2-52-11-224-17.us-west-2.compute.amazonaws.com in /Users/larryeichenbaum/workshop/chef- fundamentals-repo-master/.chef/trusted_certs/ec2-52-11-224-17_us-west-2_compute_amazonaws_com.crt Exercise: Setup Trust with Chef Server $ knife ssl fetch
Chef server stores all! • If you have access, you have access to database, cookbooks, etc. http://midstatelockandsafe.com/wp-content/uploads/2010/10/Keys- To-The-Castle.jpg
your infrastructure. • Be protective… • Or explain later. http://vignette3.wikia.nocookie.net/koc/images/7/75/ Defend_victory.jpg/revision/latest?cb=20101126001303
Passwords not THAT secure • Avoid duplicate passwords on multiple hosts • More convenient • Reduces likelihood of brute- force attacks http://www.correderajorge.es/wp-content/uploads/2013/09/ public_key_cryptography_sm.png http://commons.wikimedia.org/wiki/File:Crypto-Nerd.png
of entry • Contains the code that defines your servers https://www.flickr.com/photos/bradhigham/14760496352/in/photolist-ispVZE-mNbLn-6AXqjt-89yaQo-oukpjq-prBQef-jqtAUm-8fQsXR-cBahob- anqncf-8qtez7-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr-86n2rp-89uVET-Ci1ba-2616FK-7ZiXZa-ct1ddE- cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN-uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx- a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj-bE8t-7YRahJ
of entry • Contains the code that defines your servers • Contains the code that runs on your servers http://cdn.arstechnica.net/wp-content/uploads/2014/01/falken.jpg Yea, I'm old white guy… but have you really not seen War Games?
GitHub or equivalent • grants insight to your infra code • GitHub Enterprise or equivalent • internal network • secure • control access https://connect.decknetwork.net/i/github_enterprise2.png
be a point of entry https://www.flickr.com/photos/innoxiuss/3306092527/in/photolist-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr- Ci1ba-2616FK-7ZiXZa-ct1ddE-7YRahJ-cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN- uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx-a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj- bE8t-59G4t1-9fTpzb-9gKoEX-cvzKn5-qLC9TW-8s59Mp-2cHYtX-kKVeFB-rmWHuR-9xKv5y-8iAAWZ-guvBw9-9CuSNo
be a point of entry • Read and review everything https://www.flickr.com/photos/bcgovphotos/8113334531/in/photolist-637KYD-6rCqSw- ayHV31-h4FzN1-QtXAi-bz31qy-bz31ud-bz2ZPE-bMWH56-bMWH2e-bMWFmc-bz2ZDb- bz2ZvG-bz2ZyC-bMWFyH-bMWGTi-bz312W-bMWF5B-bMWGGM-bz2ZSY-bMWFH4- aMGyv-dmWXG8-2TzBBJ-58mAsb-6bTs8x-hCCAGd-dmYcfU-akUamc-qw7r3R- b4uCGP-9fhgkr-knVUPZ-mrMJKx-mcYPuV-bMWH6P-bMWF9B-bMWGRg-bMWGMx- bMWGWa-bz31gE-bz2YrY-bz2YA9-bMWE2z-bMWFak-bz2YuE-bMWDUx-h4FodQ- h4FoUD-5B38Au
be a point of entry • Read and review everything • Test run in isolation -- if your security requires it https://www.flickr.com/photos/riacale/4230091725/in/photolist-jxpBYJ-6nssFe-fMcDHU-piQkus-dwnAtk-84xhpn-apLYmH-4H6hbz-4T59iu-q5c9ra-5BK2bU- qHjXax-5Rb5eJ-piHCcJ-6HeMaE-6HaJxV-4zQbEB-buLxSf-iWs9-GiDTU-7rNjTX-4Q2u8A-9gFVjE-bpeN7A-dJGb51- dGokV5-98ukrU-6334S-9gF9xM-7EpEWX-7vAbgF-dy4bk7-5NjDk8-4CvBbo-8TftWn-68aLxw-6v7j4D-6cuhSL-3bm4U-3bm4S-3bm4T- fz4T94-95m3Ti-6ugcDf-6NqRS7-dMPypX-p9tcgy-3bm4V-5RAXGm-5NoTru
when it makes HTTPS connections - hence the SSL error when running chef-client • Enable SSL Verification by setting ssl_verify_mode :verify_peer in your config file • If enabled, affects all SSL endpoints (eg., remote_files, etc.) • Two useful knife commands: • knife ssl check - makes an SSL connection to your Chef server or any other HTTPS server and tells you if the server presents a valid certificate • knife ssl fetch - allows you to automatically fetch a server’s certificates to your trusted certs directory
"ssl_verify_mode" => ":verify_peer", "log_level" => ":info" } } ) chef_server_url "https://api.opscode.com/organizations/MYORG" validation_client_name "MYORG-validator" ssl_verify_mode :verify_peer node_name "config-ubuntu-1204" log_level :info • For example, the following attributes (in a role) • Will render the following configuration (/etc/chef/client.rb)
is signed by a known CA • If you run your own CA, or your Chef server’s certificate is self-signed, you can use knife to import it • If you do, it is also necessary to import to your Chef server, the certificate from the server used to generate your Chef certificate
a new Chef 12 server •You’ve just installed Chef 12 client on your local system •You’ve created a user account and an organization node_name 'someNode' client_key 'clientKey.pem' validation_client_name 'org-validator' validation_key 'org-validator.pem' chef_server_url 'https://chef-server.some.domain/organization/org
certificate •Follow steps indicated to retrieve the server SSL certificate and store it locally as a ‘trusted’ certificate •Or… Connecting to host chef-server.example.com:443 ERROR: The SSL certificate of chef-server.example.com could not be verified Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/ CN=chef-server.example.com/[email protected] Configuration Info: OpenSSL Configuration: * Version: OpenSSL 1.0.1j 15 Oct 2014 * Certificate file: /opt/chefdk/embedded/ssl/cert.pem * Certificate directory: /opt/chefdk/embedded/ssl/certs Chef SSL Configuration: * ssl_ca_path: nil * ssl_ca_file: nil * trusted_certs_dir: "/Users/jtimberman/Downloads/chef-repo/.chef/ trusted_certs" TO FIX THIS ERROR: If the server you are connecting to uses a self-signed certificate, you must configure chef to trust that server's certificate. By default, the certificate is stored in the following location on the host where your chef-server runs: /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt Copy that file to your trusted_certs_dir (currently: /Users/<userName>/ Downloads/chef-repo/.chef/trusted_certs) using SSH/SCP or some other secure method, then re-run this command to confirm that the server's certificate is now trusted.
your trusted_cert directory (/Users/newUser/Downloads/chef-repo/.chef/trusted_certs). Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading. Adding certificate for chef-server.example.com in /Users/newUser/Downloads/chef-repo/.chef/ trusted_certs/chef-server.example.com.crt Chef 12 SSL $ knife ssl fetch
parallel) on a subset of nodes within an organization •Based on the results of a search query made to the Chef server •Syntax: knife ssh SEARCH_QUERY SSH_COMMAND (options)
in your trusted_cert node-output.example.com directory (/etc/chef/trusted_certs). node-output.example.com node-output.example.com Knife has no means to verify these are the correct certificates. You should node-output.example.com verify the authenticity of these certificates after downloading. node-output.example.com node-output.example.com Adding certificate for chef-server.example.com in /etc/chef/ trusted_certs/chef-server.example.com.crt Chef 12 SSL $ knife ssh 'name:*' 'sudo knife ssl fetch –c /etc/chef/client.rb
•Verify the FQDN for the Chef server $ hostname -f •Delete certificate files $ rm /var/opt/opscode/nginx/ca/$<FQDN>.crt $ rm /var/opt/opscode/nginx/ca/$<FQDN>.key
refer to the file names/location as referenced by the values: nginx['ssl_certificate'] nginx['ssl_certificate_key'] within file: /etc/opscode/chef-server.rb •Delete those files; regenerate using same CA
the SSL keys from the Chef server? •What is the key difference between the Chef 11 server and Chef 12 server with regard to SSL certificates? •How frequently do Chef server SSL certifications auto regenerate?
a RubyGem • Bundled within chef-dk • Gem needs to be installed on all workstations/servers that will either encrypt or decrypt data 103 http://meta.wikimedia.org/wiki/ File:SLNSW_17216_Sliding_grille_at_the_entrance_to_the _Rural_Banks_vault_Martin_Place.jpg http://flickr.com/photos/77947478@N00/2631203510 N ot this k ind… Chef Vault
create the item that stores the secret info… • We can directly pass the unencrypted secret without storing it • Alternatively, we can store the unencrypted secret on disk in a JSON formatted file 107 Create the Item…
vaultuser: cipher: aes-256-cbc encrypted_data: j+/ fFM7ist6I7K360GNfzSgu6ix63HGyXN2ZAd99R6H4TAJ4pQKuFNpJXYn C SXA5n68xn9frxHAJNcLuDXCkEv+F/MnW9vMlTaiuwW/jO+ +vS5mIxWU170mR EgeB7gvPH7lfUdJFURNGQzdiTSSFua9E06kAu9dcrT83PpoQQzk= iv: cu2Ugw+RpTDVRu1QaaAfug== version: 1 Examine First Item in Data Bag…
a password that is set to the value stored within our encrypted data bag. •Create a cookbook •Add cookbook to base role CD into your cookbooks directory 117 Manage a User Password
two variables: the data bag (e.g. vault) and the item • The value of the item is returned 120 Breaking It Down… vault = ChefVault::Item.load("secrets", "vaultuser")
has been re-encrypted with SHA-512 required by the operating system for user passwords. • Using a case statement you could set the re-encryption to match that required by different operating systems. 121 Breaking It Down… hashed_password = vault['vaultuser'].crypt("$6$")
as we’re referencing the local variable vault and the vault user key 122 Breaking It Down… user "vaultuser" do password hashed_password home "/home/vaultuser" supports :manage_home => true shell "/bin/bash" comment "Chef Vault User" end
Server Version 12.0.3 or higher; Enterprise Chef 11.3 or higher Chef Client Version 11 or 12 required; 12.1 required for audit-mode Reporting Version 1.2.3 suggested for full feature/functionality
then used to test the state of the system for compliance Audit Mode Execution mode of chef-client that evaluates control / control-groups that are defined within recipes •Can be run in combination with a standard run after resources converge •As a separate chef-client run with no convergence
made available Keeps track of what happened during chef-client runs across your Chef managed systems Reporting Reports can be generated for the entire organization and they can be generated for specific nodes. Rules Track specific outcomes of audits and issue notifications to various endpoints and/or services
used to test the state of the system for compliance •A single control block defines one or more validations •Contains an it statement •Each it statement contains one or more expect statements
"should do something" do expect(something).to/.to_not be_something end end control "nameTwo" do it "should do something new" do expect(something).to/.to_not be_something expect(somethingElse).to/.to_not be_something end end end
Name:: auditlab # Recipe:: default # # Copyright (c) 2015 The Authors, All Rights Reserved. control_group 'check smtp configuration' do control 'smtp port' do it 'should not be listening' do expect(port(25)).to_not be_listening end end end /cookbooks/auditlab/recipes/default.rb
rules, or audits, defined within recipes •Can be run as part of normal chef-run, at the end •Can be run separately, where no converge occurs o --audit-mode disabled (default) o --audit-mode enabled o --audit-mode audit-only
run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase Execute client-run Audit Mode user@centos$ sudo chef-client –-audit-mode audit-only
be listening (FAILED - 1) Failures: 1) check smtp configuration smtp port should not be listening Failure/Error: expect(port(25)).to_not be_listening expected Port "25" not to be listening . . . Finished in 0.05565 seconds (files took 0.27206 seconds to load) 1 example, 1 failure Failed examples: rspec # check smtp configuration smtp port should not be listening Execute client-run Audit Mode user@centos$ sudo chef-client –audit-mode audit-only
Name:: auditlab # Recipe:: default # # Copyright (c) 2015 The Authors, All Rights Reserved. control_group 'check smtp configuration' do control 'smtp port' do it 'should be listening' do expect(port(25)).to be_listening end end end /cookbooks/auditlab/recipes/default.rb
are unknown or not communicated: •Rules track the events generated by audits •Rules track specific outcomes of the audit tests •Rules send notifications about these outcomes o http webhook o email o chat services, etc.
message type •Messages sent to analytics as rules are triggered •Message types: •action (msgs about actions that occur on Chef Server) •run_control (audit related) •run_control_group (audit related) •run_converge (msgs sent at end of chef-client run) •run_resource (msgs sent during chef-client run) •run_start (msgs sent at start of chef-client run)
run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase Execute client-run Audit Mode user@centos$ sudo chef-client –audit-mode audit-only
users and clients •Server will check request against all valid keys for user or client •Enables server to know if request is from a user or client •Resolves user-client ambiguity •Optional expiration dates
the more complex data bag encryption we can accomplish via Chef Vault • We’ve learned about how to lock down specific access points to Chef Server application and how to apply industry standards for security to further secure our Chef Server infrastructure
generate, distribute and encrypt/decrypt secrets •We’ve successfully performed a security audit of a Chef managed server and learned how we can extend this functionality in a broader capacity
larryebaum
bun913
ohbarye
pingcap0315
kyohmizu
kaz29
sansantech
inductor
taishin
yajihum
masahirokawahara
recruitengineers
poteboy
jakevdp
searls
scottboms
lara
denniskardys
bkeepers
revolveconf
michaelherold
cromwellryan
holman
schacon
gr2m
![LARRY EICHENBAUM [email protected] larryebaum larryebaum @larryebaum Solutions Engineer](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_2.jpg){kind=link}
![Uploading logrotate [1.4.0] Uploading cron [1.2.8] Uploading chef-client [3.4.0] Uploaded](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_24.jpg){kind=link}
![node1: run_list: role[base] Exercise: Set run list for managed node](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_28.jpg){kind=link}
![[email protected]'s password: Last login: Sat Mar 28 11:13:08 2015 from](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_29.jpg){kind=link}
![Uploading vault [0.1.0] Uploaded 1 cookbook. Upload Vault Cookbook $](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_121.jpg){kind=link}
![node1: run_list: role[base] recipe[vault] Exercise: Edit run list for managed](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_122.jpg){kind=link}
![Compiling Cookbooks... Recipe: code_generator::cookbook * directory[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab] action create - create](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_151.jpg){kind=link}
![Uploading auditlab [0.1.0] Uploaded 1 cookbook. Upload Cookbook $ knife](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_153.jpg){kind=link}
![node1: run_list: role[base] recipe[vault] recipe[auditlab] Set Node Runlist $ knife](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_154.jpg){kind=link}
![[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_156.jpg){kind=link}
![Uploading auditlab [0.1.0] Uploaded 1 cookbook. Re-upload Cookbook $ knife](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_159.jpg){kind=link}
![[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_176.jpg){kind=link}
![Key Rotation Syntax $#chefGserverGctl#addGclientGkey#ORG_NAME#CLIENT_NAME#\#PATH_TO_KEY#[GG expirationGdate#DATE]#[GGkeyGname#NAME]# $#chefGserverGctl#addGuserGkey#USER_NAME#PATH_TO_KEY#[GGexpirationGdate# DATE]#[GGkeyGname#NAME]# $#chefGserverGctl#deleteGclientGkey#ORG_NAME#CLIENT_NAME#KEY_NAME# $#chefGserverGctl#deleteGclientGkey#USER_NAME#KEY_NAME# $#chefGserverGctl#listGuserGkeys#ORG_NAME#CLIENT_NAME#[GGhideGpublicGkeys]#](https://files.speakerdeck.com/presentations/331de09542a94632988421398dcdeff8/slide_183.jpg){kind=link}
