Using Hashicorp Vault for Secrets Management

Secrets management for application developers

Presented at Gluecon 2017 in Broomfield, Colorado

Transcript

1. Using Hashicorp Vault for Secrets Management Chris Stevens
 @stevenscg Gluecon 2017 This work is licensed under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
2. Chris Stevens @stevenscg CTO @Traxo
3. Alternate Title Secrets Management for Developers
4. @stevenscg Do you have production credentials on your laptop right now?
5. @stevenscg
6. @stevenscg TOO MANY SECRETS
7. @stevenscg Without enough oversight
8. @stevenscg No audit trail
9. @stevenscg No break-glass options
10. @stevenscg Secrets Management Concepts • Privilege Separation • Time Limited Access • Centralized Management • Break-Glass Procedures • Secure Introduction • Audit Trail
11. @stevenscg Security Responsibilities • Infrastructure • SSH Keys • VPN Certificates • External Service Credentials • Internal Service Credentials • Configuration Management Systems Operators
12. @stevenscg Security Responsibilities • Sensitive Business Data • Service Credentials • Database Credentials • AWS Services (IAM, EC2, S3) • Access Tokens Developers
13. @stevenscg Secrets - All the wrong ways
14. @stevenscg Secrets - All the wrong ways
15. @stevenscg Operational Secrets - All the wrong ways • In the project repository • On the application server • Configuration management systems • Encrypted files or data-bags • Environment variables • Password managers
16. @stevenscg Operational Secrets - The right ways • HSMs (Hardware-only and $$$$) • KMS (Proprietary from Amazon) • Keywhiz (Open-source from Square) • Knox (Open-source from Pinterest) • Confidant (Open-source from Lyft) • Red October (Open-source from Cloudflare)
17. @stevenscg https://www.vaultproject.io
18. @stevenscg Vault • Vault is an open-source tool for securely accessing secrets • Vault provides • a unified interface to any secret, • rigorous access control and • records to a detailed audit log • Highly available with certain storage backends • Active and engaged open-source community • Completed two third-party source code audits
19. @stevenscg Architecture
20. @stevenscg Storage Backends • Consul • DynamoDB • Etcd • MySQL • PostgreSQL • S3 • Filesystem • In-Memory • Azure • Google Cloud • Swift • Zookeeper Vault stores data in [choose one] ...
21. @stevenscg Secret Backends • AWS • Consul • Databases • Generic • Transit • PKI / TLS • MongoDB • RabbitMQ • SSH • TOTP Vault manages access to ...
22. @stevenscg Auth Backends • AWS • Github • LDAP • MFA • Okta • Radius • PKI
 (TLS Certificates) • Tokens • Username &
 Password Humans and machines authenticate via ...
23. @stevenscg Audit Backends • The audit log contains every interaction with Vault • Request/Response data hashed with HMAC-SHA256 • Must be available once configured • Syslog • File
24. @stevenscg Vault Features • Easy to install and manage • Privilege Separation / "Two-Man Rule" • Break-glass procedures • Secure secret storage • Encryption as a service • Dynamic secrets • PKI / TLS
25. @stevenscg Easy to install and maintain • A single Go binary for server and client • Minimal configuration • HTTP API with TLS • SDKs for most languages • Command-line client and various helpers (SSH, etc)
26. @stevenscg Privilege Separation
27. @stevenscg Privilege Separation • Once started, the Vault is in a sealed state • Vault must be unsealed before any operation can be performed • This is done by providing 1 or more unseal keys • Typically 2 or more unseal keys are required • Unseal key holders can be any member of your organization
28. @stevenscg Split Master Key Bob (Engineering) Sue (Engineering) Jim (Product) Steve (QA) Mary (Marketing) Shamir's secret sharing algorithm
29. @stevenscg Break-Glass Procedures • The "bad guys" are already in • What do you do? • Revoke specific leases or "vault seal" • Requires only a single operator with root privileges • Vault will have to be unsealed by a quorum of key holders
30. @stevenscg Secure Secret Storage • Example: "Generic" secret backend • Secret storage in a path-based hierarchy • Vault administrator • vault mount secret • vault write secret/google/credentials \
 client_id=abc1234
 client_secret=cca8a8d0d9de07e7ef • Your application • vault read secret/google/credentials
31. @stevenscg Encryption as a Service • "Transit" secret backend • Handles cryptographic functions on data in-transit using named keys • The encrypted ciphertext is stored outside of Vault in your database, S3, etc • Natively supports key rotation • Additional Features • Sign and verify data • Generate hashes and HMACs of data • Act as a source of random bytes • Data keys for storing large objects
32. @stevenscg Dynamic Secrets • Example: "Databases" secret backend (MySQL, PostgreSQL, Cassandra) • Generates database credentials dynamically • Unique credentials for each service instance • Vault administrator • vault mount database • vault write database/config/mysql • vault write database/roles/myapp • Your application • vault read database/creds/myapp
33. @stevenscg Dynamic Secrets • Example: "AWS" secret backend • Generates AWS access credentials dynamically • Based on IAM policies • Vault administrator • vault mount aws • vault write aws/roles/readonly \
 arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess • Your application • vault read aws/creds/readonly
34. @stevenscg PKI / TLS • "PKI" secret backend • Generates X.509 TLS certificates dynamically for internal services • Recommandations • Generate and store your Root CA offline with air-gapped machine • Create an intermediate CA for each region (and/or environment) • Import the intermediate CA into Vault • Vault will issue certificates from the intermediate CA
35. @stevenscg Vault in Practice https://github.com/stevenscg/secrets-management-with-vault
36. @stevenscg Vault in Practice • Installation • Starting and unsealing • Roles, policies, mounts • Writing and reading secrets • Secure introduction • Leases
37. @stevenscg Installation • curl https://releases.hashicorp.com/vault/0.7.2/vault_0.7.2_linux_amd64.zip && unzip vault_0.7.2_linux_amd64.zip • Configure via vault.hcl (Consul as the storage backend, TLS, etc) • Recommendations • Run 2 or more Vault server instances • Have a documented process to keep all instances unsealed and practice it regularly • Use an HA backend like Consul • Use a configuration management system to deploy and maintain policies, roles, etc • Be very careful with restarting the vault server process
38. @stevenscg Starting and unsealing • vault init • vault status • vault unseal • Recommendations • Unseal key threshold of 2-5+ depending on organization size • Use Keybase and/or GPG when generating the unseal keys • Remove root token after administrator accounts and roles are setup • Practice sealing, unsealing, and key regeneration processes
39. @stevenscg Roles, policies, mounts • These will vary by organization • Recommendations • Store these in a separate project repository owned by SecOps • Store each role and policy definition in a separate HCL file • Develop automation to load and prevent drift • Use a configuration management system to deploy them
40. @stevenscg Roles, policies, mounts • Example Policies • admins • operators • developers • instance/backend • service/api
41. @stevenscg Roles, policies, mounts • Example Mounts • pki • transit • mysql/db • secret/service/api • secret/external/twilio
42. @stevenscg Roles, policies, mounts • Example Roles • pki: pki/roles/vault • mysql/db: mysql/db/roles/api • token: auth/token/roles/myapp • aws-ec2: auth/aws-ec2/role/nomad-server
43. @stevenscg Writing and reading secrets • vault write secret/myapp/acces_token token=abcde-1234-5678 • vault read secret/myapp/access_token • Recommendations • Develop automation to load data and prevent drift • Think about access patterns for humans and machines • Develop policies that expose the paths needed
44. @stevenscg Application Configuration • Vault-Aware Applications • Receive initial token via runtime environment • Able to renew or replace token on a schedule (typically 1/2 of TTL) • Communicate with Vault via HTTP API for secrets • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Secrets are made available via environment variables • Configuration files are templated by management process
45. @stevenscg Application Configuration // config.ctmpl 'app' => [ 'cdn_host' => '{{ key "config/service/myapp/cdn_host" }}', ], 'smtp' => [ 'host' => '{{ key "config/external/smtp/host" }}', 'port' => '{{ key "config/external/smtp/port" }}', 'username' => '{{ key "config/external/smtp/username" }}', 'password' => '{{ with secret "ops/external/smtp" }}{{ .Data.password }}{{ end }}', ] Vault Consul
46. @stevenscg Application Configuration // config.php (rendered from config.ctmpl) 'app' => [ 'cdn_host' => 'cdn.myapp.io', ], 'smtp' => [ 'host' => 'smtp.gmail.com', 'port' => '465', 'username' => 'example@gmail.com', 'password' => 'a super secret smtp password', ] Vault Consul
47. @stevenscg Database Access • Vault-Aware Applications • Request database credentials on startup via Vault API • Periodically renew their own credentials (typically 1/2 of TTL) • Revoke credentials on shutdown • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Database configuration files are templated by management process
48. @stevenscg Database Access // database.php (static) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp', 'password' => 'test1234', 'host' => '10.0.2.15', 'port' => '3306', ];{{ end }} Vault Consul
49. @stevenscg Database Access // database.ctmpl (with dynamic credentials from vault) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
50. @stevenscg Database Access // database.ctmpl (also with the current "db" service from consul) {{ range service "db" }} $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '{{ .Address }}', 'port' => '{{ .Port }}', ];{{ end }} Vault Consul
51. @stevenscg Database Access // database.php (rendered from database.ctmpl) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp-token-f85d1', 'password' => 'a9a48ba6-c366-a473-5fe1-00ee6856b0ab', 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
52. @stevenscg Using Secrets in Applications • For a scripting language like PHP... • Application configuration • Typically rely on vault-aware management processes • Accessing secrets • Access the API directly using the VAULT_TOKEN provided by the management process • Or rely on vault-aware management processes
53. @stevenscg Using Secrets in Applications use Jippi\Vault; // A valid vault token should be supplied via the app configuration // or the VAULT_TOKEN environment variable. $vaultSettings = [ 'headers' => [ 'X-Vault-Token' => getenv('VAULT_TOKEN') ] ]; $this->vault = new Vault\ServiceFactory($vaultSettings);
54. @stevenscg Using Secrets in Applications // Using the Vault API via SDK to decrypt ciphertext with the transit backend try { $resp = $this->vault->get('data')->write('transit/decrypt/tokens', [ 'ciphertext' => 'vault:v1:abcdefgh' // from the "myapp" database ]); $tokenData = json_decode($resp->getBody(), true); } catch (\Exception $e) { \Log::error("vault_decrypt_failed msg=" . $e->getMessage()); return null; } $token = json_decode(base64_decode($tokenData['data']['plaintext']), true);
55. @stevenscg Using Secrets in Applications • For a threaded language like Go.... • Use a go-routine to renew token on a schedule • Use a go-routine to request secrets and access database • 12-Factor Hashiapp talk by Kelsey Hightower • He is awesome. Watch it. Run it. Learn it. • Video and Repo
56. @stevenscg Secure Introduction • The ability to deliver a secret to a process with reasonable assurance of privacy • Vault provides "response wrapping" for almost all operations • Request a wrapped secret valid for 30 seconds • vault read -wrap-ttl=30s secret/myapp/access_token • wrapping_token: 9ac83bda-96d9-c27c-bdc9-700a3ee879ca • Unwrap the real secret within 30 seconds • vault unwrap 9ac83bda-96d9-c27c-bdc9-700a3ee879ca
57. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron within AWS // The parameters pkcs7 and nonce derived from AWS instance metadata result=$( curl -Ss -XPOST "${vault_addr}/v1/auth/aws-ec2/login" \ -d '{ "role":"'"$role_name"'", "pkcs7":"'"$pkcs7"'", "nonce":"'"$nonce"'"}"' ) token=$( jq -r .auth.client_token <<< "$result" )
58. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron within AWS // write $token to tmpfs // readable only to vault-users group token_file=/var/run/vault/instance_token touch $token_file chown root:vault-users $token_file chmod 0640 $token_file echo "$token" > $token_file
59. @stevenscg Secure Introduction // /etc/systemd/system/nomad.service [Service] User=root Group=root ExecStart=/opt/nomad/bin/nomad agent -config /etc/nomad // For nomad servers only: ExecStartPre=/usr/bin/bash -c \ "/usr/bin/systemctl set-environment VAULT_TOKEN=$(cat /var/run/vault/ instance_token)" ExecStartPost=/usr/bin/bash -c \ "/usr/bin/systemctl unset-environment VAULT_TOKEN"
60. @stevenscg Leases and TTLs • All dynamic secrets and tokens are leased • Administrators can configure roles to generate "periodic" tokens • A periodic token that is renewed before it expires can live forever • Long-running services maintain vault access as long as they remain healthy
61. @stevenscg Using Vault with Nomad • Nomad - A distributed job scheduler from HashiCorp • First-class Docker support, but much more • Nomad servers fully manage Vault tokens for tasks • Nomad handles templating and service discovery via Consul • Project website https://nomadproject.io
62. @stevenscg Using Vault with Nomad // myapp nomad job file for the "api" service job "myapp-api" { type = "service" region = "global" datacenters = ["local"] vault { policies = ["service/myapp"] } group "php-fpm" {} group "nginx" {} }
63. @stevenscg Innovative Uses • Grant access to resources during on-call (threatstack/deputize) • HMAC sign API payloads or webhooks (Transit backend) • Tokenize data with convergent encryption (Transit backend) • Massive internal certificate authority (like LetsEncrypt)
64. @stevenscg Summary • Never roll your own crypto • Outsource crypto to Vault and your security teams • Leases will force changes in how applications are developed and deployed • Secure introduction is hard, but worth it • Well-positioned for information security audits & certifications
65. @stevenscg Summary • Start simple • Basic static secrets and credentials • Add dynamic secrets for databases and services • Add Transit to encrypt data stored in your databases
66. @stevenscg Summary • Adopting a proper secrets management strategy will take time • Involve executives, operators, developers, security personnel • Contribute to Vault and related open-source projects • Join us in the Gitter chat rooms
67. Using Hashicorp Vault for Secrets Management This work is licensed under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
68. @stevenscg Image Credits 3 Flickr / oscarcn1 https://flic.kr/p/5Kxkn6 5 http://www.glondon.com/setec.gif 12 Flickr / reidrac https://flic.kr/p/8a3QUS 13 Flickr / jadijadi https://flic.kr/p/7uWh2d 16 HashiCorp https://vaultproject.io 18 Flickr / airmanmagazine https://flic.kr/p/dSUMBa 27 HashiCorp https://vaultproject.io