Using Hashicorp Vault for Secrets Management

Using Hashicorp Vault for Secrets Management

Secrets management for application developers

Presented at Gluecon 2017 in Broomfield, Colorado

733c60b9662dea37bc29e2d758e3e9f9?s=128

Chris Stevens

May 25, 2017
Tweet

Transcript

  1. Using Hashicorp Vault for Secrets Management Chris Stevens
 @stevenscg Gluecon

    2017 This work is licensed under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
  2. Chris Stevens @stevenscg CTO @Traxo

  3. Alternate Title Secrets Management for Developers

  4. @stevenscg Do you have production credentials on your laptop right

    now?
  5. @stevenscg

  6. @stevenscg TOO MANY SECRETS

  7. @stevenscg Without enough oversight

  8. @stevenscg No audit trail

  9. @stevenscg No break-glass options

  10. @stevenscg Secrets Management Concepts • Privilege Separation • Time Limited

    Access • Centralized Management • Break-Glass Procedures • Secure Introduction • Audit Trail
  11. @stevenscg Security Responsibilities • Infrastructure • SSH Keys • VPN

    Certificates • External Service Credentials • Internal Service Credentials • Configuration Management Systems Operators
  12. @stevenscg Security Responsibilities • Sensitive Business Data • Service Credentials

    • Database Credentials • AWS Services (IAM, EC2, S3) • Access Tokens Developers
  13. @stevenscg Secrets - All the wrong ways

  14. @stevenscg Secrets - All the wrong ways

  15. @stevenscg Operational Secrets - All the wrong ways • In

    the project repository • On the application server • Configuration management systems • Encrypted files or data-bags • Environment variables • Password managers
  16. @stevenscg Operational Secrets - The right ways • HSMs (Hardware-only

    and $$$$) • KMS (Proprietary from Amazon) • Keywhiz (Open-source from Square) • Knox (Open-source from Pinterest) • Confidant (Open-source from Lyft) • Red October (Open-source from Cloudflare)
  17. @stevenscg https://www.vaultproject.io

  18. @stevenscg Vault • Vault is an open-source tool for securely

    accessing secrets • Vault provides • a unified interface to any secret, • rigorous access control and • records to a detailed audit log • Highly available with certain storage backends • Active and engaged open-source community • Completed two third-party source code audits
  19. @stevenscg Architecture

  20. @stevenscg Storage Backends • Consul • DynamoDB • Etcd •

    MySQL • PostgreSQL • S3 • Filesystem • In-Memory • Azure • Google Cloud • Swift • Zookeeper Vault stores data in [choose one] ...
  21. @stevenscg Secret Backends • AWS • Consul • Databases •

    Generic • Transit • PKI / TLS • MongoDB • RabbitMQ • SSH • TOTP Vault manages access to ...
  22. @stevenscg Auth Backends • AWS • Github • LDAP •

    MFA • Okta • Radius • PKI
 (TLS Certificates) • Tokens • Username &
 Password Humans and machines authenticate via ...
  23. @stevenscg Audit Backends • The audit log contains every interaction

    with Vault • Request/Response data hashed with HMAC-SHA256 • Must be available once configured • Syslog • File
  24. @stevenscg Vault Features • Easy to install and manage •

    Privilege Separation / "Two-Man Rule" • Break-glass procedures • Secure secret storage • Encryption as a service • Dynamic secrets • PKI / TLS
  25. @stevenscg Easy to install and maintain • A single Go

    binary for server and client • Minimal configuration • HTTP API with TLS • SDKs for most languages • Command-line client and various helpers (SSH, etc)
  26. @stevenscg Privilege Separation

  27. @stevenscg Privilege Separation • Once started, the Vault is in

    a sealed state • Vault must be unsealed before any operation can be performed • This is done by providing 1 or more unseal keys • Typically 2 or more unseal keys are required • Unseal key holders can be any member of your organization
  28. @stevenscg Split Master Key Bob (Engineering) Sue (Engineering) Jim (Product)

    Steve (QA) Mary (Marketing) Shamir's secret sharing algorithm
  29. @stevenscg Break-Glass Procedures • The "bad guys" are already in

    • What do you do? • Revoke specific leases or "vault seal" • Requires only a single operator with root privileges • Vault will have to be unsealed by a quorum of key holders
  30. @stevenscg Secure Secret Storage • Example: "Generic" secret backend •

    Secret storage in a path-based hierarchy • Vault administrator • vault mount secret • vault write secret/google/credentials \
 client_id=abc1234
 client_secret=cca8a8d0d9de07e7ef • Your application • vault read secret/google/credentials
  31. @stevenscg Encryption as a Service • "Transit" secret backend •

    Handles cryptographic functions on data in-transit using named keys • The encrypted ciphertext is stored outside of Vault in your database, S3, etc • Natively supports key rotation • Additional Features • Sign and verify data • Generate hashes and HMACs of data • Act as a source of random bytes • Data keys for storing large objects
  32. @stevenscg Dynamic Secrets • Example: "Databases" secret backend (MySQL, PostgreSQL,

    Cassandra) • Generates database credentials dynamically • Unique credentials for each service instance • Vault administrator • vault mount database • vault write database/config/mysql • vault write database/roles/myapp • Your application • vault read database/creds/myapp
  33. @stevenscg Dynamic Secrets • Example: "AWS" secret backend • Generates

    AWS access credentials dynamically • Based on IAM policies • Vault administrator • vault mount aws • vault write aws/roles/readonly \
 arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess • Your application • vault read aws/creds/readonly
  34. @stevenscg PKI / TLS • "PKI" secret backend • Generates

    X.509 TLS certificates dynamically for internal services • Recommandations • Generate and store your Root CA offline with air-gapped machine • Create an intermediate CA for each region (and/or environment) • Import the intermediate CA into Vault • Vault will issue certificates from the intermediate CA
  35. @stevenscg Vault in Practice https://github.com/stevenscg/secrets-management-with-vault

  36. @stevenscg Vault in Practice • Installation • Starting and unsealing

    • Roles, policies, mounts • Writing and reading secrets • Secure introduction • Leases
  37. @stevenscg Installation • curl https://releases.hashicorp.com/vault/0.7.2/vault_0.7.2_linux_amd64.zip && unzip vault_0.7.2_linux_amd64.zip • Configure

    via vault.hcl (Consul as the storage backend, TLS, etc) • Recommendations • Run 2 or more Vault server instances • Have a documented process to keep all instances unsealed and practice it regularly • Use an HA backend like Consul • Use a configuration management system to deploy and maintain policies, roles, etc • Be very careful with restarting the vault server process
  38. @stevenscg Starting and unsealing • vault init • vault status

    • vault unseal • Recommendations • Unseal key threshold of 2-5+ depending on organization size • Use Keybase and/or GPG when generating the unseal keys • Remove root token after administrator accounts and roles are setup • Practice sealing, unsealing, and key regeneration processes
  39. @stevenscg Roles, policies, mounts • These will vary by organization

    • Recommendations • Store these in a separate project repository owned by SecOps • Store each role and policy definition in a separate HCL file • Develop automation to load and prevent drift • Use a configuration management system to deploy them
  40. @stevenscg Roles, policies, mounts • Example Policies • admins •

    operators • developers • instance/backend • service/api
  41. @stevenscg Roles, policies, mounts • Example Mounts • pki •

    transit • mysql/db • secret/service/api • secret/external/twilio
  42. @stevenscg Roles, policies, mounts • Example Roles • pki: pki/roles/vault

    • mysql/db: mysql/db/roles/api • token: auth/token/roles/myapp • aws-ec2: auth/aws-ec2/role/nomad-server
  43. @stevenscg Writing and reading secrets • vault write secret/myapp/acces_token token=abcde-1234-5678

    • vault read secret/myapp/access_token • Recommendations • Develop automation to load data and prevent drift • Think about access patterns for humans and machines • Develop policies that expose the paths needed
  44. @stevenscg Application Configuration • Vault-Aware Applications • Receive initial token

    via runtime environment • Able to renew or replace token on a schedule (typically 1/2 of TTL) • Communicate with Vault via HTTP API for secrets • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Secrets are made available via environment variables • Configuration files are templated by management process
  45. @stevenscg Application Configuration // config.ctmpl 'app' => [ 'cdn_host' =>

    '{{ key "config/service/myapp/cdn_host" }}', ], 'smtp' => [ 'host' => '{{ key "config/external/smtp/host" }}', 'port' => '{{ key "config/external/smtp/port" }}', 'username' => '{{ key "config/external/smtp/username" }}', 'password' => '{{ with secret "ops/external/smtp" }}{{ .Data.password }}{{ end }}', ] Vault Consul
  46. @stevenscg Application Configuration // config.php (rendered from config.ctmpl) 'app' =>

    [ 'cdn_host' => 'cdn.myapp.io', ], 'smtp' => [ 'host' => 'smtp.gmail.com', 'port' => '465', 'username' => 'example@gmail.com', 'password' => 'a super secret smtp password', ] Vault Consul
  47. @stevenscg Database Access • Vault-Aware Applications • Request database credentials

    on startup via Vault API • Periodically renew their own credentials (typically 1/2 of TTL) • Revoke credentials on shutdown • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Database configuration files are templated by management process
  48. @stevenscg Database Access // database.php (static) $this->default = [ 'database'

    => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp', 'password' => 'test1234', 'host' => '10.0.2.15', 'port' => '3306', ];{{ end }} Vault Consul
  49. @stevenscg Database Access // database.ctmpl (with dynamic credentials from vault)

    $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
  50. @stevenscg Database Access // database.ctmpl (also with the current "db"

    service from consul) {{ range service "db" }} $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '{{ .Address }}', 'port' => '{{ .Port }}', ];{{ end }} Vault Consul
  51. @stevenscg Database Access // database.php (rendered from database.ctmpl) $this->default =

    [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp-token-f85d1', 'password' => 'a9a48ba6-c366-a473-5fe1-00ee6856b0ab', 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
  52. @stevenscg Using Secrets in Applications • For a scripting language

    like PHP... • Application configuration • Typically rely on vault-aware management processes • Accessing secrets • Access the API directly using the VAULT_TOKEN provided by the management process • Or rely on vault-aware management processes
  53. @stevenscg Using Secrets in Applications use Jippi\Vault; // A valid

    vault token should be supplied via the app configuration // or the VAULT_TOKEN environment variable. $vaultSettings = [ 'headers' => [ 'X-Vault-Token' => getenv('VAULT_TOKEN') ] ]; $this->vault = new Vault\ServiceFactory($vaultSettings);
  54. @stevenscg Using Secrets in Applications // Using the Vault API

    via SDK to decrypt ciphertext with the transit backend try { $resp = $this->vault->get('data')->write('transit/decrypt/tokens', [ 'ciphertext' => 'vault:v1:abcdefgh' // from the "myapp" database ]); $tokenData = json_decode($resp->getBody(), true); } catch (\Exception $e) { \Log::error("vault_decrypt_failed msg=" . $e->getMessage()); return null; } $token = json_decode(base64_decode($tokenData['data']['plaintext']), true);
  55. @stevenscg Using Secrets in Applications • For a threaded language

    like Go.... • Use a go-routine to renew token on a schedule • Use a go-routine to request secrets and access database • 12-Factor Hashiapp talk by Kelsey Hightower • He is awesome. Watch it. Run it. Learn it. • Video and Repo
  56. @stevenscg Secure Introduction • The ability to deliver a secret

    to a process with reasonable assurance of privacy • Vault provides "response wrapping" for almost all operations • Request a wrapped secret valid for 30 seconds • vault read -wrap-ttl=30s secret/myapp/access_token • wrapping_token: 9ac83bda-96d9-c27c-bdc9-700a3ee879ca • Unwrap the real secret within 30 seconds • vault unwrap 9ac83bda-96d9-c27c-bdc9-700a3ee879ca
  57. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron

    within AWS // The parameters pkcs7 and nonce derived from AWS instance metadata result=$( curl -Ss -XPOST "${vault_addr}/v1/auth/aws-ec2/login" \ -d '{ "role":"'"$role_name"'", "pkcs7":"'"$pkcs7"'", "nonce":"'"$nonce"'"}"' ) token=$( jq -r .auth.client_token <<< "$result" )
  58. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron

    within AWS // write $token to tmpfs // readable only to vault-users group token_file=/var/run/vault/instance_token touch $token_file chown root:vault-users $token_file chmod 0640 $token_file echo "$token" > $token_file
  59. @stevenscg Secure Introduction // /etc/systemd/system/nomad.service [Service] User=root Group=root ExecStart=/opt/nomad/bin/nomad agent

    -config /etc/nomad // For nomad servers only: ExecStartPre=/usr/bin/bash -c \ "/usr/bin/systemctl set-environment VAULT_TOKEN=$(cat /var/run/vault/ instance_token)" ExecStartPost=/usr/bin/bash -c \ "/usr/bin/systemctl unset-environment VAULT_TOKEN"
  60. @stevenscg Leases and TTLs • All dynamic secrets and tokens

    are leased • Administrators can configure roles to generate "periodic" tokens • A periodic token that is renewed before it expires can live forever • Long-running services maintain vault access as long as they remain healthy
  61. @stevenscg Using Vault with Nomad • Nomad - A distributed

    job scheduler from HashiCorp • First-class Docker support, but much more • Nomad servers fully manage Vault tokens for tasks • Nomad handles templating and service discovery via Consul • Project website https://nomadproject.io
  62. @stevenscg Using Vault with Nomad // myapp nomad job file

    for the "api" service job "myapp-api" { type = "service" region = "global" datacenters = ["local"] vault { policies = ["service/myapp"] } group "php-fpm" {} group "nginx" {} }
  63. @stevenscg Innovative Uses • Grant access to resources during on-call

    (threatstack/deputize) • HMAC sign API payloads or webhooks (Transit backend) • Tokenize data with convergent encryption (Transit backend) • Massive internal certificate authority (like LetsEncrypt)
  64. @stevenscg Summary • Never roll your own crypto • Outsource

    crypto to Vault and your security teams • Leases will force changes in how applications are developed and deployed • Secure introduction is hard, but worth it • Well-positioned for information security audits & certifications
  65. @stevenscg Summary • Start simple • Basic static secrets and

    credentials • Add dynamic secrets for databases and services • Add Transit to encrypt data stored in your databases
  66. @stevenscg Summary • Adopting a proper secrets management strategy will

    take time • Involve executives, operators, developers, security personnel • Contribute to Vault and related open-source projects • Join us in the Gitter chat rooms
  67. Using Hashicorp Vault for Secrets Management This work is licensed

    under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
  68. @stevenscg Image Credits 3 Flickr / oscarcn1 https://flic.kr/p/5Kxkn6 5 http://www.glondon.com/setec.gif

    12 Flickr / reidrac https://flic.kr/p/8a3QUS 13 Flickr / jadijadi https://flic.kr/p/7uWh2d 16 HashiCorp https://vaultproject.io 18 Flickr / airmanmagazine https://flic.kr/p/dSUMBa 27 HashiCorp https://vaultproject.io