Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Hashicorp Vault for Secrets Management

Using Hashicorp Vault for Secrets Management

Secrets management for application developers

Presented at Gluecon 2017 in Broomfield, Colorado

Chris Stevens

May 25, 2017
Tweet

More Decks by Chris Stevens

Other Decks in Technology

Transcript

  1. Using Hashicorp Vault for Secrets Management Chris Stevens
 @stevenscg Gluecon

    2017 This work is licensed under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
  2. Chris Stevens @stevenscg CTO @Traxo

  3. Alternate Title Secrets Management for Developers

  4. @stevenscg Do you have production credentials on your laptop right

    now?
  5. @stevenscg

  6. @stevenscg TOO MANY SECRETS

  7. @stevenscg Without enough oversight

  8. @stevenscg No audit trail

  9. @stevenscg No break-glass options

  10. @stevenscg Secrets Management Concepts • Privilege Separation • Time Limited

    Access • Centralized Management • Break-Glass Procedures • Secure Introduction • Audit Trail
  11. @stevenscg Security Responsibilities • Infrastructure • SSH Keys • VPN

    Certificates • External Service Credentials • Internal Service Credentials • Configuration Management Systems Operators
  12. @stevenscg Security Responsibilities • Sensitive Business Data • Service Credentials

    • Database Credentials • AWS Services (IAM, EC2, S3) • Access Tokens Developers
  13. @stevenscg Secrets - All the wrong ways

  14. @stevenscg Secrets - All the wrong ways

  15. @stevenscg Operational Secrets - All the wrong ways • In

    the project repository • On the application server • Configuration management systems • Encrypted files or data-bags • Environment variables • Password managers
  16. @stevenscg Operational Secrets - The right ways • HSMs (Hardware-only

    and $$$$) • KMS (Proprietary from Amazon) • Keywhiz (Open-source from Square) • Knox (Open-source from Pinterest) • Confidant (Open-source from Lyft) • Red October (Open-source from Cloudflare)
  17. @stevenscg https://www.vaultproject.io

  18. @stevenscg Vault • Vault is an open-source tool for securely

    accessing secrets • Vault provides • a unified interface to any secret, • rigorous access control and • records to a detailed audit log • Highly available with certain storage backends • Active and engaged open-source community • Completed two third-party source code audits
  19. @stevenscg Architecture

  20. @stevenscg Storage Backends • Consul • DynamoDB • Etcd •

    MySQL • PostgreSQL • S3 • Filesystem • In-Memory • Azure • Google Cloud • Swift • Zookeeper Vault stores data in [choose one] ...
  21. @stevenscg Secret Backends • AWS • Consul • Databases •

    Generic • Transit • PKI / TLS • MongoDB • RabbitMQ • SSH • TOTP Vault manages access to ...
  22. @stevenscg Auth Backends • AWS • Github • LDAP •

    MFA • Okta • Radius • PKI
 (TLS Certificates) • Tokens • Username &
 Password Humans and machines authenticate via ...
  23. @stevenscg Audit Backends • The audit log contains every interaction

    with Vault • Request/Response data hashed with HMAC-SHA256 • Must be available once configured • Syslog • File
  24. @stevenscg Vault Features • Easy to install and manage •

    Privilege Separation / "Two-Man Rule" • Break-glass procedures • Secure secret storage • Encryption as a service • Dynamic secrets • PKI / TLS
  25. @stevenscg Easy to install and maintain • A single Go

    binary for server and client • Minimal configuration • HTTP API with TLS • SDKs for most languages • Command-line client and various helpers (SSH, etc)
  26. @stevenscg Privilege Separation

  27. @stevenscg Privilege Separation • Once started, the Vault is in

    a sealed state • Vault must be unsealed before any operation can be performed • This is done by providing 1 or more unseal keys • Typically 2 or more unseal keys are required • Unseal key holders can be any member of your organization
  28. @stevenscg Split Master Key Bob (Engineering) Sue (Engineering) Jim (Product)

    Steve (QA) Mary (Marketing) Shamir's secret sharing algorithm
  29. @stevenscg Break-Glass Procedures • The "bad guys" are already in

    • What do you do? • Revoke specific leases or "vault seal" • Requires only a single operator with root privileges • Vault will have to be unsealed by a quorum of key holders
  30. @stevenscg Secure Secret Storage • Example: "Generic" secret backend •

    Secret storage in a path-based hierarchy • Vault administrator • vault mount secret • vault write secret/google/credentials \
 client_id=abc1234
 client_secret=cca8a8d0d9de07e7ef • Your application • vault read secret/google/credentials
  31. @stevenscg Encryption as a Service • "Transit" secret backend •

    Handles cryptographic functions on data in-transit using named keys • The encrypted ciphertext is stored outside of Vault in your database, S3, etc • Natively supports key rotation • Additional Features • Sign and verify data • Generate hashes and HMACs of data • Act as a source of random bytes • Data keys for storing large objects
  32. @stevenscg Dynamic Secrets • Example: "Databases" secret backend (MySQL, PostgreSQL,

    Cassandra) • Generates database credentials dynamically • Unique credentials for each service instance • Vault administrator • vault mount database • vault write database/config/mysql • vault write database/roles/myapp • Your application • vault read database/creds/myapp
  33. @stevenscg Dynamic Secrets • Example: "AWS" secret backend • Generates

    AWS access credentials dynamically • Based on IAM policies • Vault administrator • vault mount aws • vault write aws/roles/readonly \
 arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess • Your application • vault read aws/creds/readonly
  34. @stevenscg PKI / TLS • "PKI" secret backend • Generates

    X.509 TLS certificates dynamically for internal services • Recommandations • Generate and store your Root CA offline with air-gapped machine • Create an intermediate CA for each region (and/or environment) • Import the intermediate CA into Vault • Vault will issue certificates from the intermediate CA
  35. @stevenscg Vault in Practice https://github.com/stevenscg/secrets-management-with-vault

  36. @stevenscg Vault in Practice • Installation • Starting and unsealing

    • Roles, policies, mounts • Writing and reading secrets • Secure introduction • Leases
  37. @stevenscg Installation • curl https://releases.hashicorp.com/vault/0.7.2/vault_0.7.2_linux_amd64.zip && unzip vault_0.7.2_linux_amd64.zip • Configure

    via vault.hcl (Consul as the storage backend, TLS, etc) • Recommendations • Run 2 or more Vault server instances • Have a documented process to keep all instances unsealed and practice it regularly • Use an HA backend like Consul • Use a configuration management system to deploy and maintain policies, roles, etc • Be very careful with restarting the vault server process
  38. @stevenscg Starting and unsealing • vault init • vault status

    • vault unseal • Recommendations • Unseal key threshold of 2-5+ depending on organization size • Use Keybase and/or GPG when generating the unseal keys • Remove root token after administrator accounts and roles are setup • Practice sealing, unsealing, and key regeneration processes
  39. @stevenscg Roles, policies, mounts • These will vary by organization

    • Recommendations • Store these in a separate project repository owned by SecOps • Store each role and policy definition in a separate HCL file • Develop automation to load and prevent drift • Use a configuration management system to deploy them
  40. @stevenscg Roles, policies, mounts • Example Policies • admins •

    operators • developers • instance/backend • service/api
  41. @stevenscg Roles, policies, mounts • Example Mounts • pki •

    transit • mysql/db • secret/service/api • secret/external/twilio
  42. @stevenscg Roles, policies, mounts • Example Roles • pki: pki/roles/vault

    • mysql/db: mysql/db/roles/api • token: auth/token/roles/myapp • aws-ec2: auth/aws-ec2/role/nomad-server
  43. @stevenscg Writing and reading secrets • vault write secret/myapp/acces_token token=abcde-1234-5678

    • vault read secret/myapp/access_token • Recommendations • Develop automation to load data and prevent drift • Think about access patterns for humans and machines • Develop policies that expose the paths needed
  44. @stevenscg Application Configuration • Vault-Aware Applications • Receive initial token

    via runtime environment • Able to renew or replace token on a schedule (typically 1/2 of TTL) • Communicate with Vault via HTTP API for secrets • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Secrets are made available via environment variables • Configuration files are templated by management process
  45. @stevenscg Application Configuration // config.ctmpl 'app' => [ 'cdn_host' =>

    '{{ key "config/service/myapp/cdn_host" }}', ], 'smtp' => [ 'host' => '{{ key "config/external/smtp/host" }}', 'port' => '{{ key "config/external/smtp/port" }}', 'username' => '{{ key "config/external/smtp/username" }}', 'password' => '{{ with secret "ops/external/smtp" }}{{ .Data.password }}{{ end }}', ] Vault Consul
  46. @stevenscg Application Configuration // config.php (rendered from config.ctmpl) 'app' =>

    [ 'cdn_host' => 'cdn.myapp.io', ], 'smtp' => [ 'host' => 'smtp.gmail.com', 'port' => '465', 'username' => 'example@gmail.com', 'password' => 'a super secret smtp password', ] Vault Consul
  47. @stevenscg Database Access • Vault-Aware Applications • Request database credentials

    on startup via Vault API • Periodically renew their own credentials (typically 1/2 of TTL) • Revoke credentials on shutdown • Traditional Applications • Require a vault-aware management process like
 consul-template, envconsul, or Nomad • Database configuration files are templated by management process
  48. @stevenscg Database Access // database.php (static) $this->default = [ 'database'

    => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp', 'password' => 'test1234', 'host' => '10.0.2.15', 'port' => '3306', ];{{ end }} Vault Consul
  49. @stevenscg Database Access // database.ctmpl (with dynamic credentials from vault)

    $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
  50. @stevenscg Database Access // database.ctmpl (also with the current "db"

    service from consul) {{ range service "db" }} $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '{{ .Address }}', 'port' => '{{ .Port }}', ];{{ end }} Vault Consul
  51. @stevenscg Database Access // database.php (rendered from database.ctmpl) $this->default =

    [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp-token-f85d1', 'password' => 'a9a48ba6-c366-a473-5fe1-00ee6856b0ab', 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul
  52. @stevenscg Using Secrets in Applications • For a scripting language

    like PHP... • Application configuration • Typically rely on vault-aware management processes • Accessing secrets • Access the API directly using the VAULT_TOKEN provided by the management process • Or rely on vault-aware management processes
  53. @stevenscg Using Secrets in Applications use Jippi\Vault; // A valid

    vault token should be supplied via the app configuration // or the VAULT_TOKEN environment variable. $vaultSettings = [ 'headers' => [ 'X-Vault-Token' => getenv('VAULT_TOKEN') ] ]; $this->vault = new Vault\ServiceFactory($vaultSettings);
  54. @stevenscg Using Secrets in Applications // Using the Vault API

    via SDK to decrypt ciphertext with the transit backend try { $resp = $this->vault->get('data')->write('transit/decrypt/tokens', [ 'ciphertext' => 'vault:v1:abcdefgh' // from the "myapp" database ]); $tokenData = json_decode($resp->getBody(), true); } catch (\Exception $e) { \Log::error("vault_decrypt_failed msg=" . $e->getMessage()); return null; } $token = json_decode(base64_decode($tokenData['data']['plaintext']), true);
  55. @stevenscg Using Secrets in Applications • For a threaded language

    like Go.... • Use a go-routine to renew token on a schedule • Use a go-routine to request secrets and access database • 12-Factor Hashiapp talk by Kelsey Hightower • He is awesome. Watch it. Run it. Learn it. • Video and Repo
  56. @stevenscg Secure Introduction • The ability to deliver a secret

    to a process with reasonable assurance of privacy • Vault provides "response wrapping" for almost all operations • Request a wrapped secret valid for 30 seconds • vault read -wrap-ttl=30s secret/myapp/access_token • wrapping_token: 9ac83bda-96d9-c27c-bdc9-700a3ee879ca • Unwrap the real secret within 30 seconds • vault unwrap 9ac83bda-96d9-c27c-bdc9-700a3ee879ca
  57. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron

    within AWS // The parameters pkcs7 and nonce derived from AWS instance metadata result=$( curl -Ss -XPOST "${vault_addr}/v1/auth/aws-ec2/login" \ -d '{ "role":"'"$role_name"'", "pkcs7":"'"$pkcs7"'", "nonce":"'"$nonce"'"}"' ) token=$( jq -r .auth.client_token <<< "$result" )
  58. @stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron

    within AWS // write $token to tmpfs // readable only to vault-users group token_file=/var/run/vault/instance_token touch $token_file chown root:vault-users $token_file chmod 0640 $token_file echo "$token" > $token_file
  59. @stevenscg Secure Introduction // /etc/systemd/system/nomad.service [Service] User=root Group=root ExecStart=/opt/nomad/bin/nomad agent

    -config /etc/nomad // For nomad servers only: ExecStartPre=/usr/bin/bash -c \ "/usr/bin/systemctl set-environment VAULT_TOKEN=$(cat /var/run/vault/ instance_token)" ExecStartPost=/usr/bin/bash -c \ "/usr/bin/systemctl unset-environment VAULT_TOKEN"
  60. @stevenscg Leases and TTLs • All dynamic secrets and tokens

    are leased • Administrators can configure roles to generate "periodic" tokens • A periodic token that is renewed before it expires can live forever • Long-running services maintain vault access as long as they remain healthy
  61. @stevenscg Using Vault with Nomad • Nomad - A distributed

    job scheduler from HashiCorp • First-class Docker support, but much more • Nomad servers fully manage Vault tokens for tasks • Nomad handles templating and service discovery via Consul • Project website https://nomadproject.io
  62. @stevenscg Using Vault with Nomad // myapp nomad job file

    for the "api" service job "myapp-api" { type = "service" region = "global" datacenters = ["local"] vault { policies = ["service/myapp"] } group "php-fpm" {} group "nginx" {} }
  63. @stevenscg Innovative Uses • Grant access to resources during on-call

    (threatstack/deputize) • HMAC sign API payloads or webhooks (Transit backend) • Tokenize data with convergent encryption (Transit backend) • Massive internal certificate authority (like LetsEncrypt)
  64. @stevenscg Summary • Never roll your own crypto • Outsource

    crypto to Vault and your security teams • Leases will force changes in how applications are developed and deployed • Secure introduction is hard, but worth it • Well-positioned for information security audits & certifications
  65. @stevenscg Summary • Start simple • Basic static secrets and

    credentials • Add dynamic secrets for databases and services • Add Transit to encrypt data stored in your databases
  66. @stevenscg Summary • Adopting a proper secrets management strategy will

    take time • Involve executives, operators, developers, security personnel • Contribute to Vault and related open-source projects • Join us in the Gitter chat rooms
  67. Using Hashicorp Vault for Secrets Management This work is licensed

    under a
 Creative Commons Attribution-ShareAlike 3.0 United States License
  68. @stevenscg Image Credits 3 Flickr / oscarcn1 https://flic.kr/p/5Kxkn6 5 http://www.glondon.com/setec.gif

    12 Flickr / reidrac https://flic.kr/p/8a3QUS 13 Flickr / jadijadi https://flic.kr/p/7uWh2d 16 HashiCorp https://vaultproject.io 18 Flickr / airmanmagazine https://flic.kr/p/dSUMBa 27 HashiCorp https://vaultproject.io