Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Hashicorp Vault for Secrets Management

Using Hashicorp Vault for Secrets Management

Secrets management for application developers

Presented at Gluecon 2017 in Broomfield, Colorado

Chris Stevens

May 25, 2017
Tweet

More Decks by Chris Stevens

Other Decks in Technology

Transcript

  1. Using Hashicorp Vault for
    Secrets Management
    Chris Stevens

    @stevenscg
    Gluecon 2017
    This work is licensed under a

    Creative Commons Attribution-ShareAlike 3.0 United States License

    View Slide

  2. Chris Stevens
    @stevenscg
    CTO @Traxo

    View Slide

  3. Alternate Title
    Secrets Management for Developers

    View Slide

  4. @stevenscg
    Do you have production credentials
    on your laptop right now?

    View Slide

  5. @stevenscg

    View Slide

  6. @stevenscg
    TOO MANY SECRETS

    View Slide

  7. @stevenscg
    Without enough oversight

    View Slide

  8. @stevenscg
    No audit trail

    View Slide

  9. @stevenscg
    No break-glass options

    View Slide

  10. @stevenscg
    Secrets Management Concepts
    • Privilege Separation
    • Time Limited Access
    • Centralized Management
    • Break-Glass Procedures
    • Secure Introduction
    • Audit Trail

    View Slide

  11. @stevenscg
    Security Responsibilities
    • Infrastructure
    • SSH Keys
    • VPN Certificates
    • External Service Credentials
    • Internal Service Credentials
    • Configuration Management Systems
    Operators

    View Slide

  12. @stevenscg
    Security Responsibilities
    • Sensitive Business Data
    • Service Credentials
    • Database Credentials
    • AWS Services (IAM, EC2, S3)
    • Access Tokens
    Developers

    View Slide

  13. @stevenscg
    Secrets - All the wrong ways

    View Slide

  14. @stevenscg
    Secrets - All the wrong ways

    View Slide

  15. @stevenscg
    Operational Secrets - All the wrong ways
    • In the project repository
    • On the application server
    • Configuration management systems
    • Encrypted files or data-bags
    • Environment variables
    • Password managers

    View Slide

  16. @stevenscg
    Operational Secrets - The right ways
    • HSMs (Hardware-only and $$$$)
    • KMS (Proprietary from Amazon)
    • Keywhiz (Open-source from Square)
    • Knox (Open-source from Pinterest)
    • Confidant (Open-source from Lyft)
    • Red October (Open-source from Cloudflare)

    View Slide

  17. @stevenscg
    https://www.vaultproject.io

    View Slide

  18. @stevenscg
    Vault
    • Vault is an open-source tool for securely accessing secrets
    • Vault provides
    • a unified interface to any secret,
    • rigorous access control and
    • records to a detailed audit log
    • Highly available with certain storage backends
    • Active and engaged open-source community
    • Completed two third-party source code audits

    View Slide

  19. @stevenscg
    Architecture

    View Slide

  20. @stevenscg
    Storage Backends
    • Consul
    • DynamoDB
    • Etcd
    • MySQL
    • PostgreSQL
    • S3
    • Filesystem
    • In-Memory
    • Azure
    • Google Cloud
    • Swift
    • Zookeeper
    Vault stores data in [choose one] ...

    View Slide

  21. @stevenscg
    Secret Backends
    • AWS
    • Consul
    • Databases
    • Generic
    • Transit
    • PKI / TLS
    • MongoDB
    • RabbitMQ
    • SSH
    • TOTP
    Vault manages access to ...

    View Slide

  22. @stevenscg
    Auth Backends
    • AWS
    • Github
    • LDAP
    • MFA
    • Okta
    • Radius
    • PKI

    (TLS Certificates)
    • Tokens
    • Username &

    Password
    Humans and machines authenticate via ...

    View Slide

  23. @stevenscg
    Audit Backends
    • The audit log contains every interaction with Vault
    • Request/Response data hashed with HMAC-SHA256
    • Must be available once configured
    • Syslog
    • File

    View Slide

  24. @stevenscg
    Vault Features
    • Easy to install and manage
    • Privilege Separation / "Two-Man Rule"
    • Break-glass procedures
    • Secure secret storage
    • Encryption as a service
    • Dynamic secrets
    • PKI / TLS

    View Slide

  25. @stevenscg
    Easy to install and maintain
    • A single Go binary for server and client
    • Minimal configuration
    • HTTP API with TLS
    • SDKs for most languages
    • Command-line client and various helpers (SSH, etc)

    View Slide

  26. @stevenscg
    Privilege Separation

    View Slide

  27. @stevenscg
    Privilege Separation
    • Once started, the Vault is in a sealed state
    • Vault must be unsealed before any operation can be performed
    • This is done by providing 1 or more unseal keys
    • Typically 2 or more unseal keys are required
    • Unseal key holders can be any member of your organization

    View Slide

  28. @stevenscg
    Split Master Key
    Bob (Engineering)
    Sue (Engineering)
    Jim (Product)
    Steve (QA)
    Mary (Marketing)
    Shamir's secret sharing algorithm

    View Slide

  29. @stevenscg
    Break-Glass Procedures
    • The "bad guys" are already in
    • What do you do?
    • Revoke specific leases or "vault seal"
    • Requires only a single operator with root privileges
    • Vault will have to be unsealed by a quorum of key holders

    View Slide

  30. @stevenscg
    Secure Secret Storage
    • Example: "Generic" secret backend
    • Secret storage in a path-based hierarchy
    • Vault administrator
    • vault mount secret
    • vault write secret/google/credentials \

    client_id=abc1234

    client_secret=cca8a8d0d9de07e7ef
    • Your application
    • vault read secret/google/credentials

    View Slide

  31. @stevenscg
    Encryption as a Service
    • "Transit" secret backend
    • Handles cryptographic functions on data in-transit using named keys
    • The encrypted ciphertext is stored outside of Vault in your database, S3, etc
    • Natively supports key rotation
    • Additional Features
    • Sign and verify data
    • Generate hashes and HMACs of data
    • Act as a source of random bytes
    • Data keys for storing large objects

    View Slide

  32. @stevenscg
    Dynamic Secrets
    • Example: "Databases" secret backend (MySQL, PostgreSQL, Cassandra)
    • Generates database credentials dynamically
    • Unique credentials for each service instance
    • Vault administrator
    • vault mount database
    • vault write database/config/mysql
    • vault write database/roles/myapp
    • Your application
    • vault read database/creds/myapp

    View Slide

  33. @stevenscg
    Dynamic Secrets
    • Example: "AWS" secret backend
    • Generates AWS access credentials dynamically
    • Based on IAM policies
    • Vault administrator
    • vault mount aws
    • vault write aws/roles/readonly \

    arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess
    • Your application
    • vault read aws/creds/readonly

    View Slide

  34. @stevenscg
    PKI / TLS
    • "PKI" secret backend
    • Generates X.509 TLS certificates dynamically for internal services
    • Recommandations
    • Generate and store your Root CA offline with air-gapped machine
    • Create an intermediate CA for each region (and/or environment)
    • Import the intermediate CA into Vault
    • Vault will issue certificates from the intermediate CA

    View Slide

  35. @stevenscg
    Vault in Practice
    https://github.com/stevenscg/secrets-management-with-vault

    View Slide

  36. @stevenscg
    Vault in Practice
    • Installation
    • Starting and unsealing
    • Roles, policies, mounts
    • Writing and reading secrets
    • Secure introduction
    • Leases

    View Slide

  37. @stevenscg
    Installation
    • curl https://releases.hashicorp.com/vault/0.7.2/vault_0.7.2_linux_amd64.zip && unzip
    vault_0.7.2_linux_amd64.zip
    • Configure via vault.hcl (Consul as the storage backend, TLS, etc)
    • Recommendations
    • Run 2 or more Vault server instances
    • Have a documented process to keep all instances unsealed and practice it regularly
    • Use an HA backend like Consul
    • Use a configuration management system to deploy and maintain policies, roles, etc
    • Be very careful with restarting the vault server process

    View Slide

  38. @stevenscg
    Starting and unsealing
    • vault init
    • vault status
    • vault unseal
    • Recommendations
    • Unseal key threshold of 2-5+ depending on organization size
    • Use Keybase and/or GPG when generating the unseal keys
    • Remove root token after administrator accounts and roles are setup
    • Practice sealing, unsealing, and key regeneration processes

    View Slide

  39. @stevenscg
    Roles, policies, mounts
    • These will vary by organization
    • Recommendations
    • Store these in a separate project repository owned by SecOps
    • Store each role and policy definition in a separate HCL file
    • Develop automation to load and prevent drift
    • Use a configuration management system to deploy them

    View Slide

  40. @stevenscg
    Roles, policies, mounts
    • Example Policies
    • admins
    • operators
    • developers
    • instance/backend
    • service/api

    View Slide

  41. @stevenscg
    Roles, policies, mounts
    • Example Mounts
    • pki
    • transit
    • mysql/db
    • secret/service/api
    • secret/external/twilio

    View Slide

  42. @stevenscg
    Roles, policies, mounts
    • Example Roles
    • pki: pki/roles/vault
    • mysql/db: mysql/db/roles/api
    • token: auth/token/roles/myapp
    • aws-ec2: auth/aws-ec2/role/nomad-server

    View Slide

  43. @stevenscg
    Writing and reading secrets
    • vault write secret/myapp/acces_token token=abcde-1234-5678
    • vault read secret/myapp/access_token
    • Recommendations
    • Develop automation to load data and prevent drift
    • Think about access patterns for humans and machines
    • Develop policies that expose the paths needed

    View Slide

  44. @stevenscg
    Application Configuration
    • Vault-Aware Applications
    • Receive initial token via runtime environment
    • Able to renew or replace token on a schedule (typically 1/2 of TTL)
    • Communicate with Vault via HTTP API for secrets
    • Traditional Applications
    • Require a vault-aware management process like

    consul-template, envconsul, or Nomad
    • Secrets are made available via environment variables
    • Configuration files are templated by management process

    View Slide

  45. @stevenscg
    Application Configuration
    // config.ctmpl
    'app' => [
    'cdn_host' => '{{ key "config/service/myapp/cdn_host" }}',
    ],
    'smtp' => [
    'host' => '{{ key "config/external/smtp/host" }}',
    'port' => '{{ key "config/external/smtp/port" }}',
    'username' => '{{ key "config/external/smtp/username" }}',
    'password' => '{{ with secret "ops/external/smtp" }}{{ .Data.password }}{{ end }}',
    ]
    Vault
    Consul

    View Slide

  46. @stevenscg
    Application Configuration
    // config.php (rendered from config.ctmpl)
    'app' => [
    'cdn_host' => 'cdn.myapp.io',
    ],
    'smtp' => [
    'host' => 'smtp.gmail.com',
    'port' => '465',
    'username' => '[email protected]',
    'password' => 'a super secret smtp password',
    ]
    Vault
    Consul

    View Slide

  47. @stevenscg
    Database Access
    • Vault-Aware Applications
    • Request database credentials on startup via Vault API
    • Periodically renew their own credentials (typically 1/2 of TTL)
    • Revoke credentials on shutdown
    • Traditional Applications
    • Require a vault-aware management process like

    consul-template, envconsul, or Nomad
    • Database configuration files are templated by management process

    View Slide

  48. @stevenscg
    Database Access
    // database.php (static)
    $this->default = [
    'database' => 'Myapp',
    'datasource' => 'Database/Mysql',
    'login' => 'myapp',
    'password' => 'test1234',
    'host' => '10.0.2.15',
    'port' => '3306',
    ];{{ end }}
    Vault
    Consul

    View Slide

  49. @stevenscg
    Database Access
    // database.ctmpl (with dynamic credentials from vault)
    $this->default = [
    'database' => 'Myapp',
    'datasource' => 'Database/Mysql',
    {{- with secret mysql/db/creds/myapp }}
    'login' => '{{ index .Data "username" }}',
    'password' => '{{ index .Data "password" }}',{{- end }}
    'host' => '10.0.2.15',
    'port' => '3306',
    ];
    Vault
    Consul

    View Slide

  50. @stevenscg
    Database Access
    // database.ctmpl (also with the current "db" service from consul)
    {{ range service "db" }}
    $this->default = [
    'database' => 'Myapp',
    'datasource' => 'Database/Mysql',
    {{- with secret mysql/db/creds/myapp }}
    'login' => '{{ index .Data "username" }}',
    'password' => '{{ index .Data "password" }}',{{- end }}
    'host' => '{{ .Address }}',
    'port' => '{{ .Port }}',
    ];{{ end }}
    Vault
    Consul

    View Slide

  51. @stevenscg
    Database Access
    // database.php (rendered from database.ctmpl)
    $this->default = [
    'database' => 'Myapp',
    'datasource' => 'Database/Mysql',
    'login' => 'myapp-token-f85d1',
    'password' => 'a9a48ba6-c366-a473-5fe1-00ee6856b0ab',
    'host' => '10.0.2.15',
    'port' => '3306',
    ];
    Vault
    Consul

    View Slide

  52. @stevenscg
    Using Secrets in Applications
    • For a scripting language like PHP...
    • Application configuration
    • Typically rely on vault-aware management processes
    • Accessing secrets
    • Access the API directly using the VAULT_TOKEN provided by the
    management process
    • Or rely on vault-aware management processes

    View Slide

  53. @stevenscg
    Using Secrets in Applications
    use Jippi\Vault;
    // A valid vault token should be supplied via the app configuration
    // or the VAULT_TOKEN environment variable.
    $vaultSettings = [
    'headers' => [
    'X-Vault-Token' => getenv('VAULT_TOKEN')
    ]
    ];
    $this->vault = new Vault\ServiceFactory($vaultSettings);

    View Slide

  54. @stevenscg
    Using Secrets in Applications
    // Using the Vault API via SDK to decrypt ciphertext with the transit backend
    try {
    $resp = $this->vault->get('data')->write('transit/decrypt/tokens', [
    'ciphertext' => 'vault:v1:abcdefgh' // from the "myapp" database
    ]);
    $tokenData = json_decode($resp->getBody(), true);
    } catch (\Exception $e) {
    \Log::error("vault_decrypt_failed msg=" . $e->getMessage());
    return null;
    }
    $token = json_decode(base64_decode($tokenData['data']['plaintext']), true);

    View Slide

  55. @stevenscg
    Using Secrets in Applications
    • For a threaded language like Go....
    • Use a go-routine to renew token on a schedule
    • Use a go-routine to request secrets and access database
    • 12-Factor Hashiapp talk by Kelsey Hightower
    • He is awesome. Watch it. Run it. Learn it.
    • Video and Repo

    View Slide

  56. @stevenscg
    Secure Introduction
    • The ability to deliver a secret to a process with reasonable assurance of privacy
    • Vault provides "response wrapping" for almost all operations
    • Request a wrapped secret valid for 30 seconds
    • vault read -wrap-ttl=30s secret/myapp/access_token
    • wrapping_token: 9ac83bda-96d9-c27c-bdc9-700a3ee879ca
    • Unwrap the real secret within 30 seconds
    • vault unwrap 9ac83bda-96d9-c27c-bdc9-700a3ee879ca

    View Slide

  57. @stevenscg
    Secure Introduction
    // ec2-auth.sh - runs daily via cron within AWS
    // The parameters pkcs7 and nonce derived from AWS instance metadata
    result=$(
    curl -Ss -XPOST "${vault_addr}/v1/auth/aws-ec2/login" \
    -d '{
    "role":"'"$role_name"'",
    "pkcs7":"'"$pkcs7"'",
    "nonce":"'"$nonce"'"}"'
    )
    token=$( jq -r .auth.client_token <<< "$result" )

    View Slide

  58. @stevenscg
    Secure Introduction
    // ec2-auth.sh - runs daily via cron within AWS
    // write $token to tmpfs
    // readable only to vault-users group
    token_file=/var/run/vault/instance_token
    touch $token_file
    chown root:vault-users $token_file
    chmod 0640 $token_file
    echo "$token" > $token_file

    View Slide

  59. @stevenscg
    Secure Introduction
    // /etc/systemd/system/nomad.service
    [Service]
    User=root
    Group=root
    ExecStart=/opt/nomad/bin/nomad agent -config /etc/nomad
    // For nomad servers only:
    ExecStartPre=/usr/bin/bash -c \
    "/usr/bin/systemctl set-environment VAULT_TOKEN=$(cat /var/run/vault/
    instance_token)"
    ExecStartPost=/usr/bin/bash -c \
    "/usr/bin/systemctl unset-environment VAULT_TOKEN"

    View Slide

  60. @stevenscg
    Leases and TTLs
    • All dynamic secrets and tokens are leased
    • Administrators can configure roles to generate "periodic" tokens
    • A periodic token that is renewed before it expires can live forever
    • Long-running services maintain vault access as long as they remain
    healthy

    View Slide

  61. @stevenscg
    Using Vault with Nomad
    • Nomad - A distributed job scheduler from HashiCorp
    • First-class Docker support, but much more
    • Nomad servers fully manage Vault tokens for tasks
    • Nomad handles templating and service discovery via Consul
    • Project website https://nomadproject.io

    View Slide

  62. @stevenscg
    Using Vault with Nomad
    // myapp nomad job file for the "api" service
    job "myapp-api" {
    type = "service"
    region = "global"
    datacenters = ["local"]
    vault {
    policies = ["service/myapp"]
    }
    group "php-fpm" {}
    group "nginx" {}
    }

    View Slide

  63. @stevenscg
    Innovative Uses
    • Grant access to resources during on-call (threatstack/deputize)
    • HMAC sign API payloads or webhooks (Transit backend)
    • Tokenize data with convergent encryption (Transit backend)
    • Massive internal certificate authority (like LetsEncrypt)

    View Slide

  64. @stevenscg
    Summary
    • Never roll your own crypto
    • Outsource crypto to Vault and your security teams
    • Leases will force changes in how applications are developed and
    deployed
    • Secure introduction is hard, but worth it
    • Well-positioned for information security audits & certifications

    View Slide

  65. @stevenscg
    Summary
    • Start simple
    • Basic static secrets and credentials
    • Add dynamic secrets for databases and services
    • Add Transit to encrypt data stored in your databases

    View Slide

  66. @stevenscg
    Summary
    • Adopting a proper secrets management strategy will take time
    • Involve executives, operators, developers, security personnel
    • Contribute to Vault and related open-source projects
    • Join us in the Gitter chat rooms

    View Slide

  67. Using Hashicorp Vault for
    Secrets Management
    This work is licensed under a

    Creative Commons Attribution-ShareAlike 3.0 United States License

    View Slide

  68. @stevenscg
    Image Credits
    3 Flickr / oscarcn1 https://flic.kr/p/5Kxkn6
    5 http://www.glondon.com/setec.gif
    12 Flickr / reidrac https://flic.kr/p/8a3QUS
    13 Flickr / jadijadi https://flic.kr/p/7uWh2d
    16 HashiCorp https://vaultproject.io
    18 Flickr / airmanmagazine https://flic.kr/p/dSUMBa
    27 HashiCorp https://vaultproject.io

    View Slide