Using Hashicorp Vault for Secrets Management

Slide 1

Slide 1 text

Using Hashicorp Vault for Secrets Management Chris Stevens  @stevenscg Gluecon 2017 This work is licensed under a  Creative Commons Attribution-ShareAlike 3.0 United States License

Slide 2

Slide 2 text

Chris Stevens @stevenscg CTO @Traxo

Slide 3

Slide 3 text

Alternate Title Secrets Management for Developers

Slide 4

Slide 4 text

@stevenscg Do you have production credentials on your laptop right now?

Slide 5

Slide 5 text

@stevenscg

Slide 6

Slide 6 text

@stevenscg TOO MANY SECRETS

Slide 7

Slide 7 text

@stevenscg Without enough oversight

Slide 8

Slide 8 text

@stevenscg No audit trail

Slide 9

Slide 9 text

@stevenscg No break-glass options

Slide 10

Slide 10 text

@stevenscg Secrets Management Concepts • Privilege Separation • Time Limited Access • Centralized Management • Break-Glass Procedures • Secure Introduction • Audit Trail

Slide 11

Slide 11 text

@stevenscg Security Responsibilities • Infrastructure • SSH Keys • VPN Certiﬁcates • External Service Credentials • Internal Service Credentials • Conﬁguration Management Systems Operators

Slide 12

Slide 12 text

@stevenscg Security Responsibilities • Sensitive Business Data • Service Credentials • Database Credentials • AWS Services (IAM, EC2, S3) • Access Tokens Developers

Slide 13

Slide 13 text

@stevenscg Secrets - All the wrong ways

Slide 14

Slide 14 text

@stevenscg Secrets - All the wrong ways

Slide 15

Slide 15 text

@stevenscg Operational Secrets - All the wrong ways • In the project repository • On the application server • Conﬁguration management systems • Encrypted ﬁles or data-bags • Environment variables • Password managers

Slide 16

Slide 16 text

@stevenscg Operational Secrets - The right ways • HSMs (Hardware-only and $$$$) • KMS (Proprietary from Amazon) • Keywhiz (Open-source from Square) • Knox (Open-source from Pinterest) • Conﬁdant (Open-source from Lyft) • Red October (Open-source from Cloudﬂare)

Slide 17

Slide 17 text

@stevenscg https://www.vaultproject.io

Slide 18

Slide 18 text

@stevenscg Vault • Vault is an open-source tool for securely accessing secrets • Vault provides • a uniﬁed interface to any secret, • rigorous access control and • records to a detailed audit log • Highly available with certain storage backends • Active and engaged open-source community • Completed two third-party source code audits

Slide 19

Slide 19 text

@stevenscg Architecture

Slide 20

Slide 20 text

@stevenscg Storage Backends • Consul • DynamoDB • Etcd • MySQL • PostgreSQL • S3 • Filesystem • In-Memory • Azure • Google Cloud • Swift • Zookeeper Vault stores data in [choose one] ...

Slide 21

Slide 21 text

@stevenscg Secret Backends • AWS • Consul • Databases • Generic • Transit • PKI / TLS • MongoDB • RabbitMQ • SSH • TOTP Vault manages access to ...

Slide 22

Slide 22 text

@stevenscg Auth Backends • AWS • Github • LDAP • MFA • Okta • Radius • PKI  (TLS Certiﬁcates) • Tokens • Username &  Password Humans and machines authenticate via ...

Slide 23

Slide 23 text

@stevenscg Audit Backends • The audit log contains every interaction with Vault • Request/Response data hashed with HMAC-SHA256 • Must be available once conﬁgured • Syslog • File

Slide 24

Slide 24 text

@stevenscg Vault Features • Easy to install and manage • Privilege Separation / "Two-Man Rule" • Break-glass procedures • Secure secret storage • Encryption as a service • Dynamic secrets • PKI / TLS

Slide 25

Slide 25 text

@stevenscg Easy to install and maintain • A single Go binary for server and client • Minimal conﬁguration • HTTP API with TLS • SDKs for most languages • Command-line client and various helpers (SSH, etc)

Slide 26

Slide 26 text

@stevenscg Privilege Separation

Slide 27

Slide 27 text

@stevenscg Privilege Separation • Once started, the Vault is in a sealed state • Vault must be unsealed before any operation can be performed • This is done by providing 1 or more unseal keys • Typically 2 or more unseal keys are required • Unseal key holders can be any member of your organization

Slide 28

Slide 28 text

@stevenscg Split Master Key Bob (Engineering) Sue (Engineering) Jim (Product) Steve (QA) Mary (Marketing) Shamir's secret sharing algorithm

Slide 29

Slide 29 text

@stevenscg Break-Glass Procedures • The "bad guys" are already in • What do you do? • Revoke speciﬁc leases or "vault seal" • Requires only a single operator with root privileges • Vault will have to be unsealed by a quorum of key holders

Slide 30

Slide 30 text

@stevenscg Secure Secret Storage • Example: "Generic" secret backend • Secret storage in a path-based hierarchy • Vault administrator • vault mount secret • vault write secret/google/credentials \  client_id=abc1234  client_secret=cca8a8d0d9de07e7ef • Your application • vault read secret/google/credentials

Slide 31

Slide 31 text

@stevenscg Encryption as a Service • "Transit" secret backend • Handles cryptographic functions on data in-transit using named keys • The encrypted ciphertext is stored outside of Vault in your database, S3, etc • Natively supports key rotation • Additional Features • Sign and verify data • Generate hashes and HMACs of data • Act as a source of random bytes • Data keys for storing large objects

Slide 32

Slide 32 text

@stevenscg Dynamic Secrets • Example: "Databases" secret backend (MySQL, PostgreSQL, Cassandra) • Generates database credentials dynamically • Unique credentials for each service instance • Vault administrator • vault mount database • vault write database/conﬁg/mysql • vault write database/roles/myapp • Your application • vault read database/creds/myapp

Slide 33

Slide 33 text

@stevenscg Dynamic Secrets • Example: "AWS" secret backend • Generates AWS access credentials dynamically • Based on IAM policies • Vault administrator • vault mount aws • vault write aws/roles/readonly \  arn=arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess • Your application • vault read aws/creds/readonly

Slide 34

Slide 34 text

@stevenscg PKI / TLS • "PKI" secret backend • Generates X.509 TLS certificates dynamically for internal services • Recommandations • Generate and store your Root CA offline with air-gapped machine • Create an intermediate CA for each region (and/or environment) • Import the intermediate CA into Vault • Vault will issue certificates from the intermediate CA

Slide 35

Slide 35 text

@stevenscg Vault in Practice https://github.com/stevenscg/secrets-management-with-vault

Slide 36

Slide 36 text

@stevenscg Vault in Practice • Installation • Starting and unsealing • Roles, policies, mounts • Writing and reading secrets • Secure introduction • Leases

Slide 37

Slide 37 text

@stevenscg Installation • curl https://releases.hashicorp.com/vault/0.7.2/vault_0.7.2_linux_amd64.zip && unzip vault_0.7.2_linux_amd64.zip • Conﬁgure via vault.hcl (Consul as the storage backend, TLS, etc) • Recommendations • Run 2 or more Vault server instances • Have a documented process to keep all instances unsealed and practice it regularly • Use an HA backend like Consul • Use a conﬁguration management system to deploy and maintain policies, roles, etc • Be very careful with restarting the vault server process

Slide 38

Slide 38 text

@stevenscg Starting and unsealing • vault init • vault status • vault unseal • Recommendations • Unseal key threshold of 2-5+ depending on organization size • Use Keybase and/or GPG when generating the unseal keys • Remove root token after administrator accounts and roles are setup • Practice sealing, unsealing, and key regeneration processes

Slide 39

Slide 39 text

@stevenscg Roles, policies, mounts • These will vary by organization • Recommendations • Store these in a separate project repository owned by SecOps • Store each role and policy definition in a separate HCL file • Develop automation to load and prevent drift • Use a configuration management system to deploy them

Slide 40

Slide 40 text

@stevenscg Roles, policies, mounts • Example Policies • admins • operators • developers • instance/backend • service/api

Slide 41

Slide 41 text

@stevenscg Roles, policies, mounts • Example Mounts • pki • transit • mysql/db • secret/service/api • secret/external/twilio

Slide 42

Slide 42 text

@stevenscg Roles, policies, mounts • Example Roles • pki: pki/roles/vault • mysql/db: mysql/db/roles/api • token: auth/token/roles/myapp • aws-ec2: auth/aws-ec2/role/nomad-server

Slide 43

Slide 43 text

@stevenscg Writing and reading secrets • vault write secret/myapp/acces_token token=abcde-1234-5678 • vault read secret/myapp/access_token • Recommendations • Develop automation to load data and prevent drift • Think about access patterns for humans and machines • Develop policies that expose the paths needed

Slide 44

Slide 44 text

@stevenscg Application Configuration • Vault-Aware Applications • Receive initial token via runtime environment • Able to renew or replace token on a schedule (typically 1/2 of TTL) • Communicate with Vault via HTTP API for secrets • Traditional Applications • Require a vault-aware management process like  consul-template, envconsul, or Nomad • Secrets are made available via environment variables • Configuration files are templated by management process

Slide 45

Slide 45 text

@stevenscg Application Conﬁguration // config.ctmpl 'app' => [ 'cdn_host' => '{{ key "config/service/myapp/cdn_host" }}', ], 'smtp' => [ 'host' => '{{ key "config/external/smtp/host" }}', 'port' => '{{ key "config/external/smtp/port" }}', 'username' => '{{ key "config/external/smtp/username" }}', 'password' => '{{ with secret "ops/external/smtp" }}{{ .Data.password }}{{ end }}', ] Vault Consul

Slide 46

Slide 46 text

@stevenscg Application Conﬁguration // config.php (rendered from config.ctmpl) 'app' => [ 'cdn_host' => 'cdn.myapp.io', ], 'smtp' => [ 'host' => 'smtp.gmail.com', 'port' => '465', 'username' => '[email protected]', 'password' => 'a super secret smtp password', ] Vault Consul

Slide 47

Slide 47 text

@stevenscg Database Access • Vault-Aware Applications • Request database credentials on startup via Vault API • Periodically renew their own credentials (typically 1/2 of TTL) • Revoke credentials on shutdown • Traditional Applications • Require a vault-aware management process like  consul-template, envconsul, or Nomad • Database conﬁguration ﬁles are templated by management process

Slide 48

Slide 48 text

@stevenscg Database Access // database.php (static) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp', 'password' => 'test1234', 'host' => '10.0.2.15', 'port' => '3306', ];{{ end }} Vault Consul

Slide 49

Slide 49 text

@stevenscg Database Access // database.ctmpl (with dynamic credentials from vault) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul

Slide 50

Slide 50 text

@stevenscg Database Access // database.ctmpl (also with the current "db" service from consul) {{ range service "db" }} $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', {{- with secret mysql/db/creds/myapp }} 'login' => '{{ index .Data "username" }}', 'password' => '{{ index .Data "password" }}',{{- end }} 'host' => '{{ .Address }}', 'port' => '{{ .Port }}', ];{{ end }} Vault Consul

Slide 51

Slide 51 text

@stevenscg Database Access // database.php (rendered from database.ctmpl) $this->default = [ 'database' => 'Myapp', 'datasource' => 'Database/Mysql', 'login' => 'myapp-token-f85d1', 'password' => 'a9a48ba6-c366-a473-5fe1-00ee6856b0ab', 'host' => '10.0.2.15', 'port' => '3306', ]; Vault Consul

Slide 52

Slide 52 text

@stevenscg Using Secrets in Applications • For a scripting language like PHP... • Application conﬁguration • Typically rely on vault-aware management processes • Accessing secrets • Access the API directly using the VAULT_TOKEN provided by the management process • Or rely on vault-aware management processes

Slide 53

Slide 53 text

@stevenscg Using Secrets in Applications use Jippi\Vault; // A valid vault token should be supplied via the app configuration // or the VAULT_TOKEN environment variable. $vaultSettings = [ 'headers' => [ 'X-Vault-Token' => getenv('VAULT_TOKEN') ] ]; $this->vault = new Vault\ServiceFactory($vaultSettings);

Slide 54

Slide 54 text

@stevenscg Using Secrets in Applications // Using the Vault API via SDK to decrypt ciphertext with the transit backend try { $resp = $this->vault->get('data')->write('transit/decrypt/tokens', [ 'ciphertext' => 'vault:v1:abcdefgh' // from the "myapp" database ]); $tokenData = json_decode($resp->getBody(), true); } catch (\Exception $e) { \Log::error("vault_decrypt_failed msg=" . $e->getMessage()); return null; } $token = json_decode(base64_decode($tokenData['data']['plaintext']), true);

Slide 55

Slide 55 text

@stevenscg Using Secrets in Applications • For a threaded language like Go.... • Use a go-routine to renew token on a schedule • Use a go-routine to request secrets and access database • 12-Factor Hashiapp talk by Kelsey Hightower • He is awesome. Watch it. Run it. Learn it. • Video and Repo

Slide 56

Slide 56 text

@stevenscg Secure Introduction • The ability to deliver a secret to a process with reasonable assurance of privacy • Vault provides "response wrapping" for almost all operations • Request a wrapped secret valid for 30 seconds • vault read -wrap-ttl=30s secret/myapp/access_token • wrapping_token: 9ac83bda-96d9-c27c-bdc9-700a3ee879ca • Unwrap the real secret within 30 seconds • vault unwrap 9ac83bda-96d9-c27c-bdc9-700a3ee879ca

Slide 57

Slide 57 text

@stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron within AWS // The parameters pkcs7 and nonce derived from AWS instance metadata result=$( curl -Ss -XPOST "${vault_addr}/v1/auth/aws-ec2/login" \ -d '{ "role":"'"$role_name"'", "pkcs7":"'"$pkcs7"'", "nonce":"'"$nonce"'"}"' ) token=$( jq -r .auth.client_token <<< "$result" )

Slide 58

Slide 58 text

@stevenscg Secure Introduction // ec2-auth.sh - runs daily via cron within AWS // write $token to tmpfs // readable only to vault-users group token_file=/var/run/vault/instance_token touch $token_file chown root:vault-users $token_file chmod 0640 $token_file echo "$token" > $token_file

Slide 59

Slide 59 text

@stevenscg Secure Introduction // /etc/systemd/system/nomad.service [Service] User=root Group=root ExecStart=/opt/nomad/bin/nomad agent -config /etc/nomad // For nomad servers only: ExecStartPre=/usr/bin/bash -c \ "/usr/bin/systemctl set-environment VAULT_TOKEN=$(cat /var/run/vault/ instance_token)" ExecStartPost=/usr/bin/bash -c \ "/usr/bin/systemctl unset-environment VAULT_TOKEN"

Slide 60

Slide 60 text

@stevenscg Leases and TTLs • All dynamic secrets and tokens are leased • Administrators can conﬁgure roles to generate "periodic" tokens • A periodic token that is renewed before it expires can live forever • Long-running services maintain vault access as long as they remain healthy

Slide 61

Slide 61 text

@stevenscg Using Vault with Nomad • Nomad - A distributed job scheduler from HashiCorp • First-class Docker support, but much more • Nomad servers fully manage Vault tokens for tasks • Nomad handles templating and service discovery via Consul • Project website https://nomadproject.io

Slide 62

Slide 62 text

@stevenscg Using Vault with Nomad // myapp nomad job file for the "api" service job "myapp-api" { type = "service" region = "global" datacenters = ["local"] vault { policies = ["service/myapp"] } group "php-fpm" {} group "nginx" {} }

Slide 63

Slide 63 text

@stevenscg Innovative Uses • Grant access to resources during on-call (threatstack/deputize) • HMAC sign API payloads or webhooks (Transit backend) • Tokenize data with convergent encryption (Transit backend) • Massive internal certiﬁcate authority (like LetsEncrypt)

Slide 64

Slide 64 text

@stevenscg Summary • Never roll your own crypto • Outsource crypto to Vault and your security teams • Leases will force changes in how applications are developed and deployed • Secure introduction is hard, but worth it • Well-positioned for information security audits & certiﬁcations

Slide 65

Slide 65 text

@stevenscg Summary • Start simple • Basic static secrets and credentials • Add dynamic secrets for databases and services • Add Transit to encrypt data stored in your databases

Slide 66

Slide 66 text

@stevenscg Summary • Adopting a proper secrets management strategy will take time • Involve executives, operators, developers, security personnel • Contribute to Vault and related open-source projects • Join us in the Gitter chat rooms

Slide 67

Slide 67 text

Using Hashicorp Vault for Secrets Management This work is licensed under a  Creative Commons Attribution-ShareAlike 3.0 United States License

Slide 68

Slide 68 text

@stevenscg Image Credits 3 Flickr / oscarcn1 https://flic.kr/p/5Kxkn6 5 http://www.glondon.com/setec.gif 12 Flickr / reidrac https://flic.kr/p/8a3QUS 13 Flickr / jadijadi https://flic.kr/p/7uWh2d 16 HashiCorp https://vaultproject.io 18 Flickr / airmanmagazine https://flic.kr/p/dSUMBa 27 HashiCorp https://vaultproject.io