Vault at Hootsuite

Presented at Vancouver HashiCorp User Group, Aug 23 2017
https://www.meetup.com/Vancouver-HashiCorp-User-Group/events/241966250/

Transcript

1. Bill Monkman Senior Staff Developer @bmonkman Vault at Hootsuite +
2. • The most widely used social media management platform. • Over 15 million users including more than 800 of the Fortune 1000 companies • Written primarily in PHP, Scala, and Go • Running in AWS, with almost 2000 instances What is Hootsuite?
3. • A tool for managing secrets. • Released by Hashicorp about 2 years ago • Written in Go • Controls access to secrets, certificates, etc. • Handles leases, revocation, expiry, key rolling. What is Vault?
4. • Time-consuming process to create certificates • No unified method for secret management ◦ ansible vault, git crypt, ad-hoc ◦ Not great security policy ◦ Poor developer experience What was the problem?
5. • Time-consuming process to create certificates • No unified method for secret management ◦ ansible vault, git crypt, ad-hoc ◦ Not great security policy ◦ Poor developer experience What was the problem?
6. • Unify process for retrieving secrets • Add access-control and auditability • Integrate with existing tools ◦ LDAP/Okta, Consul • Simple, secure way to store Root CA, generate certs How can Vault solve it?
7. 3 stages of secret management • Credentials in Vault, render with Consul template • Credentials in Vault, accessed by applications • Credentials generated by Vault, accessed by applications What did we do?
8. • Implemented for our core (legacy) PHP codebase ◦ Replaced manually encrypted configuration stored in Git • LDAP access for developers • aws-ec2 auth method for instances • “Blessed” by Jenkins • Downside - plain text config Credentials in Vault, render with Consul template
9. Credentials in Vault, render with Consul template {{- with secret "secret/application-config/myapp/db/mysql" }} define("APP_DB_USERNAME", '{{ .Data.username }}'); define("APP_DB_PASSWORD", '{{ .Data.password }}'); {{- end }}
10. • Same LDAP access for users • Added app-role login flow for containers • vault token cron for kubernetes nodes • Downside - still need to manually create and store credentials Credentials in Vault, accessed by applications
11. • “Database” backend allows us to delegate creation of credentials to Vault ◦ Short-lived credentials ◦ Renewal Credentials generated by Vault, accessed by applications
12. • New Root CA created, injected into Vault by our security team ◦ Not globally trusted due to cost • Cert generation now as easy as running 1 Vault command • Kubernetes - create multiple intermediate CAs per subdomain Certificate Generation
13. • Collect GPG keys of keyholders, generate and distribute • Create root token ◦ grant permissions ◦ set up system ◦ hide it! Unseal Keys / Root Token
14. • Collect GPG keys of keyholders, generate and distribute • Create root token ◦ grant permissions ◦ set up system ◦ hide it! Unseal Keys / Root Token
15. • Consul for auto-discovery of Vault servers ◦ Servers already auto-forward requests to leader • Retries in case of network issues, leader election, etc. Vault Binary Wrapper
16. • Consul for auto-discovery of Vault servers ◦ Servers already auto-forward requests to leader • Retries in case of network issues, leader election, etc. Vault Binary Wrapper $ dig +short +tcp -p 8600 @127.0.0.1 vault.service.consul SRV 1 1 8200 vault1.vault.somehostname.com.node.us-east-1.consul. 1 1 8200 vault2.vault.somehostname.com.node.us-east-1.consul.
17. • PKCS#7 validation issues ◦ Problem with a go lib, workaround exists • Keeping track of commands ◦ Document them ◦ Create makefiles ◦ Keep environments in sync • Vault is a toolbox What problems did we run into?
18. • Database backend • Vault SSH • More use of cubbyhole / wrapping • Transit backend Next Steps
19. Thank you! Senior Staff Developer @bmonkman Bill Monkman +