Deploying Consul and Vault on AWS ECS

Deploying Consul and Vault in an automated fashion can be challenging. Automation and operational challenges of deploying Consul and Vault in AWS ECS come with some additional challenges, but also some additional benefits. This session describes those obstacles, benefits, some of the tradeoffs, and how you might deploy Consul and Vault using AWS ECS as a POC or in Production.

Transcript

1. Deploying Consul and Vault on ECS Tim Hartmann Steve Huff
2. Getting Started • Start Here: https://github.com/asicsdigital/consul-vault-ecs-workshop Steve Huff @hakamadare Tim Hartmann @paxindustria
3. Consul? Steve Huff @hakamadare Tim Hartmann @paxindustria
4. Show me, Show me, Show me... • Contextual information consul kv get -recurse aws/vpc/VpcId aws/vpc/VpcId:vpc-e1234567 • Reporting on stuff like - Terraform! consul kv get -recurse app app/dashboard/tf/last-run:2018-08-08T19:15:44Z app/dashboard/tf/terraform:true app/dashboard/tf/version:0.11.7 Steve Huff @hakamadare Tim Hartmann @paxindustria
5. • gliderlabs/registrator makes for easy container discovery ◦ Run sidecar on every ECS Node • Use Lambdas to add external services, like RDS Complete Service Inventory with Consul • Inventory RDS Assets with Service Discovery Steve Huff @hakamadare Tim Hartmann @paxindustria • Enrich the data with the KV store
6. We use a healthcheck to measure task availability. We also use a health check to test Instance Status, and set the node to draining! We also use a healthcheck to verify that the running AMI matches the AMI in the launch config. Speaking of Dog Food... Steve Huff @hakamadare Tim Hartmann @paxindustria
7. Vault?? Steve Huff @hakamadare Tim Hartmann @paxindustria
8. Show me, Show me, Show me... • AWS Secret Engine is AMAZE vault read aws/creds/ci • Revoke! vault lease revoke aws/creds/ci/116b71a9-99fd-ef6f-ee3a-6f530b93c22a • Secure KV is *super* useful ◦ vault read kv/secret • DB Secrets Engine ◦ vault read app/rds/creds/db-ro Steve Huff @hakamadare Tim Hartmann @paxindustria
9. ..scripts! Dynamically Load Secrets … aka - Vault + direnv is pretty amazing. ( sample code in asicsdigital/terraform-reference ) Steve Huff @hakamadare Tim Hartmann @paxindustria
10. AWS ECS??? Steve Huff @hakamadare Tim Hartmann @paxindustria
11. What problems are we trying to solve - what we wanted • Containerized Deployment • Upstream / public images • Automated Deployment ◦ With Autoscaling • Automated Initialization • Dynamic Secrets • No more secrets in repos! • No more secrets in repos! Steve Huff @hakamadare Tim Hartmann @paxindustria
12. What problems are we trying to solve - what we didn’t want • Manage a library of private docker images • We did not want to fight the tools • Manage AMI’s ◦ No, No I do not want to manage AMI’s Steve Huff @hakamadare Tim Hartmann @paxindustria
13. Consul! Steve Huff @hakamadare Tim Hartmann @paxindustria
14. Challenges we ran into deploying Consul • Bootstrapping • Tags, and how we discovered too many ◦ ec2:DescribeTags, I curse thee • Encryption, or lack thereof • ACL system • Setting ECS nodes to DRAINING • Configuring Consul ◦ Sidecar containers and Consul Check with Docker checks Steve Huff @hakamadare Tim Hartmann @paxindustria
15. Challenges we ran into deploying Consul • Bootstrapping • AMI’s! (aka, send more cops) ◦ Multi ECS Cluster support ◦ pin the AMI version of the 2nd cluster • Exposing the Consul API on the WAN ◦ Docker host networking ◦ OAUTH Proxy and basic auth :sad: ◦ Potential solution: Vault! • How do clients discover Consul? (we need a service discovery tool!) • Backups (who needs backups?) Steve Huff @hakamadare Tim Hartmann @paxindustria
16. Vault! Steve Huff @hakamadare Tim Hartmann @paxindustria
17. Challenges we ran into deploying Vault • Initializing the Vault ◦ Autoscaling ◦ Unseal key storage?!! (we need a secret store!) • Shamir's Secret Sharing ◦ Shamir’s what now? ◦ ...and our many conversations about unsealing the vault • AMI’s …(send more cops) • Manual Process to encrypt a list of unseal keys Steve Huff @hakamadare Tim Hartmann @paxindustria
18. Things I dream about doing... • TLS.. TLS..TLS.. • No more host mode (awsvpc mode) • Completing the migration to upstream consul image • Better vault initialization • Consul ACL’s, and Initializing the ACL system Steve Huff @hakamadare Tim Hartmann @paxindustria
19. • Terraform Consul Module ◦ https://github.com/asicsdigital/terraform-aws-consul-cluster • Terraform Vault Module ◦ https://github.com/asicsdigital/terraform-aws-vault • Terraform ECS Module ◦ https://github.com/terraform-community-modules/tf_aws_ecs • Demo! https://github.com/asicsdigital/consul-vault-ecs-workshop • Direnv + Vault sample ◦ https://github.com/asicsdigital/terraform-reference/blob/master/.envrc Steve Huff @hakamadare Tim Hartmann @paxindustria
20. Workshop!