Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deploying Consul and Vault on ECS

Steve Huff
September 24, 2018

Deploying Consul and Vault on ECS

Given at DevOpsDays Boston 2018. Associated workshop content is at https://github.com/asicsdigital/consul-vault-ecs-workshop

Steve Huff

September 24, 2018
Tweet

More Decks by Steve Huff

Other Decks in Technology

Transcript

  1. Show me, Show me, Show me... • Contextual information consul

    kv get -recurse aws/vpc/VpcId aws/vpc/VpcId:vpc-e1234567 • Reporting on stuff like - Terraform! consul kv get -recurse app app/dashboard/tf/last-run:2018-08-08T19:15:44Z app/dashboard/tf/terraform:true app/dashboard/tf/version:0.11.7 Steve Huff @hakamadare Tim Hartmann @paxindustria
  2. • gliderlabs/registrator makes for easy container discovery ◦ Run sidecar

    on every ECS Node • Use Lambdas to add external services, like RDS Complete Service Inventory with Consul • Inventory RDS Assets with Service Discovery Steve Huff @hakamadare Tim Hartmann @paxindustria • Enrich the data with the KV store
  3. We use a healthcheck to measure task availability. We also

    use a health check to test Instance Status, and set the node to draining! We also use a healthcheck to verify that the running AMI matches the AMI in the launch config. Speaking of Dog Food... Steve Huff @hakamadare Tim Hartmann @paxindustria
  4. Show me, Show me, Show me... • AWS Secret Engine

    is AMAZE vault read aws/creds/ci • Revoke! vault lease revoke aws/creds/ci/116b71a9-99fd-ef6f-ee3a-6f530b93c22a • Secure KV is *super* useful ◦ vault read kv/secret • DB Secrets Engine ◦ vault read app/rds/creds/db-ro Steve Huff @hakamadare Tim Hartmann @paxindustria
  5. ..scripts! Dynamically Load Secrets … aka - Vault + direnv

    is pretty amazing. ( sample code in asicsdigital/terraform-reference ) Steve Huff @hakamadare Tim Hartmann @paxindustria
  6. What problems are we trying to solve - what we

    wanted • Containerized Deployment • Upstream / public images • Automated Deployment ◦ With Autoscaling • Automated Initialization • Dynamic Secrets • No more secrets in repos! • No more secrets in repos! Steve Huff @hakamadare Tim Hartmann @paxindustria
  7. What problems are we trying to solve - what we

    didn’t want • Manage a library of private docker images • We did not want to fight the tools • Manage AMI’s ◦ No, No I do not want to manage AMI’s Steve Huff @hakamadare Tim Hartmann @paxindustria
  8. Challenges we ran into deploying Consul • Bootstrapping • Tags,

    and how we discovered too many ◦ ec2:DescribeTags, I curse thee • Encryption, or lack thereof • ACL system • Setting ECS nodes to DRAINING • Configuring Consul ◦ Sidecar containers and Consul Check with Docker checks Steve Huff @hakamadare Tim Hartmann @paxindustria
  9. Challenges we ran into deploying Consul • Bootstrapping • AMI’s!

    (aka, send more cops) ◦ Multi ECS Cluster support ◦ pin the AMI version of the 2nd cluster • Exposing the Consul API on the WAN ◦ Docker host networking ◦ OAUTH Proxy and basic auth :sad: ◦ Potential solution: Vault! • How do clients discover Consul? (we need a service discovery tool!) • Backups (who needs backups?) Steve Huff @hakamadare Tim Hartmann @paxindustria
  10. Challenges we ran into deploying Vault • Initializing the Vault

    ◦ Autoscaling ◦ Unseal key storage?!! (we need a secret store!) • Shamir's Secret Sharing ◦ Shamir’s what now? ◦ ...and our many conversations about unsealing the vault • AMI’s …(send more cops) • Manual Process to encrypt a list of unseal keys Steve Huff @hakamadare Tim Hartmann @paxindustria
  11. Things I dream about doing... • TLS.. TLS..TLS.. • No

    more host mode (awsvpc mode) • Completing the migration to upstream consul image • Better vault initialization • Consul ACL’s, and Initializing the ACL system Steve Huff @hakamadare Tim Hartmann @paxindustria
  12. • Terraform Consul Module ◦ https://github.com/asicsdigital/terraform-aws-consul-cluster • Terraform Vault Module

    ◦ https://github.com/asicsdigital/terraform-aws-vault • Terraform ECS Module ◦ https://github.com/terraform-community-modules/tf_aws_ecs • Demo! https://github.com/asicsdigital/consul-vault-ecs-workshop • Direnv + Vault sample ◦ https://github.com/asicsdigital/terraform-reference/blob/master/.envrc Steve Huff @hakamadare Tim Hartmann @paxindustria