kv get -recurse aws/vpc/VpcId aws/vpc/VpcId:vpc-e1234567 • Reporting on stuff like - Terraform! consul kv get -recurse app app/dashboard/tf/last-run:2018-08-08T19:15:44Z app/dashboard/tf/terraform:true app/dashboard/tf/version:0.11.7 Steve Huff @hakamadare Tim Hartmann @paxindustria
on every ECS Node • Use Lambdas to add external services, like RDS Complete Service Inventory with Consul • Inventory RDS Assets with Service Discovery Steve Huff @hakamadare Tim Hartmann @paxindustria • Enrich the data with the KV store
use a health check to test Instance Status, and set the node to draining! We also use a healthcheck to verify that the running AMI matches the AMI in the launch config. Speaking of Dog Food... Steve Huff @hakamadare Tim Hartmann @paxindustria
wanted • Containerized Deployment • Upstream / public images • Automated Deployment ◦ With Autoscaling • Automated Initialization • Dynamic Secrets • No more secrets in repos! • No more secrets in repos! Steve Huff @hakamadare Tim Hartmann @paxindustria
didn’t want • Manage a library of private docker images • We did not want to fight the tools • Manage AMI’s ◦ No, No I do not want to manage AMI’s Steve Huff @hakamadare Tim Hartmann @paxindustria
and how we discovered too many ◦ ec2:DescribeTags, I curse thee • Encryption, or lack thereof • ACL system • Setting ECS nodes to DRAINING • Configuring Consul ◦ Sidecar containers and Consul Check with Docker checks Steve Huff @hakamadare Tim Hartmann @paxindustria
(aka, send more cops) ◦ Multi ECS Cluster support ◦ pin the AMI version of the 2nd cluster • Exposing the Consul API on the WAN ◦ Docker host networking ◦ OAUTH Proxy and basic auth :sad: ◦ Potential solution: Vault! • How do clients discover Consul? (we need a service discovery tool!) • Backups (who needs backups?) Steve Huff @hakamadare Tim Hartmann @paxindustria
◦ Autoscaling ◦ Unseal key storage?!! (we need a secret store!) • Shamir's Secret Sharing ◦ Shamir’s what now? ◦ ...and our many conversations about unsealing the vault • AMI’s …(send more cops) • Manual Process to encrypt a list of unseal keys Steve Huff @hakamadare Tim Hartmann @paxindustria
more host mode (awsvpc mode) • Completing the migration to upstream consul image • Better vault initialization • Consul ACL’s, and Initializing the ACL system Steve Huff @hakamadare Tim Hartmann @paxindustria