Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Year with ECS

A Year with ECS

When we first began evaluating solutions for managing containers, we knew very little about our options. We evaluated ECS, Fleet, and Kubernetes--eventually deciding to use Fleet. ECS had only just been released to general availability, and we were very unfamiliar with it. Kubernetes was significantly more complexity than we wanted to own. Ultimately, Fleet was the easiest option. Once we got to the double digits in services and had multiple developers launching and maintaining services, we found that Fleet was lacking in many ways, so we decided to reevaluate our options. Migrating to ECS allowed us to consolidate responsibility for managing containers in AWS--allowing us to focus on shipping features instead of building infrastructure tooling.

In this talk, we will explore the migration process, lessons learned, and what we plan to do to improve our internal developer experience for ECS.

F1d0ab801e972d49592bf75d69b7bf65?s=128

Greg Poirier

May 10, 2016
Tweet

Transcript

  1. A Year With ECS Greg Poirier CTO - Opsee

  2. Origin Story • Fifteen years of operations experience • Observability,

    infrastructure architecture • Operations/Systems/Software engineer
  3. Opsee • Effortless monitoring for your AWS environment • Reacts

    to changes in your infrastructure • Monitoring that stays out of the way
  4. Infrastructure: What do we want? • As little code as

    possible • Reproducibility • Easy engineer onboarding • Automate all the things
  5. Development Process • Open PR • Run build against PR

    • Merge to master • Build master and publish artifacts • Deploy
  6. Reproducibility • If it builds in CI, it should build

    locally. • CI should not be a magical, mystical place that makes tests pass. • In case of emergency, break glass and deploy from my laptop to production with confidence.
  7. I kind of like Docker.

  8. Docker? • "Docker is great for local dev." • "I

    don't do anything in production with Docker yet." • "I'm not sure Docker is ready for production."
  9. Really?

  10. One year of Docker and ECS later… • A year

    of Docker in production • Started with CoreOS's Fleet • Migrated to ECS a year ago
  11. Why Fleet? • It was there (we use CoreOS) •

    Trivial to setup/use (already running) • It did a thing (deployed services to VMs)
  12. What could go wrong?

  13. Highly problematic. • Etcd outage -> cannot deploy • Etcd

    and Docker share same disk space by default • Systemd • (Very) Bad default configurations
  14. How do I even Docker? • Container fills up /var/lib/docker

    • Node crashes • Naïve scheduling in Fleet causes containers to be scheduled to new nodes • Next node crashes…
  15. Ffffffffffffffleet

  16. We could fix Fleet, but… • Seed-stage startup • Must

    focus solely on our own product • Just Make it Work Mode
  17. What are we trying to do here?

  18. Make Building and Deploying Easy • Commit • Push •

    Merge • Deploy
  19. Make building and deploying easy. • Commit • Push •

    Merge • Done.
  20. How do we do it?

  21. Components we need. • Container scheduler • Continuous integration •

    Automated deploys
  22. Don't build what you don't sell.

  23. What did we use? • Container scheduler - ECS •

    Continuous integration - CircleCI • Automated deploys - Ansible
  24. Allons-y!

  25. How do we ECS? • Deploy ecs-agent with CoreOS cloud-config

    • Service and Task definitions in YAML w/ Ansible • Ansible playbook to deploy Task definitions / trigger service updates
  26. ECS on CoreOS w/ cloud-config • Cloud-config w/ EC2 instance

    userdata • Add a systemd unit and configuration data • We do this with Ansible • https://coreos.com/os/docs/latest/booting- on-ecs.htmlECS in Ansible
  27. ECS and Ansible • Task and service definitions in YAML

    • Small Ansible libraries using boto3 for task/ service definition • Deploy with a simple playbook that creates a task definition and updates service
  28. But not everything is totally automated…

  29. It’s true. • Commit • Push • Merge • Push-button

    deploy
  30. What did we do Wrong? • Didn’t use CloudFormation •

    Deployed 'latest' tag • Used default configurations • Poor management of service configuration material
  31. If I could turn back time… • Tag Docker image

    w/ Git SHA • Use CloudFormation: stack all the things • Remember that defaults are usually bad • Configuration…
  32. What about configuration? • Configuration data stored in S3 with

    KMS encryption • Startup script pulls config data from S3 • Sources the config data (all in env vars) • Starts the service
  33. What to do instead? • Separate container for config. •

    Export config file as a volume. • Mount config volume in service’s container.
  34. Mostly, I am happy… Mostly.

  35. What does ECS do well? • Easy to deploy •

    “Transactional” changes to ECS cluster • ELB Integration • Keeps pace with Docker
  36. Let ECS tell you its secrets. • Task-specific information •

    Exposed ports • Service information • ELB name, Service name, desired count • Metrics • CPU Utilization, Memory utilization
  37. Go a step further. • CloudWatch -> Lambda -> Autoscale

    ECS Services • Compute nodes in ASG • CloudWatch on cluster utilization to scale up compute ASG
  38. What is ECS missing? • IP-per-Container w/ ELB integration •

    Persistent storage • Per-Task/Service IAM roles • Per-Task/Service security groups
  39. IP per Container • Don’t have to worry about port

    collisions. • Directly addressable containers are useful. • Plenty of private IP space to go around.
  40. Least-Privileged Access • Every service has the same IAM privileges

    as the service with the most IAM privileges. • This goes directly against AWS best practices.
  41. I dreamed a dream…

  42. What do I want, though? • I want to think

    less about infrastructure. • No more “instances.” • I want a framework for building and deploying applications.
  43. Questions?

  44. Thank you! • Thanks AdvAWS and sponsors. • Thank you

    for attending. • Greg Poirier - @grepory • Opsee - https://opsee.com - @GetOpsee