Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fancy Containers (Velocity SF 2018)

E5b21a91ebcda83d51e9d59fdc0883cc?s=47 Abby Fuller
June 14, 2018

Fancy Containers (Velocity SF 2018)


Abby Fuller

June 14, 2018


  1. Containers on AWS: let’s get fancy Abby Fuller, Developer Relations,

    AWS @abbyfuller
  2. What we’re going to talk about • First things first

    • You’ve deployed your cluster- now what? • Speeding up deployments • Cleaning up after yourself • Fun with user-data • A little bit on monitoring/observability/debugging
  3. First things first…

  4. We hear a lot about “how to get started with

  5. This leaves you with a (hopefully) working cluster, and not

    much else.
  6. So what happens after?

  7. None
  8. Speeding up deployments

  9. In a lot of cases, first problem that crops up

    is deployments. As in, they’re SLOW. Lots of reasons this happens!
  10. Before you blame your orchestration tool…

  11. Check your image sizes Larger images mean slower deploys (slower

    to build/push/pull) Image size is determined (mostly), by the number of layers your image has, and how large those layers are. A few tips: • Use shared base images wherever possible • Limit the data written to the container layer • Chain RUN statements • Prevent cache misses at build for as long as possible
  12. CACHE Most importantly for images, understand the Docker cache

  13. Smaller images means understanding the cache Starting from the parent

    instruction, Docker will look at each following instruction to see if it matches the cached version. Only ADD and COPY will look at checksums for a match Other than ADD and COPY, only the string of the command is used, not the contents of the files. Once the cache is broken, every subsequent layer is built again.
  14. How quickly your images are pulling from your registry is

    important for speeding up your deploys: smaller, lighter images can build and push more quickly during your CI/CD process, and pull more quickly during your deploys.
  15. Next up for faster deployments: EC2 settings

  16. “Secret”* ASG settings • There are some less intuitive settings

    (UX! ) that can impact how quickly your group can a) autoscale, b) how long it takes to mark your containers healthy.
  17. What are these? Default cooldown: how long the autoscaling group

    will wait before evaluating the rule again Health check grace period: amount of time, in seconds, that the autoscaling group will wait before health checking the service.
  18. Since services run through ECS and EKS are backed by

    autoscaling groups, changing these settings can cut down the amount of time it takes your service to a) scale, b) become healthy after starting.
  19. You can also customize settings at the load balancer health

    check level!
  20. None
  21. Advanced health check settings can also influence your deployments! This

    happens through a few settings: Healthy and unhealthy thresholds determine how many times your health check endpoint is tried before your container is declared healthy or unhealthy. The shorter you can make the interval between these checks, the faster your deployment will be. Be careful though- you need to give your application enough time to actually pass.
  22. Moving off deployments, let’s get more efficient

  23. Garbage collection for fun and profit

  24. Garbage collection for containers is used to remove things like

    dangling/untagged images, containers, and volumes. Most orchestration platforms (like ECS/EKS, or Kubernetes) will do some garbage collection for you, but it’s not always enough. Too many unused images, containers and volumes will: • Steal your disk space • Wake you up • Cost you cash money
  25. Tune the image cleanup parameters with ECS • With ECS,

    you can tune the options you have available for garbage collection. • If you want to cleanup more aggressively, you can look at setting these ecs-agent options*: ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION ECS_IMAGE_CLEANUP_INTERVAL ECS_NUM_IMAGES_DELETE_PER_CYCLE * don’t know how to set ecs-agent options? We’ll talk about it in a second!
  26. Tune the image cleanup parameters with kubelet • Kubelet also

    has a cleanup function! • Can clean up images and containers, controlled through kubelet flags • Image collection is based on disk usage • Container collection is based on flags (or the defaults) • minimum-container-ttl-duration • maximum-dead-containers-per-container • maximum-dead-containers Just like with ECS, be careful not to collect so aggressively that you lose useful containers
  27. What happens when the defaults aren’t enough? You can: A)

    tune (like we just talked about), B) use a 3rd party tool (like spotify-gc) C) All of the above
  28. It’s also worth mentioning that unused images (and volumes and

    containers, you know the drill), aren’t the only way to lose your disk space. Log rotate is also configurable. Rotate away more files, or more often, or both. With Ubuntu, you can do that with /etc/logrotate.conf. For example, you could change monthly to daily.
  29. *Configuring the ecs-agent, a belated footnote For ECS, you can

    customize quite a bit through the ecs-agent. This is good for things like customizing image cleanup, and changing how your instances interact with Docker and AWS (for example, changing Docker flags, or resource usage). Full list of options is available here. Options can be set in /etc/ecs/ecs.config
  30. Bringing your own AMI

  31. Sometimes, you might want to bring your own AMI to

    ECS or EKS (or Fargate, but that’s not possible yet). With EKS, this is easy! Just select your own when you start the cluster. For ECS, this is a little trickier. Fun fact: this is the icon for AMI
  32. The easiest way to use ECS is with the ECS-optimized

    AMI, but life is too short to not have custom AMIs. A few steps to registering your own AMI with ECS: 0. Make sure your AMI has the right requirements! 1. Install ecs-agent 2. Install Docker daemon (if it’s not already installed) 3. Register instance with cluster 4. Optional (but good): some sort of init process to manage the ecs-agent Most of these options can be set in EC2 user-data (more on that in a sec)
  33. Your instance needs a few things before you can use

    it If you’re using Amazon Linux, you need the correct role, and Docker installed. If you’re not using Amazon Linux, you have a few more steps to follow: documentation is here.
  34. Install and start the ecs-agent and Docker daemon $ yum

    install -y ecs-init $ service docker start $ start ecs Full instructions here.
  35. Register your instance with the cluster With Amazon Linux, in

    /etc/ecs/ecs.config: ECS_ENABLE_TASK_IAM_ROLE=true ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true ECS_LOGFILE=/log/ecs-agent.log ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"] ECS_LOGLEVEL=info ECS_CLUSTER=default Full instructions here.
  36. User-data is where the magic happens

  37. Little known fact: user-data is amazing. You can do ALL

  38. User-data: literally just bash™

  39. All of those options we just talked about (plus many

    more) can be scripted in EC2 user-data. To start with, here is a walkthrough on bootstrapping ECS container instances with user-data.
  40. You can do all kinds of stuff here, both with

    shell scripts, and cloud-init directives. Or MIME multi-part files. Wild. User-data is good for everything from starting services, to configuring your environment, to enabling options and flags that aren’t supported in the UI. Scripts run at boot, and you can modify them through the EC2 Console.
  41. Speaking of unsupported flags, what happens when I want to

    use a Docker flag that’s not supported in the UI? $ echo ”WHATEVER_FLAG=hi” >> /etc/sysconfig/docker An actual, candid reaction from the ECS team when I told them I do this.
  42. You’ve done some cool stuff with your containers: how do

    you know when things are working?
  43. Monitoring and observability!

  44. Hang on: you said monitoring AND observability

  45. With both observability and monitoring, most people start out like

  46. None
  47. But rather than building this:

  48. None
  49. They end up with this:

  50. None
  51. If it doesn’t make sense to you at 3am, it’s

    probably not helping much: a memoir.
  52. You don’t have to go it alone! Lots of tool

    out there to help
  53. Just a few things to think about: • Reduce the

    noise • Page only on issues that require immediate, emergency attention • You need all of it- both monitoring AND observability (more than just logs!) • Make sure you’re asking and answering the right questions.
  54. Hear from the experts: Charity Majors (@mipsytipsy): on Twitter, and

    on the honeycomb.io blog. And a great article from Cindy Sridharan (@copyconstruct) on the differences between monitoring and observability
  55. Thank you! @abbyfuller