Docker In Production - A War Story

C65d18a43152b199ee94aad2b79b70c4?s=47 Seb
July 23, 2015

Docker In Production - A War Story

Docker Meetup Vancouver - 23 July 2015

C65d18a43152b199ee94aad2b79b70c4?s=128

Seb

July 23, 2015
Tweet

Transcript

  1. Docker in Production A War Story

  2. Kyle Young github.com/kyoung @ksgyoung Sebastian Vetter github.com/elbaschid @elbaschid Who We

    Are
  3. Rackspace Setup • Many many Rackspace servers • A lot

    of tooling around legacy infrastructure • Running a lot of services ourselves: RabbitMQ, MySQL (in replication), and more
  4. None
  5. None
  6. New Setup • AWS Beanstalk • Docker • Quay.io: Docker

    Registry • External services: RDS, ElastiCache, hosted RabbitMQ
  7. None
  8. None
  9. Development Environment • How are we running the dev env

    ◦ docker-machine ◦ docker-compose ◦ much bash
  10. docker-compose • Container per service • Full async setup •

    Controlled by our own CLI docker ├── cli │ ├── bootstrap.sh │ ├── circle-run-tests-in-parallel.sh │ ├── cleanup.sh │ ├── deploy.sh │ ├── dev.sh │ ├── hotfix.sh │ └── promote.sh
  11. Deployment • Testing on Circle CI • Auto-deploy to Staging

    • ./portal.sh promote
  12. None
  13. Problems

  14. Beanstalk Tooling

  15. Beanstalk Tooling • Very basic tooling around Beanstalk • Needs

    a bit of a nudge some of the times • Modifying AMI using .ebextensions
  16. Auto-Mounting Volumes

  17. Auto-Mounting Volumes • Having VOLUME in Dockerfile • Creates auto-mounted

    AUFS volume • Doesn’t get cleaned up • Don’t use VOLUME
  18. Containers & Images

  19. Containers & Images • Failed deployments due to lack of

    disk space • Pulling images adds data • No automatic cleanup • Adding some cleanup commands to . ebextensions
  20. Centralized Logging

  21. Centralized Logging • Docker logs to stdout & stderr •

    Beanstalk pulls them into log files • Forwarding to syslog is tricky ◦ dynamic naming of log files: /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log ◦ old version of rsyslog that doesn't allow globbing
  22. Solution • Using gliderlabs’ logspout: https://github. com/gliderlabs/logspout • Best of

    both worlds: ◦ Beanstalk logging ◦ syslog forwarding
  23. Monitoring

  24. Monitoring with Datadog • Better insight: ◦ containers running ◦

    CPU & memory usage ◦ triggering alerts • Using the dd-agent container from Datadog https://github.com/DataDog/dd-agent
  25. Monitoring with Datadog

  26. Migrations

  27. Database and migrations • Currently has to be run manually

    • Better strategy needed long-term But: • No sophisticated migration strategy • No high-traffic, high-performance service
  28. The Future

  29. Possible Improvements • AWS ECS for more efficient containers •

    Possibly investigating "Project Orca" at some point (maybe)
  30. Questions?