Docker In Production - A War Story

Transcript

1. Docker in Production A War Story

2. Kyle Young github.com/kyoung @ksgyoung Sebastian Vetter github.com/elbaschid @elbaschid Who We

   Are

3. Rackspace Setup • Many many Rackspace servers • A lot

   of tooling around legacy infrastructure • Running a lot of services ourselves: RabbitMQ, MySQL (in replication), and more
4. None
5. None

6. New Setup • AWS Beanstalk • Docker • Quay.io: Docker

   Registry • External services: RDS, ElastiCache, hosted RabbitMQ
7. None
8. None

9. Development Environment • How are we running the dev env

   ◦ docker-machine ◦ docker-compose ◦ much bash

10. docker-compose • Container per service • Full async setup •

    Controlled by our own CLI docker ├── cli │ ├── bootstrap.sh │ ├── circle-run-tests-in-parallel.sh │ ├── cleanup.sh │ ├── deploy.sh │ ├── dev.sh │ ├── hotfix.sh │ └── promote.sh

11. Deployment • Testing on Circle CI • Auto-deploy to Staging

    • ./portal.sh promote
12. None

13. Problems

14. Beanstalk Tooling

15. Beanstalk Tooling • Very basic tooling around Beanstalk • Needs

    a bit of a nudge some of the times • Modifying AMI using .ebextensions

16. Auto-Mounting Volumes

17. Auto-Mounting Volumes • Having VOLUME in Dockerfile • Creates auto-mounted

    AUFS volume • Doesn’t get cleaned up • Don’t use VOLUME

18. Containers & Images

19. Containers & Images • Failed deployments due to lack of

    disk space • Pulling images adds data • No automatic cleanup • Adding some cleanup commands to . ebextensions

20. Centralized Logging

21. Centralized Logging • Docker logs to stdout & stderr •

    Beanstalk pulls them into log files • Forwarding to syslog is tricky ◦ dynamic naming of log files: /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log ◦ old version of rsyslog that doesn't allow globbing

22. Solution • Using gliderlabs’ logspout: https://github. com/gliderlabs/logspout • Best of

    both worlds: ◦ Beanstalk logging ◦ syslog forwarding

23. Monitoring

24. Monitoring with Datadog • Better insight: ◦ containers running ◦

    CPU & memory usage ◦ triggering alerts • Using the dd-agent container from Datadog https://github.com/DataDog/dd-agent

25. Monitoring with Datadog

26. Migrations

27. Database and migrations • Currently has to be run manually

    • Better strategy needed long-term But: • No sophisticated migration strategy • No high-traffic, high-performance service

28. The Future

29. Possible Improvements • AWS ECS for more efficient containers •

    Possibly investigating "Project Orca" at some point (maybe)

30. Questions?