Docker In Production - A War Story

Docker Meetup Vancouver - 23 July 2015

Transcript

1. Docker in Production A War Story
2. Kyle Young github.com/kyoung @ksgyoung Sebastian Vetter github.com/elbaschid @elbaschid Who We Are
3. Rackspace Setup • Many many Rackspace servers • A lot of tooling around legacy infrastructure • Running a lot of services ourselves: RabbitMQ, MySQL (in replication), and more
6. New Setup • AWS Beanstalk • Docker • Quay.io: Docker Registry • External services: RDS, ElastiCache, hosted RabbitMQ
9. Development Environment • How are we running the dev env ◦ docker-machine ◦ docker-compose ◦ much bash
10. docker-compose • Container per service • Full async setup • Controlled by our own CLI docker ├── cli │ ├── bootstrap.sh │ ├── circle-run-tests-in-parallel.sh │ ├── cleanup.sh │ ├── deploy.sh │ ├── dev.sh │ ├── hotfix.sh │ └── promote.sh
11. Deployment • Testing on Circle CI • Auto-deploy to Staging • ./portal.sh promote
13. Problems
14. Beanstalk Tooling
15. Beanstalk Tooling • Very basic tooling around Beanstalk • Needs a bit of a nudge some of the times • Modifying AMI using .ebextensions
16. Auto-Mounting Volumes
17. Auto-Mounting Volumes • Having VOLUME in Dockerfile • Creates auto-mounted AUFS volume • Doesn’t get cleaned up • Don’t use VOLUME
18. Containers & Images
19. Containers & Images • Failed deployments due to lack of disk space • Pulling images adds data • No automatic cleanup • Adding some cleanup commands to . ebextensions
20. Centralized Logging
21. Centralized Logging • Docker logs to stdout & stderr • Beanstalk pulls them into log files • Forwarding to syslog is tricky ◦ dynamic naming of log files: /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log ◦ old version of rsyslog that doesn't allow globbing
22. Solution • Using gliderlabs’ logspout: https://github. com/gliderlabs/logspout • Best of both worlds: ◦ Beanstalk logging ◦ syslog forwarding
23. Monitoring
24. Monitoring with Datadog • Better insight: ◦ containers running ◦ CPU & memory usage ◦ triggering alerts • Using the dd-agent container from Datadog https://github.com/DataDog/dd-agent
25. Monitoring with Datadog
26. Migrations
27. Database and migrations • Currently has to be run manually • Better strategy needed long-term But: • No sophisticated migration strategy • No high-traffic, high-performance service
28. The Future
29. Possible Improvements • AWS ECS for more efficient containers • Possibly investigating "Project Orca" at some point (maybe)
30. Questions?