Docker In Production - A War Story

Docker Meetup Vancouver - 23 July 2015

Transcript

1. Docker in Production
   A War Story
   

   View Slide

2. Kyle Young
   github.com/kyoung
   @ksgyoung
   Sebastian Vetter
   github.com/elbaschid
   @elbaschid
   Who We Are
   

   View Slide

3. Rackspace Setup
   ● Many many Rackspace servers
   ● A lot of tooling around legacy infrastructure
   ● Running a lot of services ourselves: RabbitMQ,
   MySQL (in replication), and more
   

   View Slide

4. View Slide

5. View Slide

6. New Setup
   ● AWS Beanstalk
   ● Docker
   ● Quay.io: Docker Registry
   ● External services: RDS, ElastiCache, hosted
   RabbitMQ
   

   View Slide

7. View Slide

8. View Slide

9. Development Environment
   ● How are we running the dev env
   ○ docker-machine
   ○ docker-compose
   ○ much bash
   

   View Slide

10. docker-compose
    ● Container per service
    ● Full async setup
    ● Controlled by our own CLI
    docker
    ├── cli
    │ ├── bootstrap.sh
    │ ├── circle-run-tests-in-parallel.sh
    │ ├── cleanup.sh
    │ ├── deploy.sh
    │ ├── dev.sh
    │ ├── hotfix.sh
    │ └── promote.sh
    

    View Slide

11. Deployment
    ● Testing on Circle CI
    ● Auto-deploy to Staging
    ● ./portal.sh promote
    

    View Slide

12. View Slide

13. Problems
    

    View Slide

14. Beanstalk Tooling
    

    View Slide

15. Beanstalk Tooling
    ● Very basic tooling around Beanstalk
    ● Needs a bit of a nudge some of the times
    ● Modifying AMI using .ebextensions
    

    View Slide

16. Auto-Mounting Volumes
    

    View Slide

17. Auto-Mounting Volumes
    ● Having VOLUME in Dockerfile
    ● Creates auto-mounted AUFS volume
    ● Doesn’t get cleaned up
    ● Don’t use VOLUME
    

    View Slide

18. Containers & Images
    

    View Slide

19. Containers & Images
    ● Failed deployments due to lack of disk space
    ● Pulling images adds data
    ● No automatic cleanup
    ● Adding some cleanup commands to .
    ebextensions
    

    View Slide

20. Centralized Logging
    

    View Slide

21. Centralized Logging
    ● Docker logs to stdout & stderr
    ● Beanstalk pulls them into log files
    ● Forwarding to syslog is tricky
    ○ dynamic naming of log files:
    /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log
    ○ old version of rsyslog that doesn't allow globbing
    

    View Slide

22. Solution
    ● Using gliderlabs’ logspout: https://github.
    com/gliderlabs/logspout
    ● Best of both worlds:
    ○ Beanstalk logging
    ○ syslog forwarding
    

    View Slide

23. Monitoring
    

    View Slide

24. Monitoring with Datadog
    ● Better insight:
    ○ containers running
    ○ CPU & memory usage
    ○ triggering alerts
    ● Using the dd-agent container from Datadog
    https://github.com/DataDog/dd-agent
    

    View Slide

25. Monitoring with Datadog
    

    View Slide

26. Migrations
    

    View Slide

27. Database and migrations
    ● Currently has to be run manually
    ● Better strategy needed long-term
    But:
    ● No sophisticated migration strategy
    ● No high-traffic, high-performance service
    

    View Slide

28. The Future
    

    View Slide

29. Possible Improvements
    ● AWS ECS for more efficient containers
    ● Possibly investigating "Project Orca" at some
    point (maybe)
    

    View Slide

30. Questions?
    

    View Slide