Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Docker In Production - A War Story

Seb
July 23, 2015

Docker In Production - A War Story

Docker Meetup Vancouver - 23 July 2015

Seb

July 23, 2015
Tweet

More Decks by Seb

Other Decks in Technology

Transcript

  1. Docker in Production
    A War Story

    View Slide

  2. Kyle Young
    github.com/kyoung
    @ksgyoung
    Sebastian Vetter
    github.com/elbaschid
    @elbaschid
    Who We Are

    View Slide

  3. Rackspace Setup
    ● Many many Rackspace servers
    ● A lot of tooling around legacy infrastructure
    ● Running a lot of services ourselves: RabbitMQ,
    MySQL (in replication), and more

    View Slide

  4. View Slide

  5. View Slide

  6. New Setup
    ● AWS Beanstalk
    ● Docker
    ● Quay.io: Docker Registry
    ● External services: RDS, ElastiCache, hosted
    RabbitMQ

    View Slide

  7. View Slide

  8. View Slide

  9. Development Environment
    ● How are we running the dev env
    ○ docker-machine
    ○ docker-compose
    ○ much bash

    View Slide

  10. docker-compose
    ● Container per service
    ● Full async setup
    ● Controlled by our own CLI
    docker
    ├── cli
    │ ├── bootstrap.sh
    │ ├── circle-run-tests-in-parallel.sh
    │ ├── cleanup.sh
    │ ├── deploy.sh
    │ ├── dev.sh
    │ ├── hotfix.sh
    │ └── promote.sh

    View Slide

  11. Deployment
    ● Testing on Circle CI
    ● Auto-deploy to Staging
    ● ./portal.sh promote

    View Slide

  12. View Slide

  13. Problems

    View Slide

  14. Beanstalk Tooling

    View Slide

  15. Beanstalk Tooling
    ● Very basic tooling around Beanstalk
    ● Needs a bit of a nudge some of the times
    ● Modifying AMI using .ebextensions

    View Slide

  16. Auto-Mounting Volumes

    View Slide

  17. Auto-Mounting Volumes
    ● Having VOLUME in Dockerfile
    ● Creates auto-mounted AUFS volume
    ● Doesn’t get cleaned up
    ● Don’t use VOLUME

    View Slide

  18. Containers & Images

    View Slide

  19. Containers & Images
    ● Failed deployments due to lack of disk space
    ● Pulling images adds data
    ● No automatic cleanup
    ● Adding some cleanup commands to .
    ebextensions

    View Slide

  20. Centralized Logging

    View Slide

  21. Centralized Logging
    ● Docker logs to stdout & stderr
    ● Beanstalk pulls them into log files
    ● Forwarding to syslog is tricky
    ○ dynamic naming of log files:
    /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log
    ○ old version of rsyslog that doesn't allow globbing

    View Slide

  22. Solution
    ● Using gliderlabs’ logspout: https://github.
    com/gliderlabs/logspout
    ● Best of both worlds:
    ○ Beanstalk logging
    ○ syslog forwarding

    View Slide

  23. Monitoring

    View Slide

  24. Monitoring with Datadog
    ● Better insight:
    ○ containers running
    ○ CPU & memory usage
    ○ triggering alerts
    ● Using the dd-agent container from Datadog
    https://github.com/DataDog/dd-agent

    View Slide

  25. Monitoring with Datadog

    View Slide

  26. Migrations

    View Slide

  27. Database and migrations
    ● Currently has to be run manually
    ● Better strategy needed long-term
    But:
    ● No sophisticated migration strategy
    ● No high-traffic, high-performance service

    View Slide

  28. The Future

    View Slide

  29. Possible Improvements
    ● AWS ECS for more efficient containers
    ● Possibly investigating "Project Orca" at some
    point (maybe)

    View Slide

  30. Questions?

    View Slide