Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Docker In Production - A War Story
Seb
July 23, 2015
Technology
1
260
Docker In Production - A War Story
Docker Meetup Vancouver - 23 July 2015
Seb
July 23, 2015
Tweet
Share
More Decks by Seb
See All by Seb
Double Click - Continue Building Better CLIs
elbaschid
0
170
I Can Be A Speaker, So Can You
elbaschid
0
170
Click - PyCaribbean 2017 - Puerto Rico
elbaschid
0
160
Conferencing - Engineering Meeting
elbaschid
1
27
Show & Tell - PyCon US 2016 Summary
elbaschid
1
31
Click: A Pleasure To Write, A Pleasure To Use
elbaschid
0
320
Hunting for Treasure in Django
elbaschid
1
340
Moby & The Beanstalk
elbaschid
1
270
Hunting For Treasure In Django
elbaschid
0
120
Other Decks in Technology
See All in Technology
tfcon-2022-cpp
cpp
5
5k
Stripe Search APIを利用した、LINEとStripeの顧客情報連携/line-dc-202205
stripehideokamoto
0
120
Devに力を授けたいSREのあゆみ / SRE that wants to empower developers
tocyuki
3
470
一人から始めるプロダクトSRE / How to start SRE in a product team, all by yourself
vtryo
4
2.5k
Who owns the Service Level?
chaspy
5
890
ソフトウェアテストで参考にしている67のモノ #scrumniigata / 67 things for software testing
kyonmm
PRO
1
260
JAWS-UG 朝会 #33 登壇資料
takakuni
0
380
スタートアップ入社4日目までに考えたAWSのセキュリティ向上/ Startup AWS Security
shonansurvivors
3
2.9k
LINE WORKS API 2.0について
mmclsntr
0
110
220510 プロセスマイニングを学ぶ PLAY与田さん
comucal
PRO
0
780
1,000万人以上が利用する「家族アルバム みてね」のSRE組織は4年間でどのように作られてきたのか/SRE NEXT 2022
isaoshimizu
4
2.8k
Graph API について
miyakemito
0
200
Featured
See All Featured
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
38
12k
Build The Right Thing And Hit Your Dates
maggiecrowley
19
1.1k
Art, The Web, and Tiny UX
lynnandtonic
280
17k
5 minutes of I Can Smell Your CMS
philhawksworth
196
18k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
226
15k
Bootstrapping a Software Product
garrettdimon
295
110k
What the flash - Photography Introduction
edds
61
9.9k
Designing for humans not robots
tammielis
241
23k
A Philosophy of Restraint
colly
192
14k
Agile that works and the tools we love
rasmusluckow
319
19k
The Power of CSS Pseudo Elements
geoffreycrofte
46
3.9k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
910
Transcript
Docker in Production A War Story
Kyle Young github.com/kyoung @ksgyoung Sebastian Vetter github.com/elbaschid @elbaschid Who We
Are
Rackspace Setup • Many many Rackspace servers • A lot
of tooling around legacy infrastructure • Running a lot of services ourselves: RabbitMQ, MySQL (in replication), and more
None
None
New Setup • AWS Beanstalk • Docker • Quay.io: Docker
Registry • External services: RDS, ElastiCache, hosted RabbitMQ
None
None
Development Environment • How are we running the dev env
◦ docker-machine ◦ docker-compose ◦ much bash
docker-compose • Container per service • Full async setup •
Controlled by our own CLI docker ├── cli │ ├── bootstrap.sh │ ├── circle-run-tests-in-parallel.sh │ ├── cleanup.sh │ ├── deploy.sh │ ├── dev.sh │ ├── hotfix.sh │ └── promote.sh
Deployment • Testing on Circle CI • Auto-deploy to Staging
• ./portal.sh promote
None
Problems
Beanstalk Tooling
Beanstalk Tooling • Very basic tooling around Beanstalk • Needs
a bit of a nudge some of the times • Modifying AMI using .ebextensions
Auto-Mounting Volumes
Auto-Mounting Volumes • Having VOLUME in Dockerfile • Creates auto-mounted
AUFS volume • Doesn’t get cleaned up • Don’t use VOLUME
Containers & Images
Containers & Images • Failed deployments due to lack of
disk space • Pulling images adds data • No automatic cleanup • Adding some cleanup commands to . ebextensions
Centralized Logging
Centralized Logging • Docker logs to stdout & stderr •
Beanstalk pulls them into log files • Forwarding to syslog is tricky ◦ dynamic naming of log files: /var/log/eb-docker/.../6fb0153a0b99-stdouterr.log ◦ old version of rsyslog that doesn't allow globbing
Solution • Using gliderlabs’ logspout: https://github. com/gliderlabs/logspout • Best of
both worlds: ◦ Beanstalk logging ◦ syslog forwarding
Monitoring
Monitoring with Datadog • Better insight: ◦ containers running ◦
CPU & memory usage ◦ triggering alerts • Using the dd-agent container from Datadog https://github.com/DataDog/dd-agent
Monitoring with Datadog
Migrations
Database and migrations • Currently has to be run manually
• Better strategy needed long-term But: • No sophisticated migration strategy • No high-traffic, high-performance service
The Future
Possible Improvements • AWS ECS for more efficient containers •
Possibly investigating "Project Orca" at some point (maybe)
Questions?