Docker in Practice FOSSASIA 2015 

About me • Kuan-Yen Heng (Chris) • Software Engineer at Pie • [email protected] • https://github.com/gigablah • @gigablah 

About Docker • Started as an internal project in dotCloud, a PaaS company • Open sourced in Mar 2013 • Huge success; company pivots • Docker, Inc in Oct 2013 • Docker Machine, Swarm and Compose announced in Dec 2014 

What’s Docker? lxc / libcontainer cgroups golang aufs selinux btrfs devicemapper chroot namespaces images containers volumes libvirt 

Images • What Docker also brings to the table: image management and distribution • Incremental builds via a layered filesystem • Flattening the layers produces a snapshot • Docker Hub: central public image repository 

Source: https://docs.docker.com/terms/layer 

Containers • Runtime instances of images • Spawn multiple containers from an image with individual parameters • Can be started and stopped • Containers will retain filesystem changes until they are removed using docker rm • Changes to a container can be persisted to a new image layer using docker commit 

docker push docker pull registry FROM debian:wheezy MAINTAINER blah RUN apt-get install rabbitmq-server EXPOSE 5672 15672 ENTRYPOINT ["/bin/bash", "-c"] CMD ["/usr/sbin/rabbitmq-server"] Dockerfile docker build docker tag image docker run container docker commit 

Volumes • Mount external directories from the host machine • Can be a bind mount or a volume attached to a container • Typically used to share data across containers • Volumes are local to the host machine; they cannot be distributed like images 

Background 

Building a production cluster • API backend for realtime chat • Autoscaling cluster on AWS • Load balancing • Multiple availability zones • Job queue / scalable worker system • Rapid develop-build-test-deploy cycle 

Architecture Source: http://aws.amazon.com/architecture/ 

Deployment pipeline 

bastion registry build api cluster api api api worker cluster worker worker worker 

Container orchestration 

Amazon EC2 

CoreOS: fleetd + etcd fleetd etcd fleetd etcd fleetctl fleetd etcd systemd systemd docker docker 

Scheduling units • Basically writing systemd units • Fleet specific metadata [X-Fleet] • Schedule global units, specify constraints and dependencies, restart policies • Deploy units based on machine fleet metadata, e.g. role=api and role=worker 

MachineMetadata=role=api Global=true Global=true single instance 

> fleetctl list-units UNIT MACHINE ACTIVE SUB api-discovery@master_123.service 75e1c8bd.../10.0.10.xxx active running api-discovery@master_123.service f54a4d78.../10.0.11.xxx active running api-discovery@master_123.service 320af1d0.../10.0.12.xxx active running api-proxy.service 75e1c8bd.../10.0.10.xxx active running api-proxy.service f54a4d78.../10.0.11.xxx active running api-proxy.service 320af1d0.../10.0.12.xxx active running api@master_123.service 75e1c8bd.../10.0.10.xxx active running api@master_123.service f54a4d78.../10.0.11.xxx active running api@master_123.service 320af1d0.../10.0.12.xxx active running logspout.service 75e1c8bd.../10.0.10.xxx active running logspout.service 17291bf6.../10.0.11.xxx active running logspout.service 320af1d0.../10.0.12.xxx active running logspout.service e1c8ca4c.../10.0.10.xxx active running logspout.service f54a4d78.../10.0.11.xxx active running logspout.service d28b5a20.../10.0.12.xxx active running logspout.service db206400.../10.0.10.xxx active running rabbitmq.service e1c8ca4c.../10.0.10.xxx active running rabbitmq.service 17291bf6.../10.0.11.xxx active running rabbitmq.service d28b5a20.../10.0.12.xxx active running [email protected] e1c8ca4c.../10.0.10.xxx active running [email protected] 17291bf6.../10.0.11.xxx active running [email protected] d28b5a20.../10.0.12.xxx active running 

RancherOS: Docker as init Source: http://rancher.com 

Logging and monitoring 

docker host containers 

collector container gliderlabs/logspout 

monitoring / metrics container etsy/statsd datadog/docker-dd-agent scoutapp/docker-scout logentries/docker-logentries 

Lessons learnt 

CoreOS • btrfs is painful • http://marc.merlins.org/perso/btrfs/ post_2014-05-04_Fixing-Btrfs-Filesystem-Full- Problems.html • Be wary of CoreOS automatic updates • Use a reboot strategy you can stomach • Think twice about using global units 

Build process • Differentiate between “build” and “runtime” • Short-lived (scripts) vs long-lived (daemons) • Compilation tools and libraries should not be present in your production environment • Build your app in a “dev” or “builder” container and transfer it to a “runtime” container • Dedicated containers for common utilities (e.g. git) 

Build pipeline • How do I write my Dockerfile now that I need another container (or more) to build my app? • Script a build pipeline! • Process your source files in a shared volume with your build container(s) before loading it into your base runtime image as the final step • A popular approach is to use Makefiles 

Image Makefiles GIT = pie/git BUILD = pie/builder IMAGE = pie/hubot hubot: docker run --rm -v $(pwd):/opt:rw -e GPG=$$GPG $(GIT) /bin/bash -c “[…]” hubot.tar: | hubot docker run --rm -v $(pwd):/opt:rw $(BUILD) /bin/bash -c “npm […] && tar […]” build: hubot.tar Dockerfile docker build -t $(IMAGE):latest --rm --no-cache . clean: rm -rf hubot && rm -f hubot.tar Credentials Git container Builder container Shared folder (alternatively, make a data container) 

Testing • “Works on my machine” • “Tests pass on my machine” • “Tests pass on the CI server” • Test your code inside your container! 

Test in Docker container Push to registry 

Lean Containers 

Why lean containers? • Continuous integration / automated testing • Third party CI services boot up a fresh environment each time • Fast bootstrapping • A new host in e.g. an autoscaling cluster has to download all images from scratch • Bandwidth / transfer • Especially if you’re running a private registry 

Sample image sizes debian:wheezy 85.1 MB ubuntu:trusty 188.3 MB phusion/baseimage:0.9.16 279.7 MB wordpress:4.1.0 470 MB google/golang:latest 611.3 MB python:2.7.9 744.9 MB 

Docker containers This Not this 

Trimming the fat • phusion/baseimage is one of the most popular base images on the Docker Hub • It provides an init system (runit) so you can run multiple processes in a container • Clocks in at 280 MB • How do we pare this down? 

Removing artifacts • Already done in phusion/baseimage: • Just typical housekeeping • Without: 300.3 MB • With: 279.7 MB (-20.6 MB) apt-get clean rm -rf /tmp/* /var/tmp/* rm -rf /var/lib/apt/lists/* rm -f /etc/ssh/ssh_host_* rm -rf /usr/share/man/?? rm -rf /usr/share/man/??_* 

Removing packages • phusion/baseimage installs syslog-ng, logrotate and openssh-server (sshd) • SSH isn’t needed now that we have docker exec (addressed in a blog post) • Log management: dump process logs to stdout and use a collection container like progrium/logspout • Alternatively mount /dev/log into your container • With log management + sshd: 279 MB • Without: 244 MB (-35 MB) 

Reducing dependencies • For example: a frontend app that uses a Gulp pipeline with gulp-ruby-sass • This requires “gem install sass”, which requires “apt-get install ruby-full rubygems-integration” • OR you could switch to gulp-sass and use native bindings to libsass (C implementation) • With gulp-ruby-sass: 487.2 MB • With gulp-sass: 386 MB (-101.2 MB) 

Your image is everything base 

Switching the base image • Basing your image off Debian instead of Ubuntu results in >100 MB savings off the bat • Some tweaks needed: different packages, python3 not installed by default, etc • Example: olberger/baseimage-docker • Before: 279 MB • After: 166.8 MB (-112.2 MB) 

Reducing dependencies II • phusion/baseimage relies on a Python 3 my_init script which bootstraps runit • Replace runit with s6, a process supervisor suite designed to run as PID 1, which removes the need for certain workarounds (e.g. environment variables) • Eliminates python3 as a dependency • Before: 166.8 MB • After: 144.3 MB (-22.5 MB) 

How low can you go? • Build Linux from scratch! (LFS) • The hard work has been done for you: Buildroot and BusyBox • You could also compile a statically linked binary, e.g. a Golang app and load it into the scratch image (0 MB) 

Switching the base image II • BusyBox weighs in at 2.4 MB (!!) • Batteries not included • A popular setup is to include opkg and piggyback on the OpenWrt package index • An example being progrium/busybox (4.8 MB) • Roll your own using progrium/rootbuilder • Before: 183 MB • After: 56 MB (-127 MB) 

However… • OpenWrt packages are intended for routers and embedded systems, hence it has a rather limited selection • Packages not available in OpenWrt (nodejs, redis, nginx, etc) usually have to be compiled from source, often with manual tweaks • Lots and lots of annoyances (tar missing -z, git missing https support, no python-dev, etc) 

Alpine Linux • docker-alpine is based on Alpine Linux and comes with a more general purpose package index (using apk) • Most package woes solved • App container: nodejs, php, python, etc available • Build container: apk-install build-base 

In short… Original image (nodejs app) 426 MB Without ruby dependency 325 MB Without sshd and syslog-ng 290 MB With Debian as base 183 MB With s6 as init system 166 MB With BusyBox / Alpine as base 56 MB 

A tiny baseimage • https://registry.hub.docker.com/u/gigablah/baseimage/ • Result: 6 MB FROM gliderlabs/alpine:3.1 MAINTAINER Chris Heng ADD s6-2.0.0.1.tar.gz / ADD service /etc/service RUN mkdir -p /var/spool/cron/crontabs ENTRYPOINT ["/usr/bin/s6-svscan", "-t0"] CMD ["/etc/service"] 

Thank you [email protected] https://github.com/gigablah @gigablah