Docker in Practice

As Docker rapidly gains adoption, an ecosystem of tools, patterns and practices has emerged. Here are some of the lessons we've learnt from optimizing our Dockerized build and deployment pipeline.

Transcript

1. Docker in Practice FOSSASIA 2015
2. About me • Kuan-Yen Heng (Chris) • Software Engineer at Pie • kuanyen@pie.co • https://github.com/gigablah • @gigablah
3. About Docker • Started as an internal project in dotCloud, a PaaS company • Open sourced in Mar 2013 • Huge success; company pivots • Docker, Inc in Oct 2013 • Docker Machine, Swarm and Compose announced in Dec 2014
4. What’s Docker? lxc / libcontainer cgroups golang aufs selinux btrfs devicemapper chroot namespaces images containers volumes libvirt
5. Images • What Docker also brings to the table: image management and distribution • Incremental builds via a layered filesystem • Flattening the layers produces a snapshot • Docker Hub: central public image repository
6. Source: https://docs.docker.com/terms/layer
7. Containers • Runtime instances of images • Spawn multiple containers from an image with individual parameters • Can be started and stopped • Containers will retain filesystem changes until they are removed using docker rm • Changes to a container can be persisted to a new image layer using docker commit
8. docker push docker pull registry FROM debian:wheezy MAINTAINER blah <me@blah.co> RUN apt-get install rabbitmq-server EXPOSE 5672 15672 ENTRYPOINT ["/bin/bash", "-c"] CMD ["/usr/sbin/rabbitmq-server"] Dockerfile docker build docker tag image docker run container docker commit
9. Volumes • Mount external directories from the host machine • Can be a bind mount or a volume attached to a container • Typically used to share data across containers • Volumes are local to the host machine; they cannot be distributed like images
10. Background
11. Building a production cluster • API backend for realtime chat • Autoscaling cluster on AWS • Load balancing • Multiple availability zones • Job queue / scalable worker system • Rapid develop-build-test-deploy cycle
12. Architecture Source: http://aws.amazon.com/architecture/
13. Deployment pipeline
14. bastion registry build api cluster api api api worker cluster worker worker worker
15. Container orchestration
16. Amazon EC2
17. CoreOS: fleetd + etcd fleetd etcd fleetd etcd fleetctl fleetd etcd systemd systemd docker docker
18. Scheduling units • Basically writing systemd units • Fleet specific metadata [X-Fleet] • Schedule global units, specify constraints and dependencies, restart policies • Deploy units based on machine fleet metadata, e.g. role=api and role=worker
19. MachineMetadata=role=api Global=true Global=true single instance
20. > fleetctl list-units UNIT MACHINE ACTIVE SUB api-discovery@master_123.service 75e1c8bd.../10.0.10.xxx active running api-discovery@master_123.service f54a4d78.../10.0.11.xxx active running api-discovery@master_123.service 320af1d0.../10.0.12.xxx active running api-proxy.service 75e1c8bd.../10.0.10.xxx active running api-proxy.service f54a4d78.../10.0.11.xxx active running api-proxy.service 320af1d0.../10.0.12.xxx active running api@master_123.service 75e1c8bd.../10.0.10.xxx active running api@master_123.service f54a4d78.../10.0.11.xxx active running api@master_123.service 320af1d0.../10.0.12.xxx active running logspout.service 75e1c8bd.../10.0.10.xxx active running logspout.service 17291bf6.../10.0.11.xxx active running logspout.service 320af1d0.../10.0.12.xxx active running logspout.service e1c8ca4c.../10.0.10.xxx active running logspout.service f54a4d78.../10.0.11.xxx active running logspout.service d28b5a20.../10.0.12.xxx active running logspout.service db206400.../10.0.10.xxx active running rabbitmq.service e1c8ca4c.../10.0.10.xxx active running rabbitmq.service 17291bf6.../10.0.11.xxx active running rabbitmq.service d28b5a20.../10.0.12.xxx active running worker@master.service e1c8ca4c.../10.0.10.xxx active running worker@master.service 17291bf6.../10.0.11.xxx active running worker@master.service d28b5a20.../10.0.12.xxx active running
21. RancherOS: Docker as init Source: http://rancher.com
22. Logging and monitoring
23. docker host containers
24. collector container gliderlabs/logspout
25. monitoring / metrics container etsy/statsd datadog/docker-dd-agent scoutapp/docker-scout logentries/docker-logentries
26. Lessons learnt
27. CoreOS • btrfs is painful • http://marc.merlins.org/perso/btrfs/ post_2014-05-04_Fixing-Btrfs-Filesystem-Full- Problems.html • Be wary of CoreOS automatic updates • Use a reboot strategy you can stomach • Think twice about using global units
28. Build process • Differentiate between “build” and “runtime” • Short-lived (scripts) vs long-lived (daemons) • Compilation tools and libraries should not be present in your production environment • Build your app in a “dev” or “builder” container and transfer it to a “runtime” container • Dedicated containers for common utilities (e.g. git)
29. Build pipeline • How do I write my Dockerfile now that I need another container (or more) to build my app? • Script a build pipeline! • Process your source files in a shared volume with your build container(s) before loading it into your base runtime image as the final step • A popular approach is to use Makefiles
30. Image Makefiles GIT = pie/git BUILD = pie/builder IMAGE = pie/hubot hubot: docker run --rm -v $(pwd):/opt:rw -e GPG=$$GPG $(GIT) /bin/bash -c “[…]” hubot.tar: | hubot docker run --rm -v $(pwd):/opt:rw $(BUILD) /bin/bash -c “npm […] && tar […]” build: hubot.tar Dockerfile docker build -t $(IMAGE):latest --rm --no-cache . clean: rm -rf hubot && rm -f hubot.tar Credentials Git container Builder container Shared folder (alternatively, make a data container)
31. Testing • “Works on my machine” • “Tests pass on my machine” • “Tests pass on the CI server” • Test your code inside your container!
32. Test in Docker container Push to registry
33. Lean Containers
34. Why lean containers? • Continuous integration / automated testing • Third party CI services boot up a fresh environment each time • Fast bootstrapping • A new host in e.g. an autoscaling cluster has to download all images from scratch • Bandwidth / transfer • Especially if you’re running a private registry
35. Sample image sizes debian:wheezy 85.1 MB ubuntu:trusty 188.3 MB phusion/baseimage:0.9.16 279.7 MB wordpress:4.1.0 470 MB google/golang:latest 611.3 MB python:2.7.9 744.9 MB
36. Docker containers This Not this
37. Trimming the fat • phusion/baseimage is one of the most popular base images on the Docker Hub • It provides an init system (runit) so you can run multiple processes in a container • Clocks in at 280 MB • How do we pare this down?
38. Removing artifacts • Already done in phusion/baseimage: • Just typical housekeeping • Without: 300.3 MB • With: 279.7 MB (-20.6 MB) apt-get clean rm -rf /tmp/* /var/tmp/* rm -rf /var/lib/apt/lists/* rm -f /etc/ssh/ssh_host_* rm -rf /usr/share/man/?? rm -rf /usr/share/man/??_*
39. Removing packages • phusion/baseimage installs syslog-ng, logrotate and openssh-server (sshd) • SSH isn’t needed now that we have docker exec (addressed in a blog post) • Log management: dump process logs to stdout and use a collection container like progrium/logspout • Alternatively mount /dev/log into your container • With log management + sshd: 279 MB • Without: 244 MB (-35 MB)
40. Reducing dependencies • For example: a frontend app that uses a Gulp pipeline with gulp-ruby-sass • This requires “gem install sass”, which requires “apt-get install ruby-full rubygems-integration” • OR you could switch to gulp-sass and use native bindings to libsass (C implementation) • With gulp-ruby-sass: 487.2 MB • With gulp-sass: 386 MB (-101.2 MB)
41. Your image is everything base
42. Switching the base image • Basing your image off Debian instead of Ubuntu results in >100 MB savings off the bat • Some tweaks needed: different packages, python3 not installed by default, etc • Example: olberger/baseimage-docker • Before: 279 MB • After: 166.8 MB (-112.2 MB)
43. Reducing dependencies II • phusion/baseimage relies on a Python 3 my_init script which bootstraps runit • Replace runit with s6, a process supervisor suite designed to run as PID 1, which removes the need for certain workarounds (e.g. environment variables) • Eliminates python3 as a dependency • Before: 166.8 MB • After: 144.3 MB (-22.5 MB)
44. How low can you go? • Build Linux from scratch! (LFS) • The hard work has been done for you: Buildroot and BusyBox • You could also compile a statically linked binary, e.g. a Golang app and load it into the scratch image (0 MB)
45. Switching the base image II • BusyBox weighs in at 2.4 MB (!!) • Batteries not included • A popular setup is to include opkg and piggyback on the OpenWrt package index • An example being progrium/busybox (4.8 MB) • Roll your own using progrium/rootbuilder • Before: 183 MB • After: 56 MB (-127 MB)
46. However… • OpenWrt packages are intended for routers and embedded systems, hence it has a rather limited selection • Packages not available in OpenWrt (nodejs, redis, nginx, etc) usually have to be compiled from source, often with manual tweaks • Lots and lots of annoyances (tar missing -z, git missing https support, no python-dev, etc)
47. Alpine Linux • docker-alpine is based on Alpine Linux and comes with a more general purpose package index (using apk) • Most package woes solved • App container: nodejs, php, python, etc available • Build container: apk-install build-base
48. In short… Original image (nodejs app) 426 MB Without ruby dependency 325 MB Without sshd and syslog-ng 290 MB With Debian as base 183 MB With s6 as init system 166 MB With BusyBox / Alpine as base 56 MB
49. A tiny baseimage • https://registry.hub.docker.com/u/gigablah/baseimage/ • Result: 6 MB FROM gliderlabs/alpine:3.1 MAINTAINER Chris Heng <bigblah@gmail.com> ADD s6-2.0.0.1.tar.gz / ADD service /etc/service RUN mkdir -p /var/spool/cron/crontabs ENTRYPOINT ["/usr/bin/s6-svscan", "-t0"] CMD ["/etc/service"]
50. Thank you kuanyen@pie.co https://github.com/gigablah @gigablah