Docker at Shopify Simon Eskildsen, WebScale Team @ Shopify @Sirupsen

Why? — Cyber monday and black friday — Equip Shopify with future-proof infrastructure (heavy sharding, multi dc, services, 1000s of servers, millions of RPM) — Develop an operating system for the datacenter — Operations provide a platform for developers

Ops should be a product team, not consultants. [..] ops can provide [a platform] to developers to power production, testing, and development, freeing developers to focus. — Flynn.io

— Summer 2013 Investigated Mesos and Docker, still very immature. — Fall 2013 Worked on deploying Kafka to wait for technologies to mature. — Winter 2014 Started looking at Mesos and Docker again. — Winter 2014 Everything is terrible and it's fuck off cold. — March 2014 Ran Shopify production traffic on Mesos and Docker.

— April 2014 Ragequit. Started looking into CoreOS. — June 2014 Everything is terrible. — June 2014 Chef, runit and Capistrano. — July 2014 1% -> 10% -> 20% -> 100% as of end of July 2014. — August 2014 ! and "

Try this at home. Think twice before trying it in production.

Current setup

Image builds 1. Commit lands in master. 2. Build server builds image for new SHA (~2m). 3. Pushed to internal registry: registry.shopify.com/shopify:.

# recipe container "shopify" do role "unicorn" instances node[:containers].fetch(:shopify,{})[:unicorn] || 0 health_check true http_health_check "/services/ping" revision "file" docker_args [ "-e RAILS_ENV=production" ] end # role name 'app--borg--shopify-unicorn-32' run_list 'role[app--borg--shopify-unicorn]' default_attributes(containers: { shopify: { unicorn: 23 }})

/usr/bin/docker -d ... \_ borginit:[email protected] | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:[email protected] | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:[email protected] | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:[email protected] | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co..

Deploys 1. Capistrano updates revision in /etc/borg/ shopify/current/revision. 2. ls -d /etc/sv/borg-shopify-unicorn* | xargs sudo sv restart — New image layer (~2 MiB) is downloaded from registry. — Subset of containers (~20%) are restarted with new image.

Process layout root@borg-shopify-unicorn-11:/app# ps auxf borginit:[email protected] . /sbin/ppidshim /bin/sh -c config/borgconfig/borgc... \_ unicorn_shopify master -c /app/unicorn.conf.rb.. \_ Rails uri_path=/services/ping request_id=2.. \_ Rails uri_path=/services/ping request_id=9.. \_ Rails uri_path=/services/ping request_id=b.. \_ Rails uri_path=/services/ping request_id=5..

No content

??