DevOps Masterclass: Docker at Shopify

40-minute talk about what we learned when putting Docker into production at Shopify.

Transcript

1. Docker at Shopify Simon Eskildsen, WebScale Team @ Shopify @Sirupsen
2. Why? — Cyber monday and black friday — Equip Shopify with future-proof infrastructure (heavy sharding, multi dc, services, 1000s of servers, millions of RPM) — Develop an operating system for the datacenter — Operations provide a platform for developers
3. Ops should be a product team, not consultants. [..] ops can provide [a platform] to developers to power production, testing, and development, freeing developers to focus. — Flynn.io
4. — Summer 2013 Investigated Mesos and Docker, still very immature. — Fall 2013 Worked on deploying Kafka to wait for technologies to mature. — Winter 2014 Started looking at Mesos and Docker again. — Winter 2014 Everything is terrible and it's fuck off cold. — March 2014 Ran Shopify production traffic on Mesos and Docker.
5. — April 2014 Ragequit. Started looking into CoreOS. — June 2014 Everything is terrible. — June 2014 Chef, runit and Capistrano. — July 2014 1% -> 10% -> 20% -> 100% as of end of July 2014. — August 2014 ! and "
6. Try this at home. Think twice before trying it in production.
7. Current setup
8. Image builds 1. Commit <sha> lands in master. 2. Build server builds image for new SHA (~2m). 3. Pushed to internal registry: registry.shopify.com/shopify:<sha>.
9. # recipe container "shopify" do role "unicorn" instances node[:containers].fetch(:shopify,{})[:unicorn] || 0 health_check true http_health_check "/services/ping" revision "file" docker_args [ "-e RAILS_ENV=production" ] end # role name 'app--borg--shopify-unicorn-32' run_list 'role[app--borg--shopify-unicorn]' default_attributes(containers: { shopify: { unicorn: 23 }})
10. /usr/bin/docker -d ... \_ borginit:shopify--unicorn@19.service | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:shopify--unicorn@16.service | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:shopify--unicorn@9.service | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co.. \_ borginit:shopify--unicorn@13.service | \_ /sbin/ppidshim /bin/sh -c config/borgconfig/borgconf.. | \_ unicorn_shopify master -c /app/unicorn.conf.rb .. | \_ unicorn_shopify worker[0] -c /app/unicorn.co.. | \_ unicorn_shopify worker[1] -c /app/unicorn.co.. | \_ unicorn_shopify worker[2] -c /app/unicorn.co.. | \_ unicorn_shopify worker[3] -c /app/unicorn.co..
11. Deploys 1. Capistrano updates revision in /etc/borg/ shopify/current/revision. 2. ls -d /etc/sv/borg-shopify-unicorn* | xargs sudo sv restart — New image layer (~2 MiB) is downloaded from registry. — Subset of containers (~20%) are restarted with new image.
12. Process layout root@borg-shopify-unicorn-11:/app# ps auxf borginit:shopify--unicorn@11.service . /sbin/ppidshim /bin/sh -c config/borgconfig/borgc... \_ unicorn_shopify master -c /app/unicorn.conf.rb.. \_ Rails uri_path=/services/ping request_id=2.. \_ Rails uri_path=/services/ping request_id=9.. \_ Rails uri_path=/services/ping request_id=b.. \_ Rails uri_path=/services/ping request_id=5..
14. ??