Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

30m presentation at DockerCon EU 2014 in Amsterdam.


Simon Hørup Eskildsen

December 05, 2014

More Decks by Simon Hørup Eskildsen

Other Decks in Technology


  1. Shopiscale • 200-300K RPM during day, > 500K during large

    sales • ~100 large metal servers running containers • 100+ product developers, ~10 infrastructure, ~10 core ops • 10 year old monolith • Ruby on Rails, MySQL, Redis, memcached, ..
  2. Timeline to production • Mid 2013. Investigated Mesos and Docker

    • Winter 2014 Legwork for containerization, failed attempt to deploy Mesos • Spring 2014 Containerization work and no success with CoreOS either • Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it
  3. 1. Play stage • Develop feel for Docker • Study

    namespaces, companies, cgroups, .. • Prototyping • Don't get stuck or starstruck
  4. 2. Preparation stage • Secrets (github.com/Shopify/ejson) • Assets • Logs

    • Graceful shutdown • Monitoring • Roles/Procfile • Containerization patterns
  5. 3. Containerization stage • Don't add new features to your

    infrastructure • Building images/CI/dev • Deployment • Static orchestration • Union filesystems • init in container, Linux namespaces, cgroups, routing, monitoring, security updates, registry..
  6. 4. Feature stage ! • Buttons • Distributed orchestration •

    Service discovery • Routing • CoreOS • PaaS • ..
  7. Image builder 1. Commit <sha> lands in master 2. Build

    server builds image for new SHA (~2m) 3. Pushed to internal registry: registry.shopify.com/ shopify:<sha>
  8. Builder API • script/prepare System-level packages or other infrequent changes

    on top of base image. • script/build Frequently changed dependencies, run on every build. • config/roles.json Roles for the application, our Procfile.
  9. Final image for Shopify Ubuntu ---------------- Baseimage (750 MiB) ================

    `script/prepare` (100MiB) ---------------- `script/build`: Start of week (800MiB) ---------------- `script/build`: Start of day (100MiB) ----------------- `script/build`: Current commit (80MiB)
  10. container "shopify" do role :web instances 5 health_check true http_health_check

    "/ping/ponies" revision "file" env({ RAILS_ENV: "production", UNICORN_WORKERS: node[:cores] / 5 }) end Turns into supervisor scripts: docker run -e CONTAINER=1 -e HOST_IP= \ -e ENV=production -e PORT=20003 \ -e APP_HEALTH_CHECK=/ping/ponies \ -e ENDPOINT= -e UNICORN_WORKERS=5 \ -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \ --volume=/tmp/borg-run/keys/11607:/key \ --cidfile=/tmp/borg-run/keys/11607/cidfile \ --name="borg-shopify-unicorn-4-$$" \ --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web
  11. Deploys 1. Deploy two stacks at once. 2. Capistrano updates

    revision in /etc/shopify/revision. 3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv restart • Restart ~20% of containers at a time.
  12. Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/) Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/) Panama

    canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/) Other pictures used by permissions of Florian Weingarten Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/) Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/) Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/) Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/) Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/) Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K. Video of me in walrus costume, Lily Rogers and Emma Craig