DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

30m presentation at DockerCon EU 2014 in Amsterdam.

https://www.youtube.com/watch?v=Qr0sATj9IVc

4d931adc15dea47e6e5854f745719269?s=128

Simon Hørup Eskildsen

December 05, 2014
Tweet

Transcript

  1. 3.

    Shopiscale • 200-300K RPM during day, > 500K during large

    sales • ~100 large metal servers running containers • 100+ product developers, ~10 infrastructure, ~10 core ops • 10 year old monolith • Ruby on Rails, MySQL, Redis, memcached, ..
  2. 6.

    Timeline to production • Mid 2013. Investigated Mesos and Docker

    • Winter 2014 Legwork for containerization, failed attempt to deploy Mesos • Spring 2014 Containerization work and no success with CoreOS either • Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it
  3. 9.

    1. Play stage • Develop feel for Docker • Study

    namespaces, companies, cgroups, .. • Prototyping • Don't get stuck or starstruck
  4. 10.

    2. Preparation stage • Secrets (github.com/Shopify/ejson) • Assets • Logs

    • Graceful shutdown • Monitoring • Roles/Procfile • Containerization patterns
  5. 11.

    3. Containerization stage • Don't add new features to your

    infrastructure • Building images/CI/dev • Deployment • Static orchestration • Union filesystems • init in container, Linux namespaces, cgroups, routing, monitoring, security updates, registry..
  6. 12.

    4. Feature stage ! • Buttons • Distributed orchestration •

    Service discovery • Routing • CoreOS • PaaS • ..
  7. 16.

    Image builder 1. Commit <sha> lands in master 2. Build

    server builds image for new SHA (~2m) 3. Pushed to internal registry: registry.shopify.com/ shopify:<sha>
  8. 18.

    Builder API • script/prepare System-level packages or other infrequent changes

    on top of base image. • script/build Frequently changed dependencies, run on every build. • config/roles.json Roles for the application, our Procfile.
  9. 19.

    Final image for Shopify Ubuntu ---------------- Baseimage (750 MiB) ================

    `script/prepare` (100MiB) ---------------- `script/build`: Start of week (800MiB) ---------------- `script/build`: Start of day (100MiB) ----------------- `script/build`: Current commit (80MiB)
  10. 20.

    container "shopify" do role :web instances 5 health_check true http_health_check

    "/ping/ponies" revision "file" env({ RAILS_ENV: "production", UNICORN_WORKERS: node[:cores] / 5 }) end Turns into supervisor scripts: docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227 \ -e ENV=production -e PORT=20003 \ -e APP_HEALTH_CHECK=/ping/ponies \ -e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5 \ -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \ --volume=/tmp/borg-run/keys/11607:/key \ --cidfile=/tmp/borg-run/keys/11607/cidfile \ --name="borg-shopify-unicorn-4-$$" \ --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web
  11. 21.

    Deploys 1. Deploy two stacks at once. 2. Capistrano updates

    revision in /etc/shopify/revision. 3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv restart • Restart ~20% of containers at a time.
  12. 23.
  13. 26.

    Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/) Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/) Panama

    canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/) Other pictures used by permissions of Florian Weingarten Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/) Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/) Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/) Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/) Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/) Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K. Video of me in walrus costume, Lily Rogers and Emma Craig
  14. 27.