DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

30m presentation at DockerCon EU 2014 in Amsterdam.

https://www.youtube.com/watch?v=Qr0sATj9IVc

4d931adc15dea47e6e5854f745719269?s=128

Simon Hørup Eskildsen

December 05, 2014
Tweet

Transcript

  1. Docker at Shopify From This-Looks-Fun to Production

  2. Simon Eskildsen Infrastructure @ Shopify @Sirupsen

  3. Shopiscale • 200-300K RPM during day, > 500K during large

    sales • ~100 large metal servers running containers • 100+ product developers, ~10 infrastructure, ~10 core ops • 10 year old monolith • Ruby on Rails, MySQL, Redis, memcached, ..
  4. Make it easier to solve problems at Shopify

  5. Container ------- Platform

  6. Timeline to production • Mid 2013. Investigated Mesos and Docker

    • Winter 2014 Legwork for containerization, failed attempt to deploy Mesos • Spring 2014 Containerization work and no success with CoreOS either • Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it
  7. Ship Docker as the only change

  8. Successfully shipping Docker 1. Play 2. Prepare 3. Containerize 4.

    Feature
  9. 1. Play stage • Develop feel for Docker • Study

    namespaces, companies, cgroups, .. • Prototyping • Don't get stuck or starstruck
  10. 2. Preparation stage • Secrets (github.com/Shopify/ejson) • Assets • Logs

    • Graceful shutdown • Monitoring • Roles/Procfile • Containerization patterns
  11. 3. Containerization stage • Don't add new features to your

    infrastructure • Building images/CI/dev • Deployment • Static orchestration • Union filesystems • init in container, Linux namespaces, cgroups, routing, monitoring, security updates, registry..
  12. 4. Feature stage ! • Buttons • Distributed orchestration •

    Service discovery • Routing • CoreOS • PaaS • ..
  13. Verschlimmbesserung

  14. What we took to production

  15. Image Builds

  16. Image builder 1. Commit <sha> lands in master 2. Build

    server builds image for new SHA (~2m) 3. Pushed to internal registry: registry.shopify.com/ shopify:<sha>
  17. We don't use Dockerfiles

  18. Builder API • script/prepare System-level packages or other infrequent changes

    on top of base image. • script/build Frequently changed dependencies, run on every build. • config/roles.json Roles for the application, our Procfile.
  19. Final image for Shopify Ubuntu ---------------- Baseimage (750 MiB) ================

    `script/prepare` (100MiB) ---------------- `script/build`: Start of week (800MiB) ---------------- `script/build`: Start of day (100MiB) ----------------- `script/build`: Current commit (80MiB)
  20. container "shopify" do role :web instances 5 health_check true http_health_check

    "/ping/ponies" revision "file" env({ RAILS_ENV: "production", UNICORN_WORKERS: node[:cores] / 5 }) end Turns into supervisor scripts: docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227 \ -e ENV=production -e PORT=20003 \ -e APP_HEALTH_CHECK=/ping/ponies \ -e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5 \ -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \ --volume=/tmp/borg-run/keys/11607:/key \ --cidfile=/tmp/borg-run/keys/11607/cidfile \ --name="borg-shopify-unicorn-4-$$" \ --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web
  21. Deploys 1. Deploy two stacks at once. 2. Capistrano updates

    revision in /etc/shopify/revision. 3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv restart • Restart ~20% of containers at a time.
  22. Containers are the future, but it's still hard in production

  23. None
  24. Simon Eskildsen, Infrastructure @ Shopify @Sirupsen

  25. Thank You.

  26. Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/) Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/) Panama

    canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/) Other pictures used by permissions of Florian Weingarten Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/) Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/) Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/) Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/) Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/) Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K. Video of me in walrus costume, Lily Rogers and Emma Craig
  27. None