$30 off During Our Annual Pro Sale. View Details »

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

30m presentation at DockerCon EU 2014 in Amsterdam.

https://www.youtube.com/watch?v=Qr0sATj9IVc

Simon Hørup Eskildsen

December 05, 2014
Tweet

More Decks by Simon Hørup Eskildsen

Other Decks in Technology

Transcript

  1. Docker at Shopify
    From This-Looks-Fun to Production

    View Slide

  2. Simon Eskildsen
    Infrastructure @ Shopify
    @Sirupsen

    View Slide

  3. Shopiscale
    • 200-300K RPM during day, > 500K during large sales
    • ~100 large metal servers running containers
    • 100+ product developers, ~10 infrastructure, ~10 core ops
    • 10 year old monolith
    • Ruby on Rails, MySQL, Redis, memcached, ..

    View Slide

  4. Make it easier to
    solve problems at
    Shopify

    View Slide

  5. Container
    -------
    Platform

    View Slide

  6. Timeline to production
    • Mid 2013. Investigated Mesos and Docker
    • Winter 2014 Legwork for containerization, failed attempt to
    deploy Mesos
    • Spring 2014 Containerization work and no success with
    CoreOS either
    • Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it

    View Slide

  7. Ship Docker as the only
    change

    View Slide

  8. Successfully shipping Docker
    1. Play
    2. Prepare
    3. Containerize
    4. Feature

    View Slide

  9. 1. Play stage
    • Develop feel for Docker
    • Study namespaces, companies, cgroups, ..
    • Prototyping
    • Don't get stuck or starstruck

    View Slide

  10. 2. Preparation stage
    • Secrets (github.com/Shopify/ejson)
    • Assets
    • Logs
    • Graceful shutdown
    • Monitoring
    • Roles/Procfile
    • Containerization patterns

    View Slide

  11. 3. Containerization stage
    • Don't add new features to your infrastructure
    • Building images/CI/dev
    • Deployment
    • Static orchestration
    • Union filesystems
    • init in container, Linux namespaces, cgroups, routing,
    monitoring, security updates, registry..

    View Slide

  12. 4. Feature stage !
    • Buttons
    • Distributed orchestration
    • Service discovery
    • Routing
    • CoreOS
    • PaaS
    • ..

    View Slide

  13. Verschlimmbesserung

    View Slide

  14. What we took to
    production

    View Slide

  15. Image Builds

    View Slide

  16. Image builder
    1. Commit lands in master
    2. Build server builds image for new SHA (~2m)
    3. Pushed to internal registry: registry.shopify.com/
    shopify:

    View Slide

  17. We don't use Dockerfiles

    View Slide

  18. Builder API
    • script/prepare System-level packages or other infrequent
    changes on top of base image.
    • script/build Frequently changed dependencies, run on
    every build.
    • config/roles.json Roles for the application, our
    Procfile.

    View Slide

  19. Final image for Shopify
    Ubuntu
    ----------------
    Baseimage (750 MiB)
    ================
    `script/prepare` (100MiB)
    ----------------
    `script/build`: Start of week (800MiB)
    ----------------
    `script/build`: Start of day (100MiB)
    -----------------
    `script/build`: Current commit (80MiB)

    View Slide

  20. container "shopify" do
    role :web
    instances 5
    health_check true
    http_health_check "/ping/ponies"
    revision "file"
    env({
    RAILS_ENV: "production",
    UNICORN_WORKERS: node[:cores] / 5
    })
    end
    Turns into supervisor scripts:
    docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227 \
    -e ENV=production -e PORT=20003 \
    -e APP_HEALTH_CHECK=/ping/ponies \
    -e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5 \
    -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \
    --volume=/tmp/borg-run/keys/11607:/key \
    --cidfile=/tmp/borg-run/keys/11607/cidfile \
    --name="borg-shopify-unicorn-4-$$" \
    --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web

    View Slide

  21. Deploys
    1. Deploy two stacks at once.
    2. Capistrano updates revision in /etc/shopify/revision.
    3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv
    restart
    • Restart ~20% of containers at a time.

    View Slide

  22. Containers are the future,
    but it's still hard in
    production

    View Slide

  23. View Slide

  24. Simon Eskildsen, Infrastructure @ Shopify
    @Sirupsen

    View Slide

  25. Thank You.

    View Slide

  26. Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/)
    Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/)
    Panama canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/)
    Other pictures used by permissions of Florian Weingarten
    Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/)
    Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/)
    Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/)
    Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/)
    Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/)
    Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K.
    Video of me in walrus costume, Lily Rogers and Emma Craig

    View Slide

  27. View Slide