DockerCon EU: Docker at Shopify: From This-Looks-Fun to Production

30m presentation at DockerCon EU 2014 in Amsterdam.

https://www.youtube.com/watch?v=Qr0sATj9IVc

Transcript

1. Docker at Shopify
   From This-Looks-Fun to Production
   

   View Slide

2. Simon Eskildsen
   Infrastructure @ Shopify
   @Sirupsen
   

   View Slide

3. Shopiscale
   • 200-300K RPM during day, > 500K during large sales
   • ~100 large metal servers running containers
   • 100+ product developers, ~10 infrastructure, ~10 core ops
   • 10 year old monolith
   • Ruby on Rails, MySQL, Redis, memcached, ..
   

   View Slide

4. Make it easier to
   solve problems at
   Shopify
   

   View Slide

5. Container
   -------
   Platform
   

   View Slide

6. Timeline to production
   • Mid 2013. Investigated Mesos and Docker
   • Winter 2014 Legwork for containerization, failed attempt to
   deploy Mesos
   • Spring 2014 Containerization work and no success with
   CoreOS either
   • Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it
   

   View Slide

7. Ship Docker as the only
   change
   

   View Slide

8. Successfully shipping Docker
   1. Play
   2. Prepare
   3. Containerize
   4. Feature
   

   View Slide

9. 1. Play stage
   • Develop feel for Docker
   • Study namespaces, companies, cgroups, ..
   • Prototyping
   • Don't get stuck or starstruck
   

   View Slide

10. 2. Preparation stage
    • Secrets (github.com/Shopify/ejson)
    • Assets
    • Logs
    • Graceful shutdown
    • Monitoring
    • Roles/Procfile
    • Containerization patterns
    

    View Slide

11. 3. Containerization stage
    • Don't add new features to your infrastructure
    • Building images/CI/dev
    • Deployment
    • Static orchestration
    • Union filesystems
    • init in container, Linux namespaces, cgroups, routing,
    monitoring, security updates, registry..
    

    View Slide

12. 4. Feature stage !
    • Buttons
    • Distributed orchestration
    • Service discovery
    • Routing
    • CoreOS
    • PaaS
    • ..
    

    View Slide

13. Verschlimmbesserung
    

    View Slide

14. What we took to
    production
    

    View Slide

15. Image Builds
    

    View Slide

16. Image builder
    1. Commit lands in master
    2. Build server builds image for new SHA (~2m)
    3. Pushed to internal registry: registry.shopify.com/
    shopify:
    

    View Slide

17. We don't use Dockerfiles
    

    View Slide

18. Builder API
    • script/prepare System-level packages or other infrequent
    changes on top of base image.
    • script/build Frequently changed dependencies, run on
    every build.
    • config/roles.json Roles for the application, our
    Procfile.
    

    View Slide

19. Final image for Shopify
    Ubuntu
    ----------------
    Baseimage (750 MiB)
    ================
    `script/prepare` (100MiB)
    ----------------
    `script/build`: Start of week (800MiB)
    ----------------
    `script/build`: Start of day (100MiB)
    -----------------
    `script/build`: Current commit (80MiB)
    

    View Slide

20. container "shopify" do
    role :web
    instances 5
    health_check true
    http_health_check "/ping/ponies"
    revision "file"
    env({
    RAILS_ENV: "production",
    UNICORN_WORKERS: node[:cores] / 5
    })
    end
    Turns into supervisor scripts:
    docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227 \
    -e ENV=production -e PORT=20003 \
    -e APP_HEALTH_CHECK=/ping/ponies \
    -e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5 \
    -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \
    --volume=/tmp/borg-run/keys/11607:/key \
    --cidfile=/tmp/borg-run/keys/11607/cidfile \
    --name="borg-shopify-unicorn-4-$$" \
    --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web
    

    View Slide

21. Deploys
    1. Deploy two stacks at once.
    2. Capistrano updates revision in /etc/shopify/revision.
    3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv
    restart
    • Restart ~20% of containers at a time.
    

    View Slide

22. Containers are the future,
    but it's still hard in
    production
    

    View Slide

23. View Slide

24. Simon Eskildsen, Infrastructure @ Shopify
    @Sirupsen
    

    View Slide

25. Thank You.
    

    View Slide

26. Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/)
    Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/)
    Panama canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/)
    Other pictures used by permissions of Florian Weingarten
    Flight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/)
    Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/)
    Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/)
    Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/)
    Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/)
    Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K.
    Video of me in walrus costume, Lily Rogers and Emma Craig
    

    View Slide

27. View Slide