Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Delivery at Shopify

A5f3383a1a0c7e6d3df7f06361e39a5c?s=47 John Arthorne
September 07, 2017

Continuous Delivery at Shopify

Talk for DevOps Ottawa Meetup, September 2017

A5f3383a1a0c7e6d3df7f06361e39a5c?s=128

John Arthorne

September 07, 2017
Tweet

Transcript

  1. Continuous Delivery @ Shopify DevOps Ottawa Meetup John Arthorne Shopify

    Production Engineering @jarthorne
  2. 2 Data Center Host Web Server Load Balancers Host Job

    Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby The Internet Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router Data Center Host Web Server Load Balancers Host Job Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router Shopify Architecture CDN
  3. 3 Dev Time Architecture

  4. A tale of two environments

  5. More environments Dev Test Stage Prod App Code Parity +OS

    +Container +Hardware +Database +Middleware +Traffic Volume +Credentials
  6. Enter continuous delivery • You can’t be sure your code

    works until it is in production • Minimize time to production for all changes • Small batch sizes keep the risk low • Dark launches, beta flags, ...
  7. Shopify style continuous delivery • Code handoffs slow us down

    and hurt problem determination • Everyone in Shopify R&D can deploy • Everyone in Shopify R&D must deploy • Dedicated team to build the tools to enable everyone to ship their changes with confidence
  8. Continuous delivery culture • There is a higher level of

    chaos with CD • Every dev takes ownership of ensuring their change lands safely • Every dev needs access and permission to act • ATC role is very helpful for herding the chaos
  9. Mechanics of Shipping • Develop in a localhost environment •

    Push changes in a branch, make the test suite pass • Code review • Add to merge queue (or manual git merge) • Deploy to production (usually automatic) • Monitor/verify your changes
  10. Local Development • Big investment in tools to automate local

    dev setup • Ensure it is easy to set up an env locally that is as close as possible to production
  11. Getting ready to ship: Push to GitHub

  12. None
  13. Getting ready to ship: Make a pull request

  14. Value of code review • Extra eyes catch mistakes missed

    during development • Pushes code towards cultural/style norms • Shared understanding of code - reduced bus factor
  15. Getting things deployed: a pipeline built for speed Image Build

    Git Merge Automated Tests Deploy 5s 5m 5m 5m Goal: Merged to deployed in 15 minutes Pull Request
  16. Deploy speed: webscale It required some considerable feats of engineering

    to make this pipeline fast. Why is this important? • Less wasted time for developers • Faster time to a fix for merchants • Fewer changes per deploy, so it’s safer
  17. Batch Size vs Pipeline Speed 200 commits merged to shopify

    master on a busy day Commit every 2.4 minutes assuming 8 hour work day 3 minute deploy required for smallest batch size Builds have to keep getting faster to keep batch size down
  18. Container Build Automated Tests Deploy Git Merge

  19. As soon as you merge, Pipa will start building 2

    Docker images, one for production, and one for the automated tests. Automated Tests Deploy Container Build Git Merge
  20. Buildkite will run the 70,000+ automated tests. If the test

    succeeded on your branch, they will likely succeed on master after merging as well. If not, the failure has to be investigated, and potentially your merge has to be reverted. Automated Tests Deploy Container Build Git Merge
  21. Buildkite Hosted build and test orchestration service Test agents run

    in parallel on our own GKE boxes Agents pull tests from Redis queue Ruby tests + Browser tests run with Selenium/Chrome 330 N1-standard-16 VMs 7000 Peak agents 73k Tests/Build
  22. Shipit automatically deploys code to production. Changes deployed in parallel

    across 4 data centres, ~800 servers, and 500,000+ merchants. Automated Tests Deploy Container Build Git Merge
  23. Chat notifications

  24. Deploy Dashboard

  25. Deploy Dashboard

  26. None
  27. None
  28. A successful deploy

  29. A failed deploy

  30. None
  31. • Lock automatic deploys • Roll back to previously deployed

    version using shipit. • Revert change in Git • Always be communicating • ATC and incident response team standing by to help What if shit hits the fan?
  32. • It is impossible to simulate a production environment •

    Strive to keep environment differences to a minimum • Push smallest possible units of change to production continuously in order to validate code • Invest in tools to keep it flowing smoothly Summary
  33. Continuous Delivery @ Shopify DevOps Ottawa Meetup John Arthorne Shopify

    Production Engineering @jarthorne