Continuous Delivery at Shopify

Talk for DevOps Ottawa Meetup, September 2017

Transcript

1. Continuous Delivery @ Shopify DevOps Ottawa Meetup John Arthorne Shopify Production Engineering @jarthorne
2. 2 Data Center Host Web Server Load Balancers Host Job Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby The Internet Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router Data Center Host Web Server Load Balancers Host Job Server Host Web Server Hosts Web Servers Host Job Server Hosts Job Servers Host DB Standby Host DB Reader Load Balancers Host DB Writer Edge Router Edge Router Shopify Architecture CDN
3. 3 Dev Time Architecture
4. A tale of two environments
5. More environments Dev Test Stage Prod App Code Parity +OS +Container +Hardware +Database +Middleware +Traffic Volume +Credentials
6. Enter continuous delivery • You can’t be sure your code works until it is in production • Minimize time to production for all changes • Small batch sizes keep the risk low • Dark launches, beta flags, ...
7. Shopify style continuous delivery • Code handoffs slow us down and hurt problem determination • Everyone in Shopify R&D can deploy • Everyone in Shopify R&D must deploy • Dedicated team to build the tools to enable everyone to ship their changes with confidence
8. Continuous delivery culture • There is a higher level of chaos with CD • Every dev takes ownership of ensuring their change lands safely • Every dev needs access and permission to act • ATC role is very helpful for herding the chaos
9. Mechanics of Shipping • Develop in a localhost environment • Push changes in a branch, make the test suite pass • Code review • Add to merge queue (or manual git merge) • Deploy to production (usually automatic) • Monitor/verify your changes
10. Local Development • Big investment in tools to automate local dev setup • Ensure it is easy to set up an env locally that is as close as possible to production
11. Getting ready to ship: Push to GitHub
13. Getting ready to ship: Make a pull request
14. Value of code review • Extra eyes catch mistakes missed during development • Pushes code towards cultural/style norms • Shared understanding of code - reduced bus factor
15. Getting things deployed: a pipeline built for speed Image Build Git Merge Automated Tests Deploy 5s 5m 5m 5m Goal: Merged to deployed in 15 minutes Pull Request
16. Deploy speed: webscale It required some considerable feats of engineering to make this pipeline fast. Why is this important? • Less wasted time for developers • Faster time to a fix for merchants • Fewer changes per deploy, so it’s safer
17. Batch Size vs Pipeline Speed 200 commits merged to shopify master on a busy day Commit every 2.4 minutes assuming 8 hour work day 3 minute deploy required for smallest batch size Builds have to keep getting faster to keep batch size down
18. Container Build Automated Tests Deploy Git Merge
19. As soon as you merge, Pipa will start building 2 Docker images, one for production, and one for the automated tests. Automated Tests Deploy Container Build Git Merge
20. Buildkite will run the 70,000+ automated tests. If the test succeeded on your branch, they will likely succeed on master after merging as well. If not, the failure has to be investigated, and potentially your merge has to be reverted. Automated Tests Deploy Container Build Git Merge
21. Buildkite Hosted build and test orchestration service Test agents run in parallel on our own GKE boxes Agents pull tests from Redis queue Ruby tests + Browser tests run with Selenium/Chrome 330 N1-standard-16 VMs 7000 Peak agents 73k Tests/Build
22. Shipit automatically deploys code to production. Changes deployed in parallel across 4 data centres, ~800 servers, and 500,000+ merchants. Automated Tests Deploy Container Build Git Merge
23. Chat notifications
24. Deploy Dashboard
25. Deploy Dashboard
28. A successful deploy
29. A failed deploy
31. • Lock automatic deploys • Roll back to previously deployed version using shipit. • Revert change in Git • Always be communicating • ATC and incident response team standing by to help What if shit hits the fan?
32. • It is impossible to simulate a production environment • Strive to keep environment differences to a minimum • Push smallest possible units of change to production continuously in order to validate code • Invest in tools to keep it flowing smoothly Summary
33. Continuous Delivery @ Shopify DevOps Ottawa Meetup John Arthorne Shopify Production Engineering @jarthorne