Moving Shopify Core to the Cloud

Presented at SREcon Asia 2018 by Scott Francis.

This talk covers Shopify's move from a DC-based infrastructure to a Kubernetes cloud-based architecture.

Transcript

1. Moving Shopify Core to the Scott Francis scott.francis@shopify.com @planetscott
4. 80k 600k+ requests per second peak active merchants $26b 40+ gmv (2017) daily deploys 800+ developers
6. Part One: Pre-
7. Shopify early 2013
8. Sharding - 2013
9. Multi DC - 2015
12. A (Shopify) pod is an isolated unit of one or more shops.
13. Pods - 2016
15. + =
16. Pod Routing
17. Part Two:
18. Building for the long term.
19. Compliance
22. Performance
26. Consistency
29. Simplicity
32. Execution
34. Pod Routing 2.0
35. “We’re going to move to the cloud one shop at a time.” - Nameless Shopify Production Engineer
36. Shop Mover • Data migration uses Ghostferry • Moving a shop is NOT trivial • Randomized shop selection
37. Shop Mover New Shop Move Begin binlog Copy SELECT and INSERT Wait for binlog Stop Shop Writes Complete Replication Switch Shard ID Allow Shop Writes Shop Move Complete Perform Sanity Checks Wait for Jobs
38. Cloud Shards
41. Activefailover Start Pod Failover Update Pod Location Pause Requests Failover MySQL Clean Up Transfer Jobs Pod Failover Complete Perform Sanity Checks Resume Requests
42. Problems
43. Fear, Uncertainty, Doubt • Database Performance • Kubernetes • Cloud Infrastructure
44. Scaling Kubernetes Clusters
45. Shopify Core k8s v1
46. k8s Scaling Problems • Running out of routes?? • How to communicate across clusters?? • Cross-cluster service discovery??
47. k8s Scaling Solutions • Running out of routes?? • Cross-cluster service discovery?? • IP Aliasing • External kube-dns ILB
48. External kube-dns ILB
49. Shopify Core k8s v2
50. kubectl is a really big
51. kubernetes-deploy • Pre-deploys dependent resources • Watches changes to make sure they’re successful • Runs tasks at the beginning of the deploy
52. kubernetes-deploy
53. Lessons Learned
54. Maintaining two separate infrastructures is really hard.
55. Lift and Shift, or: don’t do too much at the same time
56. Iterate
57. Automate
58. Thanks! p.s. we’re hiring
59. Links - Ghostferry: https://github.com/Shopify/ghostferry - kubernetes-deploy: https://github.com/Shopify/kubernetes-deploy - Shipit: https://github.com/Shopify/shipit-engine - Chat Ops at Shopify: https://engineering.shopify.com/blogs/engineering/ implementing-chatops-into-our-incident-management-procedure
60. Image Credits - https://i.ytimg.com/vi/u19mQlQeMmE/maxresdefault.jpg - http://static.tvtropes.org/pmwiki/pub/images/and_mission_control_rejoiced_7367.jpg - https://blogs.voanews.com/tedlandphairsamerica/files/2011/06/03-GM_Exhibition_Assembly_Line.jpg - https://technofaq.org/wp-content/uploads/2015/10/cabling.jpg - https://eliteequestrianmagazine.com/wp-content/uploads/2012/09/Roman-Riding-crp.jpeg