Rapid Release at Massive Scale

The study of Facebook's release architect: Rapid Release at Massive Scale

1. Rapid Release at Massive Scale 20171106 Proko
2. About Author
3. Agenda • Canary Deployment • The reasons why adopt rapid release at massive scale • Solutions • Limitations
4. Canary Deployment
5. Canary Deployment (cont.) • Canaries were once regularly used in coal mining as an early warning system • Distributed to a small number of users • As an early warning indicator with less impact on downtime
6. Canary Deployment • Deploy to one or more canary servers • Test, or wait until satisfied • Deploy to the remaining servers
7. Why
8. Previous deployment process Three times a day Cherry-picks
9. Weekly Web Branch
10. Time Zone Issue Release duration: 14 hours every single week London New York Menlo Park
11. How
12. Quasi-continuous “push from master” system
13. Tool and Process
14. Improve tools and process
15. Continuous Integration
16. Gatekeeper System
17. Test Tool Deployment (cont.)
18. Test Tool Deployment (cont.)
19. Test Tool Deployment (cont.)
20. Test Tool Deployment (cont.)
21. Test Tool Deployment
22. Continuous Delivery at Scale
23. Continuous Delivery at Scale (cont.)
24. Continuous Delivery at Scale: C1 (internal)
25. Canary Deployment Continuous Delivery at Scale: C2 (production)
26. Continuous Delivery at Scale: C3
27. Result
28. System
29. No More Hotfix
30. Deploy anytime, anywhere All engineers everywhere in the world can develop and deliver their code when it makes sense for them.
31. Forcing Function
32. Make the user experience better, faster
33. Limitation
36. Android OS Version
37. Solution
38. What’s more • Use Puffin for Facebook (For Facebook) • Add more checking point which can be executed automatically • The result can be checked by developers if it is failed
39. Reference 1. Article: https://code.facebook.com/posts/270314900139291/rapid-release-at-mas sive-scale/ 2. Video: https://atscaleconference.com/videos/rapid-release-at-massive-scale/ 3. Canary Deployment: https://blog.ccjeng.com/2015/06/canary-deployment.html https://octopus.com/docs/patterns/canary-deployments
40. Q&A