High Fidelity Deployment of Micro-services

This was first presented at GovTech's STACK2022, on the 16 Nov 2022.

A recording of the full presentation may be found at GovTech's Developer Portal [here]{https://www.developer.tech.gov.sg/communities/events/conferences/stack-2022-developer-conference/recordings/all-recordings.html} and on YouTube [here]{https://www.youtube.com/watch?v=s-QZ3rR8WXU}.

Transcript

1. High-Fidelity Deployment of Microservices Samantha Wong, GovTech
2. Thought Experiment @sshinnee on GitHub
4. You might have: deployment fatigue @sshinnee on GitHub
5. Common Complaints • Feature Lockstep: All features under development must be deployed at the same time to any new environment • Feature Dysmorphia: Features that worked on lower environments do not work on Production • Bug Proliferation: New features introduce new bugs • Environment Failure: Deployment of new features prevents existing users from accessing application @sshinnee on GitHub
6. Common Complaints • Feature Lockstep: All features under development must be deployed at the same time to any new environment • Feature Dysmorphia: Features that worked on lower environments do not work on Production • Bug Proliferation: New features introduce new bugs • Environment Failure: Deployment of new features prevents existing users from accessing application @sshinnee on GitHub
7. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
8. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
9. Problem Solving • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment • Environment Branching • Canary Deployment • Blue-Green Deployment • Feature Lockstep • Feature Dysmorphia • Bug Proliferation • Environment Failure
10. Environment Branching @sshinnee on GitHub
11. @sshinnee on GitHub
12. Happy Confluence with our Card Workflow Ready for Dev In Dev Ready for Test In Test Ready for PO Sign-Off Ready for Production Deploy Done dev uat staging prep prod @sshinnee on GitHub
13. Why do we care? - Ability to work on multiple cards - At different stages/environments of development - Ability to choose which cards go out - Ability to react to production issues @sshinnee on GitHub
14. What are our goals? - Reduce complexity - Production state is the same as states on at least one lower environment - Testing of any state that would go to Production on lower environments @sshinnee on GitHub
15. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
16. Rather than explicit Release Cycles, the Sprint Cycle is the Release Cycle @sshinnee on GitHub
17. Canary (Holding) Deployment @sshinnee on GitHub
18. A new feature or product version is rolled out to a subset of users in Production @sshinnee on GitHub
19. Typically, it looks like so @sshinnee on GitHub
20. Problems come when you have microservices @sshinnee on GitHub
21. Typical Microservice Mesh @sshinnee on GitHub
22. prep prod @sshinnee on GitHub
23. prep prod @sshinnee on GitHub
24. @sshinnee on GitHub
25. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
26. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
27. Blue-Green Deployment @sshinnee on GitHub
28. Start with your original load @sshinnee on GitHub
29. Deploy a new stack, hook up with live load balancer @sshinnee on GitHub
30. Swap routing @sshinnee on GitHub
31. Delete previous stack @sshinnee on GitHub
32. Blue-Green Lifecycle From A to Z @sshinnee on GitHub
33. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
34. Group Goals • Select what features go out, and when • Confidence about deployment into Production environment ◦ Ability to test in multiple environments ◦ Ability to rollback to one previous state • Zero downtime during deployment @sshinnee on GitHub
35. Acknowledgements Behind every presentation is a team. My bosses Steven, Chris and Saiful for providing the opportunity to tackle these questions. My team, Dashboard in GoBusiness for working through these problems alongside me. @sshinnee on GitHub
36. That’s it, folks Questions? @sshinnee on GitHub @sshinnee on Speaker Deck @wong-samantha-shin-nee on Medium Samantha Wong on LinkedIn.