A DevOps story - Turning our most significant production outage into a driver for positive lasting change. A DevOps story.

Transcript

1. A DevOps story How we turned our most significant outage into lasting positive change! @martincronje
3. 💡 Cohesive collaborative teams with end-to-end ownership of their services (build it, run it)
4. 2000’s First online product launched Second major iteration launched
5. Writing code during the early 2000’s
6. ❤ The heart of the product was built in the 2000’s
7. 💡 All companies have heritage systems that enabled them to become successful businesses!
8. Question: What is the heritage in your environment?
9. 2000’s First online product launched Second major iteration launched Starts building innovative user experience 2010’s Moved to the cloud | Disrupts market and expands globally
10. System Context S Orignal UI Core Product (Monolith) 🥰
11. System Context Z S New API Orignal UI UI UI UI Service Service Service Core Product (Monolith) 🥰
12. Environment Context Z S New API Orignal UI UI UI UI Service Service Service Core Product (Monolith) •New Product • Cloud-native • Continuous deployment • Some auto-infrastructure provisioning •Heritage Product • Virtual servers • Manual deployments • Manual infrastructure provisioning
13. Environment Context Z S New API Orignal UI UI UI UI Service Service Service Core Product (Monolith) • Agile delivery • Low test-automation • Incidents highly disruptive • Reliance on long-tenured engineers • Traditional ops team (HIGH TOIL)
14. 40+ major incidents per month
15. The perfect storm was brewing!
16. Question: Do you have a perfect storms brewing?
17. 2000’s First online product launched Second major iteration launched Starts building innovative user experience 2010’s Moved to the cloud | Disrupts market and expands globally 💥 Super outage
18. The super outage > Regular deployment with hundreds unique changes > Unusable for numerous weeks > Busy time of the year > Most customers affected > System performance continued to degrade
19. Unable to isolate the problem
20. Unable to restore a previous version
21. Unable to failover
22. Unable to troubleshoot
23. Unable to create a new environment
24. 🤕 😴 🤬
25. Opportunity Strikes! > Reduce the size of changes > Make rollback effortless > Reduce configuration change risk > Make it easy to setup a new environment > Make it easy to analyse and troubleshoot problems > Give everyone the skills to fix problems > Handle production outages cool, calm and controlled
26. With 40+ incidents/month a catastrophic outage was only a matter of time
27. Question: What would you do address this risk?
28. 😱 We needed to remove fear
30. Technology Improvement Plan > Incident Response > Deep Ownership > Production Insights > Deploy Monolith Safely > Decouple Monolith Z S New API Orignal UI UI UI UI Service Service Service Core Product (Monolith)
31. Pitching our plan…. > Economic argument on supporting scale > Proposed a stop to delivery to focus on resilience > Ended up securing about 1/5 of engineering capacity https://www.wired.com/2013/04/linkedin-software-revolution/
32. 💡 Initial objective was to reduce the impact of failure on customers
33. Technology Improvement Plan > Incident Response > Deep Ownership > Production Insights > Deploy Monolith Safely > Decouple Monolith Z S New API Orignal UI UI UI UI Service Service Service Core Product (Monolith)
34. Planning approach • Outcomes focus charting (Auftragsklärung) • 1-pager plan outlining - Context - Problem - Objective - Outputs - Success measures • Incremental approach where possible https://auftragsklaerung.com/
35. Enabling us to break-up the monolith
36. Incident Response > Primary Objective: Shorten repair and recovery times > Secondary Objective: Make sure they don’t happen again
37. Y1Q1 Y1Q2 Y1Q3 Y1Q4 Y2Q1 New incident manager (ITIL) Incident response process and prioritisation matrix Engineers on-call PagerDuty 🤯 🤑 Marked reduction in incidents 💥 Incident Command Weekly Ops Review 💥 Disestablish tech-support 🎯 Incident recovery up to 95% SLA Incident recovery at <80% SLA Engineering doing post-mortems for Sev2’s 💥 Engineering doing post-mortems for Sev0 and Sev1 incidents 💡 Make sure incidents don’t happen again 💡 Time to acknowledge incidents 💡 Awareness of sensitive customers
38. 💡 Started with old-school incident management, because it was the right thing to do.
39. Production Insights > Primary Objective: Standard measurement of health > Secondary Objective: Don’t call us, we’ll call you
40. Service Level Objectives Availability = Successful requests Total requests
41. Implementing SLO’s > Top down approach > Key interactions workflow > Failed requests included slow and errors > Differentiated between desired, current and failing > Teams implemented SLI dashboards to back these up
42. Y1Q1 Y1Q2 Y1Q3 Y1Q4 Y2Q1 🤯 Manual SLO recording in Excel! 🎯 First basic dashboards internally (incl. support) 💥 Dashboard released to some customers Prioritised improvement SLOs to 75% compliance 🤯 Engineering site-visits 🎯 Customers stopped complaining Dedicated work-streams to dramatically improve performance
43. 💡 We created a standard definition of what good looks like for the customer
44. Deployment Safety > Primary Objective: Reduce the impact of bad deployments > Secondary Objective: Reduce undifferentiated heavy-lifting
45. Canary deployments on monolith C1 v2 C3 v2 C2 v2 C4 v2 C1 v1 C3 v1 C2 v1 C4 v1 Active Passive
46. Canary deployments C1 v2 C3 v2 C2 v2 C1 v1 C3 v1 C2 v1 C4 v1 Active Passive Easy rollback C4 v2
47. Canary deployments C1 v2 C3 v2 C2 v2 C4 v2 C1 v3 C3 v1 C2 v1 C4 v1 Active Passive Deploy v3
48. Canary deployments C3 v2 C2 v2 C4 v2 C1 v3 C3 v1 C2 v1 C4 v1 Active Passive C1 v2 Make live
49. Canary deployments C3 v2 C2 v2 C4 v2 C1 v3 C3 v1 C2 v3 C4 v1 Active Passive C1 v2 Deploy v3
50. Canary deployments C3 v2 C2 v2 C4 v2 C1 v3 C3 v1 C2 v3 C4 v1 Active Passive C1 v2 Make live
51. Database Blue / Green Schema v2 Schema v1
52. Other things • Moved SQL Server to management instances • Moved all config management into source-control • Eliminate toil where possible
53. 💡 Not only did we reduce impact of issues but we also reduced snowflakes and toil.
54. This is was only the start…
55. Lessons > Spend A LOT of time with stakeholders > Need technical executive support > Back out of failed experiments early > Getting this right is hard
56. Achievements • Single definition of good using Service Level Objectives • We started detecting issues before out customers! • Incidents resolved within SLA (improved from 80% to 95%+) • Mostly automated deployment and provisioning on our monolith • Reducing the impact of bad deployments • Rollback within minutes instead of hours (or even days) • Enabled modernisation roadmap • Happy customers!!
57. Question: What is your key takeaway?
58. 💡 There’s no perfect environment. Find the right DevOps topology for you.
59. 💡 Cohesive collaborative teams with end-to-end ownership of their services (build it, run it)
60. 💡 All companies have heritage systems that enabled them to become successful businesses!
61. 💡 Find incremental gains by looking at the biggest fires in your environment
62. 💡 Find your initial objective! We reduced failure impact
63. 💡 Started with old-school processes because it was the right thing to do. …and then we matured
64. @martincronje