Design Orgs to Prevent Incidents CNS Munich 2026

Transcript

1. Adelina Stanciu Designing Organizations to Prevent Incidents Senior Engineering Manager - Finastra
2. AGENDA Designing Organizations to Prevent Incidents 1 Introduction 2 Incident Story 1 3 Incident Story 2 4 What Both Revealed 5 Organizational Models, Design Teams 6 Practical Frameworks 7 Conclusion
3. P A R T O N E WHY INCIDENTS ARE A PROBLEM Outages, breaches and signals lost in silos. The hidden tax every engineering team pays. Designing Organizations to Prevent Incidents
4. THE PROBLEM Our biggest operational risk is organizational Operational risk emerges from organizational design, not just technical systems. Outages break continuity Disrupt operations, breach SLAs, burn revenue. Breaches reset trust Expose data, trigger fines, erase customer confidence. Signals lost in silos Warnings stay fragmented, issues slip through. Designing Organizations to Prevent Incidents
5. WHY IT MATTERS Incidents impose a hidden tax on engineering ~40% time lost to incidents vs planned delivery ~10–50+ hours lost per incident total effort to resolve (incl. context switching & RCA) 3-10 engineers pulled into each incident cost multiplies fast And it compounds: more products, more teams, more complexity, more incidents. Designing Organizations to Prevent Incidents
6. WHY ORGANIZATIONS FAIL Systems fail the way organizations fail. When incidents occur, postmortems typically focus on: Technical root causes The specific component or code path that triggered the failure Software defects Defects in logic, edge cases or untested conditions in the codebase Infrastructure failures Hardware, network or capacity issues in the underlying systems Designing Organizations to Prevent Incidents
7. P A R T T W O TWO INCIDENT STORIES Different products. Different years. The same organizational fault lines. Designing Organizations to Prevent Incidents
8. INCIDENT 01 / SEVERITY ONE THE 18-HOUR OUTAGE It began as a routine change: ❑ Certificate renewal on load balancer. ❑ Pre-approved. ❑ Previously implemented twice. ❑ First failure due to password format P L A N N E D S C O P E Certificate renewal N E T W O R K C H A N G E Load-balancer update Designing Organizations to Prevent Incidents
11. INCIDENT 01 / POST-MORTEM WHAT IT REVEALED Hidden dependencies Critical links no single team had mapped. Fragmented ownership No one accountable end to end. Incomplete rollback A change shipped before it could be undone. Weak cross-team planning Coordination happened during the outage, not before. Designing Organizations to Prevent Incidents
12. INCIDENT 02 / INTEGRATION FAILURE THE MISSING PUBLIC KEY A partner integration failed in UAT. Requests that worked yesterday were suddenly rejected. Authentication broke with no obvious cause on either side. O B S E R V E D S Y M P T O M Authentication failed The error said nothing about why. Designing Organizations to Prevent Incidents
13. INCIDENT 02 / ROOT CAUSE THE ROOT CAUSE O U R S I D E The public key was renewed on schedule → T H E G A P It was never shared with the client → T H E I R S I D E Still verifying against the old key Designing Organizations to Pevent Incidentsr
14. INCIDENT 02 / POST-MORTEM WHAT IT REVEALED No clear owner The key rotation belonged to no one in particular. No process Nothing required notifying the other side. Communication gap A change on one side, invisible to the other. No lifecycle management Credentials renewed, but never tracked. Designing Organizations to Prevent Incidents
15. P A R T T H R E E COMMON ROOT CAUSES Look past the stack traces and the same pattern appears every time. Designing Organizations to Prevent Incidents
16. ANALYSIS — CROSS INCIDENT THE SHARED PATTERN 01 Communication gaps 02 Planning gaps 03 Missing ownership 04 Hidden dependencies 05 Tribal knowledge 06 Siloed teams Designing Organizations to Prevent Incidents
17. P A R T F O U R DESIGNING ORGANIZATIONS LIKE SYSTEMS Borrow the discipline we already apply to software. Designing Organizations to Prevent Incidents
18. MODEL - DOMAIN-DRIVEN ORG DESIGN O R G A N I Z A T I O N S A S S Y S T E M S THE DOMAIN-DRIVEN ANALOGY Ownership Every capability has one team that owns it. Boundaries Clear edges define where responsibility ends. Contracts Interfaces between teams are explicit and stable. Communication paths How information crosses boundaries is designed, not assumed. Designing Organizations to Prevent Incidents Critical questions Who owns this capability? Who owns this process? What are the dependencies? How are changes coordinated?
20. P A R T F I V E ORGANIZATIONAL FRAMEWORKS Turning good structure into repeatable change. Designing Organizations to Prevent Incidents
22. THE QUALITY PROCEDURE The procedure exists to force the right conversations EVERY CHANGE REQUIRES • Planning & Dependency Review • Cross-team Involvement • Risk Assessment • Supporting Team Tickets RELEASE READINESS • Testing Completed • Rollback Documented and Tested • Monitoring Configured • Stakeholders Informed Designing Organizations to Prevent Incidents
25. P A R T S I X CONTINUOUS LEARNING Every incident is data. Build the loop that turns it into design. Designing Organizations to Prevent Incidents
27. LEARNING LOOP — PRODUCT'S ROLE WHAT PRODUCT BRINGS Reliability is a product decision too. A clear error message turns a multi-hour investigation into a five-minute fix. B E F O R E “Authentication failed” ↓ A F T E R “Client public key not found” Designing Organizations to Prevent Incidents
28. LEARNING LOOP - ENG & ARCHITECTURE WHAT ENGINEERING AND ARCHITECTURE BRING DEBT AND SYSTEMIC RISK Engineering Which technical debt is quietly raising the odds of the next incident and what it costs to retire it. Architecture The systemic risks and hidden dependencies no single team can see from the inside. Designing Organizations to Prevent Incidents
29. REDUCING THE BLAST RADIUS OPERATIONAL EXCELLENCE Automated deploy & test Monitoring & Alerts Blue-green deployments Disaster recovery drills Multi-region failover Tested rollback Designing Organizations to Prevent Incidents Organizational design reduces likelihood of incidents. Engineering practices reduce impact of incidents.
30. WHAT SRE, DEVOPS & SECURITY BRING SRE Where the system actually strains under load and which alerts predict real failure. DevOps Where the pipeline is fragile, and which manual steps keep causing the same mistakes. Security Where the system gets probed and abused, and which weaknesses attackers reach first. Designing Organizations to Prevent Incidents LEARNING LOOP — DEVSECOPS & SRE ROLE
33. Thank You!