I have a dumb theory, but I can't find any counter-example

A post-mortem/fishing story about diagnosing and resolving a major site latency issue in the days leading up to a major relaunch of VA.gov, the home page for the Department of Veterans’ Affairs.

This incident displayed interesting characteristics, mainly that the symptom we observed caused us to over-fit our mental model of the system and spend valuable time looking in the wrong place for possible root causes.

We discuss other takeaways such as the importance of careful incident command structure, how organizational boundaries in IT organizations impact operations, and the virtuous cycle between automation and monitoring - monitoring giving you the insight to know what to change, and automation giving you the confidence to make changes, even on a live production site right before a high-profile launch.

Transcript

1. I have a dumb theory, but I can't find any counter-example Patrick Vinograd Monitorama Baltimore 2019 patrick@adhoc.team
4. Vets.gov VA.gov
7. (Dramatic Pause)
14. Deployment version? Availability zone? Serving different content? GZIP compression? Caching? Availability zone?
20. (Plus a mis-configuration)
22. Takeaways
23. You are your dependencies
24. Be careful during incidents
25. Splitting operations dilutes responsibility
26. Monitoring + Automation =
27. Thank you