Disaster Recovery: A Process, Not a Tool

As presented at PGDay Boston 2026

Transcript

1. Disaster Recovery A Process, not a Tool June 9, 2026 Richard Yen
2. The Changed Landscape
4. The Changed Landscape •99.99% Uptime •p95, p99 metrics •Status pages •Social Media affects reputation
5. Agenda 1. Where We Are 2. Where We Need to Be 3. How We’ll Get There 4. Some Stories Along the Way
6. Where We Are
7. A disaster is any sustained event that compromises the system’s availability, correctness, or business trust
8. How DR is Usually Done 1. Prepare 2. Prevent
9. How DR is Usually Done 1. Prepare 2. Prevent “An ounce of prevention is worth a pound of cure”
10. Disaster Recovery is the act of restoring business operations
11. Where We Need to Be
12. Postgres Makes Recovery Easy • pg_dump/pg_restore • pg_basebackup • pg_stat_replication • pg_stat_activity • Point-In-Time Recovery • repmgr/efm • Third-party backup tools
13. RPO & RTO
14. RPO & RTO – It’s going to cost you
15. RPO & RTO Talk to your leadership, and you’ll discover how much it’s really worth to them
16. RPO 1. 24-hour RPO -- $ 2. 15-minute RPO -- $$ 3. Near-zero RPO -- $$$
17. RTO is your team’s ability to execute the DR plan
18. How We’ll Get There
19. 3 Layers of DR Planning 1. Infrastructure failure 2. Procedural failure 3. Human failure
20. 3 Layers of DR Planning 1. Infrastructure failure 2. Procedural failure 3. Human failure Recovery is not always about failing over
21. Runbook Engineering Should Assume 1. Stress 2. Chaos 3. Confusion 4. Exhaustion 5. Ambiguity
22. Runbook Engineering Should Assume 1. Stress 2. Chaos 3. Confusion 4. Exhaustion 5. Ambiguity
23. Runbook Engineering: Anti-patterns 1. Wiki Pages 2. Stale documents 3. Unclear owner 4. Vague instructions
24. Runbook Engineering: Non-Technical Essentials 1. Incident Commander 2. Communications Owner 3. Notification Cadence 4. Escalation Chain 5. Risk Authorization
25. Runbook Validation 1. Can a new engineer follow it? 2. Does it assume access? 3. Are commands and names current? 4. Does it get regular playtime?
26. Runbook Validation: Level Up Your Ability 1. Prove that your Runbook works 2. Reduce the time it takes to complete 3. Simulate failure 4. Test with unavailable human resources
27. Runbook Validation: Level Up Your Ability 1. Prove that your Runbook works 2. Reduce the time it takes to complete 3. Simulate failure 4. Test with unavailable human resources This is how you reduce RTO
28. Validation Metrics 1. Did recovery succeed? 2. How long did each section take? 3. What vagueness needs to be clarified? 4. Identify documentation gaps
29. Validation Metrics 1. Did recovery succeed? 2. How long did each section take? 3. What vagueness needs to be clarified? 4. Identify documentation gaps 5. Be Encouraging! Go out for dinner!
30. Don’t Blame, or You’ll Feel Lame 1. Communication is Key 2. People hide when they feel shame 3. When people don’t feel safe to ask, they guess 4. Guessing hurts your RTO
31. Make your RPO worth it by investing in your RTO
32. © Copyright Microsoft Corporation. All rights reserved.