Lessons Learned From A Data Center Meltdown

No amount of planning can prevent routine hardware failures or the occasional site-wide hardware disaster. Many times your advanced planning will be an exact match for the problems the world throws at you. Every once in awhile, something unexpected happens. Tragically, our human made plans often have human fallibilities that don't handle the unexpected very well. This presentation will discuss some general practices you can use during a major disruption in your primary data center to make the best of a bad situation. We will also discuss what you can do right now to make the inevitable next occurrence less disruptive. The information presented will be based on lessons learned from multiple real world incidents, both big and small.

Transcript

1. Lessons Learned from a Data Center Meltdown Albert Crowley, Principal Engineer
2. 2 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc. WHERE AM I COMING FROM? •Grant management, research data, budget data •Low user volume, high complexity systems. •University and government provided hosting
4. 4 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc. “Be prepared.” - Every Boy Scout Ever
5. JUSTIFYING COSTS • Emergency repairs • Lost productivity • Lost goodwill • Data replacement 5 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc. Lack of investment or unwillingness to invest
6. 6 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc. “No plan survives contact with the enemy.” - Helmuth Von Moltke
7. WHAT CAN YOU DO? Develop guidelines that make the best of a bad situation Try to make the inevitable less disruptive 7 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc.
8. First step: standard maintenance message Second step: deploy a read only copy 8 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc.
9. LESSON LEARNED #1 9 Getting everyone on the phone may take longer than you like.
10. LESSON LEARNED: TIME Quality and frequency of your alerts Decision making 10
11. LESSON LEARNED: TIME Time to get the right folks to the table Escalation procedures Response times in your SLA? 11
12. LESSON LEARNED #2 12 Verify everything now. Then verify again later.
13. LESSON LEARNED: VERIFY AGAIN Architectural changes happen Back-up procedures change SLAs get updated 13
14. LESSON LEARNED: VERIFY AGAIN Regular validation of architectural diagrams and SLA is key. 14
15. LESSON LEARNED #3 15 Test your Continuity of Operations Plan
16. LESSON LEARNED: TEST YOUR COOP Test periodically New issues frequently come up over time as your main site evolves 16
17. LESSON LEARNED: TEST YOUR COOP Brainstorm failure scenarios 17
18. LESSON LEARNED #4 18 Don’t just do something, STAND THERE!
19. LESSON LEARNED: PATIENCE Think about the future. 19
20. LESSON LEARNED #5 20 Communicate. Get the word out.
21. LESSON LEARNED: COMMUNICATE Communication is hard. 21
22. LESSON LEARNED: COMMUNICATE Communication is hard. Communication when your web site is offline is much harder. 22
23. SURPRISE FEAR ANGER INTERROGATION LESSON LEARNED: COMMUNICATE 23 The Four Stages of Communication
24. POSTMORTEM Blameless Focus on improvements 24 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc.
25. POSTMORTEM What did we do at NITRC? 25 This presentation contains proprietary data that may not be disclosed or released to any individual or party without written consent from TCG, Inc. “Plan B” became our “Plan A”
26. CONTACT INFORMATION 26 Albert Crowley Principal Engineer 202-742-8477 Albert.Crowley@tcg.com