devopsdays MSP 2018: Staying Alive

Transcript

1. STAYING ALIVE PATTERNS FOR FAILURE MANAGEMENT FROM THE BOTTOM OF THE OCEAN 1 — devopsdays MSP 2018 @rondoftw
2. Ronnie Chen @rondoftw 2 — devopsdays MSP 2018 @rondoftw
3. 3 — devopsdays MSP 2018 @rondoftw
5. 5 — devopsdays MSP 2018 @rondoftw
6. 6 — devopsdays MSP 2018 @rondoftw
7. 7 — devopsdays MSP 2018 @rondoftw
8. 8 — devopsdays MSP 2018 @rondoftw
9. TECHNICAL DIVING ▸ longer dive times ▸ deeper dives ▸ overhead ceiling ▸ decompression obligations ▸ more gear. a lot more. ▸ higher pressure ▸ more risks 9 — devopsdays MSP 2018 @rondoftw
10. RISKS MAY INCLUDE... 1. hypoxia 2. hyperoxia 3. nitrogen narcosis 4. carbon dioxide buildup 5. oxygen sensor failure 6. deep tissue isobaric counterdiffusion (ICD) 7. high pressure nervous syndrome (HPNS) 8. exhausting your carbon dioxide scrubber 9. carbon dioxide channeling from a poorly packed scrubber 10. carbon buildup causing an spark leading to an oxygen fire. underwater. 11. flooding of breathing loop or circuitry 12. water mixing with the scrubbing agent to produce a toxic caustic soda that will give you chemical burns on your mouth, airway, and lungs 13. plain old decompression sickness 10 — devopsdays MSP 2018 @rondoftw
11.   If you own a rebreather for five years, two percent of you are going to die on it. — Jill Heinerth, underwater explorer 11 — devopsdays MSP 2018 @rondoftw
12. 12 — devopsdays MSP 2018 @rondoftw
13. 13 — devopsdays MSP 2018 @rondoftw
14. 14 — devopsdays MSP 2018 @rondoftw
15. HOLD UP! 15 — devopsdays MSP 2018 @rondoftw
16. THIS IS A TALK ABOUT COMMUNICATION AND PROCESS ? ? ? 16 — devopsdays MSP 2018 @rondoftw
17. (it was a trap) 17 — devopsdays MSP 2018 @rondoftw
18. YOU CAME TO HEAR COOL STORIES... 18 — devopsdays MSP 2018 @rondoftw
19. BUT YOU'RE GETTING A MEANDERING MEDITATION ON BEST PRACTICES* WHEN DEALING WITH COMPLEX SYSTEMS INSTEAD * These guidelines have only been shown to work for life or death situations under the ocean. They have not been proven to work for tech. 19 — devopsdays MSP 2018 @rondoftw
20. How failures really happen 20 — devopsdays MSP 2018 @rondoftw
21. Complex systems are designed to protect against simple failures. 21 — devopsdays MSP 2018 @rondoftw
22. But accidents still happen. 22 — devopsdays MSP 2018 @rondoftw
23. CATASTROPHES ARE CAUSED BY A FAILURE CASCADE ▸ you have a rebreather malfunction ▸ which you would have caught it if you were testing your equipment on a regular basis ▸ your backup tank had a leak and is running low and that wasn't caught either ▸ and your buddy is too far away and isn't checking in with you ▸ and your dive light that you use to communicate at a distance is out of power ▸ and in the excitement you kick up silt and the visibility drops ▸ and in your panic your air consumption goes up and then you breathe through the last of the air in your tank ▸ so you swim for the surface even though you have a decompression obligation 23 — devopsdays MSP 2018 @rondoftw
24. A post-mortem that blames this incident on a simple mechanical malfunction would only cover 12.5% of the issues that led up to this accident. 24 — devopsdays MSP 2018 @rondoftw
25. Complex system failures don't happen because a single part of the system fails. They happen because all the safety procedures that are supposed to stop the failure from cascading didn't work. 25 — devopsdays MSP 2018 @rondoftw
26. CORE RULES OF SAFETY SYSTEMS 1. An unused safety system doesn't exist. 26 — devopsdays MSP 2018 @rondoftw
27. NORMALIZATION OF DEVIANCE That natural human tendency, particularly in pressure circumstances, to take a safety shortcut. — Colonel Mike Mullane, astronaut 27 — devopsdays MSP 2018 @rondoftw
28. FALSE FEEDBACK the absence of something bad happening means that it was safe ADAPTATION experience is no longer a suitable gauge of risk SOCIAL PRESSURE this is just how we do things 28 — devopsdays MSP 2018 @rondoftw
29. CORE RULES OF SAFETY SYSTEMS 2. An untested safety system doesn't exist either! 29 — devopsdays MSP 2018 @rondoftw
30. CORE RULES OF SAFETY SYSTEMS 3.Unused or untested safety systems are more dangerous than not having one at all. Therefore, safety systems must be tested at regular intervals. 30 — devopsdays MSP 2018 @rondoftw
31. The length of this interval should be determined not only by how likely it is for this system to fail but also how great the impact will be if it does. 31 — devopsdays MSP 2018 @rondoftw
32. A QUICK SIDENOTE ON ASSESSING RISK ▸ Make assessments based on likelihood of occurrence. ▸ Make assessments based on magnitude of regret. If you are only evaluating risk based on the chance of it happening, you must be prepared to experience the corresponding level of regret if it does. 32 — devopsdays MSP 2018 @rondoftw
33. failures will happen 33 — devopsdays MSP 2018 @rondoftw
34. WHAT IS SAFETY? 34 — devopsdays MSP 2018 @rondoftw
35. FAILURE MANAGEMENT ▸ A framework for resiliency ▸ The training and judgment to use it 35 — devopsdays MSP 2018 @rondoftw
36. FAILURE MANAGEMENT FOR SYSTEMS ▸ Have redundancy for systems that you cannot survive without. ▸ Have a redundant pathway to success: a procedure for graceful degradation for systems that are important but not critical. ▸ Have a process for changing over from primary to redundant systems. 36 — devopsdays MSP 2018 @rondoftw
37. FAILURE MANAGEMENT FOR SYSTEMS (CONT) ▸ Keep failures contained so that they don't bring down other systems ▸ Make it easy to do the right thing and hard to do the dangerous things 37 — devopsdays MSP 2018 @rondoftw
38. FAILURE MANAGEMENT FOR HUMAN SYSTEMS 38 — devopsdays MSP 2018 @rondoftw
39. TRAIN FOR PRESSURE 39 — devopsdays MSP 2018 @rondoftw
40. TRAINING: INEXPERIENCED PEOPLE TO THE FRONT ▸ Most inexperienced person leads ▸ Experienced person advises, only intervening when necessary ▸ Team is invested in personal success to ensure mission success 40 — devopsdays MSP 2018 @rondoftw
41. TRAINING: INEXPERIENCED PEOPLE TO THE FRONT (CONT) ▸ One of the best ways to equalize a gap in experience ▸ Opportunity to revise and improve problematic systems ▸ Help build good judgment 41 — devopsdays MSP 2018 @rondoftw
42. GOOD JUDGMENT Good judgment enables the reshaping of rules and frameworks to adapt to a changing environment. 42 — devopsdays MSP 2018 @rondoftw
43. REFINING JUDGMENT ▸ Post-Mortems ▸ Pre-Mortems ▸ Fire Drills ▸ Revisit Past Decisions 43 — devopsdays MSP 2018 @rondoftw
44. POST-MORTEMS ▸ Look at the safety procedures that failed to stop the cascade ▸ Look for opportunities to create new safety systems at critical points 44 — devopsdays MSP 2018 @rondoftw
45. PRE-MORTEMS ▸ Don't wait for failures to build safety frameworks ▸ Identify potential avenues of of failure and make plans for them ▸ Include both likely failures and high regret failures 45 — devopsdays MSP 2018 @rondoftw
46. FIRE DRILLS ▸ Vet your plans and safety systems ▸ Perform targeted training ▸ Evaluate effectiveness of tools and documentation 46 — devopsdays MSP 2018 @rondoftw
47. REVISIT PAST DECISIONS ▸ Examine successful operations to see what key insights were helpful ▸ Identify any dependency on luck in previous projects ▸ Share rationale for decisions 47 — devopsdays MSP 2018 @rondoftw
48. RECOGNIZING SUCCESS 48 — devopsdays MSP 2018 @rondoftw
49. I WANT TO LEARN MORE! 1. Richard I. Cook - How Complex Systems Fail 2. Astronaut Mike Mullane - https://www.youtube.com/watch?v=Ljzj9Msli5o 3. Steve Lewis aka decodoppler (Technical Diving Instructor) - Staying Alive 4. Sidney Dekker - Drift into Failure 5. Diane Vaughn - The Challenger Launch Decision 49 — devopsdays MSP 2018 @rondoftw
50. 50 — devopsdays MSP 2018 @rondoftw
51. Questions? @rondoftw 51 — devopsdays MSP 2018 @rondoftw