Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ChaosConf 2018

Ronnie Chen
September 28, 2018

ChaosConf 2018

Ronnie Chen

September 28, 2018
Tweet

More Decks by Ronnie Chen

Other Decks in Technology

Transcript

  1. STAYING ALIVE PATTERNS FOR FAILURE MANAGEMENT FROM THE BOTTOM OF

    THE OCEAN 1 — ChaosConf 2018 @rondoftw
  2. Ronnie Chen Engineering Manager at Twitter @rondoftw 2 — ChaosConf

    2018 @rondoftw
  3. 3 — ChaosConf 2018 @rondoftw

  4. None
  5. 5 — ChaosConf 2018 @rondoftw

  6. 6 — ChaosConf 2018 @rondoftw

  7. 7 — ChaosConf 2018 @rondoftw

  8. 8 — ChaosConf 2018 @rondoftw

  9. RISKS MAY INCLUDE... 1. hypoxia 2. hyperoxia 3. nitrogen narcosis

    4. carbon dioxide buildup 5. oxygen sensor failure 6. deep tissue isobaric counterdiffusion (ICD) 7. high pressure nervous syndrome (HPNS) 8. exhausting your carbon dioxide scrubber 9. carbon dioxide channeling from a poorly packed scrubber 10. carbon buildup causing an spark leading to an oxygen fire. underwater. 11. flooding of breathing loop or circuitry 12. water mixing with the scrubbing agent to produce a toxic caustic soda that will give you chemical burns on your mouth, airway, and lungs 13. plain old decompression sickness 9 — ChaosConf 2018 @rondoftw
  10.   If you own a rebreather for five years, two

    percent of you are going to die on it. — Jill Heinerth, underwater explorer 10 — ChaosConf 2018 @rondoftw
  11. How failures really happen 11 — ChaosConf 2018 @rondoftw

  12. 12 — ChaosConf 2018 @rondoftw

  13. CATASTROPHES ARE CAUSED BY A FAILURE CASCADE ▸ you have

    a rebreather malfunction ▸ which you might have caught if you didn't skip your pre-dive check ▸ your backup tank had a leak and is running low because of a faulty o-ring ▸ your buddy is far away and distracted by something else ▸ the dive light doesn't get their attention because the beam has less contrast in the day ▸ you accidentally kick up some silt and the visibility drops ▸ your air consumption goes up and then you breathe through the last of the air in your tank ▸ so you swim for the surface even though you have a decompression obligation 13 — ChaosConf 2018 @rondoftw
  14. A post-mortem that blames this incident on a simple mechanical

    malfunction would only cover 12.5% of the issues that led up to this accident. 14 — ChaosConf 2018 @rondoftw
  15. WHAT IS SAFETY? 15 — ChaosConf 2018 @rondoftw

  16. 16 — ChaosConf 2018 @rondoftw

  17. TRAIN FOR PRESSURE 17 — ChaosConf 2018 @rondoftw

  18. TEAM TRAINING: INEXPERIENCED PEOPLE TO THE FRONT ▸ Most inexperienced

    person leads ▸ Experienced person advises, only intervening when necessary 18 — ChaosConf 2018 @rondoftw
  19. INEXPERIENCED PEOPLE TO THE FRONT ▸ One of the best

    ways to equalize a gap in experience ▸ Opportunity to revise and improve problematic systems 19 — ChaosConf 2018 @rondoftw
  20. 20 — ChaosConf 2018 @rondoftw

  21. Judgment is what enables you to adapt to a changing

    environment 21 — ChaosConf 2018 @rondoftw
  22. REFINING JUDGMENT ▸ Post-Mortems ▸ Pre-Mortems ▸ Fire Drills ▸

    Revisit Past Decisions 22 — ChaosConf 2018 @rondoftw
  23. POST-MORTEMS ▸ Look at the safety procedures that failed to

    stop the cascade ▸ Understand contributing factors ▸ Look for opportunities to create new safety systems 23 — ChaosConf 2018 @rondoftw
  24. PRE-MORTEMS ▸ Don't wait for failures to build safety frameworks

    ▸ Identify potential avenues of of failure and make plans ▸ Include both likely failures and high regret failures 24 — ChaosConf 2018 @rondoftw
  25. FIRE DRILLS ▸ Vet your plans and safety systems ▸

    Perform targeted training ▸ Evaluate effectiveness of tools and documentation 25 — ChaosConf 2018 @rondoftw
  26. REVISIT PAST DECISIONS ▸ Examine successful operations to see what

    key insights were helpful ▸ Identify any dependency on luck in previous projects ▸ Share rationale for decisions 26 — ChaosConf 2018 @rondoftw
  27. RECOGNIZING SUCCESS 27 — ChaosConf 2018 @rondoftw

  28. I WANT TO LEARN MORE! 1. Richard I. Cook -

    How Complex Systems Fail 3. Steve Lewis aka decodoppler (Technical Diving Instructor) - Staying Alive 4. Sidney Dekker - Drift into Failure 5. Diane Vaughn - The Challenger Launch Decision 28 — ChaosConf 2018 @rondoftw
  29. 29 — ChaosConf 2018 @rondoftw