OSN 2021: SIGN, SIGN, EVERYWHERE A SIGN: WHAT FIVE MAN ELECTRICAL BAND AND TODD CONKLIN TAUGHT ME ABOUT CHAOS ENGINEERING

Signs and Signals are all around us, they influence how we perceive and feel about the systems we interact with on a daily basis. Many of these signals are loud, or strong, a production outage, an angry customer, or a missed deadline. But there’s more to signals than just the ones we see or hear when something goes bang. Actively searching for and understanding Weak Signals is the key to creating the most productive Chaos Experiments. Weak Signals are small, barely noticeable, indications of new emergent behavior in the system. They’re the “tea leaves”, they trigger our gut feelings, and our hunches. In this talk I will dig into the area of Chaos Engineering that I find most interesting, how do you decide what to experiment on?

Transcript

1. @andyfleener Creative Commons Image: Eric Kilby
2. @andyfleener Sign Sign, Everywhere a Sign What Five Man Electrical Band and Todd Conklin taught me about Chaos Engineering
3. @andyfleener
4. @andyfleener
5. @andyfleener PRINCIPLES OF CHAOS ENGINEERING principlesofchaos.org
6. @andyfleener The Principles of Chaos Engineering Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.
7. @andyfleener Build a Hypothesis around Steady State Behavior
8. @andyfleener @andyfleener
9. @andyfleener Creative Commons Image: Ryan Kawailani Ozawa
10. @andyfleener Creative Commons Image: Pete Harmer @andyfleener
11. @andyfleener Creative Commons Image: Susanne Wraight
12. @andyfleener Creative Commons Image: FunGi_ (Trading) @andyfleener
13. @andyfleener Creative Commons Image: Kool Cats Photography @andyfleener
14. @andyfleener Creative Commons Image: Brenda Dobbs @andyfleener
15. @andyfleener https://www.askdifference.com/sign-vs-signal/ “The main difference between Sign and Signal is that the Sign is a semiotic concept whose presence or occurrence indicates the probable presence or occurrence of something else and Signal is a varying physical quantity that conveys information”
16. @andyfleener WHAT IS A WEAK SIGNAL?
17. @andyfleener - TODD CONKLIN “weak indicators that tell us when there’s a problem happening, not when a problem has happened: 'You’ll never hear a weak signal in failure, the signal in a failure is loud.'”
18. @andyfleener – DAVID WOODS & RICHARD COOK “A seemingly random or disconnected piece of information that at first appears to be background noise but can be recognized as part of a significant pattern by viewing it through a different frame or connecting it with other pieces of information.” Weak Signals Approach to ANSP Safety Performance
19. @andyfleener – Woods & Cook, 2002 “The future seems implausible, the past incredible.”
20. @andyfleener The Power of Foresight
21. @andyfleener The Power of Hindsight
22. @andyfleener Why should I care about weak signals?
23. @andyfleener ‘‘Going solid’’: a model of system dynamics and consequences for patient safety
24. @andyfleener ‘‘Going solid’’ is a nuclear power slang term used to describe a difficult to manage technical situation. Manageable behavior of a steam boiler depends on having both steam and liquid water present in the boiler. When a boiler becomes completely filled with liquid (goes solid), its operating characteristics shift suddenly and dramatically. The resulting situation is both hazardous and difficult to control. – RICHARD COOK & Jens Rasmussen ‘‘Going solid’’: a model of system dynamics and consequences for patient safety
25. Search for the Boundaries! What makes something a success vs. a failure is whether, as Rasmussen describes in his dynamic safety model, the operating point of the system crosses over the boundary, a tipping point, of performance failure. Success and Failure are two sides of the same coin. What was once a success can suddenly and unexpectedly drift into a state of failure. Chaos Engineering’s goal is to find the boundaries before they “go solid”
26. @andyfleener Signals in the Wild
27. Signals come from every type of system A door that opens “backwards” A stoplight that isn’t timed long enough to walk across the street A buffet line that’s out of order
28. @andyfleener
29. @andyfleener Creative Commons Image: Seth Chandler
30. @andyfleener
31. @andyfleener
32. @andyfleener
33. @andyfleener WEAK SIGNALS ARE A CRITICAL SOURCE OF INSIGHTS
34. @andyfleener – GARY KLEIN “Insight is an unexpected shift in the way we understand things” SEEING WHAT OTHERS DON’T
35. @andyfleener – GARY KLEIN “It comes without warning. It's not something that we think is going to happen and that's why it's unexpected. It feels like a gift and in fact it is.” SEEING WHAT OTHERS DON’T
36. @andyfleener Insights that came from Weak Signals
37. @andyfleener On-Call Shifts should end on Fridays
38. @andyfleener The designated “ops-support” person
39. @andyfleener “I don't know anything about this, we’ll need to talk to Emma.”
40. @andyfleener Your system is signaling constantly, finding the balance of what signal to act on, vs continuing to monitor is the truly hard part.
41. @andyfleener ULTIMATELY THE VALUE PROPOSITION OF CHAOS ENGINEERING IS IN THE INSIGHTS YOU GAIN
42. @andyfleener @andyfleener Creative Commons Image: Speg of the Pigs
43. @andyfleener Questions?