SIGN, SIGN, EVERYWHERE A SIGN: WHAT FIVE MAN ELECTRICAL BAND AND TODD CONKLIN TAUGHT ME ABOUT CHAOS ENGINEERING

Signs and Signals are all around us, they influence how we perceive and feel about the systems we interact with on a daily basis. Many of these signals are loud, or strong, a production outage, an angry customer, or a missed deadline. But there’s more to signals than just the ones we see or hear when something goes bang. Actively searching for and understanding Weak Signals is the key to creating the most productive Chaos Experiments. Weak Signals are small, barely noticeable, indications of new emergent behavior in the system. They’re the “tea leaves”, they trigger our gut feelings, and our hunches. In this talk I will dig into the area of Chaos Engineering that I find most interesting, how do you decide what to experiment on?

Transcript

1. @andyfleener Creative Commons Image: Eric Kilby
2. @andyfleener Signs Signs, Everywhere a Sign What Five Man Electrical Band and Todd Conklin taught me about Chaos Engineering
3. @andyfleener
4. @andyfleener @andyfleener
5. @andyfleener Creative Commons Image: Ryan Kawailani Ozawa
6. @andyfleener Creative Commons Image: Pete Harmer @andyfleener
7. @andyfleener Creative Commons Image: Susanne Wraight
8. @andyfleener Creative Commons Image: FunGi_ (Trading) @andyfleener
9. @andyfleener Creative Commons Image: Kool Cats Photography @andyfleener
10. @andyfleener Creative Commons Image: Brenda Dobbs @andyfleener
11. @andyfleener https://www.askdifference.com/sign-vs-signal/ “The main difference between Sign and Signal is that the Sign is a semiotic concept whose presence or occurrence indicates the probable presence or occurrence of something else and Signal is a varying physical quantity that conveys information”
12. @andyfleener WHAT IS A WEAK SIGNAL?
13. @andyfleener - TODD CONKLIN “weak indicators that tell us when there’s a problem happening, not when a problem has happened: 'You’ll never hear a weak signal in failure, the signal in a failure is loud.'”
14. @andyfleener – DAVID WOODS & RICHARD COOK “A seemingly random or disconnected piece of information that at first appears to be background noise but can be recognized as part of a significant pattern by viewing it through a different frame or connecting it with other pieces of information.” Weak Signals Approach to ANSP Safety Performance
15. @andyfleener Why should I care about weak signals?
16. @andyfleener ‘‘Going solid’’: a model of system dynamics and consequences for patient safety
17. @andyfleener ‘‘Going solid’’ is a nuclear power slang term used to describe a difficult to manage technical situation. Manageable behavior of a steam boiler depends on having both steam and liquid water present in the boiler. When a boiler becomes completely filled with liquid (goes solid), its operating characteristics shift suddenly and dramatically. The resulting situation is both hazardous and difficult to control. – RICHARD COOK & Jens Rasmussen ‘‘Going solid’’: a model of system dynamics and consequences for patient safety
18. Search for the Boundaries! What makes something a success vs. a failure is whether, as Rasmussen describes in his dynamic safety model, the operating point of the system crosses over the boundary, a tipping point, of performance failure. Success and Failure are two sides of the same coin. What was once a success can suddenly and unexpectedly drift into a state of failure. Chaos Engineering’s goal is to find the boundaries before they “go solid”
19. @andyfleener Signals in the Wild
20. Signals come from every type of system A door that opens “backwards” A stoplight that isn’t timed long enough to walk across the street A buffet line that’s out of order
21. @andyfleener
22. @andyfleener Creative Commons Image: Seth Chandler
23. @andyfleener
24. @andyfleener
25. @andyfleener WEAK SIGNALS ARE A CRITICAL SOURCE OF INSIGHTS
26. @andyfleener – GARY KLEIN “Insight is an unexpected shift in the way we understand things” SEEING WHAT OTHERS DON’T
27. @andyfleener – GARY KLEIN “It comes without warning. It's not something that we think is going to happen and that's why it's unexpected. It feels like a gift and in fact it is.” SEEING WHAT OTHERS DON’T
28. @andyfleener Insights that came from Weak Signals
29. @andyfleener On-Call Shifts should end on Fridays
30. @andyfleener The designated “ops-support” person
31. @andyfleener “I don't know anything about this, we’ll need to talk to Emma.”
32. @andyfleener Your system is signaling constantly, finding the balance of what signal to act on, vs continuing to monitor is the truly hard part.
33. @andyfleener ULTIMATELY THE VALUE PROPOSITION OF CHAOS ENGINEERING IS IN THE INSIGHTS YOU GAIN
34. @andyfleener @andyfleener Creative Commons Image: Speg of the Pigs