@andyfleener Creative Commons Image: Eric Kilby

@andyfleener Sign Sign, Everywhere a Sign What Five Man Electrical Band and Todd Conklin taught me about Chaos Engineering

@andyfleener

@andyfleener

@andyfleener PRINCIPLES OF CHAOS ENGINEERING principlesofchaos.org

@andyfleener The Principles of Chaos Engineering Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

@andyfleener Build a Hypothesis around Steady State Behavior

@andyfleener @andyfleener

@andyfleener Creative Commons Image: Ryan Kawailani Ozawa

@andyfleener Creative Commons Image: Pete Harmer @andyfleener

@andyfleener Creative Commons Image: Susanne Wraight

@andyfleener Creative Commons Image: FunGi_ (Trading) @andyfleener

@andyfleener Creative Commons Image: Kool Cats Photography @andyfleener

@andyfleener Creative Commons Image: Brenda Dobbs @andyfleener

@andyfleener https://www.askdifference.com/sign-vs-signal/ “The main difference between Sign and Signal is that the Sign is a semiotic concept whose presence or occurrence indicates the probable presence or occurrence of something else and Signal is a varying physical quantity that conveys information”

@andyfleener WHAT IS A WEAK SIGNAL?

@andyfleener - TODD CONKLIN “weak indicators that tell us when there’s a problem happening, not when a problem has happened: 'You’ll never hear a weak signal in failure, the signal in a failure is loud.'”

@andyfleener – DAVID WOODS & RICHARD COOK “A seemingly random or disconnected piece of information that at first appears to be background noise but can be recognized as part of a significant pattern by viewing it through a different frame or connecting it with other pieces of information.” Weak Signals Approach to ANSP Safety Performance

@andyfleener – Woods & Cook, 2002 “The future seems implausible, the past incredible.”

@andyfleener The Power of Foresight

@andyfleener The Power of Hindsight

@andyfleener Why should I care about weak signals?

@andyfleener ‘‘Going solid’’: a model of system dynamics and consequences for patient safety

@andyfleener ‘‘Going solid’’ is a nuclear power slang term used to describe a difficult to manage technical situation. Manageable behavior of a steam boiler depends on having both steam and liquid water present in the boiler. When a boiler becomes completely filled with liquid (goes solid), its operating characteristics shift suddenly and dramatically. The resulting situation is both hazardous and difficult to control. – RICHARD COOK & Jens Rasmussen ‘‘Going solid’’: a model of system dynamics and consequences for patient safety

Search for the Boundaries! What makes something a success vs. a failure is whether, as Rasmussen describes in his dynamic safety model, the operating point of the system crosses over the boundary, a tipping point, of performance failure. Success and Failure are two sides of the same coin. What was once a success can suddenly and unexpectedly drift into a state of failure. Chaos Engineering’s goal is to find the boundaries before they “go solid”

@andyfleener Signals in the Wild

Signals come from every type of system A door that opens “backwards” A stoplight that isn’t timed long enough to walk across the street A buffet line that’s out of order

@andyfleener

@andyfleener Creative Commons Image: Seth Chandler

@andyfleener

@andyfleener

@andyfleener

@andyfleener WEAK SIGNALS ARE A CRITICAL SOURCE OF INSIGHTS

@andyfleener – GARY KLEIN “Insight is an unexpected shift in the way we understand things” SEEING WHAT OTHERS DON’T

@andyfleener – GARY KLEIN “It comes without warning. It's not something that we think is going to happen and that's why it's unexpected. It feels like a gift and in fact it is.” SEEING WHAT OTHERS DON’T

@andyfleener Insights that came from Weak Signals

@andyfleener On-Call Shifts should end on Fridays

@andyfleener The designated “ops-support” person

@andyfleener “I don't know anything about this, we’ll need to talk to Emma.”

@andyfleener Your system is signaling constantly, finding the balance of what signal to act on, vs continuing to monitor is the truly hard part.

@andyfleener ULTIMATELY THE VALUE PROPOSITION OF CHAOS ENGINEERING IS IN THE INSIGHTS YOU GAIN

@andyfleener @andyfleener Creative Commons Image: Speg of the Pigs

@andyfleener Questions?