Slide 1

Slide 1 text

@andyfleener Creative Commons Image: Eric Kilby

Slide 2

Slide 2 text

@andyfleener Sign Sign, Everywhere a Sign What Five Man Electrical Band and Todd Conklin taught me about Chaos Engineering

Slide 3

Slide 3 text

@andyfleener

Slide 4

Slide 4 text

@andyfleener

Slide 5

Slide 5 text

@andyfleener PRINCIPLES OF CHAOS ENGINEERING principlesofchaos.org

Slide 6

Slide 6 text

@andyfleener The Principles of Chaos Engineering Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Slide 7

Slide 7 text

@andyfleener Build a Hypothesis around Steady State Behavior

Slide 8

Slide 8 text

@andyfleener @andyfleener

Slide 9

Slide 9 text

@andyfleener Creative Commons Image: Ryan Kawailani Ozawa

Slide 10

Slide 10 text

@andyfleener Creative Commons Image: Pete Harmer @andyfleener

Slide 11

Slide 11 text

@andyfleener Creative Commons Image: Susanne Wraight

Slide 12

Slide 12 text

@andyfleener Creative Commons Image: FunGi_ (Trading) @andyfleener

Slide 13

Slide 13 text

@andyfleener Creative Commons Image: Kool Cats Photography @andyfleener

Slide 14

Slide 14 text

@andyfleener Creative Commons Image: Brenda Dobbs @andyfleener

Slide 15

Slide 15 text

@andyfleener https://www.askdifference.com/sign-vs-signal/ “The main difference between Sign and Signal is that the Sign is a semiotic concept whose presence or occurrence indicates the probable presence or occurrence of something else and Signal is a varying physical quantity that conveys information”

Slide 16

Slide 16 text

@andyfleener WHAT IS A WEAK SIGNAL?

Slide 17

Slide 17 text

@andyfleener - TODD CONKLIN “weak indicators that tell us when there’s a problem happening, not when a problem has happened: 'You’ll never hear a weak signal in failure, the signal in a failure is loud.'”

Slide 18

Slide 18 text

@andyfleener – DAVID WOODS & RICHARD COOK “A seemingly random or disconnected piece of information that at first appears to be background noise but can be recognized as part of a significant pattern by viewing it through a different frame or connecting it with other pieces of information.” Weak Signals Approach to ANSP Safety Performance

Slide 19

Slide 19 text

@andyfleener – Woods & Cook, 2002 “The future seems implausible, the past incredible.”

Slide 20

Slide 20 text

@andyfleener The Power of Foresight

Slide 21

Slide 21 text

@andyfleener The Power of Hindsight

Slide 22

Slide 22 text

@andyfleener Why should I care about weak signals?

Slide 23

Slide 23 text

@andyfleener ‘‘Going solid’’: a model of system dynamics and consequences for patient safety

Slide 24

Slide 24 text

@andyfleener ‘‘Going solid’’ is a nuclear power slang term used to describe a difficult to manage technical situation. Manageable behavior of a steam boiler depends on having both steam and liquid water present in the boiler. When a boiler becomes completely filled with liquid (goes solid), its operating characteristics shift suddenly and dramatically. The resulting situation is both hazardous and difficult to control. – RICHARD COOK & Jens Rasmussen ‘‘Going solid’’: a model of system dynamics and consequences for patient safety

Slide 25

Slide 25 text

Search for the Boundaries! What makes something a success vs. a failure is whether, as Rasmussen describes in his dynamic safety model, the operating point of the system crosses over the boundary, a tipping point, of performance failure. Success and Failure are two sides of the same coin. What was once a success can suddenly and unexpectedly drift into a state of failure. Chaos Engineering’s goal is to find the boundaries before they “go solid”

Slide 26

Slide 26 text

@andyfleener Signals in the Wild

Slide 27

Slide 27 text

Signals come from every type of system A door that opens “backwards” A stoplight that isn’t timed long enough to walk across the street A buffet line that’s out of order

Slide 28

Slide 28 text

@andyfleener

Slide 29

Slide 29 text

@andyfleener Creative Commons Image: Seth Chandler

Slide 30

Slide 30 text

@andyfleener

Slide 31

Slide 31 text

@andyfleener

Slide 32

Slide 32 text

@andyfleener

Slide 33

Slide 33 text

@andyfleener WEAK SIGNALS ARE A CRITICAL SOURCE OF INSIGHTS

Slide 34

Slide 34 text

@andyfleener – GARY KLEIN “Insight is an unexpected shift in the way we understand things” SEEING WHAT OTHERS DON’T

Slide 35

Slide 35 text

@andyfleener – GARY KLEIN “It comes without warning. It's not something that we think is going to happen and that's why it's unexpected. It feels like a gift and in fact it is.” SEEING WHAT OTHERS DON’T

Slide 36

Slide 36 text

@andyfleener Insights that came from Weak Signals

Slide 37

Slide 37 text

@andyfleener On-Call Shifts should end on Fridays

Slide 38

Slide 38 text

@andyfleener The designated “ops-support” person

Slide 39

Slide 39 text

@andyfleener “I don't know anything about this, we’ll need to talk to Emma.”

Slide 40

Slide 40 text

@andyfleener Your system is signaling constantly, finding the balance of what signal to act on, vs continuing to monitor is the truly hard part.

Slide 41

Slide 41 text

@andyfleener ULTIMATELY THE VALUE PROPOSITION OF CHAOS ENGINEERING IS IN THE INSIGHTS YOU GAIN

Slide 42

Slide 42 text

@andyfleener @andyfleener Creative Commons Image: Speg of the Pigs

Slide 43

Slide 43 text

@andyfleener Questions?