An Optimist's Guide to Chaos Engineering

@andyﬂeener AN OPTIMIST’S GUIDE TO CHAOS ENGINEERING BY ANDY FLEENER

@andyﬂeener HI I’M ANDY AND I’M AN OPTIMIST

@andyﬂeener OPTIMISM CAN BE HARD WHEN EVERYTHING IS BROKEN ALL
THE TIME

@andyﬂeener ABOUT ME ▸ Senior Software Engineer ▸ Ruby Developer
for 9 years ▸ In Operations for last 4 years ▸ Complexity / System Safety Nerd

@andyﬂeener

Creative Commons Image: NASA Solar System Exploration

@andyﬂeener REAL BUGS I’VE SEEN IN THE WILD ▸ Ruby’s
Timeout Library doesn’t actually timeout sometimes ▸ Node Framework miss handle’s an invalid url and crashes ▸ Ruby App server ﬂips it’s shit when a stream is closed early Creative Commons Image: Matteo X

@andyﬂeener THESE BUGS WERE AWESOME ▸ I learned a shitload
about how each of these parts of an application work ▸ We tracked down the problem and worked through solutions for each ▸ These are the kinds of challenges that get me excited to go to work Creative Commons Image: Neil Moralee

@andyﬂeener #REALTALK: I ENJOY BREAKING THINGS

@andyﬂeener

@andyﬂeener SYSTEMS THINKING ▸ It can be very counterintuitive ▸
Success and Failure are not easily deﬁned ▸ Emergent properties of a system are the feedback required to close the loop Creative Commons Image: Liam Ross

@andyﬂeener COMPLEX SYSTEMS ARE PRETTY DOPE Creative Commons Image: John
Smith

@andyﬂeener THREE TRUTHS ABOUT COMPLEX SYSTEMS ▸ They are inherently
unsafe ▸ Failure is a normal state of the system ▸ Failure can actually make a system stronger Creative Commons Image: Bjoern von Thuelen

@andyﬂeener ENTER CHAOS ENGINEERING

@andyﬂeener CHAOS ENGINEERING IS THE DISCIPLINE OF EXPERIMENTING ON A
DISTRIBUTED SYSTEM IN ORDER TO BUILD CONFIDENCE IN THE SYSTEM’S CAPABILITY TO WITHSTAND TURBULENT CONDITIONS IN PRODUCTION. PRINCIPLES OF CHAOS ENGINEERING

@andyﬂeener DISTRIBUTED SYSTEMS ARE COMPLEX SYSTEMS WE TOUCH EVERYDAY

@andyﬂeener BASIC PRINCIPLES ▸ Build a Hypothesis around Steady State
Behavior ▸ Vary Real-world Events ▸ Learn from the changes to the Steady State Creative Commons Image: Hamed Sabe

@andyﬂeener BUILD A HYPOTHESIS AROUND STEADY STATE BEHAVIOR ▸ Attempt
to understand as much of the system’s steady normal running state as you can ▸ Network is the backbone of distributed systems and it is designed to withstand failure ▸ Measure outcomes of the system by the value it provides ▸ APM is generally a good way to understanding Steady State Creative Commons Image: Seabamirum

@andyﬂeener VARY REAL-WORLD EVENTS ▸ Variables should reﬂect real world
events ▸ Prioritize by both frequency and potential impact ▸ Systems that resist failure tend to fail catastrophically ▸ Rule #1 Never Trust the Network ▸ Chaos Variables can be any event capable of disrupting steady state Creative Commons Image: Leo Fung

@andyﬂeener LEARN FROM THE CHANGES TO THE STEADY STATE ▸
How did the system react to variable change? ▸ Some times it’s obvious ▸ Hopefully more frequently your system can tolerate the failure Creative Commons Image: Simon_sees

@andyﬂeener CHAOS ENGINEERING WAS FOUNDED ON RUNNING EXPERIMENTS IN PRODUCTION

@andyﬂeener

@andyﬂeener WHY WOULD YOU DO THAT? ▸ It’s impossible to
create a duplicate complex system ▸ Failure in a production like environment won’t be exactly the same ▸ Fake it till you make it ▸ Do not run a Chaos Experiment on something you never tested ▸ Running experiments on production like systems is better than not running them at all Creative Commons Image: Toms River Fire Dept

@andyﬂeener COOL SO HOW DO I DO THIS?

@andyﬂeener NETWORK LATENCY IS A DISTRIBUTED SYSTEM’S KRYPTONITE

@andyﬂeener USE TOOLS THAT EXIST ▸ Toxiproxy by Shopify ▸
Comcast by tylertreat ▸ Chaos Monkey by Netﬂix Creative Commons Image: Toms River Fire Dept

@andyﬂeener DISTRIBUTED SYSTEMS NEED TO TALK TO EACH OTHER, SLOW
THAT DOWN AND I BET YOU’LL SEE SOMETHING INTERESTING

@andyﬂeener

@andyﬂeener GOOGLE “CIRCUIT BREAKER PATTERN”

@andyﬂeener NETFLIX HAS YOUR BACK CHECK OUT HYSTRIX

@andyﬂeener BUT WHY SHOULD I EVEN DO THIS?

@andyﬂeener TO DISCOVER LATENT FAILURES

@andyﬂeener IT’S EASY TO ASSUME THE SYSTEM IS SAFE

@andyﬂeener THE PRIME DIRECTIVE OF CHAOS ENGINEERING IS TO LEARN

@andyﬂeener BY LEARNING, OUR SYSTEM CAN CONTINUE TO GROW AND
IMPROVE

@andyﬂeener WE’RE ACTIVELY ADDING CAPACITY TO THE SYSTEM TO FAIL

@andyﬂeener IS THE KEY TO CREATING NOT JUST RELIABLE SYSTEMS
BUT RESILIENT ONES ADDING CAPACITY TO FAIL

@andyﬂeener AND NOW FOR SOMETHING COMPLETELY DIFFERENT

@andyﬂeener ORGANIZATIONS ARE DISTRIBUTED SYSTEMS

@andyﬂeener WAIT WHAT?

@andyﬂeener

@andyﬂeener HOW DO WE APPLY CHAOS ENGINEERING TO OTHER SOCIOTECHNICAL
SYSTEMS?

@andyﬂeener THAT’S NOT A THING

@andyﬂeener ORGANIZATIONS WHICH DESIGN SYSTEMS ARE CONSTRAINED TO PRODUCE DESIGNS
WHICH ARE COPIES OF THE COMMUNICATION STRUCTURES OF THESE ORGANIZATIONS CONWAY’S LAW

@andyﬂeener CHAOS ENGINEERING AN ORGANIZATION ▸ Resilient Teams create Resilient
Systems ▸ Teams are like services ▸ Communication is the Network of Teams Creative Commons Image: Meenakshi Madhavan

@andyﬂeener LEARNING ORGANIZATIONS HAVE CAPACITY TO FAIL

@andyﬂeener FAILURE INJECTION IS ONE OF THE BEST WAYS TO
LEARN ABOUT A SYSTEM

@andyﬂeener WHAT DOES “FAILURE” MEAN HERE

@andyﬂeener MOVE PEOPLE BETWEEN TEAMS

@andyﬂeener YOUR TEAM LEADS MAY NOT LIKE THIS

@andyﬂeener HIGHLY COLLABORATIVE TEAMS CAN FALL APART IN COMMUNICATION BREAKDOWNS

@andyﬂeener CREATE PATHWAYS FOR COMMUNICATION

@andyﬂeener ADDING MORE AVENUES OF COMMUNICATION MAKES ONE FAILURE LESS
IMPACTFUL

@andyﬂeener HOW ELSE CAN WE BREAK OUR ORGANIZATIONS?

@andyﬂeener STAY AWESOME! THANKS!

An Optimist's Guide to Chaos Engineering

An Optimist's Guide to Chaos Engineering

More Decks by Andy Fleener

Other Decks in Technology

Featured

Transcript