Game Days: Failing for Fun and Profit

@andyﬂeener GAME DAYS FAILING FOR FUN AND PROFIT BY ANDY
FLEENER

HI I’M ANDY

@andyﬂeener

@andyﬂeener ABOUT ME ▸ Platform Operations Manager ▸ Ruby Developer
for 10 years ▸ In Operations for last 5 years ▸ New View Safety Nerd

@andyﬂeener WHAT’S A “NEW VIEW SAFETY NERD”

@andyﬂeener BAD IDEA: BLAME AND PUNISH GOOD IDEA: LEARN AND
GROW

@andyﬂeener SYSTEMS THINKING!

@andyﬂeener WHAT’S SYSTEMS THINKING? ▸ It can be very counterintuitive
▸ Success and Failure are not easily deﬁned ▸ Emergent properties of a system are the feedback required to close the loop Creative Commons Image: Liam Ross

@andyﬂeener COMPLEX SYSTEMS ARE PRETTY DOPE Creative Commons Image: John
Smith

@andyﬂeener #REALTALK: I ENJOY BREAKING THINGS

@andyﬂeener

@andyﬂeener WHICH IS GOOD BECAUSE

@andyﬂeener LIVE LOOK AT THE INTERNET

@andyﬂeener HERE’S THE BILLIONS AND BILLIONS OF E-COINS QUESTION

@andyﬂeener IF THE SYSTEM IS UNSAFE HOW DO WE KEEP
IT RUNNING?

@andyﬂeener FAILURE INJECTION!

@andyﬂeener ENTER GAME DAYS

@andyﬂeener A PROGRAM DESIGNED TO INCREASE RESILIENCE BY PURPOSELY INJECTING
MAJOR FAILURES INTO CRITICAL SYSTEMS SEMI-REGULARLY TO DISCOVER FLAWS AND SUBTLE DEPENDENCIES. ACM Queue Volume 10, issue 9 Resilience Engineering: Learning to Embrace Failure(September 13, 2012)

@andyﬂeener CHANGE A BASIC ASSUMPTION AND YOU HAVE CHANGED THE
SYSTEM ITSELF. Eli Goldratt THEORY OF CONSTRAINTS

@andyﬂeener IT’S THE SYSTEM, MAN

THE VALUE OF GAME DAYS ▸ Find latent failures ▸
Practice Incident Response ▸Learn about your systems

@andyﬂeener GAME DAYS A BRIEF INCOMPLETE HISTORY

@andyﬂeener IT IS SAID THAT IF YOU KNOW YOUR ENEMIES
AND KNOW YOURSELF, YOU WILL NOT BE IMPERILED IN A HUNDRED BATTLES Sun Tzu ART OF WAR

THIS CONCEPT IS OLD ▸ Security has been doing “Red
Team” Exercises for decades ▸ Military organizations have been doing war games since the 1800s including predicting how the Japanese would attack Pearl Harbor 9 years before it happened

THE GAME DAY AS WE KNOW THEM ▸Amazon and the
Master of disaster Jesse Robbins ▸Game Days at Etsy ▸PagerDuty and Failure Fridays

@andyﬂeener

@andyﬂeener WHAT ABOUT SPORTSENGINE?

GAME DAYS AT SPORTSENGINE: A HISTORY ▸ Started doing Game
Days in 2013 ▸ The ﬁrst Game Day was just 3 Operations Engineers busting our staging environment ▸ We’ve been running them quarterlyish since ▸ Our last game day crossed 4 teams with an attacking team of 5 and a responding team of 6

@andyﬂeener LOGISTICS

@andyﬂeener PICK A RED TEAM

@andyﬂeener ENSURE YOU HAVE A BLUE TEAM

@andyﬂeener TREAT GAMES AS PRODUCTION INCIDENTS BY FOLLOWING THE FULL
INCIDENT RESPONSE LIFECYCLE

@andyﬂeener GET MULTIPLE TEAMS INVOLVED!

@andyﬂeener OPERATIONAL CONCERNS ARE BUSINESS CONCERNS

@andyﬂeener TAKE IT SERIOUSLY BUT DON’T FORGET TO HAVE FUN

ADVICE FOR THE RED TEAM ▸ Have more games prepared
than you think you’ll actually get to ▸ Some Games won’t have the impact you expect ▸ Fake it until you make it(alerts, monitors, support requests) ▸ Advanced technique: Apply constraints to responders

@andyﬂeener USE TOOLS THAT EXIST ▸ Toxiproxy by Shopify ▸
Comcast by tylertreat ▸ Chaos Monkey by Netﬂix Creative Commons Image: Toms River Fire Dept

ADVICE FOR THE BLUE TEAM ▸ Treat your response like
it’s PRODUCTION ▸ Use this as a way to train New/Junior Engineers on your world class Incident Response ▸ Time the response to create urgency

@andyﬂeener EXAMPLE GAME CASE STUDIES

THE FAKE DDOS ATTACK ▸ A Classic real world concern
▸ We’ve done this game multiple times in different ways ▸ The easiest way is to leave a “D” off ▸ It might be harder than you think ▸ Fake this by outlawing an IP block as a mitigation technique

THE MISCONFIGURED NETWORK ▸ This is a super easy game
adjust a ﬁrewall rule in a critical location ▸ Another Classic that happens throughout the internet ▸ This can be an easy way to see the devastating effects of high network latency

@andyﬂeener

THE SSL NEGOTIATION FAILURE ▸ Changed permissions on the ssl
cert ﬁles ▸ Cause weird negotiation failure state that was hard to debug ▸ This actually happened later due to a failed chef conﬁguration ▸ Big win because we understood the behavior when it happened

THE LATENT BUG BOMB ▸ These are fun and super
common real world scenarios ▸ This is the perfect way to get a dev team involved ▸ I’ve done things like add command injection endpoints ▸ These are most effective by ﬁnding the biggest blast radius

THE FORK BOMB ▸ This is a great one to
run if you want to seriously trash some servers ▸ Fork bombs are super easy to write ▸ You can write a fork bomb in any language ▸ Heres Ruby: loop { fork { load(__FILE__) } }

@andyﬂeener AS YOU CAN TELL I ENJOY BEING ON THE
RED TEAM

@andyﬂeener

@andyﬂeener THE PRIME DIRECTIVE : LEARN

@andyﬂeener BY LEARNING, OUR SYSTEM CAN CONTINUE TO GROW AND
IMPROVE

@andyﬂeener WE’RE ACTIVELY ADDING CAPACITY TO THE SYSTEM TO FAIL

@andyﬂeener IS THE KEY TO CREATING NOT JUST RELIABLE SYSTEMS
BUT RESILIENT ONES ADDING CAPACITY TO FAIL

@andyﬂeener TOOLS AND TOOLING BREAKOUT SESSION

@andyﬂeener STAY AWESOME AND THANKS!

Game Days: Failing for Fun and Profit

Game Days: Failing for Fun and Profit

More Decks by Andy Fleener

Other Decks in Technology

Featured

Transcript