@andyfleener HERE’S YOUR PAGER; GOOD LUCK HAVE FUN

@andyfleener

@andyfleener I’M NEW VIEW SAFETY NERD • Humanist • Systems Thinker • Wannabe Resilience Engineering Enthusiast

@andyfleener I’VE BEEN IN THE SAME ON-CALL ROTATION FOR 7 YEARS

@andyfleener BASIC PREMISE OF THIS TALK

@andyfleener THERE IS NO SUBSTITUTION FOR EXPERTISE

@andyfleener

@andyfleener

@andyfleener COMPLEX SYSTEMS ARE INTRINSICALLY HAZARDOUS SYSTEMS. — Richard Cook MD, How Complex Systems Fail

@andyfleener HUMAN OPERATORS HAVE DUAL ROLES: AS PRODUCERS & AS DEFENDERS AGAINST FAILURE. — Richard Cook MD, How Complex Systems Fail

@andyfleener ALL PRACTITIONER ACTIONS ARE GAMBLES. — Richard Cook MD, How Complex Systems Fail

@andyfleener HUMAN PRACTITIONERS ARE THE ADAPTABLE ELEMENT OF COMPLEX SYSTEMS. — Richard Cook MD, How Complex Systems Fail

@andyfleener PEOPLE CONTINUOUSLY CREATE SAFETY. — Richard Cook MD, How Complex Systems Fail

@andyfleener SO WHAT? ¯\_(ツ)_/¯

@andyfleener HOW DO YOU BUILD EXPERTISE ‽‽‽

@andyfleener Baiyin Yang, Karen E. Watkins, Victoria J. Marsick, The Construct of the Learning Organization: Dimensions, Measurement, and Validation THE CONSTRUCT OF THE LEARNING ORGANIZATION

@andyfleener “LEADERSHIP AS AN EMERGENT PROPERTY OF THE SYSTEM THAT MOVES THE ORGANIZATION FORWARD” — James Barker, PhD, Dalhousie University

@andyfleener BIG DISCLAIMER

@andyfleener ON-CALL SHOULD BE COMPENSATED

@andyfleener • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG DEFINITIONS OF YOU

@andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG

@andyfleener BUILD EXPERTISE TAKE CARE OF YOURSELF THE ENGINEER’S PERSPECTIVE

@andyfleener TAKE CARE OF YOURSELF THE ENGINEER’S PERSPECTIVE • You are not a robot, you are a human • Clear your calendar or trade conflicts away • Understand your backup situation • Keep your laptop and phone charged! • Monitor your stress and energy levels closely

@andyfleener ON-CALL CAN BE STRESSFUL. WHETHER YOU'RE GETTING HAMMERED WITH PAGES OR YOU ONLY GOT ONE (BUT IT WAS AT 3 AM) GETTING INTO A RELAXED AND SLEEPY FRAME OF MIND CAN BE DIFFICULT. — Alice Goldfuss, “The On-Call Handbook”

@andyfleener BUILD EXPERTISE BE CURIOUS THE ENGINEER’S PERSPECTIVE

@andyfleener “CHANGE A BASIC ASSUMPTION AND YOU HAVE CHANGED THE SYSTEM ITSELF.” — Eliyahu M. Goldratt, Essays on the Theory of Constraints

@andyfleener BE CURIOUS THE ENGINEER’S PERSPECTIVE • Being uncomfortable with a system or set of systems is useful feedback, embrace it as an opportunity to learn. • Take notes - you may not have time to ask in the moment but use your notes and come back to it. • Read any material you can find on previous incidents! • Pair on normal every day work and ask lots of questions! • Don’t be afraid to make mistakes!

@andyfleener “WHEN WE PUT TOO MUCH ENERGY INTO ELIMINATING MISTAKES, WE’RE LESS LIKELY TO GAIN INSIGHTS. HAVING INSIGHTS IS A DIFFERENT MATTER FROM PREVENTING MISTAKES.” — GARY KLEIN, Seeing What Others Don’t

@andyfleener BUILD EXPERTISE SEEK TO UNDERSTAND THE IMPACT THE ENGINEER’S PERSPECTIVE

@andyfleener SEEK TO UNDERSTAND THE IMPACT THE ENGINEER’S PERSPECTIVE • What does this alert mean for our customers? • Who do I need to tell about this incident? • Is something currently sideways? Is it about to go sideways? Or is this thing just a dumpster fire?

@andyfleener LIVE LOOK AT THE INTERNET

@andyfleener BUILD EXPERTISE THE ENGINEER’S PERSPECTIVE Charity Majors, Observability for Emerging Infra (what got you here won't get you there)

@andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG

@andyfleener BUILD EXPERTISE BE INTENTIONAL THE TEAM’S PERSPECTIVE

@andyfleener BUILD EXPERTISE SHADOWING THE TEAM’S PERSPECTIVE

@andyfleener BUILD EXPERTISE ESCALATION SYSTEMS THE TEAM’S PERSPECTIVE

@andyfleener BUILD EXPERTISE THE TEAM’S PERSPECTIVE

@andyfleener BUILD EXPERTISE EXPERTISE IS NOT EVENLY DISTRIBUTED THE TEAM’S PERSPECTIVE

@andyfleener BUILD EXPERTISE REVIEW YOUR ALERTS THE TEAM’S PERSPECTIVE

@andyfleener REVIEW YOUR ALERTS THE TEAM’S PERSPECTIVE

@andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG

@andyfleener BUILD EXPERTISE INCIDENT RESPONSE PLAN THE ORG’S PERSPECTIVE

@andyfleener BUILD EXPERTISE WHAT’S AN INCIDENT? THE ORG’S PERSPECTIVE

@andyfleener BUILD EXPERTISE CREATE YOUR RESPONSE VOLTRON THE ORG’S PERSPECTIVE

@andyfleener BUILD EXPERTISE SOME SAGE ADVICE THE ORG’S PERSPECTIVE

@andyfleener BUILD EXPERTISE GAME DAYS THE ORG’S PERSPECTIVE

@andyfleener GAME DAYS TREAT GAMES AS PRODUCTION INCIDENTS BY FOLLOWING THE FULL INCIDENT RESPONSE LIFECYCLE THE ORG’S PERSPECTIVE

@andyfleener GAME DAYS THE ORG’S PERSPECTIVE

@andyfleener BUILD EXPERTISE INCIDENT REVIEWS THE ORG’S PERSPECTIVE

@andyfleener TELL STORIES THE ORG’S PERSPECTIVE

@andyfleener AVOID THE TRAPS THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation

@andyfleener COUNTERFACTUAL LANGUAGE THE ORG’S PERSPECTIVE Sidney Dekker, The Field Guide to Understanding Human Error

@andyfleener COUNTERFACTUALS THE ORG’S PERSPECTIVE

@andyfleener NORMATIVE LANGUAGE THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation

@andyfleener NORMATIVE LANGUAGE “INSUFFICIENT SERVICE MONITORING OF DATABASE PERFORMANCE DURING THE DEPLOY RESULTED FROM EXPECTANCY, INCREASED WORKLOAD, FATIGUE AND AUTOMATION RELIANCE.” THE ORG’S PERSPECTIVE

@andyfleener MECHANISTIC REASONING THE ORG’S PERSPECTIVE

@andyfleener MECHANISTIC REASONING THE ORG’S PERSPECTIVE

@andyfleener AVOID THE TRAPS THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation

@andyfleener THESE IDEAS ARE NOT ALL MY OWN • How Complex Systems Fail, Richard Cook MD • The Construct of the Learning Organization: Dimensions, Measurement, and Validation, Baiyin Yang, Karen E. Watkins, Victoria J. Marsick • Leading a complex organization, James Barker PhD • “The On-Call Handbook”, Alice Goldfuss • Essays on the Theory of Constraints, Eliyahu M. Goldratt • Seeing What Others Don’t, Gary Klein PhD • Observability for Emerging Infra (what got you here won't get you there), Charity Majors • Three analytical traps in accident investigation, Johan Bergström PhD • The Field Guide to Understanding Human Error, Sidney Dekker PhD • The problem with counterfactuals, Lorin Hochstein PhD

@andyfleener THANKS!