Here's Your Pager; Good Luck Have Fun!

As a manager of a team that has On-Call responsibilities, I am personally accountable for ensuring engineers are prepared to receive pages. As a systems thinker, I’ve broken this down into many different layered approaches that leave room for adaptation in the face of new and emergent system behavior.

Ok, great… what does that mean? Troubleshooting problems within a complex system requires enormous amounts of context-specific expertise. It is next to impossible to provide adequate training to build that expertise. That expertise is created over time and I’ve found that learning on the job is the most effective way to build it.

Treat On-Call as a team sport; give an engineer the freedom to learn and grow. If it’s a team sport you can still get the best possible response out of an incident, learning along the way. How you run your On-Call on-boarding has a significant impact on creating this team sport mentality and affects team culture. If we build a team culture based on trust and learning, we can do so much better than “Good Luck Have Fun”.

Transcript

1. @andyfleener HERE’S YOUR PAGER; GOOD LUCK HAVE FUN
2. @andyfleener
3. @andyfleener I’M NEW VIEW SAFETY NERD • Humanist • Systems Thinker • Wannabe Resilience Engineering Enthusiast
4. @andyfleener I’VE BEEN IN THE SAME ON-CALL ROTATION FOR 7 YEARS
5. @andyfleener BASIC PREMISE OF THIS TALK
6. @andyfleener THERE IS NO SUBSTITUTION FOR EXPERTISE
7. @andyfleener
8. @andyfleener
9. @andyfleener COMPLEX SYSTEMS ARE INTRINSICALLY HAZARDOUS SYSTEMS. — Richard Cook MD, How Complex Systems Fail
10. @andyfleener HUMAN OPERATORS HAVE DUAL ROLES: AS PRODUCERS & AS DEFENDERS AGAINST FAILURE. — Richard Cook MD, How Complex Systems Fail
11. @andyfleener ALL PRACTITIONER ACTIONS ARE GAMBLES. — Richard Cook MD, How Complex Systems Fail
12. @andyfleener HUMAN PRACTITIONERS ARE THE ADAPTABLE ELEMENT OF COMPLEX SYSTEMS. — Richard Cook MD, How Complex Systems Fail
13. @andyfleener PEOPLE CONTINUOUSLY CREATE SAFETY. — Richard Cook MD, How Complex Systems Fail
14. @andyfleener SO WHAT? ¯\_(ツ)_/¯
15. @andyfleener HOW DO YOU BUILD EXPERTISE ‽‽‽
16. @andyfleener Baiyin Yang, Karen E. Watkins, Victoria J. Marsick, The Construct of the Learning Organization: Dimensions, Measurement, and Validation THE CONSTRUCT OF THE LEARNING ORGANIZATION
17. @andyfleener “LEADERSHIP AS AN EMERGENT PROPERTY OF THE SYSTEM THAT MOVES THE ORGANIZATION FORWARD” — James Barker, PhD, Dalhousie University
18. @andyfleener BIG DISCLAIMER
19. @andyfleener ON-CALL SHOULD BE COMPENSATED
20. @andyfleener • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG DEFINITIONS OF YOU
21. @andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG
22. @andyfleener BUILD EXPERTISE TAKE CARE OF YOURSELF THE ENGINEER’S PERSPECTIVE
23. @andyfleener TAKE CARE OF YOURSELF THE ENGINEER’S PERSPECTIVE • You are not a robot, you are a human • Clear your calendar or trade conflicts away • Understand your backup situation • Keep your laptop and phone charged! • Monitor your stress and energy levels closely
24. @andyfleener ON-CALL CAN BE STRESSFUL. WHETHER YOU'RE GETTING HAMMERED WITH PAGES OR YOU ONLY GOT ONE (BUT IT WAS AT 3 AM) GETTING INTO A RELAXED AND SLEEPY FRAME OF MIND CAN BE DIFFICULT. — Alice Goldfuss, “The On-Call Handbook”
25. @andyfleener BUILD EXPERTISE BE CURIOUS THE ENGINEER’S PERSPECTIVE
26. @andyfleener “CHANGE A BASIC ASSUMPTION AND YOU HAVE CHANGED THE SYSTEM ITSELF.” — Eliyahu M. Goldratt, Essays on the Theory of Constraints
27. @andyfleener BE CURIOUS THE ENGINEER’S PERSPECTIVE • Being uncomfortable with a system or set of systems is useful feedback, embrace it as an opportunity to learn. • Take notes - you may not have time to ask in the moment but use your notes and come back to it. • Read any material you can find on previous incidents! • Pair on normal every day work and ask lots of questions! • Don’t be afraid to make mistakes!
28. @andyfleener “WHEN WE PUT TOO MUCH ENERGY INTO ELIMINATING MISTAKES, WE’RE LESS LIKELY TO GAIN INSIGHTS. HAVING INSIGHTS IS A DIFFERENT MATTER FROM PREVENTING MISTAKES.” — GARY KLEIN, Seeing What Others Don’t
29. @andyfleener BUILD EXPERTISE SEEK TO UNDERSTAND THE IMPACT THE ENGINEER’S PERSPECTIVE
30. @andyfleener SEEK TO UNDERSTAND THE IMPACT THE ENGINEER’S PERSPECTIVE • What does this alert mean for our customers? • Who do I need to tell about this incident? • Is something currently sideways? Is it about to go sideways? Or is this thing just a dumpster fire?
31. @andyfleener LIVE LOOK AT THE INTERNET
32. @andyfleener BUILD EXPERTISE THE ENGINEER’S PERSPECTIVE Charity Majors, Observability for Emerging Infra (what got you here won't get you there)
33. @andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG
34. @andyfleener BUILD EXPERTISE BE INTENTIONAL THE TEAM’S PERSPECTIVE
35. @andyfleener BUILD EXPERTISE SHADOWING THE TEAM’S PERSPECTIVE
36. @andyfleener BUILD EXPERTISE ESCALATION SYSTEMS THE TEAM’S PERSPECTIVE
37. @andyfleener BUILD EXPERTISE THE TEAM’S PERSPECTIVE
38. @andyfleener BUILD EXPERTISE EXPERTISE IS NOT EVENLY DISTRIBUTED THE TEAM’S PERSPECTIVE
39. @andyfleener BUILD EXPERTISE REVIEW YOUR ALERTS THE TEAM’S PERSPECTIVE
40. @andyfleener REVIEW YOUR ALERTS THE TEAM’S PERSPECTIVE
41. @andyfleener DEFINITIONS OF YOU • YOU - THE ENGINEER • YOU - THE TEAM • YOU - THE ORG
42. @andyfleener BUILD EXPERTISE INCIDENT RESPONSE PLAN THE ORG’S PERSPECTIVE
43. @andyfleener BUILD EXPERTISE WHAT’S AN INCIDENT? THE ORG’S PERSPECTIVE
44. @andyfleener BUILD EXPERTISE CREATE YOUR RESPONSE VOLTRON THE ORG’S PERSPECTIVE
45. @andyfleener BUILD EXPERTISE SOME SAGE ADVICE THE ORG’S PERSPECTIVE
46. @andyfleener BUILD EXPERTISE GAME DAYS THE ORG’S PERSPECTIVE
47. @andyfleener GAME DAYS TREAT GAMES AS PRODUCTION INCIDENTS BY FOLLOWING THE FULL INCIDENT RESPONSE LIFECYCLE THE ORG’S PERSPECTIVE
48. @andyfleener GAME DAYS THE ORG’S PERSPECTIVE
49. @andyfleener BUILD EXPERTISE INCIDENT REVIEWS THE ORG’S PERSPECTIVE
50. @andyfleener TELL STORIES THE ORG’S PERSPECTIVE
51. @andyfleener AVOID THE TRAPS THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation
52. @andyfleener COUNTERFACTUAL LANGUAGE THE ORG’S PERSPECTIVE Sidney Dekker, The Field Guide to Understanding Human Error
53. @andyfleener COUNTERFACTUALS THE ORG’S PERSPECTIVE
54. @andyfleener NORMATIVE LANGUAGE THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation
55. @andyfleener NORMATIVE LANGUAGE “INSUFFICIENT SERVICE MONITORING OF DATABASE PERFORMANCE DURING THE DEPLOY RESULTED FROM EXPECTANCY, INCREASED WORKLOAD, FATIGUE AND AUTOMATION RELIANCE.” THE ORG’S PERSPECTIVE
56. @andyfleener MECHANISTIC REASONING THE ORG’S PERSPECTIVE
57. @andyfleener MECHANISTIC REASONING THE ORG’S PERSPECTIVE
58. @andyfleener AVOID THE TRAPS THE ORG’S PERSPECTIVE Johan Bergström, Three analytical traps in accident investigation
59. @andyfleener THESE IDEAS ARE NOT ALL MY OWN • How Complex Systems Fail, Richard Cook MD • The Construct of the Learning Organization: Dimensions, Measurement, and Validation, Baiyin Yang, Karen E. Watkins, Victoria J. Marsick • Leading a complex organization, James Barker PhD • “The On-Call Handbook”, Alice Goldfuss • Essays on the Theory of Constraints, Eliyahu M. Goldratt • Seeing What Others Don’t, Gary Klein PhD • Observability for Emerging Infra (what got you here won't get you there), Charity Majors • Three analytical traps in accident investigation, Johan Bergström PhD • The Field Guide to Understanding Human Error, Sidney Dekker PhD • The problem with counterfactuals, Lorin Hochstein PhD
60. @andyfleener THANKS!