Detective Hat: Investigating Production Issues

It was a dark and stormy night, and our production system was out cold. It was our job to find out what had gone wrong and set it right. But how? Our developer-instincts weren't helping. Our tester's-intuition saw risk everywhere. We needed a third way; the hard-nosed, boots on the ground, skeptical mindset of the detective! Jenny Bramble, Director of Quality Engineering, and Adrian P. Dunston, Director of Backend Engineering at Papa will give you a powerful mental model for tracking down production issues in your codebase and a set of tools to help you along the way. Using noir and tv-detective stories as a backdrop, you'll learn how to: take ownership for a production behavior, hit the streets to find the relevant facts and metrics, confront any hard truths you may find, bring your tools to bear, and follow through to make production a better place for everyone. With the detective mindset, grit, and determination you can go beyond sleuthing out your systems' behaviors. You'll build better paths to observability and better processes to prevent and recover production incidents. You might even help your colleagues find detective hats of their own.

Transcript

1. The Detective Hat Investigating production issues 1 Jenny Bramble @jennydoesthings Adrian P. Dunston
2. Hey, dev. We're getting reports of timeouts in the logs. Looks like we got a dead connection. 2
3. I'm sorry to hear that. My condolences. 3
4. We seem to be getting a lot of dead connections recently. Seems like somebody should look into it. 4
5. Yeah, maybe somebody should. But I got my own problems. Besides, I don't do that sort of thing anymore. 5
6. Seems to me like, if a person lets too many dead connections go, eventually something inside them dies too. 6
7. Anyway, see you around... 7
8. I'm a developer. I've got a sweet little epic going. Lots of little tickets to take care of. But darnit she's right. It's a dirty world out there. 8
9. Time to put my detective hat on. 9
10. The Detective Hat Investigating production issues 10
11. Hi, I'm Jenny! • Director of Quality Engineering at Papa • Test-based human for most of my career • Human interfacing is my favorite thing • Two cats—Dante and Dax • My pronouns are she/her 11
12. Adrian P. Dunston adrian@bitcapulet.com @bitcapulet 12
13. Adrian P. Dunston adrian@bitcapulet.com @bitcapulet 13
15. The purpose of this presentation is to help develop a mental model for tracking down production issues in your codebase. 15
16. The Detective Hat Prologue - Mindset 1. Taking the Case 2. Hitting the Streets 3. Chase Scene! 4. Putting them away… 16
17. Prologue Mindset 17
18. Confession Time… 18
19. We have no idea what police-work is like. But we've got a soft spot for noir stories and detective shows. 19
20. Also… we love production issues. 20
21. Production issues are a game you can win. The truth is out there. You can find it. 21
22. This is also another opportunity to do what Andy Knight was saying with "Shift Right." It's a chance to learn your system better and deepen your mental model. 22
23. Okay, my confession time. I love telling about old scars. They're shared history; even if you weren't there. You can also read the future in bumps, scars, and bruises. 23
24. Testing is expansive, checking for edge cases and fanning out. Production bughunts are contractive, drawing a box around it and narrowing down. 24
25. All production issues are a failure of process. 25
26. Part 1 Taking the Case 26
27. Someone comes in asking for help. There's something wrong, but they don't have a lot to go on, and there's little reward to be had from tracking it down. 27
28. Naturally your first instinct is to tell them to buzz off. This brings us to the first rule of the detective hat. 28
29. Believe the problem is real until you can prove it isn't. 29
30. Believe them on the first report. Maybe it's not like they think it is. Maybe the problem isn't really in your area. But there IS a problem. 30
31. Follow up on the impact and the reach of the problem. Maybe it doesn't need to be addressed. But remember, the decision isn't "is this really impacting people?" The decision is, "can I live with that impact?" 31
32. What if it's a false positive? Well then, the problem is that your system is sending false positives. 32
33. If they have a problem, and you tell them they don't, well now they have 2 problems. 33
34. Now that you've taken the case, it's time to take a deep breath. They're not going to give you much to go on. There aren't footprints leading to the killer, and there's no DNA evidence to follow up on. This is the crux of why detective hat is different. 34
35. There's no design, no well-crafted tickets. It's not as simple as upper or lower bound. It's up to you, and you'd rather this wasn't happening at all. Assume no one is going to help you. 35
36. So we took the case. We took a deep breath. Let's conduct some interviews. 36
37. Somebody reported this problem. Ask them questions. When? How often? What does it look like to them? How can we make it happen again? Who else should I talk to? 37
38. If there are no clear clues there, ask around the neighborhood. Rule 1 is believe there is a problem, so start there. If this was real, how would I prove it? Who would see it? Be creative. Be bold. Knock on doors. Check the data, the logs, the event history. 38
39. We'll talk about your detective notebook later.. Write down everything they tell you. 39
40. Write down everything they tell you. Don't believe a word of it. We always believe that there is a problem. We never believe anything else about it until we can prove it. On X-Files they say, Trust No One. 40
41. Write down everything they tell you. Don't believe a word of it. Verify the important facts. Hard-data, that's what we want. Friday from Dragnet doesn't want to know how people feel about something. "Just the facts, ma'am." Find them. 41
42. Write down everything they tell you. Don't believe a word of it. Verify the important facts. We'll be talking more about this in the next section. Collect, Doubt, and Rule-it-out 42
43. Part 2 Hitting the Streets 43
44. Keep a record of the issue. Start it as soon as you start tracking. This is your incident narrative or "detective's notebook." Detective's Notebook 44
45. Start with the problem as your users are seeing it. They reported something; write down specifically what that was. Also note the reach. This can evolve over time. • What's the problem? 45
46. Here's where that collect, doubt, and rule it out comes in. You'll come up with a bunch of things that MIGHT be causing this. Other people will come assuming they know what it is. Consider it all "potential" and put it in your notes. • What's the problem? • Potential causes 46
47. Similarly, there may be co-occuring oddities. Things that don't seem like a cause, but do seem related. This may help with triangulation later. Put it in the notes. • What's the problem? • Potential causes • Potential accessories (after-the-fact) 47
48. Every fact is treasure, and you don’t know what it will be worth. Write them down as you go. Each step you take, each fact you find, each unanswered question. • What's the problem? • Potential causes • Potential accessories (after-the-fact) • Unanswered questions 48
49. At the bottom of your notebook, put in the things that aren't relevant, but will be later. When tracking one bug, we often find others. • Tomorrow's problems ◦ unrelated bugs 49
50. Maybe you can see the logs and the data but there's some crucial bit of observability you're lacking. Write it down and ask your team to make it available for next time. • Tomorrow's problems ◦ unrelated bugs ◦ blind spots 50
51. "I'm not sure what the problem is yet, but this definitely wouldn't have happened if we'd only…" Sure. Good. Write it down, and head off that conversation. That's tomorrow's problem. • Tomorrow's problems ◦ unrelated bugs ◦ blind spots ◦ preventative steps 51
52. Here's your detective's notebook • What's the problem? • Potential causes • Potential accessories (after-the-fact) • Unanswered questions • Tomorrow's problems 52
53. Part 3 Chase Scene 53
54. Collect, Doubt, and Rule it Out 54
55. Compare, Contrast, and Find-out Fast 55
56. Round up the Usual Suspects (then let them go) 56
57. Learn about the users' situation and motivations Hercule Poirot solves a mystery by understanding the psychology of the people he's investigating. Very often a production issue is a misunderstanding between user's expectation and what we THINK they're expecting. Ask "what does this mean to you?" 57
58. Use a made-up example to illustrate where the problem might be. Go back and talk with your witnesses. Provide possible examples. What if it was like this? How would we know? 58
59. Build up your logging and observability. Surveillance You know where it's failing, but not why. Logging is cheap. Be creative and put some thought into how it would be most effective. 59
60. Make a change, throw the switch, and watch the metrics. The stakeout It's your codebase. If you have a theory about what's going on, why not act on that theory. Feature flags are your friends here. Design an experiment. Even if you don't find out where the bug is, you'll at least find out where it isn't. 60
61. Consider that some things are only possible at certain times or in certain conditions 61
62. Go back and ask the incisive follow-up question "Oh and ahh, one more thing" Go back and re-interview the witness. Once you have verified facts in hand, the story they gave before will have new perspective. Listen again, and this time ask the follow up questions. Lots of solutions to bugs start with "Oh, and one more thing…" 62
63. All of these techniques can be called modeling the incident. Modeling the Incident 63
64. In the noir stories, there's often a moment, when the detective is warned off the case. "I'm telling you, Dick. Don't follow through on this one…" 64
65. There are things about this case you don't want to know. 65
66. Some of your team's code is broken in ways you never imagined. 66
67. Nope, I checked everything. You can't stop thinking you know what's right. But you can start adding "unless there's something I don't know about." ...unless there's something I don't know about. 67
68. The detective usually gets a little banged up during the chase. Be okay with being wrong. This is where those scars come from. This is what builds seniors. 68
69. Communicating Status In a slow-burn issue, you want to report where you're at once or twice a day. In a major production incident, this is every half hour. Whether you have something new to report or not. 69
70. Part 4 Putting them away for a looong time… 70
71. Sherlock Holmes has a brother Mycroft. "He has no ambition and no energy. He will not even go out of his way to verify his own solutions, and would rather be considered wrong than take the trouble to prove himself right." Don't be Mycroft. 71
72. Don't stop until you can tell the whole story with supported facts. Handing it off to the D.A. 72
73. It's no use fixing a production issue if we don't learn from it. A production issue costs money. We paid for the thing. We may as well get our money's worth. 73
74. Sometimes this means holding a post-mortem. Sometimes it means writing documentation or adding process or guardrails to ensure this doesn't happen again. 74
75. Also review your blind-spots. Where were those holes in observability? What are the tools you WISH you had while you were in the thick of it? 75
76. The Detective Hat Prologue - Mindset 1. Taking the Case 2. Hitting the Streets 3. Chase Scene! 4. Putting them away… 76
77. There is a certain joy in having been there, done that, and being too old for this. 77
78. Because all it takes is that one case your folks can't solve, and you're back in the game! 78
79. Good work, detective. Our users will sleep better at night knowing their connections are safe. 79
80. Thanks, I appreciate your support. There's just one thing where you're wrong. I quit the detective game. I'm regular dev with regular dev problems. 80
81. Of course you are, dev. Of course you are... 81
82. …until the next time. 82
83. Fin? 83
85. The answer to "How would we know if this works in prod?" is VERY OFTEN "We wouldn't." and "It doesn't." 85
86. And the answer to "Why didn't you tell me?!" is often "We did." and "You weren't ready to hear it." 86
87. Don't trust your devs. Don't trust your unit tests. Don't trust your ability to run something and say "there's no problem here." Prove it. Verify in prod. Check prod data. Keep asking, "What if I'm wrong? How would I know?" We are wrong all the time. 87
88. Nobody expects code you tested in one environment to run the same in the world it's deployed to. And if that's your expectation, you're only going to get hurt. 88
89. But take heart! Every production bug is a failure in process. Whether designer, reviewer, tester, or operations expert; we're all here to make quality software. And if the system doesn't work, we're all responsible. This isn't about you. 89
90. We've been on the case a while now. We've asked questions, we've looked at logs, we've accepted hard truths. Now we're starting to get the picture, narrowing down suspects, closing in on the culprit. 90
91. There are a generations of techniques we can bring to bear to flush them out from here. 91
92. Rubber Ducking 92
93. Continuous integration, Continuous Delivery, and Observability Tools. What about SQL? Tech Tools 93