How post-mortems can close the loop on IT metrics

The point of a postmortem is to learn. In this presentation, Jason Hand demonstrates how they also help close the (OODA) loop on IT Metrics. Originally a webinar, you can view and hear the recording here: https://victorops.com/knowledge-drop/webinars/post-mortems-can-close-loop-metrics/

Transcript

1. How Postmortems can close the loop on IT metrics @jasonhand 1
2. Jason Hand victorops devops evangelist @jasonhand @jasonhand 2
3. automate measure share learn @jasonhand 3
4. automate @jasonhand 4
5. measure @jasonhand 5
6. share @jasonhand 6
7. learn @jasonhand 7
8. increase feedback Shortening feedback loops leads to... @jasonhand 8
9. continuous... integration (of code) deployment (of software/product) and ... @jasonhand 9
10. improvement @jasonhand 10
11. OODA loop Observe Orient Decide Act by: John Boyd @jasonhand 11
12. why we measure @jasonhand 12
13. observe & orient @jasonhand 13
14. which leads to deciding & acting @jasonhand 14
15. What metrics should we be looking at? CPU, Memory, Network, and Disk metrics (duh) @jasonhand 15
16. but wait! there's more @jasonhand 16
17. look between the spaces @jasonhand 17
18. understand what we're building @jasonhand 18
19. between the spaces look for metrics to help: —weed out edge case scenarios —confirm assumptions @jasonhand 19
20. should we? @jasonhand 20
21. actually.. more like.. @jasonhand 21
22. collect all the things @jasonhand 22
23. "We don't know what the context is and we don't know what may be interesting to us in the future. But if something goes down and we don't have metrics for it, we have no perspective... and that's probably the worst-case scenario." - Jason Dixon (librato.com & monitorama) The Ship Show (podcast): Episode 56 @jasonhand 23
24. Fact: measuring and looking at data all the time isn't that helpful. But: when you need it to understand a problem, you'll want it then. @jasonhand 24
25. during an incident leverage metrics to observe & orient to help you decide & take action @jasonhand 25
26. all to help: - triage - investigate - identify what's happening. @jasonhand 26
27. why do we do postmortems? @jasonhand 27
28. Learn... which leads to decisions and action @jasonhand 28
29. get your story straight understand the story of what took place following an incident. @jasonhand 29
30. accountability & empathy By sharing an accurate account and attempting to understand and empathize around exactly what took place, teams can learn from that incident and improve their processes. @jasonhand 30
31. how do we do postmortems @jasonhand 31
32. capture everything and in one place @jasonhand 32
33. victorops timeline @jasonhand 33
34. chat slack hipchat @jasonhand 34
35. blameless remove blame and go after the facts @jasonhand 35
36. The idea is to learn @jasonhand 36
37. punish @jasonhand 37
38. remediation items "Learning from a postmortem is only as useful as what you put into practice afterwards and we realized that without any action items after the meeting, it was more or less just a Greek Senate debate" —Ben VanEvery (box.com) @jasonhand 38
39. assign ownership it's not about sharing feelings and theories without accomplishing anything @jasonhand 39
40. tie it with a bow Now you have a very accurate story @jasonhand 40
41. a story telling us ... @jasonhand 41
42. monitoring data @jasonhand 42
43. who was alerted @jasonhand 43
44. how quickly they responded @jasonhand 44
45. who was involved throughout the incident management lifecycle @jasonhand 45
46. conversations that were had @jasonhand 46
47. commands that were run (i.e. ChatOps) @jasonhand 47
48. context @jasonhand 48
49. Memorialized - monitoring data - alerts - acknowledgments - context . graphs . logs . runbooks . notes - actions - conversations - remediation @jasonhand 49
50. loop closed We've taken the important monitoring data and metrics .. start back over @jasonhand 50
51. Jason Hand @jasonhand @victorops @jasonhand 51
52. Chatops For Dummies jhand.co/ChatOps4Dummies @jasonhand 52