Dealing with Incidents using Monzo Response

Transcript

1. Dealing with Incidents using Monzo Response ⚡ Chris Evans Platform / On-call Lead @evnsio
7. If any of that sounds interesting, we’re hiring… monzo.com/careers
8. Let’s talk about incidents...
9. “When you assign an incident lead, let customer operations know who it is”
10. “...and remember to set a severity within the first 5 mins of the incident”
11. “...and if it’s a major severity, follow this procedure”
12. “...and if it’s a critical severity, here’s this other stuff you need to do”
13. “...and if you’re going to put the status page up, contact this team so they can decide whether we need to inform the regulator”
14. “...and check this spreadsheet for risk ratings to know whether you need to notify a senior exec”
15. “...oh, and the incident needs to be dealt with too”
17. Response ⚡ is the result of applying software engineering to incident response
18. Limit context switching Context switching during an incident is often unavoidable. We aim to limit this, by enabling actions to be carried out without leaving the conversation.
19. Make the easiest thing the right thing If something needs doing, bring it to the attention of the responder when it makes sense, or better still automate it away.
20. events, messages, etc Monzo Incident Backend Monzo Incident Web UI Slack Architecture view, edit incidents messages, actions
21. The Basics
24. The Headline Post
25. The Incident Doc
26. The Comms Channel
27. The Comms Channel
28. The Comms Channel
29. Building Blocks
30. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
31. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
32. Functions that are called when a user wants the tool to do something...
33. @incident <command> <action>
39. Some other commands we use at Monzo action log an action to be followed up later escalate escalate to another set of on-callers statuspage update our customer facing status page rename rename a comms channel to something descriptive
40. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
41. Functions that are called when a word or phrase is mentioned...
43. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
44. Functions that are called on a particular schedule...
47. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
48. Functions that are called when a particular Slack event happens...
51. Some other events that might be interesting... Users posting messages Users pasting links Reactions
52. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
53. Functions that are called when a Slack button is clicked...
55. @slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
56. Functions that are called when a Slack dialog is submitted...
58. “I want to be able to Page engineers without leaving Slack”
60. Page Engineer One line of code One function
61. One incident command to post the message One function to handle buttons and raise dialog One function to handle dialog submission
62. Takeaways Map out your incident handling process Understand what you’re being asked to do and when Use the building blocks of Response ⚡ to codify it Let your on-callers do what they do best
63. Thanks for listening! @evnsio github.com/monzo/response