Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dealing with Incidents using Monzo Response
Search
Christopher Evans
May 22, 2019
Technology
0
41
Dealing with Incidents using Monzo Response
Christopher Evans
May 22, 2019
Tweet
Share
More Decks by Christopher Evans
See All by Christopher Evans
How to Build a Digital Bank Using AWS
evnsio
0
180
Keeping a modern bank online
evnsio
1
430
Other Decks in Technology
See All in Technology
『AWS Distinguished Engineerに学ぶ リトライの技術』 #ARC403/Marc Brooker on Try again: The tools and techniques behind resilient systems
quiver
0
130
Datadogとともにオブザーバビリティを布教しよう
mego2221
0
130
プロセス改善による品質向上事例
tomasagi
1
1.6k
7日間でハッキングをはじめる本をはじめてみませんか?_ITエンジニア本大賞2025
nomizone
2
1.4k
個人開発から公式機能へ: PlaywrightとRailsをつなげた3年の軌跡
yusukeiwaki
11
2.7k
AWSでRAGを実現する上で感じた3つの大事なこと
ymae
3
1k
株式会社EventHub・エンジニア採用資料
eventhub
0
4.2k
アジャイル開発とスクラム
araihara
0
160
Culture Deck
optfit
0
330
目の前の仕事と向き合うことで成長できる - 仕事とスキルを広げる / Every little bit counts
soudai
22
5.8k
High Performance PHP
cmuench
0
140
オブザーバビリティの観点でみるAWS / AWS from observability perspective
ymotongpoo
7
1k
Featured
See All Featured
Thoughts on Productivity
jonyablonski
69
4.5k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
99
18k
How STYLIGHT went responsive
nonsquared
98
5.3k
YesSQL, Process and Tooling at Scale
rocio
171
14k
Git: the NoSQL Database
bkeepers
PRO
427
64k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
Building Applications with DynamoDB
mza
93
6.2k
Faster Mobile Websites
deanohume
306
31k
Rails Girls Zürich Keynote
gr2m
94
13k
A Philosophy of Restraint
colly
203
16k
Transcript
Dealing with Incidents using Monzo Response ⚡ Chris Evans Platform
/ On-call Lead @evnsio
None
None
None
None
None
If any of that sounds interesting, we’re hiring… monzo.com/careers
Let’s talk about incidents...
“When you assign an incident lead, let customer operations know
who it is”
“...and remember to set a severity within the first 5
mins of the incident”
“...and if it’s a major severity, follow this procedure”
“...and if it’s a critical severity, here’s this other stuff
you need to do”
“...and if you’re going to put the status page up,
contact this team so they can decide whether we need to inform the regulator”
“...and check this spreadsheet for risk ratings to know whether
you need to notify a senior exec”
“...oh, and the incident needs to be dealt with too”
None
Response ⚡ is the result of applying software engineering to
incident response
Limit context switching Context switching during an incident is often
unavoidable. We aim to limit this, by enabling actions to be carried out without leaving the conversation.
Make the easiest thing the right thing If something needs
doing, bring it to the attention of the responder when it makes sense, or better still automate it away.
events, messages, etc Monzo Incident Backend Monzo Incident Web UI
Slack Architecture view, edit incidents messages, actions
The Basics
None
None
The Headline Post
The Incident Doc
The Comms Channel
The Comms Channel
The Comms Channel
Building Blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a user wants the tool
to do something...
@incident <command> <action>
None
None
None
None
None
Some other commands we use at Monzo action log an
action to be followed up later escalate escalate to another set of on-callers statuspage update our customer facing status page rename rename a comms channel to something descriptive
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a word or phrase is
mentioned...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called on a particular schedule...
None
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a particular Slack event happens...
None
None
Some other events that might be interesting... Users posting messages
Users pasting links Reactions
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack button is clicked...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack dialog is submitted...
None
“I want to be able to Page engineers without leaving
Slack”
None
Page Engineer One line of code One function
One incident command to post the message One function to
handle buttons and raise dialog One function to handle dialog submission
Takeaways Map out your incident handling process Understand what you’re
being asked to do and when Use the building blocks of Response ⚡ to codify it Let your on-callers do what they do best
Thanks for listening! @evnsio github.com/monzo/response