Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dealing with Incidents using Monzo Response
Search
Christopher Evans
May 22, 2019
Technology
0
41
Dealing with Incidents using Monzo Response
Christopher Evans
May 22, 2019
Tweet
Share
More Decks by Christopher Evans
See All by Christopher Evans
How to Build a Digital Bank Using AWS
evnsio
0
180
Keeping a modern bank online
evnsio
1
430
Other Decks in Technology
See All in Technology
「海外登壇」という 選択肢を与えるために 〜Gophers EX
logica0419
0
500
管理者しか知らないOutlookの裏側のAIを覗く#AzureTravelers
hirotomotaguchi
1
240
これからSREになる人と、これからもSREをやっていく人へ
masayoshi
6
4.1k
AndroidデバイスにFTPサーバを建立する
e10dokup
0
240
スタートアップ1人目QAエンジニアが QAチームを立ち上げ、“個”からチーム、 そして“組織”に成長するまで / How to set up QA team at reiwatravel
mii3king
1
1.1k
Tech Blogを書きやすい環境づくり
lycorptech_jp
PRO
0
120
20250208_OpenAIDeepResearchがやばいという話
doradora09
PRO
0
170
目の前の仕事と向き合うことで成長できる - 仕事とスキルを広げる / Every little bit counts
soudai
22
5.8k
RSNA2024振り返り
nanachi
0
500
Ask! NIKKEI RAG検索技術の深層
hotchpotch
13
2.8k
スクラムのイテレーションを導入してチームの雰囲気がより良くなった話
eccyun
0
110
テストアーキテクチャ設計で実現する高品質で高スピードな開発の実践 / Test Architecture Design in Practice
ropqa
3
710
Featured
See All Featured
Bash Introduction
62gerente
610
210k
BBQ
matthewcrist
86
9.5k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Documentation Writing (for coders)
carmenintech
67
4.6k
Embracing the Ebb and Flow
colly
84
4.6k
Building Applications with DynamoDB
mza
93
6.2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
99
18k
Optimising Largest Contentful Paint
csswizardry
34
3.1k
Building Your Own Lightsaber
phodgson
104
6.2k
Designing for humans not robots
tammielis
250
25k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
20
2.4k
Transcript
Dealing with Incidents using Monzo Response ⚡ Chris Evans Platform
/ On-call Lead @evnsio
None
None
None
None
None
If any of that sounds interesting, we’re hiring… monzo.com/careers
Let’s talk about incidents...
“When you assign an incident lead, let customer operations know
who it is”
“...and remember to set a severity within the first 5
mins of the incident”
“...and if it’s a major severity, follow this procedure”
“...and if it’s a critical severity, here’s this other stuff
you need to do”
“...and if you’re going to put the status page up,
contact this team so they can decide whether we need to inform the regulator”
“...and check this spreadsheet for risk ratings to know whether
you need to notify a senior exec”
“...oh, and the incident needs to be dealt with too”
None
Response ⚡ is the result of applying software engineering to
incident response
Limit context switching Context switching during an incident is often
unavoidable. We aim to limit this, by enabling actions to be carried out without leaving the conversation.
Make the easiest thing the right thing If something needs
doing, bring it to the attention of the responder when it makes sense, or better still automate it away.
events, messages, etc Monzo Incident Backend Monzo Incident Web UI
Slack Architecture view, edit incidents messages, actions
The Basics
None
None
The Headline Post
The Incident Doc
The Comms Channel
The Comms Channel
The Comms Channel
Building Blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a user wants the tool
to do something...
@incident <command> <action>
None
None
None
None
None
Some other commands we use at Monzo action log an
action to be followed up later escalate escalate to another set of on-callers statuspage update our customer facing status page rename rename a comms channel to something descriptive
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a word or phrase is
mentioned...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called on a particular schedule...
None
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a particular Slack event happens...
None
None
Some other events that might be interesting... Users posting messages
Users pasting links Reactions
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack button is clicked...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack dialog is submitted...
None
“I want to be able to Page engineers without leaving
Slack”
None
Page Engineer One line of code One function
One incident command to post the message One function to
handle buttons and raise dialog One function to handle dialog submission
Takeaways Map out your incident handling process Understand what you’re
being asked to do and when Use the building blocks of Response ⚡ to codify it Let your on-callers do what they do best
Thanks for listening! @evnsio github.com/monzo/response