Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dealing with Incidents using Monzo Response
Search
Christopher Evans
May 22, 2019
Technology
0
35
Dealing with Incidents using Monzo Response
Christopher Evans
May 22, 2019
Tweet
Share
More Decks by Christopher Evans
See All by Christopher Evans
How to Build a Digital Bank Using AWS
evnsio
0
140
Keeping a modern bank online
evnsio
1
410
Other Decks in Technology
See All in Technology
Challenges - Open Farming Hackdays 2024
loleg
0
500
どう買う?Azure
kuniteru
1
190
ビジネスとコード品質の接合点 そしてコード品質がそこに及ぼす影響 / The Intersections of Business and Engineering, and The Impact of Code Quality There
mtx2s
10
1k
統計的学習理論読み Chapter 2
kmatsui
1
180
BDD(Cucumber)コミュニティが無料提供しているコンテンツの紹介と現在起きている危機
nihonbuson
4
730
オーティファイ会社紹介資料 / Autify Company Deck
autifyhq
7
100k
戦略的DDDを実践するための跳躍力 / OOC 2024
pictiny
6
3.2k
AWS アーキテクチャクイズ
yuu26
2
700
オブジェクト指向宗教史
tanakahisateru
13
11k
生成AIの不確実性と向き合うためのオブジェクト指向設計
tkikuchi1002
2
340
マイクロサービス環境におけるDB戦略 in DMMプラットフォーム
pospome
11
3k
大規模なアジャイル開発の現場と技術負債 / Technical Debt
yoshiitaka
20
4k
Featured
See All Featured
The Art of Programming - Codeland 2020
erikaheidi
40
12k
Keith and Marios Guide to Fast Websites
keithpitt
407
22k
Designing on Purpose - Digital PM Summit 2013
jponch
109
6.4k
10 Git Anti Patterns You Should be Aware of
lemiorhan
644
57k
WebSockets: Embracing the real-time Web
robhawkes
59
6.9k
What's new in Ruby 2.0
geeforr
335
31k
Docker and Python
trallard
33
2.6k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.8k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
242
20k
Making the Leap to Tech Lead
cromwellryan
122
8.4k
Why You Should Never Use an ORM
jnunemaker
PRO
50
8.6k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
124
32k
Transcript
Dealing with Incidents using Monzo Response ⚡ Chris Evans Platform
/ On-call Lead @evnsio
None
None
None
None
None
If any of that sounds interesting, we’re hiring… monzo.com/careers
Let’s talk about incidents...
“When you assign an incident lead, let customer operations know
who it is”
“...and remember to set a severity within the first 5
mins of the incident”
“...and if it’s a major severity, follow this procedure”
“...and if it’s a critical severity, here’s this other stuff
you need to do”
“...and if you’re going to put the status page up,
contact this team so they can decide whether we need to inform the regulator”
“...and check this spreadsheet for risk ratings to know whether
you need to notify a senior exec”
“...oh, and the incident needs to be dealt with too”
None
Response ⚡ is the result of applying software engineering to
incident response
Limit context switching Context switching during an incident is often
unavoidable. We aim to limit this, by enabling actions to be carried out without leaving the conversation.
Make the easiest thing the right thing If something needs
doing, bring it to the attention of the responder when it makes sense, or better still automate it away.
events, messages, etc Monzo Incident Backend Monzo Incident Web UI
Slack Architecture view, edit incidents messages, actions
The Basics
None
None
The Headline Post
The Incident Doc
The Comms Channel
The Comms Channel
The Comms Channel
Building Blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a user wants the tool
to do something...
@incident <command> <action>
None
None
None
None
None
Some other commands we use at Monzo action log an
action to be followed up later escalate escalate to another set of on-callers statuspage update our customer facing status page rename rename a comms channel to something descriptive
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a word or phrase is
mentioned...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called on a particular schedule...
None
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a particular Slack event happens...
None
None
Some other events that might be interesting... Users posting messages
Users pasting links Reactions
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack button is clicked...
None
@slack_event @notification @action_handler @keyword_handler @incident_command @dialog_handler Building blocks
Functions that are called when a Slack dialog is submitted...
None
“I want to be able to Page engineers without leaving
Slack”
None
Page Engineer One line of code One function
One incident command to post the message One function to
handle buttons and raise dialog One function to handle dialog submission
Takeaways Map out your incident handling process Understand what you’re
being asked to do and when Use the building blocks of Response ⚡ to codify it Let your on-callers do what they do best
Thanks for listening! @evnsio github.com/monzo/response