Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
GitHub Universe 2015 Talk - Your software is broken — pay attention: Rethinking production monitoring
Search
James Smith
October 02, 2015
Technology
1
100
GitHub Universe 2015 Talk - Your software is broken — pay attention: Rethinking production monitoring
My talk from GitHub Universe 2015's "Deploy" track
James Smith
October 02, 2015
Tweet
Share
More Decks by James Smith
See All by James Smith
Why Are Android Apps So Crash-Prone?
loopj
0
160
RailsConf 2016 Talk - Your software is broken — pay attention: Rethinking production monitoring
loopj
1
340
Building A Popular Open-Source Android Library - Best practices and lessons learned
loopj
4
470
Building A Popular Open-Source javascript Library
loopj
0
81
JavaScript Stack Traces: The good, the bad, and the ugly
loopj
1
170
Other Decks in Technology
See All in Technology
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
2
710
OpenTelemetry を使ったトレースエグザンプラーの活用 / otel-trace-exemplar
k6s4i53rx
2
630
o11y入門_外形監視を利用したWebアプリケーションへの最適なモニタリング_TechBrew
k5k
2
100
オーナーシップを持つ領域を明確にする
konifar
8
1.2k
普段有償でサポート業務をしているCSAが技術知見を無料で公開する理由
07jp27
1
630
A (short) History of AI
harishpillay
0
110
DevOpsDays History and my DevOps story
kawaguti
PRO
7
1.4k
Apple Vision Pro trial session
akkeylab
0
120
日本におけるデータエンジニアリングのこれまでとこれから
foursue
9
2k
Data and AI Governance: Existing Challenges and Emerging Trends
scotthsieh825
0
140
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
13
35k
「ふりかえりのふりかえり」をふりかえり、実のあるふりかえりにする
naitosatoshi
0
220
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
43
9.7k
Navigating Team Friction
lara
177
13k
How to name files
jennybc
64
92k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
226
16k
Why You Should Never Use an ORM
jnunemaker
PRO
50
8.6k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
8
8.3k
GraphQLの誤解/rethinking-graphql
sonatard
49
9.2k
Atom: Resistance is Futile
akmur
258
25k
Building Flexible Design Systems
yeseniaperezcruz
318
37k
Code Reviewing Like a Champion
maltzj
513
39k
Agile that works and the tools we love
rasmusluckow
323
20k
Automating Front-end Workflow
addyosmani
1354
200k
Transcript
RETHINKING PRODUCTION MONITORING YOUR SOFTWARE IS BROKEN — PAY ATTENTION
JAMES SMITH loopj loopj
None
CODE TEST DEPLOY YOLO ¯\_(ϑ)_/¯
CODE TEST DEPLOY YOLO CODE TEST DEPLOY CONFIDENCE ¯\_(ϑ)_/¯ :)
STABILITY PERFORMANCE AVAILABILITY
DELIVERING AN AWESOME EXPERIENCE TO CUSTOMERS
WHY MONITORING MATTERS
YOUR APP WILL LIVE OR DIE BASED ON ITS QUALITY
— CUSTOMERS HAVE A CHOICE
84% OF USERS ABANDON AFTER TWO CRASHES
49% OF ENGINEERING TIME FINDING & FIXING BUGS
SINS OF PRODUCTION MONITORING WHAT AM I DOING WRONG?
1. PRETENDING NOTHING IS WRONG
“But I’ve written tests!” “The QA Team will check that!”
“Works great for me!”
2. WAITING FOR CUSTOMERS TO COMPLAIN
“Nobody complained so everything must be OK”
3. LACK OF VISIBILITY
“We’ll just check the logs” “Did you remember to add
a log statement?”
4. LACK OF OWNERSHIP
“Not my problem!” “I’ve got a feature to ship” “My
code works fine”
HOW CAN WE DO BETTER?
ACCEPT AUTOMATE AGGREGATE NOTIFY PRIORITIZE DIAGNOSE TEND CORE PRINCIPLES OF
PRODUCTION MONITORING
1. ACCEPT ACCEPT THAT YOUR SOFTWARE WILL BREAK AFTER SHIPPING
2. AUTOMATE ADD HOOKS TO DETECT CRASHES/ERRORS/ISSUES IN PRODUCTION
3. AGGREGATE DON'T JUST HAVE A STREAM OF EVENTS -
GROUP LIKE ISSUES TOGETHER
4. NOTIFY ALERT YOUR DEV TEAM WHERE THEY ALREADY COMMUNICATE
5. PRIORITIZE YOU CAN'T FIX EVERY ERROR - SO FOCUS
ON THE MOST HARMFUL ONES
6. DIAGNOSE KNOWING ABOUT ISSUES ISN'T ENOUGH - THEY MUST
BE ACTIONABLE
7. TEND MAKE AN ORGANIZATIONAL CHANGE - SOMEONE NEEDS TO
CARE ABOUT ERRORS
TAKING ACTION
TOOLS
USES “FAILURE” HOOKS
ASSESS IMPACT
ASSESS SEVERITY
CAPTURES DIAGNOSTIC DATA
WORKFLOW
USE TEAM CHAT
EMBRACE COLLABORATION
TRACK PROGRESS OF FIXES
TEAM STRUCTURES
EMBRACE RAPID ITERATION
CREATE A “BUG TEAM”
OR CREATE A “BUG ROTATION”
OR KNOW “WHO LAST TOUCHED THIS CODE”?
TL;DR
AVOID THE SINS
EMBRACE CORE PRINCIPLES
TAKE ACTION
THANK YOU!
QUESTIONS?
IS HIRING! bugsnag.com/jobs @bugsnag