Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Production Debugging
Search
josh_robb
April 09, 2014
Technology
0
45
Production Debugging
What to do when shits on fire
josh_robb
April 09, 2014
Tweet
Share
More Decks by josh_robb
See All by josh_robb
Wellington Codecamp 2016
josh_robb
0
130
Codemania - Coupling, Cohesion, Connascence
josh_robb
1
580
Blameless Postmortems - Security by Inclusion
josh_robb
0
280
Effective Unit Testing
josh_robb
0
64
Coupling, Cohesion, Connascence
josh_robb
1
2.9k
Brown Field ASP.NET MVC with Webforms
josh_robb
1
140
Other Decks in Technology
See All in Technology
【あのMCPって、どんな処理してるの?】 AWS CDKでの開発で便利なAWS MCP Servers特集
yoshimi0227
2
150
開発生産性を測る前にやるべきこと - 組織改善の実践 / Before Measuring Dev Productivity
kaonavi
14
6.2k
american airlines®️ USA Contact Numbers: Complete 2025 Support Guide
supportflight
1
110
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
180
CDKTFについてざっくり理解する!!~CloudFormationからCDKTFへ変換するツールも作ってみた~
masakiokuda
1
180
Reach American Airlines®️ Instantly: 19 Calling Methods for Fast Support in the USA
flyamerican
1
180
面倒な作業はAIにおまかせ。Flutter開発をスマートに効率化
ruideengineer
0
350
Coinbase™®️ USA Contact Numbers: Complete 2025 Support Guide
officialcoinbasehelpcenter
0
450
高速なプロダクト開発を実現、創業期から掲げるエンタープライズアーキテクチャ
kawauso
3
9.7k
What’s new in Android development tools
yanzm
0
420
CDK Vibe Coding Fes
tomoki10
0
160
SRE不在の開発チームが障害対応と 向き合った100日間 / 100 days dealing with issues without SREs
shin1988
1
320
Featured
See All Featured
The Invisible Side of Design
smashingmag
301
51k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Optimizing for Happiness
mojombo
379
70k
KATA
mclloyd
30
14k
Building a Modern Day E-commerce SEO Strategy
aleyda
42
7.4k
Adopting Sorbet at Scale
ufuk
77
9.5k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
Transcript
Production Debugging
Production Debugging What to do when shit’s on fire
Me @josh_robb Code Nanny @ Pushpay.com “We’re Hiring” (tm)
Overview Context Tools Demos Wrap-up
Context OODA - John Boyd
OODA - Observe - Orient - Decide - Act
Stressful situations Getting shot at in an aeroplane Fuck that
Getting shot at full stop! Flying a helicopter with no engine
How to train for stress?
How to train for stress?
None
Emotions Denial Fear Anxiety Fatigue (not an emotion - related)
Bad Judgement All of these things lead to poor quality
decision making.
What would NPH Do? First do no harm!
Don’t make things worse Evaluate your options - Can you
roll back? - Can you get a new job? - Can you roll forward?
MTTR Optimize for Mean Time To Recovery!
Don’t roll back in fear Roll forward to victory -
Etsy
Failure If you work somewhere failure is unacceptable (Apart from
Avionics or medical gear) Get a new job Seriously Failure in Tech is unavoidable Learn from it
Postmortems Postmortems are important Blameless ones are best http://codeascraft.com/2012/05/22/blameless- postmortems/
Demo Scenarios - Two of them - In one app
- Introducing
BrokenApp
BrokenApp Two Scenarios - Hang - Hi CPU
Tools - Process tools - Tracing Tools - Dump analysis
Tools - Process tools - Orient - Tracing Tools -
Observe - Dump analysis - Decide - ACT?
Process Tools Windows Server 2012/2008 - Resource Monitor! Also -
Task Manager - Process Explorer - Procmon
Process Tools Demo
Tracing Tools Perfview - Process sampling tool - Great for
what's happening over time? - Live profiling
Perfview XCopy deployable Offline analysis #FTW NOTE: Enable ASP.NET tracing
(DSIM) Demo!
Tracing Tools Message analyzer - Network traffic - Packet sniffing
Dump Analysis - Windbg (for masochists these days)
windbg
windbg - yeah - no - just say no -
debugdiag FTW!