Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Production Debugging
Search
josh_robb
April 09, 2014
Technology
0
40
Production Debugging
What to do when shits on fire
josh_robb
April 09, 2014
Tweet
Share
More Decks by josh_robb
See All by josh_robb
Wellington Codecamp 2016
josh_robb
0
130
Codemania - Coupling, Cohesion, Connascence
josh_robb
1
550
Blameless Postmortems - Security by Inclusion
josh_robb
0
250
Effective Unit Testing
josh_robb
0
58
Coupling, Cohesion, Connascence
josh_robb
1
2.7k
Brown Field ASP.NET MVC with Webforms
josh_robb
1
130
Other Decks in Technology
See All in Technology
AWS学習者向けにAzureの解説スライドを作成した話
handy
3
190
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
4
880
DMM.com アルファ室採用案内資料
hsugita
1
220
コードや知識を組み込む / Incorporate Code and knowledge
ks91
PRO
0
140
データベース02: データベースの概念
trycycle
0
180
Azureの基本的な権限管理の勉強会
yhana
1
2.1k
Babylon.js JAPAN活動紹介 (2024/4)
limes2018
1
110
MapLibreとAmazon Location Service
dayjournal
1
180
20分で完全に理解するGrafanaダッシュボード
hamadakoji
5
890
Kernel MemoryでAzure OpenAI Serviceとお手軽データソース連携
mitsuzono
1
280
web-application-security
matsuihidetoshi
1
190
EM完全に理解した と思ったけど、 やっぱり何も分からなかった話 / EM Night Fukuoka #1
hirutas
0
280
Featured
See All Featured
Unsuck your backbone
ammeep
663
57k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
126
32k
What the flash - Photography Introduction
edds
64
11k
How to train your dragon (web standard)
notwaldorf
74
5.2k
Web Components: a chance to create the future
zenorocha
306
41k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Documentation Writing (for coders)
carmenintech
61
4k
Facilitating Awesome Meetings
lara
43
5.6k
Principles of Awesome APIs and How to Build Them.
keavy
121
16k
Web development in the modern age
philhawksworth
203
10k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
StorybookのUI Testing Handbookを読んだ
zakiyama
13
4.6k
Transcript
Production Debugging
Production Debugging What to do when shit’s on fire
Me @josh_robb Code Nanny @ Pushpay.com “We’re Hiring” (tm)
Overview Context Tools Demos Wrap-up
Context OODA - John Boyd
OODA - Observe - Orient - Decide - Act
Stressful situations Getting shot at in an aeroplane Fuck that
Getting shot at full stop! Flying a helicopter with no engine
How to train for stress?
How to train for stress?
None
Emotions Denial Fear Anxiety Fatigue (not an emotion - related)
Bad Judgement All of these things lead to poor quality
decision making.
What would NPH Do? First do no harm!
Don’t make things worse Evaluate your options - Can you
roll back? - Can you get a new job? - Can you roll forward?
MTTR Optimize for Mean Time To Recovery!
Don’t roll back in fear Roll forward to victory -
Etsy
Failure If you work somewhere failure is unacceptable (Apart from
Avionics or medical gear) Get a new job Seriously Failure in Tech is unavoidable Learn from it
Postmortems Postmortems are important Blameless ones are best http://codeascraft.com/2012/05/22/blameless- postmortems/
Demo Scenarios - Two of them - In one app
- Introducing
BrokenApp
BrokenApp Two Scenarios - Hang - Hi CPU
Tools - Process tools - Tracing Tools - Dump analysis
Tools - Process tools - Orient - Tracing Tools -
Observe - Dump analysis - Decide - ACT?
Process Tools Windows Server 2012/2008 - Resource Monitor! Also -
Task Manager - Process Explorer - Procmon
Process Tools Demo
Tracing Tools Perfview - Process sampling tool - Great for
what's happening over time? - Live profiling
Perfview XCopy deployable Offline analysis #FTW NOTE: Enable ASP.NET tracing
(DSIM) Demo!
Tracing Tools Message analyzer - Network traffic - Packet sniffing
Dump Analysis - Windbg (for masochists these days)
windbg
windbg - yeah - no - just say no -
debugdiag FTW!