Production Debugging

What to do when shits on fire

Transcript

1. Production Debugging
2. Production Debugging What to do when shit’s on fire
3. Me @josh_robb Code Nanny @ Pushpay.com “We’re Hiring” (tm)
4. Overview Context Tools Demos Wrap-up
5. Context OODA - John Boyd
6. OODA - Observe - Orient - Decide - Act
7. Stressful situations Getting shot at in an aeroplane Fuck that Getting shot at full stop! Flying a helicopter with no engine
8. How to train for stress?
9. How to train for stress?
11. Emotions Denial Fear Anxiety Fatigue (not an emotion - related)
12. Bad Judgement All of these things lead to poor quality decision making.
13. What would NPH Do? First do no harm!
14. Don’t make things worse Evaluate your options - Can you roll back? - Can you get a new job? - Can you roll forward?
15. MTTR Optimize for Mean Time To Recovery!
16. Don’t roll back in fear Roll forward to victory - Etsy
17. Failure If you work somewhere failure is unacceptable (Apart from Avionics or medical gear) Get a new job Seriously Failure in Tech is unavoidable Learn from it
18. Postmortems Postmortems are important Blameless ones are best http://codeascraft.com/2012/05/22/blameless- postmortems/
19. Demo Scenarios - Two of them - In one app - Introducing
20. BrokenApp
21. BrokenApp Two Scenarios - Hang - Hi CPU
22. Tools - Process tools - Tracing Tools - Dump analysis
23. Tools - Process tools - Orient - Tracing Tools - Observe - Dump analysis - Decide - ACT?
24. Process Tools Windows Server 2012/2008 - Resource Monitor! Also - Task Manager - Process Explorer - Procmon
25. Process Tools Demo
26. Tracing Tools Perfview - Process sampling tool - Great for what's happening over time? - Live profiling
27. Perfview XCopy deployable Offline analysis #FTW NOTE: Enable ASP.NET tracing (DSIM) Demo!
28. Tracing Tools Message analyzer - Network traffic - Packet sniffing
29. Dump Analysis - Windbg (for masochists these days)
30. windbg
31. windbg - yeah - no - just say no - debugdiag FTW!