GitHub Universe 2015 Talk - Your software is broken — pay attention: Rethinking production monitoring

My talk from GitHub Universe 2015's "Deploy" track

Transcript

1. RETHINKING PRODUCTION MONITORING YOUR SOFTWARE IS BROKEN — PAY ATTENTION
2. JAMES SMITH loopj loopj
4. CODE TEST DEPLOY YOLO ¯\_(ϑ)_/¯
5. CODE TEST DEPLOY YOLO CODE TEST DEPLOY CONFIDENCE ¯\_(ϑ)_/¯ :)
6. STABILITY PERFORMANCE AVAILABILITY
7. DELIVERING AN AWESOME EXPERIENCE TO CUSTOMERS
8. WHY MONITORING MATTERS
9. YOUR APP WILL LIVE OR DIE BASED ON ITS QUALITY — CUSTOMERS HAVE A CHOICE
10. 84% OF USERS ABANDON AFTER TWO CRASHES
11. 49% OF ENGINEERING TIME FINDING & FIXING BUGS
12. SINS OF PRODUCTION MONITORING WHAT AM I DOING WRONG?
13. 1. PRETENDING NOTHING IS WRONG
14. “But I’ve written tests!” “The QA Team will check that!” “Works great for me!”
15. 2. WAITING FOR CUSTOMERS TO COMPLAIN
16. “Nobody complained so everything must be OK”
17. 3. LACK OF VISIBILITY
18. “We’ll just check the logs” “Did you remember to add a log statement?”
19. 4. LACK OF OWNERSHIP
20. “Not my problem!” “I’ve got a feature to ship” “My code works fine”
21. HOW CAN WE DO BETTER?
22. ACCEPT AUTOMATE AGGREGATE NOTIFY PRIORITIZE DIAGNOSE TEND CORE PRINCIPLES OF PRODUCTION MONITORING
23. 1. ACCEPT ACCEPT THAT YOUR SOFTWARE WILL BREAK AFTER SHIPPING
24. 2. AUTOMATE ADD HOOKS TO DETECT CRASHES/ERRORS/ISSUES IN PRODUCTION
25. 3. AGGREGATE DON'T JUST HAVE A STREAM OF EVENTS - GROUP LIKE ISSUES TOGETHER
26. 4. NOTIFY ALERT YOUR DEV TEAM WHERE THEY ALREADY COMMUNICATE
27. 5. PRIORITIZE YOU CAN'T FIX EVERY ERROR - SO FOCUS ON THE MOST HARMFUL ONES
28. 6. DIAGNOSE KNOWING ABOUT ISSUES ISN'T ENOUGH - THEY MUST BE ACTIONABLE
29. 7. TEND MAKE AN ORGANIZATIONAL CHANGE - SOMEONE NEEDS TO CARE ABOUT ERRORS
30. TAKING ACTION
31. TOOLS
32. USES “FAILURE” HOOKS
33. ASSESS IMPACT
34. ASSESS SEVERITY
35. CAPTURES DIAGNOSTIC DATA
36. WORKFLOW
37. USE TEAM CHAT
38. EMBRACE COLLABORATION
39. TRACK PROGRESS OF FIXES
40. TEAM STRUCTURES
41. EMBRACE RAPID ITERATION
42. CREATE A “BUG TEAM”
43. OR CREATE A “BUG ROTATION”
44. OR KNOW “WHO LAST TOUCHED THIS CODE”?
45. TL;DR
46. AVOID THE SINS
47. EMBRACE CORE PRINCIPLES
48. TAKE ACTION
49. THANK YOU!
50. QUESTIONS?
51. IS HIRING! bugsnag.com/jobs @bugsnag