Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

By Baruch Sadogursky and Leonid Igolnik

It’s 2017 and on-call shifts don’t have to be a nightmare anymore. In this talk we’ll discuss how to structure the process in a way it won’t look like blood, guts and body parts flying around. Which tools and techniques can help us? From a knowledge base tips, via proper design of the escalation path and an overview of the tools, we’ll talk about everything that can release the pain. Also, should all the developers have access to the production?

Transcript

1. @ligolnik @jbaruch #devoops jfrog.com/shownotes Silicon valley Chainsaw massacre Or how I spend my friday night on-call …. Leonid Igolnik & Baruch Sadogursky
2. @ligolnik @jbaruch #devoops jfrog.com/shownotes Introductions
3. @ligolnik @jbaruch #devoops jfrog.com/shownotes DISCLAIMER
4. @ligolnik @jbaruch #devoops jfrog.com/shownotes shownotes Slides Video (soon!) links
5. @ligolnik @jbaruch #devoops jfrog.com/shownotes Once upon a time ...
6. @ligolnik @jbaruch #devoops jfrog.com/shownotes
7. @ligolnik @jbaruch #devoops jfrog.com/shownotes
8. @ligolnik @jbaruch #devoops jfrog.com/shownotes
9. @ligolnik @jbaruch #devoops jfrog.com/shownotes
10. @ligolnik @jbaruch #devoops jfrog.com/shownotes
11. @ligolnik @jbaruch #devoops jfrog.com/shownotes It’s a sev 1
12. @ligolnik @jbaruch #devoops jfrog.com/shownotes
13. @ligolnik @jbaruch #devoops jfrog.com/shownotes
14. @ligolnik @jbaruch #devoops jfrog.com/shownotes
15. @ligolnik @jbaruch #devoops jfrog.com/shownotes
16. @ligolnik @jbaruch #devoops jfrog.com/shownotes
17. @ligolnik @jbaruch #devoops jfrog.com/shownotes
18. @ligolnik @jbaruch #devoops jfrog.com/shownotes
19. @ligolnik @jbaruch #devoops jfrog.com/shownotes
20. @ligolnik @jbaruch #devoops jfrog.com/shownotes
21. @ligolnik @jbaruch #devoops jfrog.com/shownotes
22. @ligolnik @jbaruch #devoops jfrog.com/shownotes
23. @ligolnik @jbaruch #devoops jfrog.com/shownotes
24. @ligolnik @jbaruch #devoops jfrog.com/shownotes
25. @ligolnik @jbaruch #devoops jfrog.com/shownotes
26. @ligolnik @jbaruch #devoops jfrog.com/shownotes Sounds Familiar ?
27. @ligolnik @jbaruch #devoops jfrog.com/shownotes
28. @ligolnik @jbaruch #devoops jfrog.com/shownotes
29. @ligolnik @jbaruch #devoops jfrog.com/shownotes Sre: it’s their freaking job! DBA Messaging Other specialties And if you are lucky you have a follow the sun NOC
30. @ligolnik @jbaruch #devoops jfrog.com/shownotes
31. @ligolnik @jbaruch #devoops jfrog.com/shownotes
32. @ligolnik @jbaruch #devoops jfrog.com/shownotes
33. @ligolnik @jbaruch #devoops jfrog.com/shownotes #painisinstructional
34. @ligolnik @jbaruch #devoops jfrog.com/shownotes
35. @ligolnik @jbaruch #devoops jfrog.com/shownotes on-call enablers
36. @ligolnik @jbaruch #devoops jfrog.com/shownotes Logs
37. @ligolnik @jbaruch #devoops jfrog.com/shownotes Logs Search and aggregation tools Data masking Alerting capabilities
38. @ligolnik @jbaruch #devoops jfrog.com/shownotes SEVerity definition It’s a sev 1
39. @ligolnik @jbaruch #devoops jfrog.com/shownotes Well defined Severity definitions Who sets the severity: Support Customer Expected SLA Update frequency expectations
40. @ligolnik @jbaruch #devoops jfrog.com/shownotes Broken phone
41. @ligolnik @jbaruch #devoops jfrog.com/shownotes
42. @ligolnik @jbaruch #devoops jfrog.com/shownotes
43. @ligolnik @jbaruch #devoops jfrog.com/shownotes Effective reachability Virtual extension etc Escalation chat Virtual phone bridge Meeting point
44. @ligolnik @jbaruch #devoops jfrog.com/shownotes Escalation PATH
45. @ligolnik @jbaruch #devoops jfrog.com/shownotes Escalation PATH Who do you wake up and when? How do you reach them? All the way to CEO
46. @ligolnik @jbaruch #devoops jfrog.com/shownotes
47. @ligolnik @jbaruch #devoops jfrog.com/shownotes Manager ON-Call
48. @ligolnik @jbaruch #devoops jfrog.com/shownotes Manager ON-CALL External communications Coordination of activities Managing resources
49. @ligolnik @jbaruch #devoops jfrog.com/shownotes PRoduction access
50. @ligolnik @jbaruch #devoops jfrog.com/shownotes PRoduction access Ability to deploy hotfixes Documented steps for: Debug Log level changes
51. @ligolnik @jbaruch #devoops jfrog.com/shownotes
52. @ligolnik @jbaruch #devoops jfrog.com/shownotes Effective shift handover Pick a standard day of the week/time Schedule a 15-30 min call / meeting
53. @ligolnik @jbaruch #devoops jfrog.com/shownotes Other barriers Training Certification Knowledge base / Runbook
54. @ligolnik @jbaruch #devoops jfrog.com/shownotes OK I am convinced BuT how do you convince them ?!?!
55. @ligolnik @jbaruch #devoops jfrog.com/shownotes Getting started with on-call This will take time Start with senior folks Find initial partner in peer teams Start small
56. @ligolnik @jbaruch #devoops jfrog.com/shownotes REACTIVE Improvement Monitor Detect Fix
57. @ligolnik @jbaruch #devoops jfrog.com/shownotes purpose
58. @ligolnik @jbaruch #devoops jfrog.com/shownotes Thank you, q&a, links @jbaruch @ligolnik #devoops Shownotes: Slides Video (soon!) links