Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

By Baruch Sadogursky and Leonid Igolnik

It’s 2017 and on-call shifts don’t have to be a nightmare anymore. In this talk we’ll discuss how to structure the process in a way it won’t look like blood, guts and body parts flying around. Which tools and techniques can help us? From a knowledge base tips, via proper design of the escalation path and an overview of the tools, we’ll talk about everything that can release the pain. Also, should all the developers have access to the production?

Transcript

1. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   Silicon valley Chainsaw
   massacre
   Or how I spend my friday night on-call
   ….
   Leonid Igolnik & Baruch Sadogursky
   

   View Slide

2. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   Introductions
   

   View Slide

3. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   DISCLAIMER
   

   View Slide

4. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   shownotes
   Slides
   Video (soon!)
   links
   

   View Slide

5. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   Once upon a time ...
   

   View Slide

6. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   

   View Slide

7. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   

   View Slide

8. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   

   View Slide

9. @ligolnik @jbaruch #devoops jfrog.com/shownotes
   

   View Slide

10. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

11. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    It’s a sev 1
    

    View Slide

12. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

13. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

14. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

15. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

16. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

17. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

18. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

19. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

20. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

21. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

22. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

23. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

24. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

25. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

26. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Sounds
    Familiar ?
    

    View Slide

27. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

28. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

29. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Sre: it’s their freaking job!
    DBA
    Messaging
    Other specialties
    And if you are lucky you have a follow the sun NOC
    

    View Slide

30. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

31. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

32. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

33. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    #painisinstructional
    

    View Slide

34. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

35. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    on-call enablers
    

    View Slide

36. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Logs
    

    View Slide

37. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Logs
    Search and aggregation tools
    Data masking
    Alerting capabilities
    

    View Slide

38. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    SEVerity definition
    It’s a sev 1
    

    View Slide

39. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Well defined Severity definitions
    Who sets the severity:
    Support
    Customer
    Expected SLA
    Update frequency expectations
    

    View Slide

40. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Broken phone
    

    View Slide

41. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

42. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

43. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Effective reachability
    Virtual extension etc
    Escalation chat
    Virtual phone bridge
    Meeting point
    

    View Slide

44. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Escalation PATH
    

    View Slide

45. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Escalation PATH
    Who do you wake up and when?
    How do you reach them?
    All the way to CEO
    

    View Slide

46. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

47. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Manager ON-Call
    

    View Slide

48. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Manager ON-CALL
    External communications
    Coordination of activities
    Managing resources
    

    View Slide

49. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    PRoduction access
    

    View Slide

50. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    PRoduction access
    Ability to deploy hotfixes
    Documented steps for:
    Debug
    Log level changes
    

    View Slide

51. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    

    View Slide

52. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Effective shift handover
    Pick a standard day of the week/time
    Schedule a 15-30 min call / meeting
    

    View Slide

53. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Other barriers
    Training
    Certification
    Knowledge base / Runbook
    

    View Slide

54. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    OK I am convinced
    BuT how do you convince
    them ?!?!
    

    View Slide

55. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Getting started with on-call
    This will take time
    Start with senior folks
    Find initial partner in peer teams
    Start small
    

    View Slide

56. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    REACTIVE Improvement
    Monitor Detect
    Fix
    

    View Slide

57. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    purpose
    

    View Slide

58. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Thank you, q&a, links
    @jbaruch
    @ligolnik
    #devoops
    Shownotes:
    Slides
    Video (soon!)
    links
    

    View Slide