Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

By Baruch Sadogursky and Leonid Igolnik

It’s 2017 and on-call shifts don’t have to be a nightmare anymore. In this talk we’ll discuss how to structure the process in a way it won’t look like blood, guts and body parts flying around. Which tools and techniques can help us? From a knowledge base tips, via proper design of the escalation path and an overview of the tools, we’ll talk about everything that can release the pain. Also, should all the developers have access to the production?

Baruch Sadogursky

October 20, 2017

More Decks by Baruch Sadogursky

Other Decks in Technology


  1. @ligolnik @jbaruch #devoops jfrog.com/shownotes Silicon valley Chainsaw massacre Or how

    I spend my friday night on-call …. Leonid Igolnik & Baruch Sadogursky
  2. @ligolnik @jbaruch #devoops jfrog.com/shownotes Introductions

  3. @ligolnik @jbaruch #devoops jfrog.com/shownotes DISCLAIMER

  4. @ligolnik @jbaruch #devoops jfrog.com/shownotes shownotes Slides Video (soon!) links

  5. @ligolnik @jbaruch #devoops jfrog.com/shownotes Once upon a time ...

  6. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  7. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  8. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  9. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  10. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  11. @ligolnik @jbaruch #devoops jfrog.com/shownotes It’s a sev 1

  12. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  13. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  14. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  15. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  16. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  17. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  18. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  19. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  20. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  21. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  22. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  23. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  24. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  25. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  26. @ligolnik @jbaruch #devoops jfrog.com/shownotes Sounds Familiar ?

  27. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  28. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  29. @ligolnik @jbaruch #devoops jfrog.com/shownotes Sre: it’s their freaking job! DBA

    Messaging Other specialties And if you are lucky you have a follow the sun NOC
  30. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  31. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  32. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  33. @ligolnik @jbaruch #devoops jfrog.com/shownotes #painisinstructional

  34. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  35. @ligolnik @jbaruch #devoops jfrog.com/shownotes on-call enablers

  36. @ligolnik @jbaruch #devoops jfrog.com/shownotes Logs

  37. @ligolnik @jbaruch #devoops jfrog.com/shownotes Logs Search and aggregation tools Data

    masking Alerting capabilities
  38. @ligolnik @jbaruch #devoops jfrog.com/shownotes SEVerity definition It’s a sev 1

  39. @ligolnik @jbaruch #devoops jfrog.com/shownotes Well defined Severity definitions Who sets

    the severity: Support Customer Expected SLA Update frequency expectations
  40. @ligolnik @jbaruch #devoops jfrog.com/shownotes Broken phone

  41. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  42. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  43. @ligolnik @jbaruch #devoops jfrog.com/shownotes Effective reachability Virtual extension etc Escalation

    chat Virtual phone bridge Meeting point
  44. @ligolnik @jbaruch #devoops jfrog.com/shownotes Escalation PATH

  45. @ligolnik @jbaruch #devoops jfrog.com/shownotes Escalation PATH Who do you wake

    up and when? How do you reach them? All the way to CEO
  46. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  47. @ligolnik @jbaruch #devoops jfrog.com/shownotes Manager ON-Call

  48. @ligolnik @jbaruch #devoops jfrog.com/shownotes Manager ON-CALL External communications Coordination of

    activities Managing resources
  49. @ligolnik @jbaruch #devoops jfrog.com/shownotes PRoduction access

  50. @ligolnik @jbaruch #devoops jfrog.com/shownotes PRoduction access Ability to deploy hotfixes

    Documented steps for: Debug Log level changes
  51. @ligolnik @jbaruch #devoops jfrog.com/shownotes

  52. @ligolnik @jbaruch #devoops jfrog.com/shownotes Effective shift handover Pick a standard

    day of the week/time Schedule a 15-30 min call / meeting
  53. @ligolnik @jbaruch #devoops jfrog.com/shownotes Other barriers Training Certification Knowledge base

    / Runbook
  54. @ligolnik @jbaruch #devoops jfrog.com/shownotes OK I am convinced BuT how

    do you convince them ?!?!
  55. @ligolnik @jbaruch #devoops jfrog.com/shownotes Getting started with on-call This will

    take time Start with senior folks Find initial partner in peer teams Start small
  56. @ligolnik @jbaruch #devoops jfrog.com/shownotes REACTIVE Improvement Monitor Detect Fix

  57. @ligolnik @jbaruch #devoops jfrog.com/shownotes purpose

  58. @ligolnik @jbaruch #devoops jfrog.com/shownotes Thank you, q&a, links @jbaruch @ligolnik

    #devoops Shownotes: Slides Video (soon!) links