Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

Silicon Valley Chainsaw Massacre (how I spent my last Friday night on-call) as presented at DevOops 2017

By Baruch Sadogursky and Leonid Igolnik

It’s 2017 and on-call shifts don’t have to be a nightmare anymore. In this talk we’ll discuss how to structure the process in a way it won’t look like blood, guts and body parts flying around. Which tools and techniques can help us? From a knowledge base tips, via proper design of the escalation path and an overview of the tools, we’ll talk about everything that can release the pain. Also, should all the developers have access to the production?

Baruch Sadogursky

October 20, 2017
Tweet

More Decks by Baruch Sadogursky

Other Decks in Technology

Transcript

  1. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Silicon valley Chainsaw
    massacre
    Or how I spend my friday night on-call
    ….
    Leonid Igolnik & Baruch Sadogursky

    View Slide

  2. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Introductions

    View Slide

  3. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    DISCLAIMER

    View Slide

  4. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    shownotes
    Slides
    Video (soon!)
    links

    View Slide

  5. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Once upon a time ...

    View Slide

  6. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  7. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  8. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  9. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  10. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  11. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    It’s a sev 1

    View Slide

  12. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  13. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  14. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  15. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  16. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  17. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  18. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  19. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  20. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  21. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  22. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  23. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  24. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  25. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  26. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Sounds
    Familiar ?

    View Slide

  27. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  28. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  29. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Sre: it’s their freaking job!
    DBA
    Messaging
    Other specialties
    And if you are lucky you have a follow the sun NOC

    View Slide

  30. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  31. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  32. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  33. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    #painisinstructional

    View Slide

  34. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  35. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    on-call enablers

    View Slide

  36. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Logs

    View Slide

  37. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Logs
    Search and aggregation tools
    Data masking
    Alerting capabilities

    View Slide

  38. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    SEVerity definition
    It’s a sev 1

    View Slide

  39. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Well defined Severity definitions
    Who sets the severity:
    Support
    Customer
    Expected SLA
    Update frequency expectations

    View Slide

  40. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Broken phone

    View Slide

  41. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  42. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  43. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Effective reachability
    Virtual extension etc
    Escalation chat
    Virtual phone bridge
    Meeting point

    View Slide

  44. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Escalation PATH

    View Slide

  45. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Escalation PATH
    Who do you wake up and when?
    How do you reach them?
    All the way to CEO

    View Slide

  46. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  47. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Manager ON-Call

    View Slide

  48. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Manager ON-CALL
    External communications
    Coordination of activities
    Managing resources

    View Slide

  49. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    PRoduction access

    View Slide

  50. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    PRoduction access
    Ability to deploy hotfixes
    Documented steps for:
    Debug
    Log level changes

    View Slide

  51. @ligolnik @jbaruch #devoops jfrog.com/shownotes

    View Slide

  52. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Effective shift handover
    Pick a standard day of the week/time
    Schedule a 15-30 min call / meeting

    View Slide

  53. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Other barriers
    Training
    Certification
    Knowledge base / Runbook

    View Slide

  54. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    OK I am convinced
    BuT how do you convince
    them ?!?!

    View Slide

  55. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Getting started with on-call
    This will take time
    Start with senior folks
    Find initial partner in peer teams
    Start small

    View Slide

  56. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    REACTIVE Improvement
    Monitor Detect
    Fix

    View Slide

  57. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    purpose

    View Slide

  58. @ligolnik @jbaruch #devoops jfrog.com/shownotes
    Thank you, q&a, links
    @jbaruch
    @ligolnik
    #devoops
    Shownotes:
    Slides
    Video (soon!)
    links

    View Slide