Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Martyrs On Film: learning to hate the #oncallselfie

Martyrs On Film: learning to hate the #oncallselfie

Alice Goldfuss

May 22, 2017
Tweet

More Decks by Alice Goldfuss

Other Decks in Technology

Transcript

  1. Martyrs on Film

    View full-size slide

  2. Hi! I’m Alice
    I like systems and
    Twitter and tea.

    View full-size slide

  3. Hi! I’m Alice
    And not getting
    paged.

    View full-size slide

  4. Benefits of On-Call
    ● Hones troubleshooting skills
    ● Forces you to identify the weak points in your systems
    ● Teaches you what is and isn’t production-ready

    View full-size slide

  5. Team bonding

    View full-size slide

  6. PagerDuty
    New Year’s Eve

    View full-size slide

  7. PagerDuty
    New Year’s Eve

    View full-size slide

  8. PagerDuty
    New Year’s Eve S3 Outage

    View full-size slide

  9. A totally normal on-call routine
    ● Don’t leave house except to commute to work
    ● Clear all non-work appointments
    ● Cook all meals beforehand
    ● Have soup on hand
    ● Don’t sleep

    View full-size slide

  10. Prepare for battle

    View full-size slide

  11. Naomi Orwin
    Writer

    View full-size slide

  12. “Action scenes stop the plot.”
    - Naomi Orwin

    View full-size slide

  13. Pages stop the plot of your career

    View full-size slide

  14. Miserable on-call professionals
    ● Have terrible work/life balance
    ● Are supporting poorly-designed systems
    ● Feel powerless to solve problems
    ● Generally hate the role

    View full-size slide

  15. Red flag:
    too few owning too much

    View full-size slide

  16. Centralia infrastructure

    View full-size slide

  17. Red flag:
    bandaids

    View full-size slide

  18. ● Bump thresholds
    ● Snooze pages
    ● Delays

    View full-size slide

  19. Red flag:
    no visibility

    View full-size slide

  20. Systems visibility

    View full-size slide

  21. Team visibility

    View full-size slide

  22. Too many pages

    View full-size slide

  23. Average # of weekly pages during WORST on-call

    View full-size slide

  24. Average # of weekly pages during WORST on-call

    View full-size slide

  25. Average # of weekly pages during WORST on-call

    View full-size slide

  26. Average # of weekly pages during WORST on-call

    View full-size slide

  27. Average # of weekly pages during BEST on-call

    View full-size slide

  28. Average # of weekly pages during BEST on-call

    View full-size slide

  29. Average # of weekly pages during BEST on-call

    View full-size slide

  30. How do we get there?

    View full-size slide

  31. Notification cleanup

    View full-size slide

  32. Actionable alerts

    View full-size slide

  33. Actionable Alerts
    ● Something breaks
    ● Customers notice
    ● I am the best person to fix it
    ● I need to fix it immediately

    View full-size slide

  34. Cluster alerts

    View full-size slide

  35. Devs on-call

    View full-size slide

  36. “If a developer is good, being ‘on
    call’ just means having to fix other
    people’s problems and
    inconsequential stuff on Sat.”
    - dev on Twitter

    View full-size slide

  37. “Fixed this for him! ‘Put your
    developers on call. You’ll be
    surprised by how quickly they go
    work for someone that isn’t an
    #$%.’”
    - dev on Twitter

    View full-size slide

  38. “If your org has change control
    working properly, if code breaks,
    the jr sysadmin should simply roll
    back the update as documented.”
    - dev on Twitter

    View full-size slide

  39. The right tool

    View full-size slide

  40. Work together

    View full-size slide

  41. Your people will burn out before
    your company does

    View full-size slide

  42. Where does that leave #oncallselfie?

    View full-size slide

  43. “Why are you getting paged so much?”

    View full-size slide

  44. Thanks!
    @alicegoldfuss
    Special Thanks:
    PagerDuty
    VictorOps
    oncallselfies.com
    All of you

    View full-size slide