$30 off During Our Annual Pro Sale. View Details »

What should wake you up at night?

bob
October 18, 2017

What should wake you up at night?

On-Call can be painful but it doesn't have to be.
Learn how modern practices can make on-call better for everyone.

bob has been doing on-call for over 15 years and has opinions about what should
wake you up and who should be woken up. He intends to share them with you.

bob

October 18, 2017
Tweet

More Decks by bob

Other Decks in Technology

Transcript

  1. bob walker
    Head of Web Operations
    Government Digital Service
    @rjw1

    View Slide

  2. GDS
    My name should be lowercased
    e.g bob walker
    http://randomness.org.uk/branding/

    View Slide

  3. GDS
    My pronouns are he/him or they/them.

    View Slide

  4. A long time ago in
    company far, far
    away...

    View Slide

  5. GDS

    View Slide

  6. GDS
    Copyright : Alex Howarth

    View Slide

  7. What should wake
    you up at night?

    View Slide

  8. nothing

    View Slide

  9. Exit pursued by
    bears

    View Slide

  10. No wait!
    I suppose I should
    explain that.

    View Slide

  11. Ways to not get
    woken up

    View Slide

  12. Shifts

    View Slide

  13. GDS
    4 shifts of 8 hours
    So you can have handover

    View Slide

  14. GDS
    Just say no!
    “Long-term night shift work is associated
    with an increased risk of certain cancers, as
    well as metabolic problems, heart disease,
    ulcers, gastrointestinal problems and
    obesity. ... People who work night shifts or
    rotating shifts also often don't sleep
    enough, and long-term sleep deprivation is
    known to be bad for health.”
    - https://sleepfoundation.org/shift-work/content/living-coping-shift-work-disorder

    View Slide

  15. Follow the Sun

    View Slide

  16. GDS

    View Slide

  17. GDS
    If you are big enough this is a
    good option.

    View Slide

  18. Self healing systems

    View Slide

  19. GDS
    https://www.flickr.com/photos/johnclare/7124089493/

    View Slide

  20. GDS
    Automatic restarts:
    ● daemontools
    ● monit
    ● God
    ● SMF
    ● systemd

    View Slide

  21. GDS
    Clusters:
    ● mongodb
    ● elasticsearch

    View Slide

  22. GDS
    Auto scaling groups

    View Slide

  23. GDS
    Schedulers:
    ● Mesos
    ● Kubernetes
    ● ECS

    View Slide

  24. What if you can’t do
    all these?

    View Slide

  25. Be selective!

    View Slide

  26. GDS
    What 5 things would you monitor
    and alert on first if you had no
    monitoring?

    View Slide

  27. GDS
    ● disk

    View Slide

  28. GDS
    ● disk
    ● CPU

    View Slide

  29. GDS
    ● disk
    ● CPU
    ● Memory

    View Slide

  30. GDS
    ● disk
    ● CPU
    ● Memory
    ● A couple of other things which
    don’t matter to the user

    View Slide

  31. GDS
    https://www.flickr.com/photos/benterrett/17936132731

    View Slide

  32. Monitor your
    user journey

    View Slide

  33. GDS
    ● Availability
    ● Response times
    ● Error rates

    View Slide

  34. GDS
    Legal reasons

    View Slide

  35. GDS
    Life or death situations

    View Slide

  36. Who?

    View Slide

  37. GDS
    Which team should get called?
    ● Ops team
    ● Dev team
    ● DevOps team

    View Slide

  38. GDS
    Trick question!

    View Slide

  39. GDS
    DevOps is a culture not a team or
    job title

    View Slide

  40. Everyone!

    View Slide

  41. “Always two there are”
    - Yoda

    View Slide

  42. GDS
    We’re hiring
    - Web Operations Engineers
    - Technical Architects
    - Developers
    https://gds.blog.gov.uk/jobs/

    View Slide

  43. GDS
    No Questions!
    Come talk to me later.
    I have stickers!

    View Slide

  44. Thanks!
    bob walker
    @rjw1

    View Slide