What should wake you up at night?

On-Call can be painful but it doesn't have to be.
Learn how modern practices can make on-call better for everyone.

bob has been doing on-call for over 15 years and has opinions about what should
wake you up and who should be woken up. He intends to share them with you.

Transcript

1. bob walker
   Head of Web Operations
   Government Digital Service
   @rjw1
   

   View Slide

2. GDS
   My name should be lowercased
   e.g bob walker
   http://randomness.org.uk/branding/
   

   View Slide

3. GDS
   My pronouns are he/him or they/them.
   

   View Slide

4. A long time ago in
   company far, far
   away...
   

   View Slide

5. GDS
   

   View Slide

6. GDS
   Copyright : Alex Howarth
   

   View Slide

7. What should wake
   you up at night?
   

   View Slide

8. nothing
   

   View Slide

9. Exit pursued by
   bears
   

   View Slide

10. No wait!
    I suppose I should
    explain that.
    

    View Slide

11. Ways to not get
    woken up
    

    View Slide

12. Shifts
    

    View Slide

13. GDS
    4 shifts of 8 hours
    So you can have handover
    

    View Slide

14. GDS
    Just say no!
    “Long-term night shift work is associated
    with an increased risk of certain cancers, as
    well as metabolic problems, heart disease,
    ulcers, gastrointestinal problems and
    obesity. ... People who work night shifts or
    rotating shifts also often don't sleep
    enough, and long-term sleep deprivation is
    known to be bad for health.”
    - https://sleepfoundation.org/shift-work/content/living-coping-shift-work-disorder
    

    View Slide

15. Follow the Sun
    

    View Slide

16. GDS
    

    View Slide

17. GDS
    If you are big enough this is a
    good option.
    

    View Slide

18. Self healing systems
    

    View Slide

19. GDS
    https://www.flickr.com/photos/johnclare/7124089493/
    

    View Slide

20. GDS
    Automatic restarts:
    ● daemontools
    ● monit
    ● God
    ● SMF
    ● systemd
    

    View Slide

21. GDS
    Clusters:
    ● mongodb
    ● elasticsearch
    

    View Slide

22. GDS
    Auto scaling groups
    

    View Slide

23. GDS
    Schedulers:
    ● Mesos
    ● Kubernetes
    ● ECS
    

    View Slide

24. What if you can’t do
    all these?
    

    View Slide

25. Be selective!
    

    View Slide

26. GDS
    What 5 things would you monitor
    and alert on first if you had no
    monitoring?
    

    View Slide

27. GDS
    ● disk
    

    View Slide

28. GDS
    ● disk
    ● CPU
    

    View Slide

29. GDS
    ● disk
    ● CPU
    ● Memory
    

    View Slide

30. GDS
    ● disk
    ● CPU
    ● Memory
    ● A couple of other things which
    don’t matter to the user
    

    View Slide

31. GDS
    https://www.flickr.com/photos/benterrett/17936132731
    

    View Slide

32. Monitor your
    user journey
    

    View Slide

33. GDS
    ● Availability
    ● Response times
    ● Error rates
    

    View Slide

34. GDS
    Legal reasons
    

    View Slide

35. GDS
    Life or death situations
    

    View Slide

36. Who?
    

    View Slide

37. GDS
    Which team should get called?
    ● Ops team
    ● Dev team
    ● DevOps team
    

    View Slide

38. GDS
    Trick question!
    

    View Slide

39. GDS
    DevOps is a culture not a team or
    job title
    

    View Slide

40. Everyone!
    

    View Slide

41. “Always two there are”
    - Yoda
    

    View Slide

42. GDS
    We’re hiring
    - Web Operations Engineers
    - Technical Architects
    - Developers
    https://gds.blog.gov.uk/jobs/
    

    View Slide

43. GDS
    No Questions!
    Come talk to me later.
    I have stickers!
    

    View Slide

44. Thanks!
    bob walker
    @rjw1
    

    View Slide