What should wake you up at night?

Slide 1

Slide 1 text

bob walker Head of Web Operations Government Digital Service @rjw1

Slide 2

Slide 2 text

GDS My name should be lowercased e.g bob walker http://randomness.org.uk/branding/

Slide 3

Slide 3 text

GDS My pronouns are he/him or they/them.

Slide 4

Slide 4 text

A long time ago in company far, far away...

Slide 5

Slide 5 text

GDS

Slide 6

Slide 6 text

GDS Copyright : Alex Howarth

Slide 7

Slide 7 text

What should wake you up at night?

Slide 8

Slide 8 text

nothing

Slide 9

Slide 9 text

Exit pursued by bears

Slide 10

Slide 10 text

No wait! I suppose I should explain that.

Slide 11

Slide 11 text

Ways to not get woken up

Slide 12

Slide 12 text

Shifts

Slide 13

Slide 13 text

GDS 4 shifts of 8 hours So you can have handover

Slide 14

Slide 14 text

GDS Just say no! “Long-term night shift work is associated with an increased risk of certain cancers, as well as metabolic problems, heart disease, ulcers, gastrointestinal problems and obesity. ... People who work night shifts or rotating shifts also often don't sleep enough, and long-term sleep deprivation is known to be bad for health.” - https://sleepfoundation.org/shift-work/content/living-coping-shift-work-disorder

Slide 15

Slide 15 text

Follow the Sun

Slide 16

Slide 16 text

GDS

Slide 17

Slide 17 text

GDS If you are big enough this is a good option.

Slide 18

Slide 18 text

Self healing systems

Slide 19

Slide 19 text

GDS https://www.flickr.com/photos/johnclare/7124089493/

Slide 20

Slide 20 text

GDS Automatic restarts: ● daemontools ● monit ● God ● SMF ● systemd

Slide 21

Slide 21 text

GDS Clusters: ● mongodb ● elasticsearch

Slide 22

Slide 22 text

GDS Auto scaling groups

Slide 23

Slide 23 text

GDS Schedulers: ● Mesos ● Kubernetes ● ECS

Slide 24

Slide 24 text

What if you can’t do all these?

Slide 25

Slide 25 text

Be selective!

Slide 26

Slide 26 text

GDS What 5 things would you monitor and alert on first if you had no monitoring?

Slide 27

Slide 27 text

GDS ● disk

Slide 28

Slide 28 text

GDS ● disk ● CPU

Slide 29

Slide 29 text

GDS ● disk ● CPU ● Memory

Slide 30

Slide 30 text

GDS ● disk ● CPU ● Memory ● A couple of other things which don’t matter to the user

Slide 31

Slide 31 text

GDS https://www.flickr.com/photos/benterrett/17936132731

Slide 32

Slide 32 text

Monitor your user journey

Slide 33

Slide 33 text

GDS ● Availability ● Response times ● Error rates

Slide 34

Slide 34 text

GDS Legal reasons

Slide 35

Slide 35 text

GDS Life or death situations

Slide 36

Slide 36 text

Who?

Slide 37

Slide 37 text

GDS Which team should get called? ● Ops team ● Dev team ● DevOps team

Slide 38

Slide 38 text

GDS Trick question!

Slide 39

Slide 39 text

GDS DevOps is a culture not a team or job title

Slide 40

Slide 40 text

Everyone!

Slide 41

Slide 41 text

“Always two there are” - Yoda

Slide 42

Slide 42 text

GDS We’re hiring - Web Operations Engineers - Technical Architects - Developers https://gds.blog.gov.uk/jobs/