Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How GOV.UK does on call

D52a081f2f45871ab197ec858e06fb33?s=47 bob
May 19, 2016

How GOV.UK does on call

Talk given about how GOV.UK does on call at the first #humanops meetup http://www.meetup.com/HumanOps-London/events/229460050/



May 19, 2016


  1. bob walker Head of Web Operations && \ GOV.UK Infrastructure

    Product Owner Government Digital Service @rjw1
  2. How GOV.UK does on call.

  3. Who am I?

  4. GDS Over 17 years as a Sys Admin

  5. GDS Over 3 years on GOV.UK

  6. GDS Head of Web Operations community at GDS

  7. Is GOV.UK Scale?

  8. https://www.gov.uk/performance/site-activity GDS ~75 applications ~2000 req/s at edge at peak

    ~150 req/s at origin at peak ~13M Unique Users per week
  9. Is GOV.UK Important?

  10. How GOV.UK does on call.

  11. Second Line

  12. GDS 2 technical members of staff dealing with: • alerts

    • incidents • requests
  13. Second Line Charter

  14. GDS 1. 2nd line is a learning experience, you should

    aim to learn more about how GOV.UK operates during the week 2. Work together, be inclusive and don’t leave anyone out or alone 3. It starts at 9:30, be there 4. Feel free to go to essential meetings, but tell people ahead of time and see item 2 5. Make small improvements to make it better for the next team (e.g. documentation, automation) 6. There’s no such thing as a stupid question (but check the Ops Manual first) 7. Help others and be patient - not everyone knows the same things 8. Leave the desks clean and tidy when you finish your stint
  15. GDS 28 people

  16. On Call

  17. GDS Currently 9 people

  18. GDS 6 alerts will call people

  19. GDS VMs going down will call people

  20. GDS They probably shouldn’t

  21. GDS Mini incident reviews when we’re called

  22. GDS PagerDuty Drill

  23. Escalations

  24. GDS 3 people

  25. GDS On-call escalation Major Technical Issues Emergency Publishing

  26. Incident Reviews

  27. Make things open: it makes things better

  28. Tools

  29. GDS • Icinga • graphite • collectd • PagerDuty •

  30. What could be better?

  31. https://gds.blog.gov.uk/jobs/ GDS We’re hiring!

  32. Thanks! bob walker @rjw1