Herding Cats to a Firefight

The Evolution of an Engineering On-Call Team

Transcript

1. H E R D I N G C AT S T O A F I R E F I G H T T H E E V O L U T I O N O F A N E N G I N E E R I N G O N - C A L L T E A M G . C H A N G @ G R E Y S C H A L E E U R U K O 2 0 1 6
2. T H E Y E A R 1 B . C . ( B E F O R E C AT S )
3. In the beginning, there was only darkness.
4. But suddenly, out of the darkness, there came a sound...
5. (pager noises)
6. One person was on-call. All day. And night. Every day. Every week. Forever.
7. (not really, but close enough)
8. Why not start having a rotation? "We don't need no stinkin' on-call rotation!"
9. Bullshit.
10. "Hi, sorry to be calling at this hour.
 I'm from Yammer, I work with _____.
 Can I please speak with him?" Date: Friday, xxth of XXX, 2013
 Time: 03:00 AM GMT -0800
11. T H E Y E A R 1 A . D . ( A F T E R D I S A S T E R )
12. How to maths?!?! • Given: • Given: • Given: (1 + 15 ± 5 ) * 2 • ((4 / 1) * ((1 + 15 ± 5) * 2)) = ??? Answer:
14. How to acronyms?!?! M T B F M T T R A A R S L A
15. • MTBF: Mean Time Between Failures • MTTR: Mean Time To Recovery • SLA: Service Level Agreement • AAR: After Action Review • IR: Incident Report • OMGWTFBBQAFK
16. M T B F M T T R less frequent faster recovery requires more
 stable systems needs good 
 response training engineers interrupted 
 less often engineers gain
 broad knowledge possibly more 
 disastrous issues possibly more 
 frequent issues
18. • Google Docs Forms • Yammer Notes • JIRA ❎ (hard to read reports) ❎ (hard to analyse) ✅ (not perfect...but sort of works)
20. Hey, we're starting to get this!. . . . . . . . . . . .
21. Actually, not yet. • System grows faster than we can learn about it • Silos appear when you don't share knowledge • Who's cleaning up this mess, anyway? • Burnout is real
22. T H E R E N A I S S A N C E ( G R O W I N G PA I N S )
23. Do more by doing less • Split responsibilities by stack • Added London office for follow-the-sun coverage • Onboard everybody to the process • Practice, practice, practice
24. All hands on deck • Keep all alerts in a configuration repo • Managers aren't doing anything, anyway --
 make them Incident Managers! • Runbooks, runbooks everywhere (and a unified one) • Make the initial response as simple as possible
25. B A C K T O T H E F U T U R E ( T H E P R E S E N T )
26. Combined schedules • Fewer rotations • Team is unified, so schedules should be too
27. Post-mortems and retrospectives • What? Where? Who? Why? How? • NO blame game
28. Weekly hand-overs and monthly reviews • Previous week engineers to current week engineers • Track top alerts and resolutions (or lack of) • Focus on the noisiest services • Timezones are hard
29. Bi-monthly surveys • Summarise overall preparedness • Make sure we're improving • ...and that nobody is actually burned out
30. Fix ALL the alerts • Noisy • Flaky • Real
31. W H E R E A R E T H E C AT S N O W ? !
32. The end game • 1 alert per person per day • Service owners are on-call for those services • The world is full of kittens!
33. Isn't on-call just for Ops? • No • Responsibility for our code • Pride in our code • No pain, no gain
34. Isn't on-call just for Ops? • No • Responsibility for our code • Pride in our code • No pain, no gain
35. After all... we are all cats being herded.
36. T H A N K Y O U @ G R E Y S C H A L E G . C H A N G @ G R E Y S C H A L E E U R U K O 2 0 1 6