Don't Panic!

How to Cope Now You're Responsible for Production

---

More and more developers are expected to be on-call, provide out-of-hours support, and respond to production outages. Without much experience handling incidents, it can be scary, intimidating, and feel like being dropped in the deep end. But it doesn’t have to be that way!

Over two years on the FT’s Content team, we’ve transformed our incident response – from a number of mildly terrifying multi-hour outages, to a stable platform where team members feel comfortable on-call.

This talk will provide practical tips and advice on:

- setting up an incident response framework
- what to do when Everything Is On Fire™
- improving things afterwards
and some horror stories of our own…

Required audience experience:
Low – aimed at engineers and teams new to supporting production services

Objective of the talk:
Attendees will leave with practical ideas for setting up a standard incident framework.

I’ll cover:
- how we used to handle incidents at the FT
- what we did to improve
- standard processes to follow during an incident
- what to do afterwards to ensure problems don’t happen again

---

https://continuouslifecycle.london/sessions/dont-panic-how-to-cope-now-youre-responsible-for-production/

Transcript

1. Don't Panic! How to Cope Now You're Responsible for Production Euan Finlay @efinlay24
2. /usr/bin/whoami @efinlay24
3. /usr/bin/whodoiworkfor No such file or directory. @efinlay24
4. https://www.ft.com
5. You've just been told you're on call (and you're mildly terrified) @efinlay24
6. Obligatory audience interaction @efinlay24
7. Everyone feels the same way at the start (I still do today) @efinlay24
8. How do you get to the point where you're more comfortable? @efinlay24
9. A tenuous link to A Christmas Carol.
10. The Ghosts of Incidents... > Future Present Past
11. The Ghost of Incidents Future
12. Handling incidents is the same as any other skill @efinlay24
13. Get comfortable with your alerts
14. Get comfortable with your alerts (and bin the rubbish ones)
15. Have a plan for when things break @efinlay24
16. Practice regularly @efinlay24
17. The one where we decommissioned all our production servers
19. Keep your documentation up to date @efinlay24
20. If it's important, document and monitor it @efinlay24
21. The Office Email Relay
22. Break things and see what happens Did your system do what you expected? @efinlay24
23. The Datacenter Disconnect
24. Have a central place for reporting changes and problems @efinlay24
26. The Ghosts of Incidents... Future > Present Past
27. The Ghost of Incidents Present
28. Calm down, take a deep breath: it's (probably) ok @efinlay24
29. Don't dive straight in Go back to first principles @efinlay24
30. What's the actual impact? @efinlay24
31. "All incidents are equal, but some incidents are more equal than others." George Orwell, probably @efinlay24
32. What's already been tried? @efinlay24
33. Is there definitely a problem? @efinlay24
34. What's the minimum viable solution? @efinlay24
35. Get it running before you get it fixed
36. Check the basics first @efinlay24
39. Don't be afraid to call for help @efinlay24
40. The One Where a Manager Falls Through the Ceiling
41. The One Where a Director Falls Through the Ceiling
43. Communication is key Especially to your customers @efinlay24
45. Put someone in charge @efinlay24
47. Creating a new incident channel is usually a good idea @efinlay24
50. Communication is key It's so important, I wrote it twice @efinlay24
51. Tired people don't think good @efinlay24
52. The one where we had to serve traffic from staging
54. It wasn't great, but it wasn't the end of the world @efinlay24
55. The Ghosts of Incidents... Future Present > Past
56. The Ghost of Incidents Past
57. Congratulations! You survived It probably wasn't that bad, was it? @efinlay24
58. Run a post-mortem with everyone involved @efinlay24
59. Incident reports are important @efinlay24
61. Prioritise follow-up actions @efinlay24
62. https://blog.travis-ci.com/2018-04-03-incident-post-mortem
63. https://blog.travis-ci.com/2018-04-03-incident-post-mortem
64. Identify what can be done better next time
65. The One with the Badly Named Database
66. Don't name your pre-production database: 'pprod' Seriously, who does that @efinlay24
67. @efinlay24 Nearly the end
68. Problems will always happen (and that's ok) @efinlay24
69. The End (please clap)
70. euan.finlay@ft.com @efinlay24 FT careers: https://aboutus.ft.com/en-gb/careers/ Image links: https://goo.gl/3DeojV