Runbooks

Runbooks at HelloFresh

Transcript

1. Runbooks
2. What is a runbook? A runbook is a form of documentation which is speci c to your company and your custom con gurations. It is a way to enable everyone to perform the Ops tasks necessary to mitigate an issue.
3. Runbook is a sequence of actions
4. Why do we need one?
5. 3:47 AM Alert rings the phone You open your laptop and check the message "<service-name> IS DOWN" Only the on-call engineer and an incident manager are online What should I do? Open the runbook!
6. What shoud be in the runbook The runbook should be able to answer the following questions: What How Where Who
7. What Minimal required information about the service (purpose, dependencies, components) Description of the problem or alert, this runbook is meant to solve Known issues that could be helpful
8. How The rst action that needs to be taken to move towards mediating the problem Ways to diagnose the problem Tips on how to interpret related data
9. Where Links to all the necessary resources and tools required to mitigate the issue Link to a dashboard where service metrics can be found Link to the service log stream on graylog Link to the tracing logs
10. Who Contact Information for the responsible squad/person
11. Examples
13. GitLab HowTo
14. Bene ts Reduce MTTD and MTTR Help to spread the knowledge Increase overall transparency in the company No need to wake up other developers
15. Where can I put my runbooks https://github.com/hellofresh/runbooks/
16. Runbooks make alerts actionable Attach runbook to every alert. It makes alerts actionable, not noisy.
17. Runbooks must be up-to-date We conduct thorough post-incident reviews (post- mortems) to identify what went well and nd ways to improve and inject learnings into our runbooks. Remember to add this to the runbook “ “
18. Questions?