The No-Nonsense Guide to Runbook Best Practices

Runbooks are a key part of incident management and preserve institutional knowledge. They can be used for both incident response as well as routine tasks like db maintenance and generating a complex report. We are mostly focused on incident response runbooks here.

Transcript

1. The No-Nonsense Guide to Runbook Best Practices https://incidenthub.cloud Last Updated: November 2024
2. /02 Runbook Structure Establish a consistent format for runbooks across the organization. Get buy-in from the team on the chosen format. Structure runbooks as decision trees, keeping them concise. Each runbook should have a single purpose. incidenthub.cloud
3. /03 Runbook Content Runbooks should provide clear, actionable instructions. Keep content concise and trim unnecessary details. Include relevant architecture diagrams and links to dashboards. Be aware of the "curse of knowledge" when writing. It's okay to have some manual steps in runbooks. incidenthub.cloud
4. /04 Updating and Maintaining Runbooks Update runbooks after incidents based on observed issues. Coordinate with teams to ensure runbook updates happen. Note and fix any inaccuracies discovered during incidents. incidenthub.cloud
5. /05 Testing Runbooks Test runbooks from a "clean" machine before deployment. Involve new hires and conduct mock incident exercises. Regularly test runbooks to ensure they work as expected. incidenthub.cloud
6. /06 Store runbooks in a central, accessible location. Link alerts directly to relevant runbooks. Improve findability through descriptive naming and keywords. incidenthub.cloud Locating Runbooks
7. /07 Runbook Ownership Service teams own the runbooks for their services. SRE/Ops team owns runbooks for infrastructure/common components. Encourage collective ownership and rotating update responsibilities. incidenthub.cloud
8. /08 What Not To Do Avoid overly generic runbooks. Don't have more than one runbook per alert. Never store credentials in runbooks. incidenthub.cloud
9. /09 Dealing With the Unexpected If no runbook exists, involve the service owner and document the steps. If runbook steps are wrong or inaccessible, do not execute them blindly. Understand system interactions before executing commands. Pull in subject matter experts if runbook steps are incorrect. incidenthub.cloud
10. Thank You incidenthub.cloud hrish@incidenthub.cloud incidenthub.cloud Follow our blog for more: https://blog.incidenthub.cloud November 2024 /10