"Incident response at Heroku" by Michael Friis

Transcript

1. Incident Response at Heroku Michael Friis
2. 5+ billion requests served / day
3. My name is Michael Ex co-founder at AppHarbor (YC W11) Product Manager at Heroku I don’t wear a pager anymore
4. ~150 people total Lots of teams, lots of software, lots of change Everything on AWS (except status.heroku.com) Total ownership: “You built it, you get the pager” SRE team monitors system, tracks uptime, etc. Ops at Heroku
5. What are incidents?
6. “Bash Security Advisory” “Git Push and Build Issue” “Network issues within the EU Region” [Internal] “Cloudbees shutting down” Some Heroku incidents
7. Here we go...
8. Move to central chat room
9. Designate Incident Commander
10. Update status site
12. Send out sitrep
13. Assess problem
14. Mitigate problem
15. Coordinate response
16. Manage ongoing response
17. Post-incident cleanup
18. Post-incident follow-up
19. 6. Mitigate problem 7. Coordinate response 8. Manage response 9. Post incident clean-up 10. Post-incident follow-up 1. Move to central chat 2. Designate IC 3. Update Status Site 4. Send out sitrep 5. Assess problem Summary https://blog.heroku.com/archives/2014/5/9/incident-response-at-heroku
20. @friism friis@heroku.com www.heroku.com