Life On-Call, Availa-liberty, and the Pursuit of Happiness

First presented at Continuous Lifecycle London on 2016-05-03.

Transcript

1. Life On-Call, Availa-liberty, & the Pursuit of Happiness Runbooks Dave Cliffe @CliffeHangers
3. Incident #1: Oct 27, 2011 Incident #2: May 1, 2013 Incident #3: Nov 2, 2015
5. Collaboration/Resolution MICROSERVICES APPS & SERVICES CONTAINERS CLOUD NETWORK DATABASE SERVERS Developer NOC Helpdesk IT Ops System and User Efficiency ALERT 1 ALERT 2 ALERT 3 Correlate, Cluster and Manage EVENTS People Data Process Deployment Tools Monitoring Tools Ticketing Tools APP SYSTEM LOG WEB MOBILE APP Automatic Escalations On-Call Scheduling Your Fastest Path to Incident Resolution
7. Availability
8. Every software powered company experiences downtime http://www.evolven.com/blog/downtime-outages-and-failures-understanding-their-true-costs.html Cost of outages: $7,400,000 annual cost @175 hours downtime Gartner
9. “The most important ability is availability.” All CEOs everywhere
10. Why is Availability a terrible metric?
11. The Tyranny of the SLA credit: J. Paul Reed (@jpaulreed)
12. “ystem Availa ility means the percentage of total time during which the Hosted Service network is available to Client and Client is able to access the Hosted Service system interface. ______ warrants the following minimum levels of Hosted Service System Availability during each calendar month: 99.95% The following definitions will apply to the calculation of availability : Hosted “ervi e “ystem Availa ility means the percentage of total time during each calendar month during which the Hosted Service is available to Client, excluding Scheduled Downtime and Emergency Maintenance An actual SaaS SLA
13. Are you Available?
14. Happiness
16. Measuring (Un)Happiness
17. Responsiveness
18. Pain
19. Health Checks https://labs.spotify.com/2014/09/16/sq uad-health-check-model/
20. Happiness++
21. http://www.activestate.com/blog/2014/01/devops-hero-culture Beware the ‘Hero Culture’
22. Eliminate Single Points of Dependence
23. Reduce Alert Fatigue https://www.pinterest.com/pin/497929302524908289
25. On a regular basis, For every alert, Ask … 1) Is it actionable? 2) Is it urgent? 3) Could we consolidate? 4) Did the right person get it?
26. “The most important on-call responsibility is to understand customer impact.” Anonymous Customer (who I didn’t verify I could quote)
27. Sharing Operational Responsibility
28. Giving developers operational responsibilities has greatly enhanced the QUALITY of the services, both from a customer and a technology point of view. The TRADITIONAL model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. -Werner Vogels, CTO Amazon SHARED OPERATIONAL RESPONSIBILITY … You build it, you run it.”
29. For developers to take responsibility for the systems they create, they need support from operations to understand how to build ’reliable software that can be continuous deployed to an unreliable platform that scales horizontally’. -Jez Humble, quoting Jesse Robbins (Chef) SHARED OPERATIONAL RESPONSIBILITY
30. Thanks! Runbooks Dave Cliffe dcliffe@pagerduty.com @CliffeHangers