Reliability Theater

When things break and why managing risk is important.

Transcript

1. RELIABILITY THEATER WHEN THINGS BREAK AND WHY MANAGING RISK IS IMPORTANT
2. @peterjshan linkedin.com/in/peterjshan/ peterjshan.com Peter Shannon
3. SPOILERS • HOW AVAILABILITY AND RELIABILITY ARE RELATED BUT DIFFERENT • RISK IS A RANDOM FORCE, HOW SHOULD WE THINK ABOUT RISK • WHAT IS AN SLA AND WHY IS IT IMPORTANT • THE COST OF BEING RELIABLE AND/OR AVAILABLE • PRACTICAL WAYS TO EVALUATE RISK, THINGS TO PONDER
4. WHAT IS RELIABILITY THEATER?
5. PERHAPS YOU’VE HEARD OF SECURITY THEATER?
6. “The T.S.A.'s profession of outrage is nothing but security theater" - Bruce Schneier
8. RELIABILITY THEATER IS THE PRACTICE OF INVESTING IN COUNTERMEASURES INTENDED TO PROVIDE THE FEELING OF IMPROVED RELIABILITY WHILE DOING LITTLE OR NOTHING TO ACHIEVE IT.
9. ILLUSION OF RELIABILITY
10. SOLVING THE WRONG PROBLEM
11. RELIABILITY IS THE PROBABILITY THAT A SYSTEM WILL PRODUCE CORRECT OUTPUTS UP TO SOME LENGTH OF TIME
12. RELIABILITY AND AVAILABILITY ARE NOT THE SAME, BUT THEY ARE RELATED
13. AVAILABLE = OPERATIONAL RELIABLE = CORRECT
14. A TRIVIAL EXAMPLE OF RELIABILITY THEATER
15. UNRELIABLE SERVICE (RELIABILITY) > MULTI-REGION SUPPORT (AVAILABILITY)
16. IF YOUR SERVICE DOESN’T WORK IN ONE REGION, WHY WOULD IT WORK IN TWO?
17. RELIABILITY AVAILABILITY RISK
18. AVAILABLE = OPERATIONAL RELIABLE = CORRECT RISK = RANDOM FORCE
19. UNDERSTANDING RISK
22. “Risk comes from not knowing what you’re doing." - Warren Buffet
24. RISK = PROBABILITY * IMPACT
25. IF YOU DON’T KNOW WHAT THE HAZARDS ARE YOU CAN’T AVOID THEM
26. RISK
27. RISK IS A RANDOM FORCE
28. PROBABILITY
29. QUANTIFY RISK, WHAT IS THE LIKELIHOOD?
30. IMPACT
31. QUALIFY RISK, IMPACT HAS CONTEXT
32. RISK = PROBABILITY * IMPACT IMPACT REDUCTION STRATEGY CHANCE REDUCTION STRATEGY
33. RISK = PROBABILITY * IMPACT WHAT IS OUR RISK POTENTIAL FOR DELIVERING INCORRECT RESPONSES AND/OR NOT BEING AVAILABLE TO OUR CUSTOMERS. THE RESULTING IMPACT IS LOSS OF REVENUE. • WHAT IS A STRATEGY FOR CHANCE REDUCTION? • WHAT IS A STRATEGY FOR IMPACT REDUCTION?
34. SERVICE RETURNS 200 OK 100% OF THE TIME RISK = PROBABILITY * IMPACT
35. SERVICE RETURNS 200 OK 100% OF THE TIME RISK = PROBABILITY * IMPACT AVAILABILITY
36. SERVICE RETURNS 200 OK 100% OF THE TIME RISK = PROBABILITY * IMPACT AVAILABILITY
37. SERVICE RETURNS 200 OK 90% OF THE TIME RISK = PROBABILITY * IMPACT RELIABILITY
38. SHOULD WE PLAN FOR CATASTROPHIC EVENTS?
40. AT THE MOVIE THEATER
42. WHAT IS THE PROBABILITY SOMETHING WILL GO WRONG?
43. THE BATTLE STATION IS “INDESTRUCTIBLE”, EXCEPT FOR THIS SMALL THERMAL EXHAUST PORT.
44. HOW DO WE INFORM OR QUANTIFY OUR DECISION MAKING?
45. Assumptions: 1.Death Star is ~150 kilometers in diameter or 7.06858×10^10 m^2 2.Thermal exhaust port is 2 meters wide or 3.14 m^2
46. Probability = 0.00000000442%
47. Risk = 0.00000000442% * Impact RISK = PROBABILITY * IMPACT
48. HOW DO WE PLAN TO AVOID RISK?
53. THINGS ARE GOING TO BREAK!
56. https://istheservicedown.com/problems/amazon-web-services-aws/history
58. Service Level Agreement (SLA)
59. AVAILABILITY (TIME BASED) = UPTIME / (UPTIME + DOWNTIME)
60. HOW MANY MINUTES ARE WE ALLOWED TO BE DOWN?
61. DOWNTIME = (1 / AVAILABILITY - 1) * UPTIME
62. AVAILABILITY (AGGREGATE BASED) = SUCCESSFUL REQUESTS / TOTAL REQUESTS
63. HOW MANY BAD REQUESTS ARE WE ALLOWED TO SERVE?
64. CAN YOU GUESS THE SLA?
65. WHAT IS THE SLA FOR S3?
66. 99.9% 43.84 MINUTES / MONTH 100K BAD REQUESTS / 100M
67. WHAT IS THE SLA FOR EC2?
68. 99.99% 4.38 MINUTES / MONTH 10K BAD REQUESTS / 100M
69. WHAT IS THE SLA FOR ROUTE53?
70. 100%
71. KNOW THE SLA FOR YOUR UPSTREAM SERVICES
72. RELIABILITY AVAILABILITY RISK R$
73. WHAT DOES AN HOUR OF DOWNTIME COST?
74. “WE STRIVE TO MAKE A SERVICE RELIABLE ENOUGH, BUT NO MORE RELIABLE THAN IT NEEDS TO BE. T H AT I S , W H E N W E S E T A N AVAILABILITY TARGET OF 99.99%, WE WANT TO EXCEED IT, BUT NOT BY MUCH: THAT WOULD WASTE O P P O R T U N I T I E S T O A D D FEATURES TO THE SYSTEM, CLEAN UP TECHNICAL DEBT, OR REDUCE ITS OPERATIONAL COSTS.” – SRE BOOK
75. “A USER ON A 99% RELIABLE SMARTPHONE CANNOT TELL THE DIFFERENCE BETWEEN 99.99% AND 99.999% RELIABILITY” – SRE BOOK
76. “MANAGING SERVICE RELIABILITY IS LARGELY ABOUT MANAGING RISK, AND MANAGING RISK CAN BE COSTLY” – SRE BOOK
77. A D D I N G A D D I T I O NA L 9 ’ S O F RELIABILITY CAN EVENTUALLY SCALE COST EXPONENTIALLY. COST OF REDUNDANT COMPUTE/ MACHINE RESOURCES. OPPORTUNITY COST - DEDICATING ENGINEERS TO WORKING ON INCREASING RELIABILITY OVER BUILDING FEATURES.
80. IF YOU DON’T KNOW THE HAZARDS YOU CAN’T AVOID OR IDENTIFY THEM
81. IDENTIFY RISKS, PRACTICE HAZARD RECOGNITION
82. HELP YOURSELF MAKE BETTER DECISIONS WITH THROUGH OBSERVABILITY
83. WHEN EVALUATING RISK ASK YOURSELF “AM I COMPETENT TO EVALUATE THE SITUATION?”
84. EXPRESS YOUR LEVEL OF CONFIDENCE, COMMUNICATE
85. Things Are Going To Break!