SRE Essentials

The basics on Site Reliability Engineering.
First delivered at Slush pre-event (Microsoft Flux - 12/3)

Transcript

1. Jason Hand Senior Cloud Advocate jasonhand
3. @jasonhand | #microsoftxslush
4. #Slush18 @jasonhand | #microsoftxslush
5. #Slush18 @jasonhand | #microsoftxslush
6. @jasonhand | #microsoftxslush
7. Site Reliability Engineering? @jasonhand | #microsoftxslush
8. Site Reliability Engineering @jasonhand | #microsoftxslush
9. Site Reliability Engineering engineering @jasonhand | #microsoftxslush
10. Site Reliability Engineering engineering appropriate @jasonhand | #microsoftxslush
11. Site Reliability Engineering engineering appropriate reliability @jasonhand | #microsoftxslush
12. Tips 5 @jasonhand | #microsoftxslush Essential SRE
13. #Slush18 @jasonhand | #microsoftxslush
14. Tip#1 @jasonhand | #microsoftxslush • Observability = Reality • Set Service Level Expectations • Measure & report against specific expectations
16. Four Signals 1.Latency
17. Model 1.Utilization
18. 1.Rate Model
19. Expectations SLIs, SLOs, and SLAs … oh my! @jasonhand | #microsoftxslush
20. Service Level Indicator (SLI) (represented as a ratio or …% proportion) # of successful HTTP calls / # of HTTP calls # of operations that completed in < 10ms / # of operations # of “full quality responses” / # of responses # of records processed / # of records ratio X 100 = % proportion @jasonhand | #microsoftxslush
22. errors How many requests? Here is everything I know.
23. errors How many failed packets? Here is everything I know. NOT NEVER
24. Service Level Objective (SLO) HTTP requests (as reported by the load balancer) 95% 30-day (example) SLI @jasonhand | #microsoftxslush
25. 9’s appropriate? 99.999% ? 99.99% ? 99.9% ? @jasonhand | #microsoftxslush
26. @jasonhand | #microsoftxslush / year / quarter / month / week / day / hour 99% 3.65 days 21.6 hours 7.2 hours 1.68 hours 14.4 minutes 36 seconds 99.9% 8.76 hours 2.16 minutes 43.2 seconds 10.1 minutes 1.44 minutes 3.6 seconds 99.99% 52.6 minutes 12.96 minutes 4.32 minutes 60.5 seconds 8.64 seconds 0.36 seconds 99.999% 5.26 minutes 1.30 minutes 25.9 seconds 6.05 seconds 0.87 seconds 0.04 seconds 9’s appropriate?
27. Error Budget Measurable, objective and self-policing mechanism to balance innovation versus reliability @jasonhand | #microsoftxslush
28. @jasonhand | #microsoftxslush Tip#2 ”Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."
29. @jasonhand | #microsoftxslush Tip#3 • PIRs for ALL customer facing disruptions • PIRs are blameless
30. @jasonhand | #microsoftxslush Tip#4
31. @jasonhand | #microsoftxslush Tip#5 Mind the silos
32. depends @jasonhand | #microsoftxslush
33. Amazon Baidu Etsy GitHub Microsoft Pinterest Stack Exchange Uber Yelp @jasonhand | #microsoftxslush
34. Those companies!
35. Unknown @jasonhand | #microsoftxslush
36. @jasonhand | #microsoftxslush
37. What We Did...
38. Observability Observability Monitoring @jasonhand | #microsoftxslush
39. Engineering REAL LIFE? @jasonhand | #microsoftxslush
40. Game Days Using knowledge and structured plans routinely to rehearse incident response. Expanding and improving the current baseline. @jasonhand | #microsoftxslush Prepare - Rehearse - Respond
41. retrospective review @jasonhand | #microsoftxslush
42. SRE @ VictorOps (a startup) @jasonhand | #microsoftxslush
43. One possible @jasonhand | #microsoftxslush
44. #Slush18 @jasonhand | #microsoftxslush
45. reliability up what keeps you up at night? SLI SLO Error Budgets critically You @jasonhand | #microsoftxslush
46. @jasonhand | #microsoftxslush
47. jhand.co/SREBook culture @jasonhand | #microsoftxslush
48. @jasonhand | #Slush18
49. honeycomb.io/observability gremlin.com/blog victorops.com/blog @jasonhand | #microsoftxslush
50. Microsoft Learn @jasonhand | #microsoftxslush
51. At Microsoft, we're aspiring to have a living, learning culture with a growth mindset that allows us to learn from ourselves and our customers. These are the key attributes of the new culture at Microsoft, and I feel great about how it seems to be resonating and how it's seen as empowering. @jasonhand | #microsoftxslush
52. Jason Hand Senior Cloud Advocate @jasonhand