Ethical and Sustainable On-Call

Transcript

1. Ethical & Sustainable On-Call Jon Daniel - @binarycleric
2. Working Environment • 150+ developers • more apps than developers • some apps are owned by multiple teams • on-call rules/procedures differ by team
3. we’re hiring!
4. My Team: Core Services • builds libraries and APIs essential to business needs • mostly ruby, some clojure • 8 developers + few split between teams
5. always remember
6. things will go wrong
7. outages are as certain as death and taxes
8. many of us don’t build single applications anymore
9. we build for a Service Oriented Architecture (SOA)
10. SOA requires us to rethink how we build software
11. maybe we should rethink on-call too
12. SOA outages look different
13. we continue humming along following non- catastrophic failure
14. (at least in theory)
15. • data distribution • circuit breakers • ‘last known good’ caching • monitoring • alerting
16. but we haven’t thought of everything
17. so…
18. an outage is just a use case we have yet to account for
20. failure cases should be understood and mitigated
21. an on-call person’s job should be to prevent catastrophe not react to failures
22. Ethical & Sustainable On-Call Jon Daniel - @binarycleric
23. all* developers included rotation is 1 week defend against interruption no shipping features flexibility when systems are stable * even junior developers once they get up to speed
24. all developers are included
25. work with junior developers
26. rotation is 1 week
27. have a plan
28. an on-call backlog is acceptable
29. be prepared for the unexpected
30. defend against interruption of both services and team members
31. you protect your team’s productivity
32. escalate 
 if necessary
33. no shipping features
34. you are doing this
35. not this
36. • finding and understanding failure cases • adding failover/error recovery where necessary • writing playbooks for handling outages • updating libraries and addressing deprecations • testing alerting and diagnostic tools • applying duct-tape when necessary but documenting for the future
37. most importantly
38. communicating with consumers of your applications
39. flexibility when systems are stable
40. if you get woken up feel free to sleep in
41. use your time as you best see fit
42. if an on-call person is having a rough week cut them some slack!
43. offer to help if you have extra time
44. missing something?
45. oh yeah, your phone goes off at 2:30 AM sometimes. :(
46. your job is to make that (almost) never happen
47. why should you be woken up?
48. all other safety mechanisms have failed
49. when customers are 
 directly impacted
50. when systems are 
 losing value
51. teams should understand what critical means
52. how should you be paged?
53. • phone call • text message • push notification pick one!
54. respect acknowledgements
55. telling me there is a problem over and over only leads to slower fixes
56. what happens when you are woken up?
57. • stay calm • acknowledge any alerts • notify others you are addressing the issue
58. get things working even if you have to apply duct tape
59. • document any fixes • notify teams that the problem has been addressed
60. go back to sleep
61. the following morning
62. implement long-term fixes update playbooks publish a post mortem
63. respect the on-call rotation
64. if you suspect a problem
65. • check acks in paging system • check email/chat for alerts • offer help/advice if you are knowledgeable • own up to outages if you caused them
66. DO NOT
67. • keep asking “is it fixed yet” • start blaming others • stop communication if you have important information
68. we’re all on the same team
69. lastly
70. problems are rarely fixed correctly by exhausted people
71. it’s okay to take time off after an exhausting on-call rotation
72. however
73. exhausting on-call rotations are always a symptom of a larger problem!
74. exhausted people are a cause of downtime
75. thank you!
76. • Long Island Solar Farm - https://flic.kr/p/drsrm8 • Hurricane Sandy power outage in Lower Manhattan, New York - https:// flic.kr/p/dp4D78 • Blackouts are fun - https://flic.kr/p/3oomWN • Linesman at work - https://flic.kr/p/7Z4TKU • Power pole replacement - https://flic.kr/p/oLo3Ye • Kudankulam nuclear power plant - https://flic.kr/p/dM66tE • Blueprint - https://www.flickr.com/photos/wscullin/3770015991 • Power Plant - https://www.flickr.com/photos/52336371@N07/19638501728 Image Credits