AA261: DevOps lessons in collaborative maintenance

On January 31, 2000, Alaska Airlines Flight 261 plunged into the Pacific ocean in an extreme "nose down" position, killing all 88 crew and passengers on board. The NTSB concluded AA261's horizontal stabiliser trim system's jackscrew was inadequately maintained, causing the pilots to lose all control of the plane.

There are striking parallels with the problems we face daily in IT operations & software development, and the 30 years of give and take between the aircraft manufacturer's engineers, airline maintenance staff, and federal regulators that preceded AA261's simple mechanical failure.

In this talk, Lindsay looks at the complex interplay between the parties in the AA261 crash through a DevOps lens, investigating the collaborative approach to maintenance and operation of the MD-83 aircraft, and relating the complexities back to the complex IT systems we build and maintain.

Transcript

1. AA261 DevOps lessons in collaborative maintenance
2. Lindsay Holmwood @auxesis
3. Software Manager @ Bulletproof Networks
5. Trigger warning: death
6. Puerto Vallarta January 31, 2000
7. Seattle
8. Departed PVR at 13.37 PST
9. Ascended to 31,000ft
10. 2 hours into flight: Jammed horizontal stabiliser
11. No trim control
12. Redirected to LAX
14. Pilots unjammed horizontal stabilisers
17. 2 pilots 3 crew 83 passengers
22. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
23. hindsight != foresight
24. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
25. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
26. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
28. “poorly conceived and woefully executed”
32. DC-9 -> MD-80 -> MD-83
34. Evolutionary product development
35. Appropriated maintenance schedules
36. Jackscrew lubrication interval
38. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
39. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
40. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
41. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
42. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
43. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
44. 1965 every 300-350 hours launch of DC-9 1985 every 700 hours industry deregulation 1987 every 1000 hours industry standardisation 1991 every 1200 hours industry standardisation 1994 every 1600 hours industry standardisation 1996 every 8 months (2550 hours) Alaska Airlines policy change
45. Decrementalism
46. Complex system constraints Jens Rasmussen
48. workload
49. workload economy
50. workload economy safety
52. time
53. time cost
54. time cost quality
55. workload economy safety
56. workload economy safety
57. workload economy safety
58. workload economy safety
59. workload economy safety
60. workload economy safety
61. workload economy safety
62. workload economy safety
63. economy safety workload
64. economy failure of foresight safety outside: workload
65. economy failure of foresight trade-offs in direction of greater efficiency safety outside: inside: workload
66. trade-offs in direction of greater efficiency
67. trade-offs in direction of greater efficiency
68. Constraints on knowledge
69. Why would they make bad decisions intentionally?
70. Decisions seemed rational
71. Local rationalisation
72. “people make what they consider to be the best decision based on available knowledge at the time”
73. This is a maintenance accident. Alaska Airlines' maintenance and inspection of its horizontal stabilizer activation system was poorly conceived and woefully executed. The failure was compounded by poor oversight... had any of the managers, mechanics, inspectors, supervisors or FAA overseers whose job it was to protect this mechanism done their job conscientiously, this accident cannot happen. -- John J. Goglia, NTSB Board Member
74. economy safety workload
75. time cost quality
77. Devops constraints
78. “God, our ops team are arseholes. I just want to deploy this change and go home!”
79. “God, our ops team are arseholes. I just want to deploy this change and go home!” workload economy safety
80. “God, our ops team are arseholes. I just want to deploy this change and go home!” workload economy safety workload economy safety
81. What are the circumstances?
82. Where are the tensions?
83. Have ops been burnt before?
84. Is there deployment friction? Why?
85. Is deployment high-risk?
86. Is deployment time consuming?
87. Is deployment important to the business?
89. “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?”
90. workload economy safety “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?”
91. workload economy safety “It’s 3am an the pager has gone off again. Why can’t these devs just write code that works?” workload economy safety
92. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
93. What are the circumstances?
94. Where are the tensions?
95. Why didn’t the dev know the code would fail like this?
96. Why weren’t you involved when the code was written?
97. How is code reviewed?
98. Is the infrastructure anti-fragile?
99. Is the code anti-fragile?
101. Hindsight bias
102. [hindsight] converts a once vague, unlikely future into an immediate, certain past -- Sidney Dekker
104. What are the motivations?
105. “amoral actors”
106. workload economy safety
107. workload economy safety
108. “root cause” is simply the point you stop looking -- Sidney Dekker
109. What are the circumstances?
110. Where are the tensions?
111. Thank you!
112. Thank you! Liked the talk? Let @auxesis know!
113. Sidney Dekker [books] Field Guide to Understand Human Error Drift Into Failure Just Culture Dan Manges [blog] How incidents affect infrastructure priorities