Distributed Failure: Learning Lessons From Aviation

Transcript

1. 1.

   DISTRIBUTED FAILURE Andrew Godwin @andrewgodwin Learning lessons from aviation

2. 2.

   Hi, I’m Andrew Godwin

3. 3.

   Content Warning Aviation accidents Road accidents Discussion of death

4. 4.

   Software is difficult.

5. 5.

   Distributed is even harder.

6. 6.

   None
7. 7.

   Not unique to distributed systems

8. 8.

   None
9. 9.

   Who's solved this? Aviation.

10. 10.

    A Boeing 747 has six million parts

11. 11.

    A Boeing 747 has six million parts

12. 12.

    Airplane Car Walking Train 220 130 30.8 Deaths per billion

    hours (UK 1990-2000) 30
13. 13.

    People matter as much as machines

14. 14.

    Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other

    16% Mechanical
15. 15.

    Let's look at some aviation principles

16. 16.

    Principle #1 Hard Failure

17. 17.

    If something is wrong it turns itself off

18. 18.

    This only works if you have redundancy

19. 19.

    None
20. 20.

    These are great ways to ensure you never fix something.

21. 21.

    No accident or outage has a single cause. Stop your

    code getting into odd states.
22. 22.

    None
23. 23.

    Single points of failure can be good

24. 24.

    None
25. 25.

    Principle #2 Good Alerting

26. 26.

    Cockpits are incredibly selective about what sets off an audio

    alarm
27. 27.

    Alert fatigue is real. Avoid at all costs.

28. 28.

    Never, ever, put all errors in the same place

29. 29.

    Critical Normal Background

30. 30.

    Critical Normal Background Wakes someone up. Actionable.

31. 31.

    Critical Normal Background Wakes someone up. Actionable. Fixed over the

    next week.
32. 32.

    Critical Normal Background Wakes someone up. Actionable. Fixed over the

    next week. Metrics, not errors.
33. 33.

    Have you been ignoring an error for weeks? Then turn

    off its error reporting.
34. 34.

    Principle #3 Find your limits

35. 35.

    Everything will fail. You should know when.

36. 36.

    Copyright Boeing

37. 37.

    What's your Minimum Equipment List?

38. 38.

    REQUIRED OPTIONAL

39. 39.

    Did you load test? Did you fuzz test?

40. 40.

    You don't have to perfectly scale.

41. 41.

    Risk is fine when you're informed!

42. 42.

    Principle #4 Build for failure

43. 43.

    No single thing in an aircraft can fail and take

    it down.
44. 44.

    We all want this for our code, but the way

    to do it is to build for failure.
45. 45.

    Kill your application randomly Practice server network failures Develop on

    unreliable connections
46. 46.

    The majority of pilot training is handling emergencies.

47. 47.

    None
48. 48.

    Use checklists. Don't rely on memory.

49. 49.

    If you practice failure, you'll be ready when the inevitable

    happens.
50. 50.

    Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other

    16% Mechanical
51. 51.

    Principle #5 Communicate well

52. 52.

    Distributed software means separate teams.

53. 53.

    As you grow, communication becomes exponentially harder.

54. 54.

    None
55. 55.

    None
56. 56.

    None
57. 57.

    Clear communication is vital.

58. 58.

    Write everything down.

59. 59.

    Have a clear chain of command.

60. 60.

    Make decisions.

61. 61.

    Principle #6 No blame culture

62. 62.

    How do I know all these aviation stats?

63. 63.

    Every incident is reported and investigated.

64. 64.

    There is never a single cause of a problem.

65. 65.

    Make it very difficult to do again.

66. 66.

    None
67. 67.

    None
68. 68.

    Encourage reporting.

69. 69.

    Reward maintenance as well as firefighting

70. 70.

    None
71. 71.

    In aviation, every rule is written in blood.

72. 72.

    Software is not yet there. But we are getting closer.

73. 73.

    Margaret Hamilton Her error detection code saved Apollo 11

74. 74.

    Therac-25 Killed 3, severely injured at least 3 more

75. 75.

    None
76. 76.

    None
77. 77.

    Hard failure Good alerting Find your limits Build for failure

    Communicate well No blame culture
78. 78.

    Thanks.