When to give up, and how?

Programs with constrained states are susceptible to end up in contradictory situations. The most likely cause of this is bugs. What to do when the program is found to be in a state where it cannot function correctly? Continue doing the wrong thing? Crash? What if it the situation can be saved? Let's look at some situations and some options we have.

Transcript

1. Björn Fahller When to give up, and how?
4. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 4/88 <radicalisation/>
5. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 5/88 Terminology
6. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 6/88 Terminology Bug The program does not behave correctly due to flaws in the encoded logic.
7. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 7/88 Terminology Bug The program does not behave correctly due to flaws in the encoded logic. Disappointment The program is subjected to something undesired outside of its logic, e.g. received malformed packet, disk full, out of memory.
8. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 8/88 Terminology Bug The program does not behave correctly due to flaws in the encoded logic. Disappointment The program is subjected to something undesired outside of its logic, e.g. received malformed packet, disk full, out of memory. Error handling Logic that deals with disappointments in a useful way.
9. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 9/88 Terminology Bug The program does not behave correctly due to flaws in the encoded logic. Disappointment The program is subjected to something undesired outside of its logic, e.g. received malformed packet, disk full, out of memory. Error handling Logic that deals with disappointments in a useful way. if (::close(fd) < 0 && errno == EINVAL) { ???? }
10. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 10/88 Terminology Bug The program does not behave correctly due to flaws in the encoded logic. Disappointment The program is subjected to something undesired outside of its logic, e.g. received malformed packet, disk full, out of memory. Error handling Logic that deals with disappointments in a useful way. This presentation is about bugs
11. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 11/88 Bugs How can a program know that it has a bug?
12. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 12/88 Bugs How can a program know that it has a bug? A function can see that it has been called with invalid arguments
13. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 13/88 Bugs How can a program know that it has a bug? A function can see that it has been called with invalid arguments A function can see that it has been called in an illegal state
14. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 14/88 Bugs How can a program know that it has a bug? A function can see that it has been called with invalid arguments A function can see that it has been called in an illegal state A caller can see a return/exception that is invalid
15. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 15/88 Bugs How can a program know that it has a bug? A function can see that it has been called with invalid arguments A function can see that it has been called in an illegal state A caller can see a return/exception that is invalid A function can see that data is inconsistent or invalid
16. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 16/88 Bugs A function can see that it has been called with invalid arguments A function can see that it has been called in an illegal state A caller can see a return/exception that is invalid A function can see that data is inconsistent or invalid But in order to see these, you need to define what legal/valid/consistent means
17. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 17/88 Bugs So what to do?
18. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 18/88 Bugs So what to do?
19. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 19/88 Bugs So what to do?
20. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 20/88 Bugs So what to do?
21. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 21/88 Bugs So what to do?
22. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 22/88 Bugs So what to do?
23. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 23/88 Bugs These counter bugs hide the real issues, making them much less likely to be found and fixed
24. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 24/88 Bugs You have two goals
25. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 25/88 Bugs You have two goals • Cause as little damage as possible
26. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 26/88 Bugs You have two goals • Cause as little damage as possible • Gather information so that you can fix the bug
27. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 27/88 Bugs You have two goals • Cause as little damage as possible • Gather information so that you can fix the bug These are sometimes in conflict
28. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 28/88 Bugs
29. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 29/88 Bugs How about assert()
30. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 30/88 Bugs How about assert()
31. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 31/88 Bugs How about assert()
32. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 32/88 Bugs How about assert() assert() does one of: • nothing • calls abort
33. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 33/88 crash do wrong things Bugs
34. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 34/88 Bugs How about assert()
35. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 35/88 Bugs How about assert() When assert() is triggered, it usually does so with: • A message • Often a stack trace • If you’re lucky a core dump
36. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 36/88 Bugs How about assert() When assert() is triggered, it usually does so with: • A message • Often a stack trace • If you’re lucky a core dump If you get the info from a triggered assert, you often have good chances of hunting down the bug
37. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 37/88 Bugs How about assert() When assert() is triggered, it usually does so with: • A message • Often a stack trace • If you’re lucky a core dump If you get the info from a triggered assert, you often have good chances of hunting down the bug Many companies have their own “assert” macros with finer control
38. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 38/88 Bugs Contracts?
39. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 39/88 Bugs Contracts?
40. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 40/88 Bugs Contracts?
41. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 41/88 Bugs Contracts?
42. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 42/88 Bugs Contracts?
43. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 43/88 Bugs Contracts? Much better visibility of expectations Program wide - unclear what implementations will support regarding custom violation handlers
44. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 44/88 Story time <How a policy of “always keep running” caused many crashes, and how adding deliberate termination points helped fix it./>
45. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 45/88 You need info! If you’re going to terminate: • Gather as much information as possible, so you can learn how to fix the underlying bug • Terminate as early as possible, before the state has been contaminated by erroneous work
46. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 46/88 Can we do better? Architecture for robustness Case study – Ericsson AXE telephone exchange
47. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 47/88 AXE Digital phone exchange First deployment in 1976 Can serve a city
48. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 48/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber.
49. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 49/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber.
50. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 50/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber. Two CPUs for redundancy.
51. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 51/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber. Two CPUs for redundancy. Each continuously monitored for faults
52. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 52/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber. Two CPUs for redundancy. Each continuously monitored for faults Cycle-by-cycle parallel and continuously monitored for differences by the MAU
53. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 53/88 AXE Central Processor Handles billing, monitoring, and subscriber services like call forwarding and wakeup calls. Typically stores 10-15 words of data per subscriber. Two CPUs for redundancy. Each continuously monitored for faults Cycle-by-cycle parallel and continuously monitored for differences by the MAU When a fault is detected, the faulty CPU is shut down, and an alarm is raised
54. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 54/88 AXE Regional Processors Can connect local phone calls even if contact with central processor is down
55. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 55/88 AXE Regional Processors Can connect local phone calls even if contact with central processor is down
56. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 56/88 AXE Regional Processors Can connect local phone calls even if contact with central processor is down Two CPUs for redundancy and damage containment
57. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 57/88 AXE Regional Processors Can connect local phone calls even if contact with central processor is down Two CPUs for redundancy and damage containment Each continuously monitored for faults
58. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 58/88 AXE Regional Processors Can connect local phone calls even if contact with central processor is down Two CPUs for redundancy and damage containment Each continuously monitored for faults When a fault is detected in one CPU, its work is moved to the other, and a restart is attempted
59. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 59/88 AXE Ongoing phone calls work without any CPUs running. CPUs being down may prevent making new phone calls.
60. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 60/88 AXE Ongoing phone calls work without any CPUs running. CPUs being down may prevent making new phone calls.
61. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 61/88 AXE Ongoing phone calls work without any CPUs running. CPUs being down may prevent making new phone calls. The local most stage, capable of handling 30 subscribers, can handle phone calls within that group in total isolation. There are no CPUs involved.
62. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 62/88 AXE The autonomy of parts makes the design highly fault tolerant All fault detection is aimed at HW malfunction
63. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 63/88 AXE The autonomy of parts makes the design highly fault tolerant All fault detection is aimed at HW malfunction SW was bug free?
64. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 64/88 What can we learn from this?
65. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 65/88 What can we learn from this? • Fault tolerance is architectural
66. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 66/88 What can we learn from this? • Fault tolerance is architectural • Divide your system into parts that can work in isolation and autonomously
67. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 67/88 What can we learn from this? • Fault tolerance is architectural • Divide your system into parts that can work in isolation and autonomously • Allow parts to fail to save the whole
68. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 68/88 What can we learn from this? • Fault tolerance is architectural • Divide your system into parts that can work in isolation and autonomously • Allow parts to fail to save the whole • Information about failures must reach the developers
69. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 69/88 What can we learn from this? • Fault tolerance is architectural • Divide your system into parts that can work in isolation and autonomously • Allow parts to fail to save the whole • Information about failures must reach the developers If you can, add a mechanism that “calls home”, or gives the user a way to send info back to you
70. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 70/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest
71. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 71/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest – Fail the transaction
72. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 72/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest – Fail the transaction – Drop the video stream
73. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 73/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest – Fail the transaction – Drop the video stream – Abort the download
74. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 74/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest – Fail the transaction – Drop the video stream – Abort the download How sure are you?
75. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 75/88 What if the problem is local? If you are certain that you found the inconsistency early enough, fail the offending part to save the rest – Fail the transaction – Drop the video stream – Abort the download How sure are you? These things are extremely difficult to test
76. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 76/88 Oh, and one neat trick! if (part_of_functionality.is_inconsistent()) { if (fork() == 0) { log(MAJOR, "Remove {} due to inconsistent state", part_of_functionality.identity()); abort(); } part_of_functionality.force_kill(); }
77. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 77/88 Oh, and one neat trick! if (part_of_functionality.is_inconsistent()) { if (fork() == 0) { log(MAJOR, "Remove {} due to inconsistent state", part_of_functionality.identity()); abort(); } part_of_functionality.force_kill(); } On a unix-like system, this creates a child process that is an exact copy of the parent.
78. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 78/88 Oh, and one neat trick! if (part_of_functionality.is_inconsistent()) { if (fork() == 0) { log(MAJOR, "Remove {} due to inconsistent state", part_of_functionality.identity()); abort(); } part_of_functionality.force_kill(); } Return value of 0 means we’re the child process.
79. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 79/88 Oh, and one neat trick! if (part_of_functionality.is_inconsistent()) { if (fork() == 0) { log(MAJOR, "Remove {} due to inconsistent state", part_of_functionality.identity()); abort(); } part_of_functionality.force_kill(); } Forces termination, and saves a snapshot of the process memory as a ‘core’ file.
80. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 80/88 Oh, and one neat trick! The ‘core’ file can be read in a debugger, as if you had stopped on a breakpoint. It shows you the current state, but it doesn’t show how you got there. To improve your chances of fixing bugs, include a simple ram-log, it can be a std::vector<std::string>, for the objects. Reading the log in the debugger shows you how you got there.
81. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 81/88 Summary
82. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 82/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”?
83. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 83/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”? • Resist the temptation to sweep bugs under the rug
84. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 84/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”? • Resist the temptation to sweep bugs under the rug • If you must crash, crash with actionable information
85. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 85/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”? • Resist the temptation to sweep bugs under the rug • If you must crash, crash with actionable information • Make sure there’s a way for that information to reach the developer
86. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 86/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”? • Resist the temptation to sweep bugs under the rug • If you must crash, crash with actionable information • Make sure there’s a way for that information to reach the developer • Resilience is architectural. You can only do so much as an individual developer
87. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 87/88 Summary • No generic right answers – Sacrifice execution to save data? – Continue running doing the wrong thing? – Revert to “safe mode”? • Resist the temptation to sweep bugs under the rug • If you must crash, crash with actionable information • Make sure there’s a way for that information to reach the developer • Resilience is architectural. You can only do so much as an individual developer • Every bit of cleverness you add to keep the system running, is added complexity that makes the system more prone to bugs
88. When to give up, and how? NDC{TechTown} 2025 © Björn Fahller @rollbear@fosstodon.org 88/88 Björn Fahller bjorn@fahller.se @rollbear @rollbear@fosstodon.org When to give up, and how? @rollbear.bsky.social https://speakerdeck.com/rollbear/when-to-give-up