DLRG Software Safety, Garching, 2016-10-05 Stefan Wagner System-Theoretic Analysis and Verification of Software Safety 

You can copy, share and change, film and photograph, blog, live-blog and tweet this presentation given that you attribute it to its author and respect the rights and licences of its parts. based on slides by @SMEasterbrook und @ethanwhite 

Software systems need a new safety analysis approach! 

Assumption 1: Safety is increased by increasing system or component reliability. If components or systems do not fail, then accidents will not occur. from: Leveson. Engineering a Safer World. MIT Press, 2011 

A plane not taking off because of a software check is safe but not reliable. 

New Assumption 1: High reliability is neither necessary nor sufficient for safety. Assumption 1: Safety is increased by increasing system or component reliability. If components or systems do not fail, then accidents will not occur. from: Leveson. Engineering a Safer World. MIT Press, 2011 

Assumption 2: Most accidents are caused by operator error. Rewarding safe behavior and punishing unsafe behavior will eliminate or reduce accidents significantly. from: Leveson. Engineering a Safer World. MIT Press, 2011 

Hindsight Bias 

The influence of system design 

New Assumption 2: Operator behavior is a product of the environment in which it occurs. To reduce operator “error” we must change the environment in which the operator works. Assumption 2: Most accidents are caused by operator error. Rewarding safe behavior and punishing unsafe behavior will eliminate or reduce accidents significantly. from: Leveson. Engineering a Safer World. MIT Press, 2011 

Assumption 3: Probabilistic risk analysis based on event chains is the best way to assess and communicate safety and risk information. from: Leveson. Engineering a Safer World. MIT Press, 2011 

The Titanic Effect 

Design faults 

New Assumption 3: Risk and safety may be best understood and communicated in ways other than probabilistic risk analysis. Assumption 3: Probabilistic risk analysis based on event chains is the best way to assess and communicate safety and risk information. from: Leveson. Engineering a Safer World. MIT Press, 2011 

Assumption 4: Highly reliable software is safe. from: Leveson. Engineering a Safer World. MIT Press, 2011 

Software is reliable but unsafe when • The software correctly implements the requirements, but the specified behavior is unsafe from a system perspective. • The software requirements do not specify some particular behavior required for system safety (that is, they are incomplete). • The software has unintended (and unsafe) behavior beyond what is specified in the requirements. from: Leveson. Engineering a Safer World. MIT Press, 2011 

New Assumption 4: Highly reliable software is not necessarily safe. Increasing software reliability or reducing implementation errors will have little impact on safety. Assumption 4: Highly reliable software is safe. from: Leveson. Engineering a Safer World. MIT Press, 2011 

The primary safety problem in software- intensive systems is not software “failure” but the lack of appropriate constraints on software behavior. – Nancy Leveson 

System Theory 

STAMP Control Process Behavior Inadequate Enforcement Hazardous Process Hierarchical Safety Control Structure Hazardous System State Inadequate of Safety Constraints on from: Leveson. Engineering a Safer World. MIT Press, 2011 

Problem Reports Operating Procedures Revised operating procedures Whistleblowers Change reports Certification Info. Manufacturing Management Safety Reports Policy, stds. Work Procedures safety reports audits work logs Manufacturing inspections Hazard Analyses Documentation Design Rationale Company Resources Standards Safety Policy Operations Reports Management Operations Resources Standards Safety Policy Incident Reports Risk Assessments Status Reports Safety−Related Changes Test reports Test Requirements Standards Review Results Safety Constraints Implementation Hazard Analyses Progress Reports Safety Standards Hazard Analyses Progress Reports Design, Work Instructions Change requests Audit reports Problem reports Maintenance Congress and Legislatures Legislation Company Congress and Legislatures Legislation Legal penalties Certification Standards Regulations Government Reports Lobbying Hearings and open meetings Accidents Case Law Legal penalties Certification Standards Regulations Accidents and incidents Government Reports Lobbying Hearings and open meetings Accidents Whistleblowers Change reports Maintenance Reports Operations reports Accident and incident reports Change Requests Performance Audits Hardware replacements Software revisions Hazard Analyses Operating Process Case Law SYSTEM DEVELOPMENT Insurance Companies, Courts User Associations, Unions, Industry Associations, Government Regulatory Agencies Management Management Management Project Government Regulatory Agencies Industry Associations, User Associations, Unions, Documentation and assurance and Evolution SYSTEM OPERATIONS Insurance Companies, Courts Physical Actuator(s) Incidents Operating Assumptions Process Controller Automated Human Controller(s) Sensor(s) from: Leveson. Engineering a Safer World. MIT Press, 2011 

Problem Reports Operating Procedures Revised erating procedures Audit reports Problem reports Change Requests rdware replacements Software revisions Operating Process Physical Actuator(s) Incidents Operating Assumptions Process Controller Automated Human Controller(s) Sensor(s) 

Safety is a control problem. 

System-Theoretic Process Analysis (STPA) 

Safety-guided design Hazard Analysis Design Decisions 

Controller 2 process changes, (Flaws in creation, or adaptation) Sensor Component failures Incorrect or no Feedback Delays missing feedback Inadequate or control action ineffective or missing Inappropriate, Delayed operation Controller Actuator Controlled Process extermal information Control input or wrong or missing Changes over time Measurement Feedback delays information provided inaccuracies Inadequate Operation operation Inadequate 1 4 4 3 Algorithm Inadequate Control 2 Process Model inconsistent, incomplete, or incorrect 3 Unidentified or out−of−range Process output contributes to disturbance system hazard Process input missing or wrong Conflicting control actions incorrect modification Unsafe inputs from higher levels Unsafe algorithms Wrong model of the process Wrong process execution 

Figure by Asim Abdulkhaleq 

Figure by Asim Abdulkhaleq 

Figure by Asim Abdulkhaleq 

Example en the current distance is less than or equal to the safe distance, and the current speed is greater than the desi ed and the brake is not pressed is hazardous (i.e. it leads to the hazard H2). Based on the evaluation of each row context table, the software safety requirements are refined. For example, the software safety requirement (S ) is refined as the ACC software controller should not provide accelerating signal more than the desired speed wh CC is in cruise mode and brake pedal is not pressed. Table 2. Examples of the context table of providing the control actions Control Actions Process Model Variables Hazardous? Distance Speed Brake ACC Mode Accelerate Signal provided Distance < safe distance Speed == desired speed applied Cruise No Distance < safe distance Speed > desired speed Not applied Cruise Yes (H2), SSR3-4 Distance < safe distance Speed > desired speed Not applied follow Yes (H1), SSR1 Once we have identified the software safety requirements, the process model and the unsafe scenarios of e ntrol action using step 1, the safe behavior model can be constructed based on the process model. The safe behav del of the ACC software controller shows the relations between the process model variables (identified by ste d labeled with software safety requirements. Each transition in the safe behavior model is labeled with the synt nt [safety constraint]/control action. The event is a trigger of the transition and the safety constraint is a Bool ndition that must be true to transit to the next state. The control action describes the effect of the transition, such w the state variables are updated and what events are generated. For example, the transition t6 can be written: Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2–11, 2015 

STPA for software-intensive systems 

Connection to Verification Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2–11, 2015 

context table, the software safety requirements are refined. For example, the software safety requirement (S ) is refined as the ACC software controller should not provide accelerating signal more than the desired speed wh CC is in cruise mode and brake pedal is not pressed. Table 2. Examples of the context table of providing the control actions Control Actions Process Model Variables Hazardous? Distance Speed Brake ACC Mode Accelerate Signal provided Distance < safe distance Speed == desired speed applied Cruise No Distance < safe distance Speed > desired speed Not applied Cruise Yes (H2), SSR3-4 Distance < safe distance Speed > desired speed Not applied follow Yes (H1), SSR1 Once we have identified the software safety requirements, the process model and the unsafe scenarios of e ntrol action using step 1, the safe behavior model can be constructed based on the process model. The safe behav del of the ACC software controller shows the relations between the process model variables (identified by ste d labeled with software safety requirements. Each transition in the safe behavior model is labeled with the synt nt [safety constraint]/control action. The event is a trigger of the transition and the safety constraint is a Bool ndition that must be true to transit to the next state. The control action describes the effect of the transition, such w the state variables are updated and what events are generated. For example, the transition t6 can be written: ntrolSpeed(currentspeed) [currentSpeed < desiredSpeed && distance > safeDistance && ACCMode ==cru & Brakestatus == Notapplied ]/ accelerateSpeed(currentspeed). 0 Asim Abdulkhaleq et al. / Procedia Engineering 128 ( 2015 ) 2 – 11 The transition t6 constrains the provision of the accelerate control action under the safety constraint derived by step 1 (Table 2). To formally verify the software safety requirements of each control action (refined from Table 2), first each software safety requirement should be formalized into a formal specification such as LTL or CTL to be able to verify them against the safe behavior model of the software controller during the verification phase. For example, the refined software safety requirement SSR1.3 can be expressed as the LTL formula: G ((currentSpeed < desiredSpeed && distance > safeDistance && ACCMode == cruise && Brakestatus == Notapplied) o accelerateSpeed). This formula means that the ACC software controller must always provide an acceleration signal when the current speed of the vehicle is less than the desired speed, there is no vehicle in the lane (distance > safe distance), and the brake pedal is not pressed when the ACC system is in cruise mode. Second, the safe behavior model needs to be Example Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2– 11, 2015 Semi-automatic transformation 

Software systems fit well to system-theoretic analysis! Tool support: www.xstampp.de European STAMP Workshop: www.stamp-workshop.eu 

Prof. Dr. Stefan Wagner e-mail [email protected] phone +49 (0) 711 685-88455 WWW www.iste.uni-stuttgart.de/se Twitter prof_wagnerst ORCID 0000-0002-5256-8429 Institute of Software Technology 

Pictures used in this slide deck Safety by GotCredit (https://flic.kr/p/qHCmfo, Got Credit) Unsafe Area by Jerome Vial under CC BY-SA 2.0 (https://flic.kr/p/71Kpk7) Airplane by StockSnap (https://pixabay.com/de/flugzeug-reisen-transport- airasia-926744/) Swiss Cheese Model by Davidmack - Own work, CC BY-SA 3.0, (https:// commons.wikimedia.org/w/index.php?curid=31679759) Die Titanic im Hafen von Southhampton - Gemeinfrei (https:// commons.wikimedia.org/w/index.php?curid=19027661) Pisa by Aaron Kreis (https://flic.kr/p/wzEw5K) Looking back by Susanne Nilsson (https://flic.kr/p/niBFZo) Concorde Cockpit by Dr. Richard Murray (https://commons.wikimedia.org/wiki/ File:Concorde_Cockpit_-_geograph.org.uk_-_1357498.jpg)