Upgrade to Pro — share decks privately, control downloads, hide ads and more …

System-Theoretic Process Analysis for Software Safety

5aecc230f8e833c3e2cf2945c704c533?s=47 Stefan Wagner
October 01, 2020

System-Theoretic Process Analysis for Software Safety

A talk at the subgroup Software of ZVEI.

5aecc230f8e833c3e2cf2945c704c533?s=128

Stefan Wagner

October 01, 2020
Tweet

Transcript

  1. ZVEI UG Software, 2020-10-01 Stefan Wagner System-Theoretic Process Analysis for

    Software Safety
  2. You can copy, share and change, film and photograph, blog,

    live-blog and tweet this presentation given that you attribute it to its author and respect the rights and licences of its parts. based on slides by @SMEasterbrook und @ethanwhite
  3. Software systems need a new safety analysis approach!

  4. Questioning Assumptions System-Theoretic Process Analysis for Software Safety

  5. The primary safety problem in software- intensive systems is not

    software “ failure” but the lack of appropriate constraints on software behavior. – Nancy Leveson
  6. Assumption 1: Safety is increased by increasing system or component

    reliability. If components or systems do not fail, then accidents will not occur. from: Leveson. Engineering a Safer World. MIT Press, 2011
  7. A plane not taking off because of a software check

    is safe but not reliable.
  8. New Assumption 1: High reliability is neither necessary nor sufficient

    for safety. Assumption 1: Safety is increased by increasing system or component reliability. If components or systems do not fail, then accidents will not occur. from: Leveson. Engineering a Safer World. MIT Press, 2011
  9. Assumption 2: Most accidents are caused by operator error. Rewarding

    safe behavior and punishing unsafe behavior will eliminate or reduce accidents significantly. from: Leveson. Engineering a Safer World. MIT Press, 2011
  10. Hindsight Bias

  11. The influence of system design

  12. New Assumption 2: Operator behavior is a product of the

    environment in which it occurs. To reduce operator “error” we must change the environment in which the operator works. Assumption 2: Most accidents are caused by operator error. Rewarding safe behavior and punishing unsafe behavior will eliminate or reduce accidents significantly. from: Leveson. Engineering a Safer World. MIT Press, 2011
  13. Assumption 3: Probabilistic risk analysis based on event chains is

    the best way to assess and communicate safety and risk information. from: Leveson. Engineering a Safer World. MIT Press, 2011
  14. The Titanic Effect

  15. Design faults

  16. New Assumption 3: Risk and safety may be best understood

    and communicated in ways other than probabilistic risk analysis. Assumption 3: Probabilistic risk analysis based on event chains is the best way to assess and communicate safety and risk information. from: Leveson. Engineering a Safer World. MIT Press, 2011
  17. Assumption 4: Highly reliable software is safe. from: Leveson. Engineering

    a Safer World. MIT Press, 2011
  18. Software is reliable but unsafe when • The software correctly

    implements the requirements, but the specified behavior is unsafe from a system perspective. • The software requirements do not specify some particular behavior required for system safety (that is, they are incomplete). • The software has unintended (and unsafe) behavior beyond what is specified in the requirements. from: Leveson. Engineering a Safer World. MIT Press, 2011
  19. New Assumption 4: Highly reliable software is not necessarily safe.

    Increasing software reliability or reducing implementation errors will have little impact on safety. Assumption 4: Highly reliable software is safe. from: Leveson. Engineering a Safer World. MIT Press, 2011
  20. STAMP/STPA System-Theoretic Process Analysis for Software Safety

  21. Safety-guided design Hazard Analysis Design Decisions

  22. Safety is a control problem.

  23. Figure by Asim Abdulkhaleq

  24. Figure by Asim Abdulkhaleq

  25. Figure by Asim Abdulkhaleq

  26. Example: ACC – Step 1 Process Model Variables Control Actions

    Distance Driver Status Hazardous? Accelerate signal provided Distance < safe distance – Yes (H2) Distance > safe distance Driver on driver’s seat No Distance > safe distance Driver not on driver’s seat Yes (H3)
  27. Example: ACC – Step 2 "ACC provides accelerate command while

    driver is exiting vehicle. ACC would accelerate if ACC is engaged and the vehicle in front starts moving again within a short time period. During this time, the driver may try to exit the vehicle because the vehicle was stopped and they believed ACC was off or had timed out." This scenario was formulated by John Thomas.
  28. Verification of Unsafe Scenarios System-Theoretic Process Analysis for Software Safety

  29. Connection to Verification Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering

    Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2–11, 2015
  30. context table, the software safety requirements are refined. For example,

    the software safety requirement (S ) is refined as the ACC software controller should not provide accelerating signal more than the desired speed wh CC is in cruise mode and brake pedal is not pressed. Table 2. Examples of the context table of providing the control actions Control Actions Process Model Variables Hazardous? Distance Speed Brake ACC Mode Accelerate Signal provided Distance < safe distance Speed == desired speed applied Cruise No Distance < safe distance Speed > desired speed Not applied Cruise Yes (H2), SSR3-4 Distance < safe distance Speed > desired speed Not applied follow Yes (H1), SSR1 Once we have identified the software safety requirements, the process model and the unsafe scenarios of ea ntrol action using step 1, the safe behavior model can be constructed based on the process model. The safe behav del of the ACC software controller shows the relations between the process model variables (identified by ste d labeled with software safety requirements. Each transition in the safe behavior model is labeled with the synt ent [safety constraint]/control action. The event is a trigger of the transition and the safety constraint is a Bool ndition that must be true to transit to the next state. The control action describes the effect of the transition, such w the state variables are updated and what events are generated. For example, the transition t6 can be written: ntrolSpeed(currentspeed) [currentSpeed < desiredSpeed && distance > safeDistance && ACCMode ==cru & Brakestatus == Notapplied ]/ accelerateSpeed(currentspeed). 0 Asim Abdulkhaleq et al. / Procedia Engineering 128 ( 2015 ) 2 – 11 The transition t6 constrains the provision of the accelerate control action under the safety constraint derived by step 1 (Table 2). To formally verify the software safety requirements of each control action (refined from Table 2), first each software safety requirement should be formalized into a formal specification such as LTL or CTL to be able to verify them against the safe behavior model of the software controller during the verification phase. For example, the refined software safety requirement SSR1.3 can be expressed as the LTL formula: G ((currentSpeed < desiredSpeed && distance > safeDistance && ACCMode == cruise && Brakestatus == Notapplied) o accelerateSpeed). This formula means that the ACC software controller must always provide an acceleration signal when the current speed of the vehicle is less than the desired speed, there is no vehicle in the lane (distance > safe distance), and the brake pedal is not pressed when the ACC system is in cruise mode. Second, the safe behavior model needs to be Example Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2– 11, 2015 Semi-automatic transformation
  31. BDD for Safety Verification System-Theoretic Process Analysis for Software Safety

  32. 32 Focus on communication

  33. Based on: M. Cohn. Succeeding with Agile. Addison-Wesley, 2010 Test

    Code Refactor Test Code Refactor Test Code Refactor Passing acceptance test Refactor the test Customer acceptance Implement acceptance test(s) Failing acceptance tests Acceptance- test-driven development Test-driven development Identify conditions of satisfaction Select a user story
  34. 34 Behaviour-Driven Development (BDD) Developer Tester Product Owner Examples Scenarios

    Automated Tests
  35. Feature: Refund item Scenario: Jeff returns a faulty microwave Given

    Jeff has bought a microwave for $100 And he has a receipt When he returns the microwave Then Jeff should be refunded $100 Behaviour-Driven Development (BDD)
  36. 36 STPA-BDD

  37. 37 Example Unsafe Scenario from STPA Gherkin Scenario During auto-parking,

    the autonomous vehicle does not stop immediately when there is an obstacle up front. Given the autonomous vehicle is auto-parking When the ultrasonic sensor provides the feedback that the forward distance is smaller or equal to a threshold indicating that ther is an obstacle up front Then the autonomous vehicle stops immediately.
  38. 38 Experimental results many safety requirements can be written into

    test cases within a limited time slot. 25 But: Communication effectiveness is significantly different! The developers consider the safety requirements deeply and initiatively. The business analysts are more confident about the test cases. It becomes easier to identify conflicts in business rules and test cases. The business analysts are clear about the status of acceptance testing. The business analysts could spend less time on sprint-end acceptance tests.
  39. 39 Speeding it up with automation

  40. 40

  41. Tooling System-Theoretic Process Analysis for Software Safety

  42. 42

  43. Software systems need a new safety analysis approach!

  44. Prof. Dr. Stefan Wagner e-mail stefan.wagner@iste.uni-stuttgart.de phone +49 (0) 711

    685-88455 WWW www.iste.uni-stuttgart.de/ese Twitter prof_wagnerst ORCID 0000-0002-5256-8429 Institute of Software Engineering These slides are available at www.stefan-wagner.biz
  45. Pictures used in this slide deck Safety by GotCredit (https://flic.kr/p/qHCmfo,

    Got Credit) Scrum framework by Dr ian mitchell under CC BY-SA 4.0 (https:// en.wikipedia.org/wiki/Scrum_(software_development)#/media/ File:Scrum_Framework.png) Screenshot from http://agilemanifesto.org by Ward Cunningham sonar-dashboard by meedanphotos under CC BY 2.0 (https://flic.kr/p/8H1Sn7)