Slide 1
Slide 1 text
ZVEI UG Software,
2020-10-01
Stefan
Wagner
System-Theoretic
Process Analysis for
Software Safety
Slide 2
Slide 2 text
You can
copy, share and change,
film and photograph,
blog, live-blog and tweet
this presentation given that you attribute
it to its author and respect the rights and
licences of its parts.
based on slides by @SMEasterbrook und @ethanwhite
Slide 3
Slide 3 text
Software systems need a new safety analysis approach!
Slide 4
Slide 4 text
Questioning Assumptions
System-Theoretic Process Analysis for Software Safety
Slide 5
Slide 5 text
The primary safety problem in software-
intensive systems is not software “
failure” but the lack of appropriate
constraints on software behavior.
– Nancy Leveson
Slide 6
Slide 6 text
Assumption 1: Safety is increased by
increasing system or component reliability. If
components or systems do not fail, then
accidents will not occur.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 7
Slide 7 text
A plane not taking off because of
a software check is safe but not
reliable.
Slide 8
Slide 8 text
New Assumption 1: High reliability is neither
necessary nor sufficient for safety.
Assumption 1: Safety is increased by
increasing system or component reliability. If
components or systems do not fail, then
accidents will not occur.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 9
Slide 9 text
Assumption 2: Most accidents are caused by
operator error. Rewarding safe behavior and
punishing unsafe behavior will eliminate or
reduce accidents significantly.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 10
Slide 10 text
Hindsight Bias
Slide 11
Slide 11 text
The influence of
system design
Slide 12
Slide 12 text
New Assumption 2: Operator behavior is a
product of the environment in which it occurs.
To reduce operator “error” we must change
the environment in which the operator works.
Assumption 2: Most accidents are caused by
operator error. Rewarding safe behavior and
punishing unsafe behavior will eliminate or
reduce accidents significantly.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 13
Slide 13 text
Assumption 3: Probabilistic risk analysis
based on event chains is the best way to
assess and communicate safety and risk
information.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 14
Slide 14 text
The Titanic Effect
Slide 15
Slide 15 text
Design faults
Slide 16
Slide 16 text
New Assumption 3: Risk and safety may be
best understood and communicated in ways
other than probabilistic risk analysis.
Assumption 3: Probabilistic risk analysis
based on event chains is the best way to
assess and communicate safety and risk
information.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 17
Slide 17 text
Assumption 4: Highly reliable software is
safe.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 18
Slide 18 text
Software is reliable but unsafe when
• The software correctly implements the
requirements, but the specified behavior is
unsafe from a system perspective.
• The software requirements do not specify
some particular behavior required for
system safety (that is, they are
incomplete).
• The software has unintended (and unsafe)
behavior beyond what is specified in the
requirements.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 19
Slide 19 text
New Assumption 4: Highly reliable
software is not necessarily safe. Increasing
software reliability or reducing
implementation errors will have little impact
on safety.
Assumption 4: Highly reliable software is
safe.
from: Leveson. Engineering a Safer World. MIT Press, 2011
Slide 20
Slide 20 text
STAMP/STPA
System-Theoretic Process Analysis for Software Safety
Slide 21
Slide 21 text
Safety-guided design
Hazard
Analysis
Design
Decisions
Slide 22
Slide 22 text
Safety is a control problem.
Slide 23
Slide 23 text
Figure by Asim Abdulkhaleq
Slide 24
Slide 24 text
Figure by Asim Abdulkhaleq
Slide 25
Slide 25 text
Figure by Asim Abdulkhaleq
Slide 26
Slide 26 text
Example: ACC – Step 1
Process Model Variables
Control Actions Distance Driver Status Hazardous?
Accelerate
signal provided
Distance < safe
distance
– Yes (H2)
Distance > safe
distance
Driver on driver’s
seat
No
Distance > safe
distance
Driver not on
driver’s seat
Yes (H3)
Slide 27
Slide 27 text
Example: ACC – Step 2
"ACC provides accelerate command while driver is exiting vehicle.
ACC would accelerate if ACC is engaged and the vehicle in front
starts moving again within a short time period. During this time, the
driver may try to exit the vehicle because the vehicle was stopped and
they believed ACC was off or had timed out."
This scenario was formulated by John Thomas.
Slide 28
Slide 28 text
Verification of Unsafe
Scenarios
System-Theoretic Process Analysis for Software Safety
Slide 29
Slide 29 text
Connection to Verification
Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on
STPA. Procedia Engineering 128:2–11, 2015
Slide 30
Slide 30 text
context table, the software safety requirements are refined. For example, the software safety requirement (S
) is refined as the ACC software controller should not provide accelerating signal more than the desired speed wh
CC is in cruise mode and brake pedal is not pressed.
Table 2. Examples of the context table of providing the control actions
Control
Actions
Process Model Variables Hazardous?
Distance Speed Brake ACC
Mode
Accelerate
Signal
provided
Distance < safe distance Speed == desired speed applied Cruise No
Distance < safe distance Speed > desired speed Not applied Cruise Yes (H2), SSR3-4
Distance < safe distance Speed > desired speed Not applied follow Yes (H1), SSR1
Once we have identified the software safety requirements, the process model and the unsafe scenarios of ea
ntrol action using step 1, the safe behavior model can be constructed based on the process model. The safe behav
del of the ACC software controller shows the relations between the process model variables (identified by ste
d labeled with software safety requirements. Each transition in the safe behavior model is labeled with the synt
ent [safety constraint]/control action. The event is a trigger of the transition and the safety constraint is a Bool
ndition that must be true to transit to the next state. The control action describes the effect of the transition, such
w the state variables are updated and what events are generated. For example, the transition t6
can be written:
ntrolSpeed(currentspeed) [currentSpeed < desiredSpeed && distance > safeDistance && ACCMode ==cru
& Brakestatus == Notapplied ]/ accelerateSpeed(currentspeed).
0 Asim Abdulkhaleq et al. / Procedia Engineering 128 ( 2015 ) 2 – 11
The transition t6
constrains the provision of the accelerate control action under the safety constraint derived by step
1 (Table 2). To formally verify the software safety requirements of each control action (refined from Table 2), first
each software safety requirement should be formalized into a formal specification such as LTL or CTL to be able to
verify them against the safe behavior model of the software controller during the verification phase. For example, the
refined software safety requirement SSR1.3 can be expressed as the LTL formula:
G ((currentSpeed < desiredSpeed && distance > safeDistance && ACCMode == cruise && Brakestatus ==
Notapplied) o accelerateSpeed).
This formula means that the ACC software controller must always provide an acceleration signal when the current
speed of the vehicle is less than the desired speed, there is no vehicle in the lane (distance > safe distance), and the
brake pedal is not pressed when the ACC system is in cruise mode. Second, the safe behavior model needs to be
Example
Abdulkhaleq, Wagner, Leveson. A Comprehensive Safety Engineering Approach for Software-Intensive Systems Based on STPA. Procedia Engineering 128:2–
11, 2015
Semi-automatic transformation
Slide 31
Slide 31 text
BDD for Safety Verification
System-Theoretic Process Analysis for Software Safety
Slide 32
Slide 32 text
32
Focus on
communication
Slide 33
Slide 33 text
Based on: M. Cohn. Succeeding with Agile. Addison-Wesley, 2010
Test
Code
Refactor
Test
Code
Refactor
Test
Code
Refactor
Passing
acceptance
test
Refactor
the test
Customer
acceptance
Implement
acceptance
test(s)
Failing
acceptance
tests
Acceptance-
test-driven
development
Test-driven
development
Identify
conditions of
satisfaction
Select a
user story
Slide 34
Slide 34 text
34
Behaviour-Driven
Development (BDD)
Developer
Tester
Product Owner
Examples
Scenarios
Automated Tests
Slide 35
Slide 35 text
Feature: Refund item
Scenario: Jeff returns a faulty microwave
Given Jeff has bought a microwave for $100
And he has a receipt
When he returns the microwave
Then Jeff should be refunded $100
Behaviour-Driven Development (BDD)
Slide 36
Slide 36 text
36
STPA-BDD
Slide 37
Slide 37 text
37
Example
Unsafe Scenario from STPA Gherkin Scenario
During auto-parking, the autonomous
vehicle does not stop immediately
when there is an obstacle up front.
Given the autonomous vehicle is auto-parking
When the ultrasonic sensor provides the
feedback that the forward distance is smaller or
equal to a threshold indicating that ther is an
obstacle up front
Then the autonomous vehicle stops immediately.
Slide 38
Slide 38 text
38
Experimental results
many safety requirements can be written into test cases within a limited time slot.
25
But: Communication effectiveness is
significantly different!
The developers consider the safety
requirements deeply and initiatively.
The business analysts are more confident
about the test cases.
It becomes easier to identify conflicts in
business rules and test cases.
The business analysts are clear about the
status of acceptance testing.
The business analysts could spend less time
on sprint-end acceptance tests.
Slide 39
Slide 39 text
39
Speeding it up with automation
Slide 40
Slide 40 text
40
Slide 41
Slide 41 text
Tooling
System-Theoretic Process Analysis for Software Safety
Slide 42
Slide 42 text
42
Slide 43
Slide 43 text
Software systems need a new safety analysis approach!
Slide 44
Slide 44 text
Prof. Dr. Stefan Wagner
e-mail [email protected]
phone +49 (0) 711 685-88455
WWW www.iste.uni-stuttgart.de/ese
Twitter prof_wagnerst
ORCID 0000-0002-5256-8429
Institute of Software Engineering
These slides are available at www.stefan-wagner.biz
Slide 45
Slide 45 text
Pictures used in this slide deck
Safety by GotCredit (https://flic.kr/p/qHCmfo, Got Credit)
Scrum framework by Dr ian mitchell under CC BY-SA 4.0 (https://
en.wikipedia.org/wiki/Scrum_(software_development)#/media/
File:Scrum_Framework.png)
Screenshot from http://agilemanifesto.org by Ward Cunningham
sonar-dashboard by meedanphotos under CC BY 2.0 (https://flic.kr/p/8H1Sn7)