Lecture 4 - Foundations of Software Testing

1 Build Software to Test Software exactpro.com
Lecture 4
Foundations of Software Testing
Software Testing for
Complex Intelligent Systems
and Autonomous Vehicles
Online Course

View Slide

Lecture structure
Sources of information about testing
What is testing?
What is quality, errors, defects, failures?
What testing is not: QA, unit testing, BDD, bad testing
Principles and types of testing
Testing objectives; what constitutes good testing
Testing Components: processes, platforms, people
The testing process: phases and results
Is it the testers' fault that failures occur in production

View Slide

Exactpro
The company was founded on
Tester Day, September 9,
2009
The company’s
headquarters are in
London
Moscow Kostroma Saratov Obninsk
Tbilisi London New York

View Slide

What We Do
Our company specializes in development and implementation of high
load market infrastructures as well as systems that work with ﬁnancial
tools/platforms. Our solutions are used in more than 20 countries all
over the world.

View Slide

Sources of information
- Absence-of-errors fallacy
and other principles of
software testing.
- Types and levels of
software testing.
- Understanding of test
objectives and
methodologies in
different schools of
thought in the software
testing domain.

View Slide

Sources of Information
https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

View Slide

https://youtu.be/jZitXMQaXvE
Watch
Yegor Bugayenko

View Slide

https://ingophilipp.medium.com/excellent-software-testing-6e0fd8f3e11e

View Slide

http://cartoontester.blogspot.com/
Andy Glover

View Slide

Software Testing: Deﬁnition
● Testing is about extracting information
about software quality
● Testing is pain
● Testing is relentless learning
● Testing is about processes, platforms,
people

View Slide

West World
TEST WORLD: REBOOT

View Slide

Ray Dalio
PAIN + REFLECTION = PROGRESS

View Slide

Ray Dalio

View Slide

How We Get Information

View Slide


View Slide

https://blogs.scientificamerican.com/observations/the-line-between-science-and-journali
sm-is-getting-blurry-again/
● Journalism is communication of 'what's new'. A journalist is
anyone who can say "I’m there, you’re not, let me tell you about
it."
● Science is communication of 'how the world works'. A scientist is
anyone who can say "I understand something about the world,
you don't, let me explain it to you".
● Neither deﬁnition necessitates that what they say is True, just
what they know to the best of their ability and understanding.

View Slide


View Slide

Quality
Quality is value to some people
Quality is the degree to which a component or system satisfies
the stated and implied needs of its various stakeholders.
Quality is inherently subjective.
Different stakeholders will perceive the same product as
having different levels of quality.

View Slide

What is a Bug?
Bug is a discrepancy between expectations
and reality
An attribute of a software product that reduces its value to a
stakeholder or increases its value to a competitor.
May or may not be a coding error or a functional error
Design errors are bugs too.

View Slide

What is a Bug?
Error – a human action that produces an
incorrect result
Defect (bug) – an imperfection or deficiency
in a work product where it does not meet its
requirements or specifications
Failure – an event in which a component or
system does not perform a required function
within specified limits

View Slide

What is a Bug?
65,586 m/ h (~65.6 km/h)
error defect/bug failure

View Slide

What is a Bug?
Breaking path

View Slide

What is a Bug?
Breaking path
https://cordis.europa.eu/arti
cle/id/19509-ariane-5-explo
sion-caused-by-fault-in-mai
n-engine-cooling-system

View Slide

https://www.snopes.com/fact-check/cone-of-silence
PASS
FAIL
PASS
+ =
=
=
+
+
What is a Bug?

View Slide

What is Software Testing?
Software Testing is an
● empirical
● technical
● investigation
● conducted to provide stakeholders
● with information
● about the quality
● of the product or service under test
Software Testing is exploring software
with the intent of ﬁnding bugs

View Slide

Watch
“Software is Eating the World.”

View Slide

“Software is Eating the World.”
http://www3.weforum.org/docs/WEF_
The_Global_Risks_Report_2021.pdf

View Slide

Software Testing is not Quality Assurance
Testers:
Get Out of the Quality Assurance Business
Quality
Assurance
Quality Control
Testing
Code
review
Unit
testing
Audit
Process
Improvements
Change
management
Early reviews
and testing

View Slide

Software Testing is an Information Service

View Slide

Testing is not Quality Assurance
Testers:
Get Out of the
Quality Assurance
Business
https://www.developsense.com/presentations/2010-06-TestersGetO
utOfTheQABusiness.pdf

View Slide

Unit Testing
Development of unit tests allows you
to isolate individual parts of the
program and test their performance -
creating tests, executing them and
obtaining reports using open source
frameworks Pytest and PyUnit.

View Slide

BDD
Studying the Behavior Driven
Development (BDD) software
development methodology
Using the Gherkin in the requirements
deﬁnition process, use cases, and
acceptance testing

View Slide

Software Testing Not Software Testing

View Slide

Examples of Obtaining Information

View Slide

Examples of obtaining and disseminating
information

View Slide

Bad Testing
https://youtu.be/2S--Ge3c-yA

View Slide

Software Testing is not...
https://mrslavchev.com/2016/03/28/software-testing-not/

View Slide


View Slide

Testing Principles
Principle 1: Testing shows the presence of defects, not their absence
Principle 2: Exhaustive testing is impossible
Principle 3: Early testing saves time and money
Principle 4: Defects cluster together
Principle 5: Beware of the pesticide paradox
Principle 6: Testing is context dependent
Principle 7: Absence-of-errors is a fallacy

View Slide

Testing shows the presence of defects,
not their absence

View Slide

Exhaustive testing is impossible

View Slide

Autonomous Vehicle Testing

View Slide

LGSVL Road Simulator
- Introduction to the multi-agent
road simulator developed by LG's
Research and Development
division based on the
cross-platform environment for
creating video games - Unity.
- Overview of installation, launch,
and user interface.

View Slide

Pareto Principle

View Slide

Beware of the Pesticide Paradox

View Slide

Context Driven Testing
Consider:
● Software development lifecycle model and project
methodologies being used
● Test levels and test types being considered
● Product and project risks
● Business domain
● Operational constraints, including but not limited to:
a. Budgets and resources
b. Timescales
c. Complexity
d. Contractual and regulatory requirements
● Organizational policies and practices
● Required internal and external standards

View Slide

Context Driven Testing
The Seven Basic Principles of the
Context-Driven School
1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s
context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t
work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the
entire project, are we able to do the right things at the right times to
eﬀectively test our products.
https://context-driven-testing.com/

View Slide


View Slide

Absence-of-errors
is a fallacy

View Slide

Verification vs. Validation
Verification – checks whether the system meets the
specified requirements
Validation – checks whether the system will meet user and
other stakeholder needs
● https://techcrunch.com/2018/06/22/uber-safety-driver-of-fatal-self-driving-crash-was-watchin
g-hulu-not-the-road
● https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf

View Slide

https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf

View Slide

Types of Software Testing
● Functional testing — WHAT
● Non-functional testing — HOW
● Testing with access to the source code —
White Box

View Slide

Software Testing at
the conﬂuence
Functional Testing Nonfunctional Testing

View Slide

Black Box Testing vs. White Box Testing

View Slide

Testing Components

View Slide

● Test planning – involves deﬁning the objectives of testing and the
approach for meeting those objectives within project constraints
and contexts
● Test monitoring and control – compares actual progress against
the plan, takes actions which are necessary to meet the mission
and objectives of the project, and/or adjust the plan
● Test analysis – determines ‘what to test’
● Test design – addresses the question ‘how to test’
● Test implementation – asks ‘do we now have everything in place
to run the tests?
● Test execution – runs tests
● Test completion – summarizes the results of testing
Testing Components

View Slide

Test Objectives
1. To prevent defects by evaluating work products such as requirements,
user stories, design, and code
2. To verify whether all specified requirements have been fulfilled
3. To check whether the test object is complete and validate if it works as
the users and other stakeholders expect
4. To build confidence in the level of quality of the test object
5. To find defects and failures thus reduce the level of risk of inadequate
software quality
6. To provide sufficient information to stakeholders to allow them to
make informed decisions, especially regarding the level of quality of
the test object
7. To comply with contractual, legal, or regulatory requirements or
standards, and/or to verify the test object’s compliance with such
requirements or standards

View Slide


View Slide

Bad Testing
Doctor Fauci

View Slide

Testers vs. Activists

View Slide

Test Objectives
● To prevent defects by evaluate work products such as requirements,
user stories, design, and code
● To verify whether all specified requirements have been fulfilled
● To check whether the test object is complete and validate if it works as
the users and other stakeholders expect
● To build confidence in the level of quality of the test object
● To find defects and failures thus reduce the level of risk of inadequate
software quality
● To provide sufficient information to stakeholders to allow them to
make informed decisions, especially regarding the level of quality of
the test object
● To comply with contractual, legal, or regulatory requirements or
standards, and/or to verify the test object’s compliance with such
requirements or standards

View Slide

2021 IEEE Autonomous Driving AI Test Challenge
http://av-test-challenge.org/

View Slide

Good Testing

View Slide

Test Objectives

View Slide

Testing is not Quality Assurance
Testers:
Get Out of the Quality Assurance Business
Quality
Assurance
Quality Control
Testing
Code
review
Unit
testing
Audit
Process
Improvements
Change
management
Early reviews
and testing

View Slide

Good Testing

View Slide

Good Testing
● Ownership – taking care of your company. Treat others the way you
would like to be treated. Take responsibility for company's needs.
Remember: responsibility is power. It is not granted. It is taken. Extreme
ownership is the way to lead and win.
● Deep Impact – exitus acta probat. The outcome justifies the effort. Make
sure that your work results in a valuable contribution. If you want to go
fast, go alone. If you want to go far, go together. Involve others to achieve
scalability and increase the impact of our work.
● Hard Work – no pain, no gain. Sometimes putting in long hours does not
help achieve any result. We do not measure time spent in the office.
Having said that, we've never observed extraordinary results without hard
work and deliberate effort.

View Slide

● Learning – software testing is relentless learning. Change is the end
result of all true learning. The beautiful thing about learning is that
nobody can take it away from you. The Mistakes and problems are
inevitable. However, it is unacceptable not to learn from them. The
best way to learn something is to teach it.
● Agility – the only constant is change. We embrace the values
reﬂected in the Manifesto for Agile Software Development. Agility
allows us to turn learning into action and action into change. From its
inception, Exactpro has been able to adapt to change. It is important
to keep this ability as we scale up our business.
● Communication – the medium is the message. A relationship is
based on communication. We need to make sure others understand
us and we understand them.
● Objectivity – you have nothing to fear from knowing the truth. It is at
the core of our business. Our main service is to build software to test
software and provide objective information about it to our clients.
Our business beneﬁts from openness. Everyone in our company has
the right to understand what makes sense, and no one has the right
to hold a critical opinion without speaking up.
Good Testing

View Slide

Thank you
Exactpro Systems Exactpro Vlog

View Slide

Lecture 4 - Foundations of Software Testing

Lecture 4 - Foundations of Software Testing

Exactpro
PRO

More Decks by Exactpro

Other Decks in Technology

Featured

Transcript

Lecture 4 - Foundations of Software Testing

Lecture 4 - Foundations of Software Testing

Exactpro PRO

More Decks by Exactpro

Other Decks in Technology

Featured

Transcript

Exactpro
PRO