Lecture 7: Software defects and metrics

1 Build Software to Test Software exactpro.com
Lecture 7
Software Defects and Metrics
Software Testing for
Complex Intelligent Systems
and Autonomous Vehicles
Online Course

View Slide

Lecture Structure
Quantifying software quality
Who needs metrics and why
Metrics: deﬁnition and validity criteria
Metrics in testing
Quantitative indicators related to defects
Metrics and decision making

View Slide

Reading Materials
Cem Kaner, Walter P. Bond
Software Engineering Metrics:
What Do They Measure and How Do We Know?

View Slide

Reading Materials
Cem Kaner,
Cem Kaner,
Michael Bolton,
Michael Bolton,
James Bach,
Lisa Crispin,
Gojko Adzic,
On the Quality of Qualitative Measures
Don’t Use Bug Counts to Measure Testers
Meaningful Metrics
Three Kinds of Measurement and Two Ways to
Use Them
Assess Quality, Don’t Measure It
STAREAST: Agile testing and defect tracking
Bug statistics are a waste of time

View Slide

Reading Materials
ISTQB Foundation Level Syllabus 2018
ISTQB Advanced Level Test Manager Syllabus 2012

View Slide

Что почитать
Список проектных метрик:
https://software-testing.ru/images/stories/library/
qaa-3-metrics.pdf
Наталья Руколь Самый полный список метрик
тестирования на русском языке:
https://software-testing.ru/library/testing/test-manag
ement/3566-metrics

View Slide

Reading Materials
https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-
assurance-success/

View Slide

Good vs Bad
● Goodness, by deﬁnition, is always rooted in some person’s feeling
● Goodness is at least a three-way relationship between people,
product and context
● Goodness is a phenomenon produced by the mysterious interaction
of many variables
● Goodness judgments tend to be socially risky
https://www.satisﬁce.com/blog/archives/487091

View Slide

Metric: Deﬁnition
http://www.kaner.com/pdfs/metrics2004.pdf
Measurement is “the assignment of numbers to
objects or events according to rule.”
S. S. Stevens, "On the Theory of Scales of Measurement,"
Science, vol. 103, pp. 677-680, 1946.
Measurement is "the act or process of assigning a
number or category to an entity to describe an
attribute of that entity."
IEEE Std. 1061-1998, Standard for a Software Quality
Metrics Methodology (1998).

View Slide

Metric: Deﬁnition
Software quality metric is "a function whose inputs are
software data and whose output is a single numerical
value that can be interpreted as the degree to which
software possesses a given attribute that aﬀects its
quality."
IEEE Std. 1061-1998, Standard for a Software Quality
Metrics Methodology (1998).

View Slide

Who Needs
Metrics
Typical test manager tasks may
include:
…
- Introduce suitable metrics for
measuring test progress and
evaluating the quality of the
testing and the product
Section 5.1.2 Tasks of a Test Manager
and Tester

View Slide

“A management cliché says that
what gets measured gets done.
In addition, what does not get
measured does not get done,
because what does not get
measured is easy to ignore. ”
Section 2.6 Deﬁning and
Using Test Metrics
Who Needs
Metrics

View Slide

Quality vs Quantity
Cem Kaner:
“There are too many simplistic metrics that don't
capture the essence of whatever it is that they are
supposed to measure. There are too many uses of
simplistic measures that don't even recognize what
attributes are supposedly being measured. Starting
from a detailed analysis of the task or attribute under
study might lead to more complex, and more
qualitative, metrics, but we believe that it will also
leads to more meaningful and therefore more useful
data.”

View Slide

Quality vs Quantity
Michael Bolton:
https://twitter.com/michaelbolton/status/1151358273498570754

View Slide

Validity criteria
for quantitative
metrics
- Correlation
- Consistency
- Tracking
- Predictability
- Discriminative power
- Reliability

View Slide

Validity criteria
for qualitative
metrics
- Credibility
- Transferability
- Dependability
- Conﬁrmability
http://kaner.com/?p=409

View Slide

Cem Kaner: Construct validity

View Slide

Cem Kaner: Metrics Validation Framework
1. The purpose of the measure
2. The scope of the measurement
3. The attribute to be measured
4. The natural scale for the attribute
5. The natural variability of the attribute
6. The metric/instrument that measures the attribute
7. The natural scale of the metric/instrument
8. The natural variability of measurements made with this instrument
9. The relationship between the attribute and the metric/instrument
10. The natural and foreseeable side eﬀects of using this instrument to
measure this attribute

View Slide

1. Metric’s purpose
- Helping you evaluate your work, identifying areas for
improvement
- Assessing project status
- Evaluating the eﬀectiveness of the team
- Informing about product features
- Informing external stakeholders

View Slide

2. The Scope of the Metric Application
- one specialist's particular methodology
- one team project
- team projects during the year
- the organization's projects over the last 10 years
A metric that works well locally might fail globally.

View Slide

3. Attribute
- The logical structure behind the measurement
- The nature of the phenomenon we describe with a metric

View Slide

4 Attribute Rating Scale
What is the natural scale of the attribute
behind the metric?
Scale types:
- nominal
- ordinal
- interval
- ratio

View Slide

5. The Natural Variability of the Attribute
What are the intrinsic sources and degrees of variation of
the attribute we are trying to measure?

View Slide

6. Choosing a Metric
- Counting (human or machine)
- Matching (by a human, an algorithm, or some other
device
- Comparison (with the help of a person, an algorithm, or
some other device)
- Calculating time (using a computer, stopwatch, or some
external automated device, or by calculating the
diﬀerence between two timestamps)

View Slide

7. The Natural Scale of the Metric
-

View Slide

8. Measurement Variability
What is the natural variability of readings from this
instrument?

View Slide

9. The Attribute and its Metric
Which model links the value of a metric to the value of an
attribute?

View Slide

10. The Other Side of Metrics
If we try to improve the outcome being measured, what eﬀect
will we have on this attribute?
“People are good at tailoring their behavior to things that they
are measured against.”
http://kaner.com/pdfs/bugcount.pdf

View Slide

Quality vs Quantity
Michael Bolton:
"But management wants numbers!" I doubt that. Management
almost certainly wants *to know things* - and from testers,
knowledge about the status of the product and problems that
threaten its value. Numbers might help to illustrate a story.
They don't, can't TELL it. Words can.”
Cem Kaner:
“Qualitative measurements tell a story (or a bunch of
stories). The skilled qualitative researcher relies on
transparency in methods and data to tell persuasive
stories… Decision-makers rely on your stories to make
important decisions.”

View Slide

Статья Натальи
Руколь
Самый полный
список метрик
тестирования
на русском языке
https://software-testing.ru/library/testing/t
est-management/3566-metrics
Для чего нужны метрики?
- оценка прогресса
- промежуточные замеры
- поиск проблем
- числовые обоснования

View Slide

Справочник по метрикам
https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

View Slide

Качество тестирования

View Slide

Проектное планирование

View Slide

Качество продукта

View Slide

Эффективность тестирования

View Slide

Essential Metrics
https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-
assurance-success/

View Slide

ISTQB Advanced
Level Test
Manager
Syllabus
https://software-testing.ru/images/stories/li
brary/qaa-3-metrics.pdf
4 categories of testing
metrics:
- Project metrics
- Product metrics
- Process metrics
- People metrics

View Slide

Defects and Metrics
https://www.amazon.com/
Why-Does-Software-Cost-Much/dp/093263334X
"I can only think of one
metric that is worth collecting
now and forever: defect
count." [p. 15]
Tom DeMarco, "Mad About
Measurement," in Why Does
Software Cost So Much? New York:
Dorset House, 1995, pp. 13-44.

View Slide

Defects and Metrics

View Slide

Bug Tracking
“From a lean point of view, a DTS is a queue of rework,
which translates into waste.”
“a DTS isn’t a good way to communicate”
“You don’t need a DTS in order to prioritize defects.”
https://searchsoftwarequality.techtarget.com/tip/STAREAST-Agile-testing-and-d
efect-tracking

View Slide

Bug Tracking
https://gojko.net/2011/05/17/bug-statistics-are-a-waste-of-time/
Gojko Adzic:
“defect tracking tools are placebo, giving
people a warm and cosy feeling that they have
done something by throwing a digital coin into
a fountain of wishes, while hoping for better
software”
“A frequent excuse for bug reports is that the management needs
them to know the current quality status. Bug measures signal quality
in the same way humidity signals nice weather. There might be zero
chance of rain, but that still doesn’t mean that I’ll enjoy it if it’s -10
outside”

View Slide

The Importance of Bug Tracking
S. McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents:
The AI Incident Database. arXiv 2020, arXiv:2011.08512.
https://incidentdatabase.ai/

View Slide

Metrics and Decision Making
https://www.developsense.com/blog/2009/01/meaningful-metrics/
M. Bolton, Meaningful Metrics:
An excellent approach to software engineering metrics is based on
several points:
- Metrics as ﬁrst-order approximations (limited, fallible)
- Metrics used for estimating rather than predicting
- Inquiry metrics rather than control metrics
- Observation: people rather than printed reports
- No metrics on unimportant things
- No waste of time on collecting and managing metrics
- Optimizing the quality of work as opposed to making metrics
look good
- Metrics give partial answers that might be useful

View Slide

Spurious Correlations
http://tylervigen.com/view_correlation?id=359

View Slide

Spurious Correlations
http://tylervigen.com/view_correlation?id=79721

View Slide

Metrics and Decision Making
https://www.developsense.com/blog/2009/01/meaningful-metrics/
M. Bolton, Meaningful Metrics:
An excellent approach to software engineering metrics is based
on several points:
- Metrics as ﬁrst-order approximations (limited, fallible)
- Metrics used for estimating rather than predicting
- Inquiry metrics rather than control metrics
- Observation: people rather than printed reports
- No metrics on unimportant things
- No waste of time on collecting and managing metrics
- Optimizing the quality of work as opposed to making
metrics look good

View Slide

Thank you!
Join our Telegram chat, meet
your peers, ask questions
and get new software testing
skills with Exactpro!
Subscribe to our YouTube
channel to see all the new
videos!

View Slide

Lecture 7: Software defects and metrics

Lecture 7: Software defects and metrics

Exactpro
PRO

More Decks by Exactpro

Other Decks in Technology

Featured

Transcript

Lecture 7: Software defects and metrics

Lecture 7: Software defects and metrics

Exactpro PRO

More Decks by Exactpro

Other Decks in Technology

Featured

Transcript

Exactpro
PRO