Lecture 7: Software defects and metrics

"Software Testing for Complex Intelligent Systems and Autonomous Vehicles" online course.

The lecture is available via the link: https://youtu.be/r6v5jKOHPD8
--
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software exactpro.com Lecture 7 Software Defects and Metrics Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course
2. 2 Build Software to Test Software exactpro.com Lecture Structure Quantifying software quality Who needs metrics and why Metrics: definition and validity criteria Metrics in testing Quantitative indicators related to defects Metrics and decision making
3. 3 Build Software to Test Software exactpro.com Reading Materials Cem Kaner, Walter P. Bond Software Engineering Metrics: What Do They Measure and How Do We Know?
4. 4 Build Software to Test Software exactpro.com Reading Materials Cem Kaner, Cem Kaner, Michael Bolton, Michael Bolton, James Bach, Lisa Crispin, Gojko Adzic, On the Quality of Qualitative Measures Don’t Use Bug Counts to Measure Testers Meaningful Metrics Three Kinds of Measurement and Two Ways to Use Them Assess Quality, Don’t Measure It STAREAST: Agile testing and defect tracking Bug statistics are a waste of time
5. 5 Build Software to Test Software exactpro.com Reading Materials ISTQB Foundation Level Syllabus 2018 ISTQB Advanced Level Test Manager Syllabus 2012
6. 6 Build Software to Test Software exactpro.com Что почитать Список проектных метрик: https://software-testing.ru/images/stories/library/ qaa-3-metrics.pdf Наталья Руколь Самый полный список метрик тестирования на русском языке: https://software-testing.ru/library/testing/test-manag ement/3566-metrics
7. 7 Build Software to Test Software exactpro.com Reading Materials https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality- assurance-success/
8. 8 Build Software to Test Software exactpro.com Good vs Bad • Goodness, by definition, is always rooted in some person’s feeling • Goodness is at least a three-way relationship between people, product and context • Goodness is a phenomenon produced by the mysterious interaction of many variables • Goodness judgments tend to be socially risky https://www.satisfice.com/blog/archives/487091
9. 9 Build Software to Test Software exactpro.com Metric: Definition http://www.kaner.com/pdfs/metrics2004.pdf Measurement is “the assignment of numbers to objects or events according to rule.” S. S. Stevens, "On the Theory of Scales of Measurement," Science, vol. 103, pp. 677-680, 1946. Measurement is "the act or process of assigning a number or category to an entity to describe an attribute of that entity." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
10. 10 Build Software to Test Software exactpro.com Metric: Definition http://www.kaner.com/pdfs/metrics2004.pdf Software quality metric is "a function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
11. 11 Build Software to Test Software exactpro.com Who Needs Metrics Typical test manager tasks may include: … - Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and the product Section 5.1.2 Tasks of a Test Manager and Tester
12. 12 Build Software to Test Software exactpro.com “A management cliché says that what gets measured gets done. In addition, what does not get measured does not get done, because what does not get measured is easy to ignore. ” Section 2.6 Defining and Using Test Metrics Who Needs Metrics
13. 13 Build Software to Test Software exactpro.com Quality vs Quantity Cem Kaner: “There are too many simplistic metrics that don't capture the essence of whatever it is that they are supposed to measure. There are too many uses of simplistic measures that don't even recognize what attributes are supposedly being measured. Starting from a detailed analysis of the task or attribute under study might lead to more complex, and more qualitative, metrics, but we believe that it will also leads to more meaningful and therefore more useful data.” http://www.kaner.com/pdfs/metrics2004.pdf
14. 14 Build Software to Test Software exactpro.com Quality vs Quantity Michael Bolton: https://twitter.com/michaelbolton/status/1151358273498570754
15. 15 Build Software to Test Software exactpro.com Validity criteria for quantitative metrics - Correlation - Consistency - Tracking - Predictability - Discriminative power - Reliability http://www.kaner.com/pdfs/metrics2004.pdf
16. 16 Build Software to Test Software exactpro.com Validity criteria for qualitative metrics - Credibility - Transferability - Dependability - Confirmability http://kaner.com/?p=409
17. 17 Build Software to Test Software exactpro.com Cem Kaner: Construct validity
18. 18 Build Software to Test Software exactpro.com Cem Kaner: Metrics Validation Framework 1. The purpose of the measure 2. The scope of the measurement 3. The attribute to be measured 4. The natural scale for the attribute 5. The natural variability of the attribute 6. The metric/instrument that measures the attribute 7. The natural scale of the metric/instrument 8. The natural variability of measurements made with this instrument 9. The relationship between the attribute and the metric/instrument 10. The natural and foreseeable side effects of using this instrument to measure this attribute
19. 19 Build Software to Test Software exactpro.com 1. Metric’s purpose - Helping you evaluate your work, identifying areas for improvement - Assessing project status - Evaluating the effectiveness of the team - Informing about product features - Informing external stakeholders
20. 20 Build Software to Test Software exactpro.com 2. The Scope of the Metric Application - one specialist's particular methodology - one team project - team projects during the year - the organization's projects over the last 10 years A metric that works well locally might fail globally.
21. 21 Build Software to Test Software exactpro.com 3. Attribute - The logical structure behind the measurement - The nature of the phenomenon we describe with a metric
22. 22 Build Software to Test Software exactpro.com 4 Attribute Rating Scale What is the natural scale of the attribute behind the metric? Scale types: - nominal - ordinal - interval - ratio
23. 23 Build Software to Test Software exactpro.com 5. The Natural Variability of the Attribute What are the intrinsic sources and degrees of variation of the attribute we are trying to measure?
24. 24 Build Software to Test Software exactpro.com 6. Choosing a Metric - Counting (human or machine) - Matching (by a human, an algorithm, or some other device - Comparison (with the help of a person, an algorithm, or some other device) - Calculating time (using a computer, stopwatch, or some external automated device, or by calculating the difference between two timestamps)
25. 25 Build Software to Test Software exactpro.com 7. The Natural Scale of the Metric -
26. 26 Build Software to Test Software exactpro.com 8. Measurement Variability What is the natural variability of readings from this instrument?
27. 27 Build Software to Test Software exactpro.com 9. The Attribute and its Metric Which model links the value of a metric to the value of an attribute?
28. 28 Build Software to Test Software exactpro.com 10. The Other Side of Metrics If we try to improve the outcome being measured, what effect will we have on this attribute? “People are good at tailoring their behavior to things that they are measured against.” http://kaner.com/pdfs/bugcount.pdf
29. 29 Build Software to Test Software exactpro.com Quality vs Quantity Michael Bolton: "But management wants numbers!" I doubt that. Management almost certainly wants *to know things* - and from testers, knowledge about the status of the product and problems that threaten its value. Numbers might help to illustrate a story. They don't, can't TELL it. Words can.” http://www.kaner.com/pdfs/metrics2004.pdf https://twitter.com/michaelbolton/status/1151379020661035008 Cem Kaner: “Qualitative measurements tell a story (or a bunch of stories). The skilled qualitative researcher relies on transparency in methods and data to tell persuasive stories… Decision-makers rely on your stories to make important decisions.”
30. 30 Build Software to Test Software exactpro.com Статья Натальи Руколь Самый полный список метрик тестирования на русском языке https://software-testing.ru/library/testing/t est-management/3566-metrics Для чего нужны метрики? - оценка прогресса - промежуточные замеры - поиск проблем - числовые обоснования
31. 31 Build Software to Test Software exactpro.com Справочник по метрикам https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
32. 32 Build Software to Test Software exactpro.com Качество тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
33. 33 Build Software to Test Software exactpro.com Проектное планирование https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
34. 34 Build Software to Test Software exactpro.com Качество продукта https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
35. 35 Build Software to Test Software exactpro.com Эффективность тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
36. 36 Build Software to Test Software exactpro.com Essential Metrics https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality- assurance-success/
37. 37 Build Software to Test Software exactpro.com ISTQB Advanced Level Test Manager Syllabus https://software-testing.ru/images/stories/li brary/qaa-3-metrics.pdf 4 categories of testing metrics: - Project metrics - Product metrics - Process metrics - People metrics
38. 38 Build Software to Test Software exactpro.com Defects and Metrics https://www.amazon.com/ Why-Does-Software-Cost-Much/dp/093263334X "I can only think of one metric that is worth collecting now and forever: defect count." [p. 15] Tom DeMarco, "Mad About Measurement," in Why Does Software Cost So Much? New York: Dorset House, 1995, pp. 13-44.
39. 39 Build Software to Test Software exactpro.com Defects and Metrics https://twitter.com/michaelbolton/status/1151358273498570754
40. 40 Build Software to Test Software exactpro.com Bug Tracking “From a lean point of view, a DTS is a queue of rework, which translates into waste.” “a DTS isn’t a good way to communicate” “You don’t need a DTS in order to prioritize defects.” https://searchsoftwarequality.techtarget.com/tip/STAREAST-Agile-testing-and-d efect-tracking
41. 41 Build Software to Test Software exactpro.com Bug Tracking https://gojko.net/2011/05/17/bug-statistics-are-a-waste-of-time/ Gojko Adzic: “defect tracking tools are placebo, giving people a warm and cosy feeling that they have done something by throwing a digital coin into a fountain of wishes, while hoping for better software” “A frequent excuse for bug reports is that the management needs them to know the current quality status. Bug measures signal quality in the same way humidity signals nice weather. There might be zero chance of rain, but that still doesn’t mean that I’ll enjoy it if it’s -10 outside”
42. 42 Build Software to Test Software exactpro.com The Importance of Bug Tracking S. McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. arXiv 2020, arXiv:2011.08512. https://incidentdatabase.ai/
43. 43 Build Software to Test Software exactpro.com Metrics and Decision Making https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good - Metrics give partial answers that might be useful
44. 44 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=359
45. 45 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=79721
46. 46 Build Software to Test Software exactpro.com Metrics and Decision Making https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good
47. 47 Build Software to Test Software exactpro.com Thank you! Join our Telegram chat, meet your peers, ask questions and get new software testing skills with Exactpro! Subscribe to our YouTube channel to see all the new videos!