Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lecture 7: Software defects and metrics

Lecture 7: Software defects and metrics

"Software Testing for Complex Intelligent Systems and Autonomous Vehicles" online course.

The lecture is available via the link: https://youtu.be/r6v5jKOHPD8
--
To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

5206c19df417b8876825b5561344c1a0?s=128

Exactpro
PRO

April 16, 2021
Tweet

Transcript

  1. 1 Build Software to Test Software exactpro.com Lecture 7 Software

    Defects and Metrics Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course
  2. 2 Build Software to Test Software exactpro.com Lecture Structure Quantifying

    software quality Who needs metrics and why Metrics: definition and validity criteria Metrics in testing Quantitative indicators related to defects Metrics and decision making
  3. 3 Build Software to Test Software exactpro.com Reading Materials Cem

    Kaner, Walter P. Bond Software Engineering Metrics: What Do They Measure and How Do We Know?
  4. 4 Build Software to Test Software exactpro.com Reading Materials Cem

    Kaner, Cem Kaner, Michael Bolton, Michael Bolton, James Bach, Lisa Crispin, Gojko Adzic, On the Quality of Qualitative Measures Don’t Use Bug Counts to Measure Testers Meaningful Metrics Three Kinds of Measurement and Two Ways to Use Them Assess Quality, Don’t Measure It STAREAST: Agile testing and defect tracking Bug statistics are a waste of time
  5. 5 Build Software to Test Software exactpro.com Reading Materials ISTQB

    Foundation Level Syllabus 2018 ISTQB Advanced Level Test Manager Syllabus 2012
  6. 6 Build Software to Test Software exactpro.com Что почитать Список

    проектных метрик: https://software-testing.ru/images/stories/library/ qaa-3-metrics.pdf Наталья Руколь Самый полный список метрик тестирования на русском языке: https://software-testing.ru/library/testing/test-manag ement/3566-metrics
  7. 7 Build Software to Test Software exactpro.com Reading Materials https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-

    assurance-success/
  8. 8 Build Software to Test Software exactpro.com Good vs Bad

    • Goodness, by definition, is always rooted in some person’s feeling • Goodness is at least a three-way relationship between people, product and context • Goodness is a phenomenon produced by the mysterious interaction of many variables • Goodness judgments tend to be socially risky https://www.satisfice.com/blog/archives/487091
  9. 9 Build Software to Test Software exactpro.com Metric: Definition http://www.kaner.com/pdfs/metrics2004.pdf

    Measurement is “the assignment of numbers to objects or events according to rule.” S. S. Stevens, "On the Theory of Scales of Measurement," Science, vol. 103, pp. 677-680, 1946. Measurement is "the act or process of assigning a number or category to an entity to describe an attribute of that entity." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
  10. 10 Build Software to Test Software exactpro.com Metric: Definition http://www.kaner.com/pdfs/metrics2004.pdf

    Software quality metric is "a function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
  11. 11 Build Software to Test Software exactpro.com Who Needs Metrics

    Typical test manager tasks may include: … - Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and the product Section 5.1.2 Tasks of a Test Manager and Tester
  12. 12 Build Software to Test Software exactpro.com “A management cliché

    says that what gets measured gets done. In addition, what does not get measured does not get done, because what does not get measured is easy to ignore. ” Section 2.6 Defining and Using Test Metrics Who Needs Metrics
  13. 13 Build Software to Test Software exactpro.com Quality vs Quantity

    Cem Kaner: “There are too many simplistic metrics that don't capture the essence of whatever it is that they are supposed to measure. There are too many uses of simplistic measures that don't even recognize what attributes are supposedly being measured. Starting from a detailed analysis of the task or attribute under study might lead to more complex, and more qualitative, metrics, but we believe that it will also leads to more meaningful and therefore more useful data.” http://www.kaner.com/pdfs/metrics2004.pdf
  14. 14 Build Software to Test Software exactpro.com Quality vs Quantity

    Michael Bolton: https://twitter.com/michaelbolton/status/1151358273498570754
  15. 15 Build Software to Test Software exactpro.com Validity criteria for

    quantitative metrics - Correlation - Consistency - Tracking - Predictability - Discriminative power - Reliability http://www.kaner.com/pdfs/metrics2004.pdf
  16. 16 Build Software to Test Software exactpro.com Validity criteria for

    qualitative metrics - Credibility - Transferability - Dependability - Confirmability http://kaner.com/?p=409
  17. 17 Build Software to Test Software exactpro.com Cem Kaner: Construct

    validity
  18. 18 Build Software to Test Software exactpro.com Cem Kaner: Metrics

    Validation Framework 1. The purpose of the measure 2. The scope of the measurement 3. The attribute to be measured 4. The natural scale for the attribute 5. The natural variability of the attribute 6. The metric/instrument that measures the attribute 7. The natural scale of the metric/instrument 8. The natural variability of measurements made with this instrument 9. The relationship between the attribute and the metric/instrument 10. The natural and foreseeable side effects of using this instrument to measure this attribute
  19. 19 Build Software to Test Software exactpro.com 1. Metric’s purpose

    - Helping you evaluate your work, identifying areas for improvement - Assessing project status - Evaluating the effectiveness of the team - Informing about product features - Informing external stakeholders
  20. 20 Build Software to Test Software exactpro.com 2. The Scope

    of the Metric Application - one specialist's particular methodology - one team project - team projects during the year - the organization's projects over the last 10 years A metric that works well locally might fail globally.
  21. 21 Build Software to Test Software exactpro.com 3. Attribute -

    The logical structure behind the measurement - The nature of the phenomenon we describe with a metric
  22. 22 Build Software to Test Software exactpro.com 4 Attribute Rating

    Scale What is the natural scale of the attribute behind the metric? Scale types: - nominal - ordinal - interval - ratio
  23. 23 Build Software to Test Software exactpro.com 5. The Natural

    Variability of the Attribute What are the intrinsic sources and degrees of variation of the attribute we are trying to measure?
  24. 24 Build Software to Test Software exactpro.com 6. Choosing a

    Metric - Counting (human or machine) - Matching (by a human, an algorithm, or some other device - Comparison (with the help of a person, an algorithm, or some other device) - Calculating time (using a computer, stopwatch, or some external automated device, or by calculating the difference between two timestamps)
  25. 25 Build Software to Test Software exactpro.com 7. The Natural

    Scale of the Metric -
  26. 26 Build Software to Test Software exactpro.com 8. Measurement Variability

    What is the natural variability of readings from this instrument?
  27. 27 Build Software to Test Software exactpro.com 9. The Attribute

    and its Metric Which model links the value of a metric to the value of an attribute?
  28. 28 Build Software to Test Software exactpro.com 10. The Other

    Side of Metrics If we try to improve the outcome being measured, what effect will we have on this attribute? “People are good at tailoring their behavior to things that they are measured against.” http://kaner.com/pdfs/bugcount.pdf
  29. 29 Build Software to Test Software exactpro.com Quality vs Quantity

    Michael Bolton: "But management wants numbers!" I doubt that. Management almost certainly wants *to know things* - and from testers, knowledge about the status of the product and problems that threaten its value. Numbers might help to illustrate a story. They don't, can't TELL it. Words can.” http://www.kaner.com/pdfs/metrics2004.pdf https://twitter.com/michaelbolton/status/1151379020661035008 Cem Kaner: “Qualitative measurements tell a story (or a bunch of stories). The skilled qualitative researcher relies on transparency in methods and data to tell persuasive stories… Decision-makers rely on your stories to make important decisions.”
  30. 30 Build Software to Test Software exactpro.com Статья Натальи Руколь

    Самый полный список метрик тестирования на русском языке https://software-testing.ru/library/testing/t est-management/3566-metrics Для чего нужны метрики? - оценка прогресса - промежуточные замеры - поиск проблем - числовые обоснования
  31. 31 Build Software to Test Software exactpro.com Справочник по метрикам

    https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
  32. 32 Build Software to Test Software exactpro.com Качество тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  33. 33 Build Software to Test Software exactpro.com Проектное планирование https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  34. 34 Build Software to Test Software exactpro.com Качество продукта https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  35. 35 Build Software to Test Software exactpro.com Эффективность тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  36. 36 Build Software to Test Software exactpro.com Essential Metrics https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-

    assurance-success/
  37. 37 Build Software to Test Software exactpro.com ISTQB Advanced Level

    Test Manager Syllabus https://software-testing.ru/images/stories/li brary/qaa-3-metrics.pdf 4 categories of testing metrics: - Project metrics - Product metrics - Process metrics - People metrics
  38. 38 Build Software to Test Software exactpro.com Defects and Metrics

    https://www.amazon.com/ Why-Does-Software-Cost-Much/dp/093263334X "I can only think of one metric that is worth collecting now and forever: defect count." [p. 15] Tom DeMarco, "Mad About Measurement," in Why Does Software Cost So Much? New York: Dorset House, 1995, pp. 13-44.
  39. 39 Build Software to Test Software exactpro.com Defects and Metrics

    https://twitter.com/michaelbolton/status/1151358273498570754
  40. 40 Build Software to Test Software exactpro.com Bug Tracking “From

    a lean point of view, a DTS is a queue of rework, which translates into waste.” “a DTS isn’t a good way to communicate” “You don’t need a DTS in order to prioritize defects.” https://searchsoftwarequality.techtarget.com/tip/STAREAST-Agile-testing-and-d efect-tracking
  41. 41 Build Software to Test Software exactpro.com Bug Tracking https://gojko.net/2011/05/17/bug-statistics-are-a-waste-of-time/

    Gojko Adzic: “defect tracking tools are placebo, giving people a warm and cosy feeling that they have done something by throwing a digital coin into a fountain of wishes, while hoping for better software” “A frequent excuse for bug reports is that the management needs them to know the current quality status. Bug measures signal quality in the same way humidity signals nice weather. There might be zero chance of rain, but that still doesn’t mean that I’ll enjoy it if it’s -10 outside”
  42. 42 Build Software to Test Software exactpro.com The Importance of

    Bug Tracking S. McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. arXiv 2020, arXiv:2011.08512. https://incidentdatabase.ai/
  43. 43 Build Software to Test Software exactpro.com Metrics and Decision

    Making https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good - Metrics give partial answers that might be useful
  44. 44 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=359

  45. 45 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=79721

  46. 46 Build Software to Test Software exactpro.com Metrics and Decision

    Making https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good
  47. 47 Build Software to Test Software exactpro.com Thank you! Join

    our Telegram chat, meet your peers, ask questions and get new software testing skills with Exactpro! Subscribe to our YouTube channel to see all the new videos!