Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Лекция 7 - Программные дефекты и показатели

Лекция 7 - Программные дефекты и показатели

Курс лекций «Тестирование сложных интеллектуальных систем и беспилотных автомобилей» знакомит вас с теоретическими основами тестирования автономных ИИ-систем.

Видео лекция доступна по ссылке: https://youtu.be/jqzZQJo3MTY

---
Подписывайтесь на Exactpro в социальных сетях:

LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Подписывайтесь на YouTube канал Exactpro http://www.youtube.com/c/ExactproVlog

5206c19df417b8876825b5561344c1a0?s=128

Exactpro
PRO

April 09, 2021
Tweet

Transcript

  1. 1 Build Software to Test Software exactpro.com Лекция 7 Программные

    дефекты и показатели Онлайн-курс Тестирование сложных интеллектуальных систем и беспилотных автомобилей
  2. 2 Build Software to Test Software exactpro.com Структура лекции Количественная

    оценка качества ПО Кому и зачем нужны метрики Метрика: определение и критерии валидности Метрики в тестировании Количественные показатели, связанные с дефектами Метрики и принятие решений
  3. 3 Build Software to Test Software exactpro.com Что почитать Cem

    Kaner, Walter P. Bond Software Engineering Metrics: What Do They Measure and How Do We Know?
  4. 4 Build Software to Test Software exactpro.com Что почитать Cem

    Kaner, Cem Kaner, Michael Bolton, Michael Bolton, James Bach, Lisa Crispin, Gojko Adzic, On the Quality of Qualitative Measures Don’t Use Bug Counts to Measure Testers Meaningful Metrics Three Kinds of Measurement and Two Ways to Use Them Assess Quality, Don’t Measure It STAREAST: Agile testing and defect tracking Bug statistics are a waste of time
  5. 5 Build Software to Test Software exactpro.com Что почитать ISTQB

    Foundation Level Syllabus 2018 ISTQB Advanced Level Test Manager Syllabus 2012
  6. 6 Build Software to Test Software exactpro.com Что почитать Список

    проектных метрик: https://software-testing.ru/images/stories/library/ qaa-3-metrics.pdf Наталья Руколь Самый полный список метрик тестирования на русском языке: https://software-testing.ru/library/testing/test-manag ement/3566-metrics
  7. 7 Build Software to Test Software exactpro.com Что почитать https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-

    assurance-success/
  8. 8 Build Software to Test Software exactpro.com Что такое хорошо/плохо

    • Goodness, by definition, is always rooted in some person’s feeling • Goodness is at least a three-way relationship between people, product and context • Goodness is a phenomenon produced by the mysterious interaction of many variables • Goodness judgments tend to be socially risky https://www.satisfice.com/blog/archives/487091
  9. 9 Build Software to Test Software exactpro.com Метрика: Определение http://www.kaner.com/pdfs/metrics2004.pdf

    Measurement is “the assignment of numbers to objects or events according to rule.” S. S. Stevens, "On the Theory of Scales of Measurement," Science, vol. 103, pp. 677-680, 1946. Measurement is "the act or process of assigning a number or category to an entity to describe an attribute of that entity." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
  10. 10 Build Software to Test Software exactpro.com Метрика: Определение http://www.kaner.com/pdfs/metrics2004.pdf

    Software quality metric is "a function whose inputs are software data and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality." IEEE Std. 1061-1998, Standard for a Software Quality Metrics Methodology (1998).
  11. 11 Build Software to Test Software exactpro.com Кому нужны метрики

    Typical test manager tasks may include: … - Introduce suitable metrics for measuring test progress and evaluating the quality of the testing and the product Section 5.1.2 Tasks of a Test Manager and Tester
  12. 12 Build Software to Test Software exactpro.com “A management cliché

    says that what gets measured gets done. In addition, what does not get measured does not get done, because what does not get measured is easy to ignore. ” Section 2.6 Defining and Using Test Metrics Кому нужны метрики
  13. 13 Build Software to Test Software exactpro.com Качество vs. количество

    Cem Kaner: “There are too many simplistic metrics that don't capture the essence of whatever it is that they are supposed to measure. There are too many uses of simplistic measures that don't even recognize what attributes are supposedly being measured. Starting from a detailed analysis of the task or attribute under study might lead to more complex, and more qualitative, metrics, but we believe that it will also leads to more meaningful and therefore more useful data.” http://www.kaner.com/pdfs/metrics2004.pdf
  14. 14 Build Software to Test Software exactpro.com Качество vs. количество

    Michael Bolton: https://twitter.com/michaelbolton/status/1151358273498570754
  15. 15 Build Software to Test Software exactpro.com Критерии валидности для

    количественных метрик - Correlation - Consistency - Tracking - Predictability - Discriminative power - Reliability http://www.kaner.com/pdfs/metrics2004.pdf
  16. 16 Build Software to Test Software exactpro.com Критерии валидности для

    качественных метрик - Credibility - Transferability - Dependability - Confirmability http://kaner.com/?p=409
  17. 17 Build Software to Test Software exactpro.com Кем Канер: Construct

    validity
  18. 18 Build Software to Test Software exactpro.com Кем Канер: Фреймворк

    валидации метрик 1. The purpose of the measure 2. The scope of the measurement 3. The attribute to be measured 4. The natural scale for the attribute 5. The natural variability of the attribute 6. The metric/instrument that measures the attribute 7. The natural scale of the metric/instrument 8. The natural variability of measurements made with this instrument 9. The relationship between the attribute and the metric/instrument 10. The natural and foreseeable side effects of using this instrument to measure this attribute
  19. 19 Build Software to Test Software exactpro.com 1. Цель метрики

    - Помощь в оценке своей работы, определении областей для совершенствования - Оценивание статуса проекта - Оценка эффективности работы команды - Информирование о характеристиках продукта - Информирование внешних заинтересованных лиц
  20. 20 Build Software to Test Software exactpro.com 2. Область применения

    метрики - конкретная методика одного специалиста - проект одной команды - проекты команды в течение года - проекты организации за последние 10 лет Метрика, которая хорошо работает локально, может не работать в глобальном масштабе
  21. 21 Build Software to Test Software exactpro.com 3. Атрибут, измеряемый

    с помощью метрики - Логическая структура, которая стоит за измерением - Природа явления, которое мы описываем с помощью метрики
  22. 22 Build Software to Test Software exactpro.com 4. Естественная шкала

    оценки признака Какова естественная шкала признака, который стоит за метрикой? Типы шкал: - Номинальная = шкала наименований - Порядковая = ранговая - Интервальная - Абсолютная = шкала отношений
  23. 23 Build Software to Test Software exactpro.com 5. Естественная вариабельность

    признака Каковы внутренние источники и степень вариации атрибута, который мы пытаемся измерить?
  24. 24 Build Software to Test Software exactpro.com 6. Выбор метрики

    и инструмента измерения - Подсчет (человеком или машиной) - Сопоставление (человеком, алгоритмом или каким- либо другим устройством) - Сравнение (с помощью человека, алгоритма или другого устройства) - Расчет времени
  25. 25 Build Software to Test Software exactpro.com 7. Естественная шкала

    самой метрики -
  26. 26 Build Software to Test Software exactpro.com 8. Естественная вариативность

    измерения Какова типичная ошибка измерения с помощью данной метрики/инструмента?
  27. 27 Build Software to Test Software exactpro.com 9. Отношение между

    признаком и описывающей его метрикой Какая модель связывает значение метрики со значением атрибута?
  28. 28 Build Software to Test Software exactpro.com 10. Потенциальные недостатки

    метрики В какой мере можно повлиять на результат измерений, не затрагивая сам признак? “People are good at tailoring their behavior to things that they are measured against.” http://kaner.com/pdfs/bugcount.pdf
  29. 29 Build Software to Test Software exactpro.com Качество vs. количество

    Michael Bolton: "But management wants numbers!" I doubt that. Management almost certainly wants *to know things* - and from testers, knowledge about the status of the product and problems that threaten its value. Numbers might help to illustrate a story. They don't, can't TELL it. Words can.” http://www.kaner.com/pdfs/metrics2004.pdf https://twitter.com/michaelbolton/status/1151379020661035008 Cem Kaner: “Qualitative measurements tell a story (or a bunch of stories). The skilled qualitative researcher relies on transparency in methods and data to tell persuasive stories… Decision-makers rely on your stories to make important decisions.”
  30. 30 Build Software to Test Software exactpro.com Статья Натальи Руколь

    Самый полный список метрик тестирования на русском языке https://software-testing.ru/library/testing/t est-management/3566-metrics Для чего нужны метрики? - оценка прогресса - промежуточные замеры - поиск проблем - числовые обоснования
  31. 31 Build Software to Test Software exactpro.com Справочник по метрикам

    https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf
  32. 32 Build Software to Test Software exactpro.com Качество тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  33. 33 Build Software to Test Software exactpro.com Проектное планирование https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  34. 34 Build Software to Test Software exactpro.com Качество продукта https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  35. 35 Build Software to Test Software exactpro.com Эффективность тестирования https://software-testing.ru/images/stories/library/qaa-3-metrics.pdf

  36. 36 Build Software to Test Software exactpro.com Список метрик https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-

    assurance-success/
  37. 37 Build Software to Test Software exactpro.com ISTQB Advanced Level

    Test Manager Syllabus https://software-testing.ru/images/stories/li brary/qaa-3-metrics.pdf 4 categories of testing metrics: - Project metrics - Product metrics - Process metrics - People metrics
  38. 38 Build Software to Test Software exactpro.com Дефекты и метрики

    https://www.amazon.com/ Why-Does-Software-Cost-Much/dp/093263334X "I can only think of one metric that is worth collecting now and forever: defect count." [p. 15] Tom DeMarco, "Mad About Measurement," in Why Does Software Cost So Much? New York: Dorset House, 1995, pp. 13-44.
  39. 39 Build Software to Test Software exactpro.com Дефекты и метрики

    https://twitter.com/michaelbolton/status/1151358273498570754
  40. 40 Build Software to Test Software exactpro.com Отслеживание дефектов в

    системах баг- трекинга “From a lean point of view, a DTS is a queue of rework, which translates into waste.” “a DTS isn’t a good way to communicate” “You don’t need a DTS in order to prioritize defects.” https://searchsoftwarequality.techtarget.com/tip/STAREAST-Agile-testing-and-d efect-tracking
  41. 41 Build Software to Test Software exactpro.com Отслеживание дефектов в

    системах баг- трекинга https://gojko.net/2011/05/17/bug-statistics-are-a-waste-of-time/ Gojko Adzic: “defect tracking tools are placebo, giving people a warm and cosy feeling that they have done something by throwing a digital coin into a fountain of wishes, while hoping for better software” “A frequent excuse for bug reports is that the management needs them to know the current quality status. Bug measures signal quality in the same way humidity signals nice weather. There might be zero chance of rain, but that still doesn’t mean that I’ll enjoy it if it’s -10 outside”
  42. 42 Build Software to Test Software exactpro.com Почему важно отслеживать

    дефекты? S. McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database. arXiv 2020, arXiv:2011.08512. https://incidentdatabase.ai/
  43. 43 Build Software to Test Software exactpro.com Метрики и принятие

    решений https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good - Metrics give partial answers that might be useful
  44. 44 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=359

  45. 45 Build Software to Test Software exactpro.com Spurious Correlations http://tylervigen.com/view_correlation?id=79721

  46. 46 Build Software to Test Software exactpro.com Метрики и принятие

    решений https://www.developsense.com/blog/2009/01/meaningful-metrics/ M. Bolton, Meaningful Metrics: An excellent approach to software engineering metrics is based on several points: - Metrics as first-order approximations (limited, fallible) - Metrics used for estimating rather than predicting - Inquiry metrics rather than control metrics - Observation: people rather than printed reports - No metrics on unimportant things - No waste of time on collecting and managing metrics - Optimizing the quality of work as opposed to making metrics look good
  47. 47 Build Software to Test Software exactpro.com Спасибо! Exactpro Systems

    Exactpro Vlog