Test Design for AI Systems

Slide 1

Slide 1 text

1 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks BUILD SOFTWARE TO TEST SOFTWARE exactpro.com Test Design for AI Systems Murad Mamedov AI Researcher, Exactpro 13 MAY | 3 PM SLST

Slide 2

Slide 2 text

2 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Table of Contents - neural network architecture - ML development process overview - current QA activities in industry - why test-design is important - black-box testing - white-box testing - data-box testing - conclusion

Slide 3

Slide 3 text

3 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Neural Net Architecture

Slide 4

Slide 4 text

4 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks ML Development Process Overview

Slide 5

Slide 5 text

5 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Current Activities in Industry The emerging risk makes governments and businesses respond with quality assurance activities. The regulatory activities are also leveraging monitoring and control of AI development. - USA Data and Trust Alliance - EU AI Regulation Draft - ISO/IEC TR 29119-11:2020 Guidelines on the testing of AI-based systems

Slide 6

Slide 6 text

6 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Current Activities in Industry

Slide 7

Slide 7 text

7 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Why Test Design is Important

Slide 8

Slide 8 text

8 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Black Box Strategy: Mutational Approach It can be applied to: - to an algorithm itself - train data - test data Original Program Mutant Program Output Compare the results of both programs

Slide 9

Slide 9 text

9 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Black Box Strategy: Combinatorial Approach The picture represents a decision map application to the Boldness and Discontinuity features, in order to generate use cases from high-level scenarios

Slide 10

Slide 10 text

10 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Black Box Strategy: Business Logic Approach The approaches based on the business logic are closer to validation-level ones, since they are going directly to the question of whether a system meets the stakeholders’ expectations or not. An example of merging approaches: Model-based exploration of the frontier of behaviours for deep learning system testing Input Database

Slide 11

Slide 11 text

11 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Black Box Strategy: Business Logic Approach The approaches based on the business logic are closer to validation-level ones, since they are going directly to the question of whether a system meets the stakeholders’ expectations or not. An example of merging approaches: Model-based exploration of the frontier of behaviours for deep learning system testing Input Database Literature Features Libelled Inputs Open Coding Metric Identiﬁcation Candidate Metrics Design Metrics Validation and Correlation Metrics Initial Labelling Consensus Meeting Final Labelling Feature name Score [5pt] Candidate Metrics

Slide 12

Slide 12 text

12 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks White Box Strategy: Activation Testing What can be tested: - if a neuron is activated - which value it’s activated with - how the neurons interact - how the layers interact

Slide 13

Slide 13 text

13 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks White Box Strategy: Tools

Slide 14

Slide 14 text

14 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks White Box Strategy: What’s Missing? Activation testing focuses mostly on neurons/layers behaviour, and pays less attention to the predictions

Slide 15

Slide 15 text

15 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Talks Data Box Strategy Combinatorial EDA helps to represent the data from a use cases perspective and enhances the further ML testing activities such as oracle education, test generation, etc.