exactpro.com 1 BUILD SOFTWARE TO TEST SOFTWARE BUILD SOFTWARE TO TEST SOFTWARE exactpro.com Using AI for Testing: Stock Exchange Model Iosif Itkin CEO and co-Founder, Exactpro AI in Software Testing

exactpro.com 2 BUILD SOFTWARE TO TEST SOFTWARE Company overview A specialist firm focused on functional and non-functional testing for complex, distributed, non-deterministic and artificial intelligence platforms Headquartered in the UK and operates delivery centres in Georgia, Sri Lanka, Armenia, Lithuania, the UK and representative offices in the US, Canada, Italy and Australia. Exactpro provides software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, market data systems, collateral management and risk management systems, central data warehouses, regulatory reporting, etc. Incorporated in 2009 with 10 people, Exactpro is now employing over 600 specialists. Most of our clients are regulated market infrastructures. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.

exactpro.com 3 BUILD SOFTWARE TO TEST SOFTWARE Our Clients – 10 of the Top 20 Global Exchange Groups

exactpro.com 4 BUILD SOFTWARE TO TEST SOFTWARE EXACTPRO EXPANDS INTO GEORGIA 350+

exactpro.com 5 BUILD SOFTWARE TO TEST SOFTWARE 5 Natia Sirbiladze CEO, Exactpro, Georgia President, GeoSTQB

exactpro.com 6 BUILD SOFTWARE TO TEST SOFTWARE 6 Build Software to Test Software exactpro.com Join the Exactpro Youtube channel for regular updates on the applications of AI and Machine Learning, DLT, cloud computing, and many other technologies and solutions to software testing and development. This channel is for chief technology, information and compliance professionals, product owners and software Quality Assurance and development specialists of all levels. @exactpro

exactpro.com 7 BUILD SOFTWARE TO TEST SOFTWARE Comparing A4Q AIST and CT-AI A4Q AIST Pages Hours 1. Key Aspects of Artificial Intelligence 15 6 2. Testing Artificial Intelligence Systems 19 6.5 2.1 General Problems with Testing AI Systems 8 2.2 Machine Learning Model Training and Testing 2 2.3 Test Environments 2 2.4 Strategies to Test AI-based Systems 5 2.5 Metrics for Testing AI-based systems 2 3. Using AI to Support Testing 13 4.5 47 17 ISTQB CT-AI Pages Hours 1. Introduction to AI 20 5.5 3. Machine Learning 6. ML Neural Networks and Testing 44 17 2. Quality Characteristics 5 2 4. ML Data 7 4 10. Test Environments 3 0.5 7. Testing AI-Based Systems 8 2 8. Testing AI-Specific Quality Characteristics 8 2.5 9. Methods and Techniques for the Testing of AI-Based Systems 8 4 5. ML Functional Metrics 5 2 11. Using AI in Testing 5 3 69 25.5

exactpro.com 8 BUILD SOFTWARE TO TEST SOFTWARE Learning Objectives for Chapter 11: 11.1 AI Technologies for Testing ● AI-11.1.1 K2 Categorize the AI technologies used in software testing. ● HO-11.1.1 H2 Discuss, using examples, those activities in testing where AI is less likely to be used. 11.2 Using AI to Analyze Reported Defects ● AI-11.2.1 K2 Explain how AI can assist in supporting the analysis of new defects. 11.3 Using AI for Test Case Generation ● AI-11.3.1 K2 Explain how AI can assist in test case generation. 11.4 Using AI for the Optimization of Regression Test Suites ● AI-11.4.1 K2 Explain how AI can assist in optimization of regression test suites 11.5 Using AI for Defect Prediction ● AI-11.5.1 K2 Explain how AI can assist in defect prediction. ● HO-11.5.1 H2 Implement a simple AI-based defect prediction system. 11.6 Using AI for Testing User Interfaces ● AI-11.6.1 K2 Explain the use of AI in testing user interfaces AI-Specific Keywords Bayesian techniques, classification, clustering algorithm, defect prediction, graphical user interface (GUI) 11 Using AI for Testing Keywords Visual testing

exactpro.com 9 BUILD SOFTWARE TO TEST SOFTWARE Using AI for Testing User Interfaces

exactpro.com 10 BUILD SOFTWARE TO TEST SOFTWARE Using AI for Testing User Interfaces

exactpro.com 11 BUILD SOFTWARE TO TEST SOFTWARE ISTQB CT-AI Syllabus

exactpro.com 12 BUILD SOFTWARE TO TEST SOFTWARE Hands-On Exercise: The Use of AI in Testing

exactpro.com 13 BUILD SOFTWARE TO TEST SOFTWARE Hands-On Exercise: The Use of AI in Testing

exactpro.com 14 BUILD SOFTWARE TO TEST SOFTWARE Hands-On Exercise: The Use of AI in Testing

exactpro.com 15 BUILD SOFTWARE TO TEST SOFTWARE Hands-On Exercise: The Use of AI in Testing This article is more than 10 years old

exactpro.com 16 BUILD SOFTWARE TO TEST SOFTWARE Using AI to Analyze Reported Defects

exactpro.com 17 BUILD SOFTWARE TO TEST SOFTWARE What is Machine Learning

exactpro.com 18 BUILD SOFTWARE TO TEST SOFTWARE What is Software Testing Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service

exactpro.com 19 BUILD SOFTWARE TO TEST SOFTWARE What is Good Software Testing? ● Goodness, by definition, is always rooted in some person’s feeling ● Goodness is at least a three-way relationship between people, product and context ● Goodness is a phenomenon produced by the mysterious interaction of many variables ● Goodness judgments tend to be socially risky James Bach Assess Quality, Don’t Measure It The mysteries of measuring goodness are evident in machine learning systems that hilariously mis-classify things. Or in ourselves when we hilariously fall in love with the wrong people.

exactpro.com 20 BUILD SOFTWARE TO TEST SOFTWARE What is Good Software Testing? ● 100% Automated ● Easy to Maintain ● Very Fast ● Consistent and Repeatable ● Applicable to All Systems ● Vendor Independent ● Transparent ● Low Cost

exactpro.com 21 BUILD SOFTWARE TO TEST SOFTWARE ISTQB CT-AI Syllabus

exactpro.com 22 BUILD SOFTWARE TO TEST SOFTWARE What is Good Software Testing? Just Do Not Do Any Testing… ● 100% Automated ● Easy to Maintain ● Very Fast ● Consistent and Repeatable ● Applicable to All Systems ● Vendor Independent ● Transparent ● Low Cost

exactpro.com 23 BUILD SOFTWARE TO TEST SOFTWARE Principles The purpose of our work is to provide information - detect and describe defects in the software developed for/by and operated by our clients. The value of our work can be measured across the following three main dimensions: ● Quality - testing better (*) ● Speed - testing faster ● Cost - testing cheaper Improvements across any of these dimensions bring value to our clients (*) better at detecting and describing defects

exactpro.com 24 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Execute actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions Mechanical (Easy) The following is a list of non-exclusive possibilities: ● Experiment results match our model of the SUT ● Something is wrong with our actions ● Something is wrong with the SUT ● Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: ● Set-up actions ● Experiment actions ● Observation actions What actions are available? Which of them to choose? In what order?

exactpro.com 25 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Execute actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions What actions are available? Which of them to choose? In what order? Evaluate obtained information and the list of available actions – repeat the steps Cognitive(Medium) Mechanical (Easy) Cognitive (Medium) Cognitive (Hard) The following is a list of non-exclusive possibilities: ● Experiment results match our model of the SUT ● Something is wrong with our actions ● Something is wrong with the SUT ● Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: ● Set-up actions ● Experiment actions ● Observation actions

exactpro.com 26 BUILD SOFTWARE TO TEST SOFTWARE Using AI for Test Case Generation

exactpro.com 27 BUILD SOFTWARE TO TEST SOFTWARE Using AI for the Optimisation of Regression Test Suites

exactpro.com 28 BUILD SOFTWARE TO TEST SOFTWARE Using AI for Testing: Stock Exchange Model

exactpro.com 29 BUILD SOFTWARE TO TEST SOFTWARE Using AI for Testing: Stock Exchange Model

exactpro.com 30 BUILD SOFTWARE TO TEST SOFTWARE All Testing is Model-based Testing A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach

exactpro.com 31 BUILD SOFTWARE TO TEST SOFTWARE A narrow definition of Model-based Testing Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 ● more formal models (expressed in some machine-readable, well-defined, notation) ● more formal test generation (we are interested in test generation algorithms) ● more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement)

exactpro.com 32 BUILD SOFTWARE TO TEST SOFTWARE Digital Twin Technologies

exactpro.com 33 BUILD SOFTWARE TO TEST SOFTWARE What are Digital Twins https://ntrs.nasa.gov/citations/20210023699

exactpro.com 34 BUILD SOFTWARE TO TEST SOFTWARE Autonomous Driving and Software Testing (Online Course)

exactpro.com 35 BUILD SOFTWARE TO TEST SOFTWARE A very narrow definition of the Model

exactpro.com 36 BUILD SOFTWARE TO TEST SOFTWARE A very narrow definition of the Model

exactpro.com 37 BUILD SOFTWARE TO TEST SOFTWARE A very narrow definition of the Model Model generates expected result

exactpro.com 38 BUILD SOFTWARE TO TEST SOFTWARE George E. P. Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind…

exactpro.com 39 BUILD SOFTWARE TO TEST SOFTWARE The best material model of a cat is another, or preferably the same, cat. Arturo Rosenblueth ”

exactpro.com 40 BUILD SOFTWARE TO TEST SOFTWARE Properties of the Model Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas. "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005) mapping feature reduction feature pragmatic feature

exactpro.com 41 BUILD SOFTWARE TO TEST SOFTWARE AI Testing Important Limitations

exactpro.com 42 BUILD SOFTWARE TO TEST SOFTWARE Label All the Things #Make Predictions from pandas import read_csv … … … print(classification_report(Y_validat ion, predictions))

exactpro.com 43 BUILD SOFTWARE TO TEST SOFTWARE exactpro.com 43 BUILD SOFTWARE TO TEST SOFTWARE AI ISTQB Youtube Course https://www.youtube.com/watch?v=FowxPWKhWts&list=PL8Ql 2_5rYPjgK0HDswJeqpATbOJ7xaGkR Thank You