Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using AI for Testing: Stock Exchange Model

Exactpro
May 11, 2023
96

Using AI for Testing: Stock Exchange Model

Exactpro

May 11, 2023
Tweet

Transcript

  1. exactpro.com 1 BUILD SOFTWARE TO TEST SOFTWARE BUILD SOFTWARE TO

    TEST SOFTWARE exactpro.com Using AI for Testing: Stock Exchange Model Iosif Itkin CEO and co-Founder, Exactpro AI in Software Testing
  2. exactpro.com 2 BUILD SOFTWARE TO TEST SOFTWARE Company overview A

    specialist firm focused on functional and non-functional testing for complex, distributed, non-deterministic and artificial intelligence platforms Headquartered in the UK and operates delivery centres in Georgia, Sri Lanka, Armenia, Lithuania, the UK and representative offices in the US, Canada, Italy and Australia. Exactpro provides software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, market data systems, collateral management and risk management systems, central data warehouses, regulatory reporting, etc. Incorporated in 2009 with 10 people, Exactpro is now employing over 600 specialists. Most of our clients are regulated market infrastructures. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.
  3. exactpro.com 3 BUILD SOFTWARE TO TEST SOFTWARE Our Clients –

    10 of the Top 20 Global Exchange Groups
  4. exactpro.com 5 BUILD SOFTWARE TO TEST SOFTWARE 5 Natia Sirbiladze

    CEO, Exactpro, Georgia President, GeoSTQB
  5. exactpro.com 6 BUILD SOFTWARE TO TEST SOFTWARE 6 Build Software

    to Test Software exactpro.com Join the Exactpro Youtube channel for regular updates on the applications of AI and Machine Learning, DLT, cloud computing, and many other technologies and solutions to software testing and development. This channel is for chief technology, information and compliance professionals, product owners and software Quality Assurance and development specialists of all levels. @exactpro
  6. exactpro.com 7 BUILD SOFTWARE TO TEST SOFTWARE Comparing A4Q AIST

    and CT-AI A4Q AIST Pages Hours 1. Key Aspects of Artificial Intelligence 15 6 2. Testing Artificial Intelligence Systems 19 6.5 2.1 General Problems with Testing AI Systems 8 2.2 Machine Learning Model Training and Testing 2 2.3 Test Environments 2 2.4 Strategies to Test AI-based Systems 5 2.5 Metrics for Testing AI-based systems 2 3. Using AI to Support Testing 13 4.5 47 17 ISTQB CT-AI Pages Hours 1. Introduction to AI 20 5.5 3. Machine Learning 6. ML Neural Networks and Testing 44 17 2. Quality Characteristics 5 2 4. ML Data 7 4 10. Test Environments 3 0.5 7. Testing AI-Based Systems 8 2 8. Testing AI-Specific Quality Characteristics 8 2.5 9. Methods and Techniques for the Testing of AI-Based Systems 8 4 5. ML Functional Metrics 5 2 11. Using AI in Testing 5 3 69 25.5
  7. exactpro.com 8 BUILD SOFTWARE TO TEST SOFTWARE Learning Objectives for

    Chapter 11: 11.1 AI Technologies for Testing • AI-11.1.1 K2 Categorize the AI technologies used in software testing. • HO-11.1.1 H2 Discuss, using examples, those activities in testing where AI is less likely to be used. 11.2 Using AI to Analyze Reported Defects • AI-11.2.1 K2 Explain how AI can assist in supporting the analysis of new defects. 11.3 Using AI for Test Case Generation • AI-11.3.1 K2 Explain how AI can assist in test case generation. 11.4 Using AI for the Optimization of Regression Test Suites • AI-11.4.1 K2 Explain how AI can assist in optimization of regression test suites 11.5 Using AI for Defect Prediction • AI-11.5.1 K2 Explain how AI can assist in defect prediction. • HO-11.5.1 H2 Implement a simple AI-based defect prediction system. 11.6 Using AI for Testing User Interfaces • AI-11.6.1 K2 Explain the use of AI in testing user interfaces AI-Specific Keywords Bayesian techniques, classification, clustering algorithm, defect prediction, graphical user interface (GUI) 11 Using AI for Testing Keywords Visual testing
  8. exactpro.com 15 BUILD SOFTWARE TO TEST SOFTWARE Hands-On Exercise: The

    Use of AI in Testing This article is more than 10 years old
  9. exactpro.com 18 BUILD SOFTWARE TO TEST SOFTWARE What is Software

    Testing Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service
  10. exactpro.com 19 BUILD SOFTWARE TO TEST SOFTWARE What is Good

    Software Testing? • Goodness, by definition, is always rooted in some person’s feeling • Goodness is at least a three-way relationship between people, product and context • Goodness is a phenomenon produced by the mysterious interaction of many variables • Goodness judgments tend to be socially risky James Bach Assess Quality, Don’t Measure It The mysteries of measuring goodness are evident in machine learning systems that hilariously mis-classify things. Or in ourselves when we hilariously fall in love with the wrong people.
  11. exactpro.com 20 BUILD SOFTWARE TO TEST SOFTWARE What is Good

    Software Testing? • 100% Automated • Easy to Maintain • Very Fast • Consistent and Repeatable • Applicable to All Systems • Vendor Independent • Transparent • Low Cost
  12. exactpro.com 22 BUILD SOFTWARE TO TEST SOFTWARE What is Good

    Software Testing? Just Do Not Do Any Testing… • 100% Automated • Easy to Maintain • Very Fast • Consistent and Repeatable • Applicable to All Systems • Vendor Independent • Transparent • Low Cost
  13. exactpro.com 23 BUILD SOFTWARE TO TEST SOFTWARE Principles The purpose

    of our work is to provide information - detect and describe defects in the software developed for/by and operated by our clients. The value of our work can be measured across the following three main dimensions: • Quality - testing better (*) • Speed - testing faster • Cost - testing cheaper Improvements across any of these dimensions bring value to our clients (*) better at detecting and describing defects
  14. exactpro.com 24 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Execute

    actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions Mechanical (Easy) The following is a list of non-exclusive possibilities: • Experiment results match our model of the SUT • Something is wrong with our actions • Something is wrong with the SUT • Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: • Set-up actions • Experiment actions • Observation actions What actions are available? Which of them to choose? In what order?
  15. exactpro.com 25 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Execute

    actions against the System Under Test (SUT) Select actions to perform Interpret the outcome of the actions What actions are available? Which of them to choose? In what order? Evaluate obtained information and the list of available actions – repeat the steps Cognitive(Medium) Mechanical (Easy) Cognitive (Medium) Cognitive (Hard) The following is a list of non-exclusive possibilities: • Experiment results match our model of the SUT • Something is wrong with our actions • Something is wrong with the SUT • Something is wrong with our model Provide the information about our findings to the stakeholders for them to make decisions as to what should be done Action types: • Set-up actions • Experiment actions • Observation actions
  16. exactpro.com 27 BUILD SOFTWARE TO TEST SOFTWARE Using AI for

    the Optimisation of Regression Test Suites
  17. exactpro.com 30 BUILD SOFTWARE TO TEST SOFTWARE All Testing is

    Model-based Testing A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach
  18. exactpro.com 31 BUILD SOFTWARE TO TEST SOFTWARE A narrow definition

    of Model-based Testing Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 • more formal models (expressed in some machine-readable, well-defined, notation) • more formal test generation (we are interested in test generation algorithms) • more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement)
  19. exactpro.com 33 BUILD SOFTWARE TO TEST SOFTWARE What are Digital

    Twins https://ntrs.nasa.gov/citations/20210023699
  20. exactpro.com 37 BUILD SOFTWARE TO TEST SOFTWARE A very narrow

    definition of the Model Model generates expected result
  21. exactpro.com 38 BUILD SOFTWARE TO TEST SOFTWARE George E. P.

    Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind…
  22. exactpro.com 39 BUILD SOFTWARE TO TEST SOFTWARE The best material

    model of a cat is another, or preferably the same, cat. Arturo Rosenblueth ”
  23. exactpro.com 40 BUILD SOFTWARE TO TEST SOFTWARE Properties of the

    Model Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas. "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005) mapping feature reduction feature pragmatic feature
  24. exactpro.com 42 BUILD SOFTWARE TO TEST SOFTWARE Label All the

    Things #Make Predictions from pandas import read_csv … … … print(classification_report(Y_validat ion, predictions))
  25. exactpro.com 43 BUILD SOFTWARE TO TEST SOFTWARE exactpro.com 43 BUILD

    SOFTWARE TO TEST SOFTWARE AI ISTQB Youtube Course https://www.youtube.com/watch?v=FowxPWKhWts&list=PL8Ql 2_5rYPjgK0HDswJeqpATbOJ7xaGkR Thank You