Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using AI for Testing: Stock Exchange Model

Exactpro
May 11, 2023
71

Using AI for Testing: Stock Exchange Model

Exactpro

May 11, 2023
Tweet

Transcript

  1. exactpro.com
    1 BUILD SOFTWARE TO TEST SOFTWARE
    BUILD SOFTWARE TO TEST SOFTWARE
    exactpro.com
    Using AI for Testing:
    Stock Exchange Model
    Iosif Itkin
    CEO and co-Founder, Exactpro
    AI in Software Testing

    View full-size slide

  2. exactpro.com
    2 BUILD SOFTWARE TO TEST SOFTWARE
    Company overview
    A specialist firm focused on
    functional and non-functional
    testing for complex, distributed,
    non-deterministic and artificial
    intelligence platforms
    Headquartered in the UK and
    operates delivery centres in
    Georgia, Sri Lanka, Armenia,
    Lithuania, the UK and
    representative offices in the US,
    Canada, Italy and Australia.
    Exactpro provides software testing services for mission critical technology that underpins global financial
    markets. The firm is experienced with trading and clearing and settlement platforms, market data systems,
    collateral management and risk management systems, central data warehouses, regulatory reporting, etc.
    Incorporated in 2009 with 10
    people, Exactpro is now employing
    over 600 specialists. Most of our
    clients are regulated market
    infrastructures.
    Part of the London Stock Exchange
    Group (LSEG) from May 2015 until
    January 2018, when the Exactpro
    management proceeded through
    the buyout of the company from
    LSEG.

    View full-size slide

  3. exactpro.com
    3 BUILD SOFTWARE TO TEST SOFTWARE
    Our Clients – 10 of the Top 20 Global Exchange Groups

    View full-size slide

  4. exactpro.com
    4 BUILD SOFTWARE TO TEST SOFTWARE
    EXACTPRO EXPANDS INTO GEORGIA
    350+

    View full-size slide

  5. exactpro.com
    5 BUILD SOFTWARE TO TEST SOFTWARE
    5
    Natia
    Sirbiladze
    CEO, Exactpro, Georgia President,
    GeoSTQB

    View full-size slide

  6. exactpro.com
    6 BUILD SOFTWARE TO TEST SOFTWARE
    6 Build Software to Test Software
    exactpro.com
    Join the Exactpro Youtube channel for regular
    updates on the applications of AI and Machine
    Learning, DLT, cloud computing, and many other
    technologies and solutions to software testing
    and development.
    This channel is for chief technology, information
    and compliance professionals, product owners
    and software Quality Assurance and
    development specialists of all levels.
    @exactpro

    View full-size slide

  7. exactpro.com
    7 BUILD SOFTWARE TO TEST SOFTWARE
    Comparing A4Q AIST and CT-AI
    A4Q AIST Pages Hours
    1. Key Aspects of Artificial Intelligence 15 6
    2. Testing Artificial Intelligence Systems 19
    6.5
    2.1 General Problems with Testing AI Systems 8
    2.2 Machine Learning Model Training and Testing 2
    2.3 Test Environments 2
    2.4 Strategies to Test AI-based Systems 5
    2.5 Metrics for Testing AI-based systems 2
    3. Using AI to Support Testing 13 4.5
    47 17
    ISTQB CT-AI Pages Hours
    1. Introduction to AI
    20 5.5
    3. Machine Learning
    6. ML Neural Networks and Testing
    44 17
    2. Quality Characteristics 5 2
    4. ML Data 7 4
    10. Test Environments 3 0.5
    7. Testing AI-Based Systems 8 2
    8. Testing AI-Specific Quality Characteristics 8 2.5
    9. Methods and Techniques for the Testing of AI-Based Systems 8 4
    5. ML Functional Metrics 5 2
    11. Using AI in Testing 5 3
    69 25.5

    View full-size slide

  8. exactpro.com
    8 BUILD SOFTWARE TO TEST SOFTWARE
    Learning Objectives for Chapter 11:
    11.1 AI Technologies for Testing
    ● AI-11.1.1 K2 Categorize the AI technologies used in software testing.
    ● HO-11.1.1 H2 Discuss, using examples, those activities in testing where AI is less likely to be used.
    11.2 Using AI to Analyze Reported Defects
    ● AI-11.2.1 K2 Explain how AI can assist in supporting the analysis of new defects.
    11.3 Using AI for Test Case Generation
    ● AI-11.3.1 K2 Explain how AI can assist in test case generation.
    11.4 Using AI for the Optimization of Regression Test Suites
    ● AI-11.4.1 K2 Explain how AI can assist in optimization of regression test suites
    11.5 Using AI for Defect Prediction
    ● AI-11.5.1 K2 Explain how AI can assist in defect prediction.
    ● HO-11.5.1 H2 Implement a simple AI-based defect prediction system.
    11.6 Using AI for Testing User Interfaces
    ● AI-11.6.1 K2 Explain the use of AI in testing user interfaces
    AI-Specific Keywords
    Bayesian techniques,
    classification, clustering
    algorithm, defect prediction,
    graphical user interface (GUI)
    11 Using AI for Testing
    Keywords
    Visual testing

    View full-size slide

  9. exactpro.com
    9 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for Testing User Interfaces

    View full-size slide

  10. exactpro.com
    10 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for Testing User Interfaces

    View full-size slide

  11. exactpro.com
    11 BUILD SOFTWARE TO TEST SOFTWARE
    ISTQB CT-AI Syllabus

    View full-size slide

  12. exactpro.com
    12 BUILD SOFTWARE TO TEST SOFTWARE
    Hands-On Exercise: The Use of AI in Testing

    View full-size slide

  13. exactpro.com
    13 BUILD SOFTWARE TO TEST SOFTWARE
    Hands-On Exercise: The Use of AI in Testing

    View full-size slide

  14. exactpro.com
    14 BUILD SOFTWARE TO TEST SOFTWARE
    Hands-On Exercise: The Use of AI in Testing

    View full-size slide

  15. exactpro.com
    15 BUILD SOFTWARE TO TEST SOFTWARE
    Hands-On Exercise: The Use of AI in Testing
    This article is more
    than 10 years old

    View full-size slide

  16. exactpro.com
    16 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI to Analyze Reported Defects

    View full-size slide

  17. exactpro.com
    17 BUILD SOFTWARE TO TEST SOFTWARE
    What is Machine Learning

    View full-size slide

  18. exactpro.com
    18 BUILD SOFTWARE TO TEST SOFTWARE
    What is Software Testing
    Software Testing is exploring software with the intent of finding bugs.
    Software Testing is an empirical technical investigation conducted to provide
    stakeholders with information about the quality of the product or service under test.
    Information Service

    View full-size slide

  19. exactpro.com
    19 BUILD SOFTWARE TO TEST SOFTWARE
    What is Good Software Testing?
    ● Goodness, by definition, is always rooted in some person’s feeling
    ● Goodness is at least a three-way relationship between people,
    product and context
    ● Goodness is a phenomenon produced by the mysterious
    interaction of many variables
    ● Goodness judgments tend to be socially risky
    James Bach
    Assess Quality, Don’t
    Measure It
    The mysteries of measuring goodness are evident in machine learning
    systems that hilariously mis-classify things. Or in ourselves when we
    hilariously fall in love with the wrong people.

    View full-size slide

  20. exactpro.com
    20 BUILD SOFTWARE TO TEST SOFTWARE
    What is Good Software Testing?
    ● 100% Automated
    ● Easy to Maintain
    ● Very Fast
    ● Consistent and Repeatable
    ● Applicable to All Systems
    ● Vendor Independent
    ● Transparent
    ● Low Cost

    View full-size slide

  21. exactpro.com
    21 BUILD SOFTWARE TO TEST SOFTWARE
    ISTQB CT-AI Syllabus

    View full-size slide

  22. exactpro.com
    22 BUILD SOFTWARE TO TEST SOFTWARE
    What is Good Software Testing?
    Just Do Not Do Any Testing…
    ● 100% Automated
    ● Easy to Maintain
    ● Very Fast
    ● Consistent and Repeatable
    ● Applicable to All Systems
    ● Vendor Independent
    ● Transparent
    ● Low Cost

    View full-size slide

  23. exactpro.com
    23 BUILD SOFTWARE TO TEST SOFTWARE
    Principles
    The purpose of our work is to provide information - detect and describe defects in the
    software developed for/by and operated by our clients.
    The value of our work can be measured across the following three main dimensions:
    ● Quality - testing better (*)
    ● Speed - testing faster
    ● Cost - testing cheaper
    Improvements across any of these dimensions bring value to our clients
    (*) better at detecting and describing defects

    View full-size slide

  24. exactpro.com
    24 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing
    Execute actions against the System
    Under Test (SUT)
    Select
    actions to
    perform
    Interpret the outcome of the actions
    Mechanical
    (Easy)
    The following is a list of non-exclusive possibilities:
    ● Experiment results match our model of the SUT
    ● Something is wrong with our actions
    ● Something is wrong with the SUT
    ● Something is wrong with our model
    Provide the information about our findings to the
    stakeholders for them to make decisions as to what
    should be done
    Action types:
    ● Set-up actions
    ● Experiment actions
    ● Observation actions
    What actions are available?
    Which of them to choose?
    In what order?

    View full-size slide

  25. exactpro.com
    25 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing
    Execute actions against the System
    Under Test (SUT)
    Select
    actions to
    perform
    Interpret the outcome of the actions
    What actions are available?
    Which of them to choose?
    In what order?
    Evaluate obtained information and
    the list of available actions – repeat
    the steps
    Cognitive(Medium)
    Mechanical
    (Easy)
    Cognitive (Medium)
    Cognitive (Hard)
    The following is a list of non-exclusive possibilities:
    ● Experiment results match our model of the SUT
    ● Something is wrong with our actions
    ● Something is wrong with the SUT
    ● Something is wrong with our model
    Provide the information about our findings to the
    stakeholders for them to make decisions as to what
    should be done
    Action types:
    ● Set-up actions
    ● Experiment actions
    ● Observation actions

    View full-size slide

  26. exactpro.com
    26 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for Test Case Generation

    View full-size slide

  27. exactpro.com
    27 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for the Optimisation of Regression Test Suites

    View full-size slide

  28. exactpro.com
    28 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for Testing: Stock Exchange Model

    View full-size slide

  29. exactpro.com
    29 BUILD SOFTWARE TO TEST SOFTWARE
    Using AI for Testing: Stock Exchange Model

    View full-size slide

  30. exactpro.com
    30 BUILD SOFTWARE TO TEST SOFTWARE
    All Testing is Model-based Testing
    A model is a simplified, formal
    representation of a relationship,
    process or system. The simplification
    makes some aspects of the thing
    modeled clearer, more visible, and
    easier to work with.
    All tests are based on models, but
    many of those models are implicit.
    When the behavior of the program
    “feels wrong”, it is clashing with your
    internal model of the program and
    how it should behave.
    Black Box Software Testing
    Cem Kaner & James Bach

    View full-size slide

  31. exactpro.com
    31 BUILD SOFTWARE TO TEST SOFTWARE
    A narrow definition of Model-based Testing
    Can be both mental representation or
    a schema or a code-based artifact
    Requirement/System
    Specifications
    MBT Models
    Abstract Test Case
    Test Results SUT
    System Adapters
    Execution Environment
    Executable
    Test Case
    Mappings
    Test Selection
    Criteria
    1 2
    3
    4
    5
    ● more formal models (expressed in some
    machine-readable, well-defined, notation)
    ● more formal test generation (we are interested in test
    generation algorithms)
    ● more formal checks for mechanical execution (the
    generated tests must be sufficiently precise to
    efficiently use humans’ involvement)

    View full-size slide

  32. exactpro.com
    32 BUILD SOFTWARE TO TEST SOFTWARE
    Digital Twin Technologies

    View full-size slide

  33. exactpro.com
    33 BUILD SOFTWARE TO TEST SOFTWARE
    What are Digital Twins
    https://ntrs.nasa.gov/citations/20210023699

    View full-size slide

  34. exactpro.com
    34 BUILD SOFTWARE TO TEST SOFTWARE
    Autonomous Driving and Software Testing (Online Course)

    View full-size slide

  35. exactpro.com
    35 BUILD SOFTWARE TO TEST SOFTWARE
    A very narrow definition of the Model

    View full-size slide

  36. exactpro.com
    36 BUILD SOFTWARE TO TEST SOFTWARE
    A very narrow definition of the Model

    View full-size slide

  37. exactpro.com
    37 BUILD SOFTWARE TO TEST SOFTWARE
    A very narrow definition of the Model
    Model generates expected result

    View full-size slide

  38. exactpro.com
    38 BUILD SOFTWARE TO TEST SOFTWARE
    George E. P. Box
    ...all models are
    approximations. Essentially,
    all models are wrong, but
    some are useful. However, the
    approximate nature of the
    model must always be borne
    in mind…

    View full-size slide

  39. exactpro.com
    39 BUILD SOFTWARE TO TEST SOFTWARE
    The best material
    model of a cat is
    another, or preferably
    the same, cat.
    Arturo Rosenblueth

    View full-size slide

  40. exactpro.com
    40 BUILD SOFTWARE TO TEST SOFTWARE
    Properties of the Model
    Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas.
    "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005)
    mapping feature reduction feature pragmatic feature

    View full-size slide

  41. exactpro.com
    41 BUILD SOFTWARE TO TEST SOFTWARE
    AI Testing Important Limitations

    View full-size slide

  42. exactpro.com
    42 BUILD SOFTWARE TO TEST SOFTWARE
    Label All the Things
    #Make Predictions
    from pandas import read_csv



    print(classification_report(Y_validat
    ion, predictions))

    View full-size slide

  43. exactpro.com
    43 BUILD SOFTWARE TO TEST SOFTWARE
    exactpro.com
    43 BUILD SOFTWARE TO TEST SOFTWARE
    AI ISTQB
    Youtube Course
    https://www.youtube.com/watch?v=FowxPWKhWts&list=PL8Ql
    2_5rYPjgK0HDswJeqpATbOJ7xaGkR
    Thank You

    View full-size slide