Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ISTQB goes AI

Exactpro
May 11, 2023
71

ISTQB goes AI

Exactpro

May 11, 2023
Tweet

Transcript

  1. ISTQB goes AI
    Exactpro Webinar, Online, 10.05.2023
    Dr. Klaudia Dussa-Zieger

    View full-size slide

  2. ISTQB® 2023
    2
    To continually improve and advance the software testing profession
    by:
    Defining and maintaining a Body of Knowledge which allows testers
    to be certified based on best practices,
    connecting the international software testing community,
    and encouraging research.
    ISTQB – Vision

    View full-size slide

  3. ISTQB® 2023
    3
    1. We promote the value of software testing as a profession to individuals and organizations.
    2. We help software testers to be more efficient and effective in their work, through the certification of competencies.
    3. We enable testers to progress their career through a Professionals' Code of Ethics and a multi-level certification pathway that provides
    them with the skills and knowledge they need to fulfil their growing responsibilities and to achieve increased professionalism.
    4.We continually advance the Testing Body of Knowledge by drawing on the
    best available industry practices and the most innovative research, and we
    make this knowledge freely available to all.
    5. We set the criteria for accrediting training providers, to ensure consistent delivery of the Body of Knowledge, world-wide.
    6. We regulate the content and coverage of exam questions, the examination process, and the issuing of certifications by official
    examination bodies.
    7. We are committed to expanding software testing certifications around the world, by admitting member boards into the ISTQB®. These
    boards adhere to the constitution, bylaws, and processes defined by the ISTQB®, and participate in regular audits.
    8.We nurture an open international community, committed to sharing
    knowledge, ideas, and innovations in software testing.
    9. We foster relationships with academia, government, media, professional associations and other interested parties.
    10.We provide a reference point against which the effectiveness of testing services can be evaluated, by maintaining our prominence as a
    respected source of knowledge in software testing.
    ISTQB – Mission

    View full-size slide

  4. ISTQB® 2023
    4
    ISTQB – Worldwide Software Testing
    Practices Report (2017/18)
    ▪ Which new technologies or
    subjects will be important to
    the software testing industry
    in the following 5 years?
    ISTQB Survey

    View full-size slide

  5. ISTQB® 2023
    5
    Lots of areas of application …
    Predictive maintenance
    Production
    Medical diagnosis Autonomous driving
    Webshops
    Voice recognition

    View full-size slide

  6. ISTQB® 2023
    6
    There is a lot of money in AI
    Source: LearnBonds

    View full-size slide

  7. ISTQB® 2023
    7
    Market Guide for
    AI-Augmented Software
    Testing Tools
    *https://info.eggplantsoftware.com/gartner-report-2021-market-guide?utm_campaign=FY22%20Modern%20Application%20Campaign&utm_source=homepage-alert-
    banner&utm_content=gartner-market-guide
    And where is the trend going in testing?
    Gartner Report*
    in December 2021

    View full-size slide

  8. ISTQB® 2023
    8
    ▪ 1956 Dartmouth conference – AI as an independent field of research
    ▪ 1940 to the end of 1950s – The early days of AI
    ▪ 1960 to early 1980 – research in symbolic and subsymbolic AI
    ▪ 1980 to early 2000 – boom in knowledge-based systems in industry,
    followed by disenchantment and decline
    ▪ Since 2010 – success of machine learning (ML) led to huge interest in
    industry and vast amount of applications
    History of Artificial Intelligence (AI)
    Automatic deduction techniques
    based on mathematical logic


    ▪ Artificial neural networks
    ▪ Automatic Deduction systems
    ▪ Prolog
    ▪ Knowledge-based systems
    ▪ Mathematical models based
    on probababilitic reasoning

    View full-size slide

  9. ISTQB® 2023
    9
    Machine learning through
    ▪ statistical learning algorithms
    ▪ artificial neural networks
    Leap in development and industrial application due to
    ▪ hugh increase of processing power
    ▪ availability of large amounts of data
    ML is the name of the game!

    View full-size slide

  10. ISTQB® 2023
    10
    In 2019 three independent interest groups developed syllabi
    on software testing and AI
    A lot of interested parties

    View full-size slide

  11. ISTQB® 2023
    11
    We work together ☺
    Certified Tester AI Testing

    View full-size slide

  12. ISTQB® 2023
    12
    ▪ Agreement on seven business objectives
    ▪ Merge of 169 learning objectives and 16 hands-on objectives
    Start of work in January 2020

    View full-size slide

  13. ISTQB® 2023
    13
    Three major blocks
    I. Introduction to AI with focus on machine learning
    II. Test of AI-based systems
    III. Testing with AI
    … a lot of hard work …
    supports
    tests

    View full-size slide

  14. ISTQB® 2023
    14
    ▪ Syllabus* was released
    in October 2021
    ▪ 4 days training
    with hands-on exercises
    *https://istqb-main-web-prod.s3.amazonaws.com/media/documents/ISTQB_CT-AI_Syllabus_v1.0_mghocmT.pdf
    … even more hard work …

    View full-size slide

  15. ISTQB® 2023
    15
    Introduction to AI
    ▪ AI terminology, e.g. AI Effect, Narrow AI, General AI, Super AI, ..
    ▪ Brief overview of AI technologies, e.g. Fuzzy logic, Search algorithms,
    reasoning techniques and Machine Learning techniques, ..
    ▪ Examples for AI Development Frameworks, e.g. KERAS, PyTorch,
    Scikit-learn, ..
    ▪ Hardware for AI-Based Systems, e.g. NVIDIA, Intel, Google TPUs, ..
    Part I – Introduction to AI

    View full-size slide

  16. ISTQB® 2023
    16
    Quality Characteristics for AI-Based Systems
    ▪ Flexibility, Adaptability, Autonomy, Bias, Ethics, Transparency, …
    ▪ Side effects and reward hacking
    ▪ Safety and AI
    Part I – Introduction to AI

    View full-size slide

  17. ISTQB® 2023
    17
    Machine Learning (ML)
    ▪ Supervised learning, unsupervised learning,
    reinforcement learning
    ▪ ML Workflow
    ▪ Overfitting, Underfitting
    Part I – Introduction to AI

    View full-size slide

  18. ISTQB® 2023
    18
    ML Data
    ▪ Data Preparation
    ▪ Training, Validation, Testing Datasets
    in the ML workflow
    ▪ Dataset Quality issues
    ▪ Data labelling for Supervised Learning
    Part I – Introduction to AI

    View full-size slide

  19. ISTQB® 2023
    19
    ML Functional Performance Metrics
    ▪ Confusion Matrix
    ▪ Accuracy, Precision, Recall, F1-score
    ▪ Metrics for Classification, Regression and Clustering
    ▪ Limitations of ML Functional Performance Metrics
    ▪ Selection of ML Functional Performance Metrics
    Part I – Introduction to AI

    View full-size slide

  20. ISTQB® 2023
    20
    ML Neural Networks and Testing
    ▪ Neural Networks
    ▪ Coverage Measures
    for Neural Networks, e.g.
    Neuron coverage, Threshold
    coverage, …
    Part I – Introduction to AI

    View full-size slide

  21. ISTQB® 2023
    21
    Testing AI-Based Systems Overview
    ▪ Test Levels of AI-Based System, e.g.
    ▪ Input Data Testing, ML Model Testing, Component Testing, Component Integration
    Testing, …
    ▪ Test Data for Testing AI-Based Systems
    ▪ Testing for Automation Bias in AI-Based Systems
    ▪ Testing for Concept Drift
    ▪ Selecting a Test Approach for an ML System
    Part II – Testing of AI-Based Systems

    View full-size slide

  22. ISTQB® 2023
    22
    Testing AI-Specific Quality Characteristics
    ▪ Challenges Testing Self-Learning Systems
    ▪ Testing autonomous AI-Based Systems
    ▪ Testing for Algorithmics, Sample and Inappropriate Bias
    ▪ Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems
    ▪ Test Oracles for AI-Based Systems
    Part II – Testing of AI-Based Systems

    View full-size slide

  23. ISTQB® 2023
    23
    Methods and Techniques for the Testing of AI-Based Systems
    ▪ Adversarial Attacks
    ▪ Data Poisoning
    ▪ Pairwise Testing
    ▪ Back-to-Back Testing
    ▪ Metamorphic Testing
    ▪ Experience-Based Testing of AI-Based Systems
    Part II – Testing of AI-Based Systems

    View full-size slide

  24. ISTQB® 2023
    24
    Test Environments for AI-Based Systems
    ▪ Test Environment for AI-Based Systems
    ▪ Virtual Test Environments for Testing AI-Based Systems
    Part II – Testing of AI-Based Systems

    View full-size slide

  25. ISTQB® 2023
    25
    Using AI for Testing
    ▪ AI Technologies for Testing, e.g. classification, prediction, search & optimization
    techniques
    ▪ Using AI to Analyse Reported Defects
    ▪ Using AI for Test Case Generation
    ▪ Using AI for the Optimization of Regression Test Suites
    ▪ Using AI for Defect Prediction
    ▪ Using AI for Testing Interfaces
    Part III – Testing with AI

    View full-size slide

  26. ISTQB® 2023
    26
    ISTQB Certified Tester AI Testing
    ▪ is an excellent introduction into AI
    ▪ explains how machine learning works
    ▪ shows the challenges when testing AI-Based Systems
    ▪ introduces new test techniques
    ▪ gives examples on how AI can support testing
    ▪ provides hands-on experiences with AI-Based Systems
    Summary

    View full-size slide

  27. ISTQB® 2023
    27
    ▪ ISO/IEC TR 29119-11 „Guidelines on the Testing of AI-Based systems“
    ▪ ISO/IEC DIS 25059 „Quality model for AI systems“
    ▪ New Subcommittee on AI started in 2017: ISO/IEC JTC1/SC42
    ▪ New standards are to be expected
    Related Standards

    View full-size slide

  28. ISTQB® 2023
    28
    Last but not least
    https://www.pinterest.de/pin/560276009884038202/

    View full-size slide

  29. ISTQB® 2023
    29
    Questions?!
    [email protected]

    View full-size slide