ISTQB goes AI

Transcript

1. ISTQB goes AI Exactpro Webinar, Online, 10.05.2023 Dr. Klaudia Dussa-Zieger
2. ISTQB® 2023 2 To continually improve and advance the software testing profession by: Defining and maintaining a Body of Knowledge which allows testers to be certified based on best practices, connecting the international software testing community, and encouraging research. ISTQB – Vision
3. ISTQB® 2023 3 1. We promote the value of software testing as a profession to individuals and organizations. 2. We help software testers to be more efficient and effective in their work, through the certification of competencies. 3. We enable testers to progress their career through a Professionals' Code of Ethics and a multi-level certification pathway that provides them with the skills and knowledge they need to fulfil their growing responsibilities and to achieve increased professionalism. 4.We continually advance the Testing Body of Knowledge by drawing on the best available industry practices and the most innovative research, and we make this knowledge freely available to all. 5. We set the criteria for accrediting training providers, to ensure consistent delivery of the Body of Knowledge, world-wide. 6. We regulate the content and coverage of exam questions, the examination process, and the issuing of certifications by official examination bodies. 7. We are committed to expanding software testing certifications around the world, by admitting member boards into the ISTQB®. These boards adhere to the constitution, bylaws, and processes defined by the ISTQB®, and participate in regular audits. 8.We nurture an open international community, committed to sharing knowledge, ideas, and innovations in software testing. 9. We foster relationships with academia, government, media, professional associations and other interested parties. 10.We provide a reference point against which the effectiveness of testing services can be evaluated, by maintaining our prominence as a respected source of knowledge in software testing. ISTQB – Mission
4. ISTQB® 2023 4 ISTQB – Worldwide Software Testing Practices Report (2017/18) ▪ Which new technologies or subjects will be important to the software testing industry in the following 5 years? ISTQB Survey
5. ISTQB® 2023 5 Lots of areas of application … Predictive maintenance Production Medical diagnosis Autonomous driving Webshops Voice recognition
6. ISTQB® 2023 6 There is a lot of money in AI Source: LearnBonds
7. ISTQB® 2023 7 Market Guide for AI-Augmented Software Testing Tools *https://info.eggplantsoftware.com/gartner-report-2021-market-guide?utm_campaign=FY22%20Modern%20Application%20Campaign&utm_source=homepage-alert- banner&utm_content=gartner-market-guide And where is the trend going in testing? Gartner Report* in December 2021
8. ISTQB® 2023 8 ▪ 1956 Dartmouth conference – AI as an independent field of research ▪ 1940 to the end of 1950s – The early days of AI ▪ 1960 to early 1980 – research in symbolic and subsymbolic AI ▪ 1980 to early 2000 – boom in knowledge-based systems in industry, followed by disenchantment and decline ▪ Since 2010 – success of machine learning (ML) led to huge interest in industry and vast amount of applications History of Artificial Intelligence (AI) Automatic deduction techniques based on mathematical logic ▪ ▪ ▪ Artificial neural networks ▪ Automatic Deduction systems ▪ Prolog ▪ Knowledge-based systems ▪ Mathematical models based on probababilitic reasoning
9. ISTQB® 2023 9 Machine learning through ▪ statistical learning algorithms ▪ artificial neural networks Leap in development and industrial application due to ▪ hugh increase of processing power ▪ availability of large amounts of data ML is the name of the game!
10. ISTQB® 2023 10 In 2019 three independent interest groups developed syllabi on software testing and AI A lot of interested parties
11. ISTQB® 2023 11 We work together ☺ Certified Tester AI Testing
12. ISTQB® 2023 12 ▪ Agreement on seven business objectives ▪ Merge of 169 learning objectives and 16 hands-on objectives Start of work in January 2020
13. ISTQB® 2023 13 Three major blocks I. Introduction to AI with focus on machine learning II. Test of AI-based systems III. Testing with AI … a lot of hard work … supports tests
14. ISTQB® 2023 14 ▪ Syllabus* was released in October 2021 ▪ 4 days training with hands-on exercises *https://istqb-main-web-prod.s3.amazonaws.com/media/documents/ISTQB_CT-AI_Syllabus_v1.0_mghocmT.pdf … even more hard work …
15. ISTQB® 2023 15 Introduction to AI ▪ AI terminology, e.g. AI Effect, Narrow AI, General AI, Super AI, .. ▪ Brief overview of AI technologies, e.g. Fuzzy logic, Search algorithms, reasoning techniques and Machine Learning techniques, .. ▪ Examples for AI Development Frameworks, e.g. KERAS, PyTorch, Scikit-learn, .. ▪ Hardware for AI-Based Systems, e.g. NVIDIA, Intel, Google TPUs, .. Part I – Introduction to AI
16. ISTQB® 2023 16 Quality Characteristics for AI-Based Systems ▪ Flexibility, Adaptability, Autonomy, Bias, Ethics, Transparency, … ▪ Side effects and reward hacking ▪ Safety and AI Part I – Introduction to AI
17. ISTQB® 2023 17 Machine Learning (ML) ▪ Supervised learning, unsupervised learning, reinforcement learning ▪ ML Workflow ▪ Overfitting, Underfitting Part I – Introduction to AI
18. ISTQB® 2023 18 ML Data ▪ Data Preparation ▪ Training, Validation, Testing Datasets in the ML workflow ▪ Dataset Quality issues ▪ Data labelling for Supervised Learning Part I – Introduction to AI
19. ISTQB® 2023 19 ML Functional Performance Metrics ▪ Confusion Matrix ▪ Accuracy, Precision, Recall, F1-score ▪ Metrics for Classification, Regression and Clustering ▪ Limitations of ML Functional Performance Metrics ▪ Selection of ML Functional Performance Metrics Part I – Introduction to AI
20. ISTQB® 2023 20 ML Neural Networks and Testing ▪ Neural Networks ▪ Coverage Measures for Neural Networks, e.g. Neuron coverage, Threshold coverage, … Part I – Introduction to AI
21. ISTQB® 2023 21 Testing AI-Based Systems Overview ▪ Test Levels of AI-Based System, e.g. ▪ Input Data Testing, ML Model Testing, Component Testing, Component Integration Testing, … ▪ Test Data for Testing AI-Based Systems ▪ Testing for Automation Bias in AI-Based Systems ▪ Testing for Concept Drift ▪ Selecting a Test Approach for an ML System Part II – Testing of AI-Based Systems
22. ISTQB® 2023 22 Testing AI-Specific Quality Characteristics ▪ Challenges Testing Self-Learning Systems ▪ Testing autonomous AI-Based Systems ▪ Testing for Algorithmics, Sample and Inappropriate Bias ▪ Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems ▪ Test Oracles for AI-Based Systems Part II – Testing of AI-Based Systems
23. ISTQB® 2023 23 Methods and Techniques for the Testing of AI-Based Systems ▪ Adversarial Attacks ▪ Data Poisoning ▪ Pairwise Testing ▪ Back-to-Back Testing ▪ Metamorphic Testing ▪ Experience-Based Testing of AI-Based Systems Part II – Testing of AI-Based Systems
24. ISTQB® 2023 24 Test Environments for AI-Based Systems ▪ Test Environment for AI-Based Systems ▪ Virtual Test Environments for Testing AI-Based Systems Part II – Testing of AI-Based Systems
25. ISTQB® 2023 25 Using AI for Testing ▪ AI Technologies for Testing, e.g. classification, prediction, search & optimization techniques ▪ Using AI to Analyse Reported Defects ▪ Using AI for Test Case Generation ▪ Using AI for the Optimization of Regression Test Suites ▪ Using AI for Defect Prediction ▪ Using AI for Testing Interfaces Part III – Testing with AI
26. ISTQB® 2023 26 ISTQB Certified Tester AI Testing ▪ is an excellent introduction into AI ▪ explains how machine learning works ▪ shows the challenges when testing AI-Based Systems ▪ introduces new test techniques ▪ gives examples on how AI can support testing ▪ provides hands-on experiences with AI-Based Systems Summary
27. ISTQB® 2023 27 ▪ ISO/IEC TR 29119-11 „Guidelines on the Testing of AI-Based systems“ ▪ ISO/IEC DIS 25059 „Quality model for AI systems“ ▪ New Subcommittee on AI started in 2017: ISO/IEC JTC1/SC42 ▪ New standards are to be expected Related Standards
28. ISTQB® 2023 28 Last but not least https://www.pinterest.de/pin/560276009884038202/
29. ISTQB® 2023 29 Questions?! k.dussa-zieger@gtb.de