ISTQB: AI and Agile Software Testing

Iosif Itkin, CEO and co-founder of Exactpro

Iosif reviewed how AI can be used to support software testing, as well as addressed the challenges associated with testing AI-Based systems. Iosif also highlighted the key software testers’ skills required to successfully contribute to large-scale Agile projects.

https://exactpro.com/events/external/qa-meetup-tbilisi-georgia?utm_source=Facebook&utm_medium=Organic&utm_campaign=Iosif-Itkin-16-06

QA Meetup. 18 June 2022

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 BUILD SOFTWARE TO TEST SOFTWARE BUILD SOFTWARE TO TEST SOFTWARE exactpro.com ISTQB: AI and Agile Software Testing Iosif Itkin 18 JUNE 2022 WYNDHAM GRAND TBILISI, BALLROOM Exactpro co-CEO and co-founder
2. 2 BUILD SOFTWARE TO TEST SOFTWARE Contents • ISTQB Portfolio of Certifications • Comparing Foundation, AI and Agile Syllabi • What Skills are Specific for Agile Testers • What Challenges are Specific for AI Systems • Why Study AI and Agile
3. 3 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certifications
4. 4 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certifications
5. 5 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certifications Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages
6. 6 BUILD SOFTWARE TO TEST SOFTWARE Foundation Level Agile Tester (CTFL-AT)
7. 7 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Syllabus Agile Technical Tester (ATT)
8. 8 BUILD SOFTWARE TO TEST SOFTWARE
9. 9 BUILD SOFTWARE TO TEST SOFTWARE 9 BUILD SOFTWARE TO TEST SOFTWARE A tester on an Agile project will work differently than one working on a traditional project.
10. 10 BUILD SOFTWARE TO TEST SOFTWARE 2.3.1 Agile Tester Skills • Be positive and solution-oriented with team members and stakeholders • Display critical, quality-oriented, skeptical thinking about the product • Actively acquire information from stakeholders (rather than relying entirely on written specifications) • Accurately evaluate and report test results, test progress, and product quality • Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and stakeholders • Collaborate within the team, working in pairs with programmers and other team members • Respond to change quickly, including changing, adding, or improving test cases • Plan and organize their own work Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams
11. 11 BUILD SOFTWARE TO TEST SOFTWARE 2.3.2 The Role of a Tester in an Agile Team • Understanding, implementing, and updating the test strategy • Measuring and reporting test coverage across all applicable coverage dimensions • Ensuring proper use of testing tools • Configuring, using, and managing test environments and test data • Reporting defects and working with the team to resolve them • Coaching other team members in relevant aspects of testing • Ensuring the appropriate testing tasks are scheduled during release and iteration planning • Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of testability, consistency, and completeness • Participating proactively in team retrospectives, suggesting and implementing improvements
12. 12 BUILD SOFTWARE TO TEST SOFTWARE Agile is Context for Testing
13. 13 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development
14. 14 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story Tests (written first) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workflows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants
15. 15 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development Guinea Pig (“Sea” Pig) no relation no relation
16. 16 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Definition Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service
17. 17 BUILD SOFTWARE TO TEST SOFTWARE All Testing is Exploratory Testing https://www.satisfice.com/blog/archives/1509
18. 18 BUILD SOFTWARE TO TEST SOFTWARE Software Development and Testing Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development
19. 19 BUILD SOFTWARE TO TEST SOFTWARE Development and Testing are Two Different Processes
20. 20 BUILD SOFTWARE TO TEST SOFTWARE Software Definition Software is a collection of instructions and data that tell a computer how to work The computer (hardware) is able to execute the software. Each instruction causes the computer to carry out an operation - moving data, carrying out a computation, altering the control flow of instructions or manipulating hardware
21. 21 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story Tests (written first) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workflows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants Test Software Build Software
22. 22 BUILD SOFTWARE TO TEST SOFTWARE Build Software to Test Software
23. 23 BUILD SOFTWARE TO TEST SOFTWARE 23 BUILD SOFTWARE TO TEST SOFTWARE The team should be relatively small; successful teams have been observed with as few as three people and as many as nine.
24. 24 BUILD SOFTWARE TO TEST SOFTWARE Database Processes Software Testing at Scale is not Quality Assurance
25. 25 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS)
26. 26 BUILD SOFTWARE TO TEST SOFTWARE https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/
27. 27 BUILD SOFTWARE TO TEST SOFTWARE Software Testing in Information Service • Accuracy • Relevance • Interpretability • Accessibility http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf Data Quality Intrinsic Data Quality Contextual Data Quality Representational Data Quality Accessibility Data Quality Believability (1) Accuracy (4) Objectivity (8) Reputation (12) Value-added (2) Relevancy (3) Timeliness (9) Completeness (10) Appropriate amount of data (19) Interpretability (5) Ease of understanding (6) Representational consistency (13) Concise representation (17) Accessibility (7) Access security (18) A Conceptual Framework of Data Quality
28. 28 BUILD SOFTWARE TO TEST SOFTWARE
29. 29 BUILD SOFTWARE TO TEST SOFTWARE Three Things a Software Tester Can Learn • 1 • 2 • 3
30. 30 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI Testing
31. 31 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI Testing
32. 32 BUILD SOFTWARE TO TEST SOFTWARE
33. 33 BUILD SOFTWARE TO TEST SOFTWARE 33 BUILD SOFTWARE TO TEST SOFTWARE In a typical conventional computer system, the software is programmed by humans using an imperative language, which includes constructs such as if-then-else and loops. It is relatively easy for humans to understand how the system transforms inputs into outputs.
34. 34 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing Self-Learning Systems • Unexpected change • Complex acceptance criteria • Insufficient testing time • Resource requirements • Insufficient specifications of operational environment • Complex test environment • Undesirable behavior modifications
35. 35 BUILD SOFTWARE TO TEST SOFTWARE Data Quality Issues
36. 36 BUILD SOFTWARE TO TEST SOFTWARE Artificial Intelligence Techniques
37. 37 BUILD SOFTWARE TO TEST SOFTWARE Humans are Afraid of Artificial Intelligence
38. 38 BUILD SOFTWARE TO TEST SOFTWARE Test Oracles are Only Covered in AI Testing Syllabus Using Test Oracles in Automation Doug Hoffman (2006 Google Tech Talks) https://youtu.be/Yusd_ogkiaQ The Often Overlooked Test Oracle Doug Hoffman (2019 Association for Software Testing) https://youtu.be/hbJ1xpG7QHs http://www.kaner.com/pdfs/automation/ Auto_Arch.pdf
39. 39 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certifications Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages
40. 40 BUILD SOFTWARE TO TEST SOFTWARE Test Automation Manual Using Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance
41. 41 BUILD SOFTWARE TO TEST SOFTWARE 41 BUILD SOFTWARE TO TEST SOFTWARE NO AUTOMATION DRIVER ASSISTANCE PARTIAL AUTOMATION CONDITIONAL AUTOMATION HIGH AUTOMATION FULL AUTOMATION Steering and acceleration/ deceleration Monitoring of driving environment Fallback when automation fails Automated Control: Human driver monitors the road Automated driving system monitors the road SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES 0 1 2 3 4 5 Human driver Automated system Autonomy
42. 42 BUILD SOFTWARE TO TEST SOFTWARE Conclusion • Unit Tests and CI/CD is Software • TDD/BDD/ATDD is Software Development • Software Testing and Software Development are Two Different Processes • Testing in Agile Context is Still Testing • Artificial Intelligence is Software • Testing AI is Software Testing • Understanding Autonomy is Necessary to Understand Automation in Testing • Ideas from ISTQB AI and Agile Certifications Can Help You with Conventional Testing