ISTQB: AI and Agile Software Testing

Slide 1

Slide 1 text

1 BUILD SOFTWARE TO TEST SOFTWARE BUILD SOFTWARE TO TEST SOFTWARE exactpro.com ISTQB: AI and Agile Software Testing Iosif Itkin 18 JUNE 2022 WYNDHAM GRAND TBILISI, BALLROOM Exactpro co-CEO and co-founder

Slide 2

Slide 2 text

2 BUILD SOFTWARE TO TEST SOFTWARE Contents ● ISTQB Portfolio of Certifications ● Comparing Foundation, AI and Agile Syllabi ● What Skills are Specific for Agile Testers ● What Challenges are Specific for AI Systems ● Why Study AI and Agile

Slide 3

Slide 3 text

3 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certiﬁcations

Slide 4

Slide 4 text

4 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certiﬁcations

Slide 5

Slide 5 text

5 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certiﬁcations Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages

Slide 6

Slide 6 text

6 BUILD SOFTWARE TO TEST SOFTWARE Foundation Level Agile Tester (CTFL-AT)

Slide 7

Slide 7 text

7 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Syllabus Agile Technical Tester (ATT)

Slide 8

Slide 8 text

8 BUILD SOFTWARE TO TEST SOFTWARE

Slide 9

Slide 9 text

9 BUILD SOFTWARE TO TEST SOFTWARE 9 BUILD SOFTWARE TO TEST SOFTWARE A tester on an Agile project will work diﬀerently than one working on a traditional project.

Slide 10

Slide 10 text

10 BUILD SOFTWARE TO TEST SOFTWARE 2.3.1 Agile Tester Skills ● Be positive and solution-oriented with team members and stakeholders ● Display critical, quality-oriented, skeptical thinking about the product ● Actively acquire information from stakeholders (rather than relying entirely on written specifications) ● Accurately evaluate and report test results, test progress, and product quality ● Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and stakeholders ● Collaborate within the team, working in pairs with programmers and other team members ● Respond to change quickly, including changing, adding, or improving test cases ● Plan and organize their own work Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams

Slide 11

Slide 11 text

11 BUILD SOFTWARE TO TEST SOFTWARE 2.3.2 The Role of a Tester in an Agile Team ● Understanding, implementing, and updating the test strategy ● Measuring and reporting test coverage across all applicable coverage dimensions ● Ensuring proper use of testing tools ● Conﬁguring, using, and managing test environments and test data ● Reporting defects and working with the team to resolve them ● Coaching other team members in relevant aspects of testing ● Ensuring the appropriate testing tasks are scheduled during release and iteration planning ● Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of testability, consistency, and completeness ● Participating proactively in team retrospectives, suggesting and implementing improvements

Slide 12

Slide 12 text

12 BUILD SOFTWARE TO TEST SOFTWARE Agile is Context for Testing

Slide 13

Slide 13 text

13 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development

Slide 14

Slide 14 text

14 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story Tests (written ﬁrst) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workﬂows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants

Slide 15

Slide 15 text

15 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development Guinea Pig (“Sea” Pig) no relation no relation

Slide 16

Slide 16 text

16 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Deﬁnition Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service

Slide 17

Slide 17 text

17 BUILD SOFTWARE TO TEST SOFTWARE All Testing is Exploratory Testing https://www.satisfice.com/blog/archives/1509

Slide 18

Slide 18 text

18 BUILD SOFTWARE TO TEST SOFTWARE Software Development and Testing Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development

Slide 19

Slide 19 text

19 BUILD SOFTWARE TO TEST SOFTWARE Development and Testing are Two Diﬀerent Processes

Slide 20

Slide 20 text

20 BUILD SOFTWARE TO TEST SOFTWARE Software Deﬁnition Software is a collection of instructions and data that tell a computer how to work The computer (hardware) is able to execute the software. Each instruction causes the computer to carry out an operation - moving data, carrying out a computation, altering the control flow of instructions or manipulating hardware

Slide 21

Slide 21 text

21 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story Tests (written ﬁrst) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workﬂows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants Test Software Build Software

Slide 22

Slide 22 text

22 BUILD SOFTWARE TO TEST SOFTWARE Build Software to Test Software

Slide 23

Slide 23 text

23 BUILD SOFTWARE TO TEST SOFTWARE 23 BUILD SOFTWARE TO TEST SOFTWARE The team should be relatively small; successful teams have been observed with as few as three people and as many as nine.

Slide 24

Slide 24 text

24 BUILD SOFTWARE TO TEST SOFTWARE Database Processes Software Testing at Scale is not Quality Assurance

Slide 25

Slide 25 text

25 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS)

Slide 26

Slide 26 text

26 BUILD SOFTWARE TO TEST SOFTWARE https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

Slide 27

Slide 27 text

27 BUILD SOFTWARE TO TEST SOFTWARE Software Testing in Information Service ● Accuracy ● Relevance ● Interpretability ● Accessibility http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf Data Quality Intrinsic Data Quality Contextual Data Quality Representational Data Quality Accessibility Data Quality Believability (1) Accuracy (4) Objectivity (8) Reputation (12) Value-added (2) Relevancy (3) Timeliness (9) Completeness (10) Appropriate amount of data (19) Interpretability (5) Ease of understanding (6) Representational consistency (13) Concise representation (17) Accessibility (7) Access security (18) A Conceptual Framework of Data Quality

Slide 28

Slide 28 text

28 BUILD SOFTWARE TO TEST SOFTWARE

Slide 29

Slide 29 text

29 BUILD SOFTWARE TO TEST SOFTWARE Three Things a Software Tester Can Learn ● 1 ● 2 ● 3

Slide 30

Slide 30 text

30 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI Testing

Slide 31

Slide 31 text

31 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI Testing

Slide 32

Slide 32 text

32 BUILD SOFTWARE TO TEST SOFTWARE

Slide 33

Slide 33 text

33 BUILD SOFTWARE TO TEST SOFTWARE 33 BUILD SOFTWARE TO TEST SOFTWARE In a typical conventional computer system, the software is programmed by humans using an imperative language, which includes constructs such as if-then-else and loops. It is relatively easy for humans to understand how the system transforms inputs into outputs.

Slide 34

Slide 34 text

34 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing Self-Learning Systems ● Unexpected change ● Complex acceptance criteria ● Insufficient testing time ● Resource requirements ● Insufficient specifications of operational environment ● Complex test environment ● Undesirable behavior modifications

Slide 35

Slide 35 text

35 BUILD SOFTWARE TO TEST SOFTWARE Data Quality Issues

Slide 36

Slide 36 text

36 BUILD SOFTWARE TO TEST SOFTWARE Artiﬁcial Intelligence Techniques

Slide 37

Slide 37 text

37 BUILD SOFTWARE TO TEST SOFTWARE Humans are Afraid of Artiﬁcial Intelligence

Slide 38

Slide 38 text

38 BUILD SOFTWARE TO TEST SOFTWARE Test Oracles are Only Covered in AI Testing Syllabus Using Test Oracles in Automation Doug Hoffman (2006 Google Tech Talks) https://youtu.be/Yusd_ogkiaQ The Often Overlooked Test Oracle Doug Hoffman (2019 Association for Software Testing) https://youtu.be/hbJ1xpG7QHs http://www.kaner.com/pdfs/automation/ Auto_Arch.pdf

Slide 39

Slide 39 text

39 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certiﬁcations Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages

Slide 40

Slide 40 text

40 BUILD SOFTWARE TO TEST SOFTWARE Test Automation Manual Using Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance

Slide 41

Slide 41 text

41 BUILD SOFTWARE TO TEST SOFTWARE 41 BUILD SOFTWARE TO TEST SOFTWARE NO AUTOMATION DRIVER ASSISTANCE PARTIAL AUTOMATION CONDITIONAL AUTOMATION HIGH AUTOMATION FULL AUTOMATION Steering and acceleration/ deceleration Monitoring of driving environment Fallback when automation fails Automated Control: Human driver monitors the road Automated driving system monitors the road SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES 0 1 2 3 4 5 Human driver Automated system Autonomy

Slide 42

Slide 42 text

42 BUILD SOFTWARE TO TEST SOFTWARE Conclusion ● Unit Tests and CI/CD is Software ● TDD/BDD/ATDD is Software Development ● Software Testing and Software Development are Two Different Processes ● Testing in Agile Context is Still Testing ● Artificial Intelligence is Software ● Testing AI is Software Testing ● Understanding Autonomy is Necessary to Understand Automation in Testing ● Ideas from ISTQB AI and Agile Certifications Can Help You with Conventional Testing