ISTQB: AI and Agile Software Testing

Iosif Itkin, CEO and co-founder of Exactpro

Iosif reviewed how AI can be used to support software testing, as well as addressed the challenges associated with testing AI-Based systems. Iosif also highlighted the key software testers’ skills required to successfully contribute to large-scale Agile projects.

https://exactpro.com/events/external/qa-meetup-tbilisi-georgia?utm_source=Facebook&utm_medium=Organic&utm_campaign=Iosif-Itkin-16-06

QA Meetup. 18 June 2022

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 BUILD SOFTWARE TO TEST SOFTWARE
   BUILD SOFTWARE TO TEST SOFTWARE
   exactpro.com
   ISTQB: AI and Agile
   Software Testing
   Iosif Itkin
   18 JUNE 2022
   WYNDHAM GRAND TBILISI, BALLROOM
   Exactpro co-CEO and co-founder
   

   View Slide

2. 2 BUILD SOFTWARE TO TEST SOFTWARE
   Contents
   ● ISTQB Portfolio of Certifications
   ● Comparing Foundation, AI and Agile Syllabi
   ● What Skills are Specific for Agile Testers
   ● What Challenges are Specific for AI Systems
   ● Why Study AI and Agile
   

   View Slide

3. 3 BUILD SOFTWARE TO TEST SOFTWARE
   ISTQB Portfolio of Certifications
   

   View Slide

4. 4 BUILD SOFTWARE TO TEST SOFTWARE
   ISTQB Portfolio of Certifications
   

   View Slide

5. 5 BUILD SOFTWARE TO TEST SOFTWARE
   Comparing ISTQB Syllabi
   Certifications Oracle Pages Date
   Foundation Level (CTFL) 2 69 2018-11-11
   AI Testing (CT-AI) 43 58 2021-10-01
   Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30
   Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09
   Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13
   Test Automation Engineer (CT-TAE) 3 60 2016-10-21
   Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30
   * Body of Knowledge contains additional 30 pages
   

   View Slide

6. 6 BUILD SOFTWARE TO TEST SOFTWARE
   Foundation Level Agile Tester (CTFL-AT)
   

   View Slide

7. 7 BUILD SOFTWARE TO TEST SOFTWARE
   Advanced Level Syllabus Agile Technical Tester (ATT)
   

   View Slide

8. 8 BUILD SOFTWARE TO TEST SOFTWARE
   

   View Slide

9. 9 BUILD SOFTWARE TO TEST SOFTWARE
   9 BUILD SOFTWARE TO TEST SOFTWARE
   A tester on an Agile project will
   work differently than one
   working on a traditional project.
   

   View Slide

10. 10 BUILD SOFTWARE TO TEST SOFTWARE
    2.3.1 Agile Tester Skills
    ● Be positive and solution-oriented with team members and stakeholders
    ● Display critical, quality-oriented, skeptical thinking about the product
    ● Actively acquire information from stakeholders (rather than relying entirely on written specifications)
    ● Accurately evaluate and report test results, test progress, and product quality
    ● Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and
    stakeholders
    ● Collaborate within the team, working in pairs with programmers and other team members
    ● Respond to change quickly, including changing, adding, or improving test cases
    ● Plan and organize their own work
    Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams
    

    View Slide

11. 11 BUILD SOFTWARE TO TEST SOFTWARE
    2.3.2 The Role of a Tester in an Agile Team
    ● Understanding, implementing, and updating the test strategy
    ● Measuring and reporting test coverage across all applicable coverage dimensions
    ● Ensuring proper use of testing tools
    ● Configuring, using, and managing test environments and test data
    ● Reporting defects and working with the team to resolve them
    ● Coaching other team members in relevant aspects of testing
    ● Ensuring the appropriate testing tasks are scheduled during release and iteration planning
    ● Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of
    testability, consistency, and completeness
    ● Participating proactively in team retrospectives, suggesting and implementing improvements
    

    View Slide

12. 12 BUILD SOFTWARE TO TEST SOFTWARE
    Agile is Context for Testing
    

    View Slide

13. 13 BUILD SOFTWARE TO TEST SOFTWARE
    Test Driven Development
    

    View Slide

14. 14 BUILD SOFTWARE TO TEST SOFTWARE
    Examples
    A/B Tests
    Story Tests (written first)
    UX (user experience) testing
    Prototypes (paper or wireframes)
    Simulations
    Exploratory Testing
    Workflows
    System Integration
    (business oriented)
    Usability Testing
    UAT (user acceptance testing)
    Unit Tests
    Component Tests (code level)
    Testing Connectivity
    Performance Testing
    Load Testing
    Security Testing
    Quality Attributes (...ilities)
    Business Facing
    Critique the Product
    Guide Development
    Q2
    Q1
    Q3
    Q4
    Technology Facing
    Agile testing quadrants, from (Gregory, Crispin 2014)
    Agile Testing Quadrants
    

    View Slide

15. 15 BUILD SOFTWARE TO TEST SOFTWARE
    Test Driven Development
    Guinea Pig (“Sea” Pig)
    no relation no relation
    

    View Slide

16. 16 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing Definition
    Software Testing is exploring software with the intent of finding bugs.
    Software Testing is an empirical technical investigation conducted to provide
    stakeholders with information about the quality of the product or service under test.
    Information
    Service
    

    View Slide

17. 17 BUILD SOFTWARE TO TEST SOFTWARE
    All Testing is Exploratory Testing
    https://www.satisfice.com/blog/archives/1509
    

    View Slide

18. 18 BUILD SOFTWARE TO TEST SOFTWARE
    Software Development and Testing
    Models
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development
    

    View Slide

19. 19 BUILD SOFTWARE TO TEST SOFTWARE
    Development and Testing are Two Different Processes
    

    View Slide

20. 20 BUILD SOFTWARE TO TEST SOFTWARE
    Software Definition
    Software is a collection of instructions and data that tell a computer how to work
    The computer (hardware) is able to execute the software. Each instruction causes
    the computer to carry out an operation - moving data, carrying out a computation,
    altering the control flow of instructions or manipulating hardware
    

    View Slide

21. 21 BUILD SOFTWARE TO TEST SOFTWARE
    Examples
    A/B Tests
    Story Tests (written first)
    UX (user experience) testing
    Prototypes (paper or wireframes)
    Simulations
    Exploratory Testing
    Workflows
    System Integration
    (business oriented)
    Usability Testing
    UAT (user acceptance testing)
    Unit Tests
    Component Tests (code level)
    Testing Connectivity
    Performance Testing
    Load Testing
    Security Testing
    Quality Attributes (...ilities)
    Business Facing
    Critique the Product
    Guide Development
    Q2
    Q1
    Q3
    Q4
    Technology Facing
    Agile testing quadrants, from (Gregory, Crispin 2014)
    Agile Testing Quadrants
    Test
    Software
    Build
    Software
    

    View Slide

22. 22 BUILD SOFTWARE TO TEST SOFTWARE
    Build Software to Test Software
    

    View Slide

23. 23 BUILD SOFTWARE TO TEST SOFTWARE
    23 BUILD SOFTWARE TO TEST SOFTWARE
    The team should be relatively small;
    successful teams have been observed
    with as few as three people and as
    many as nine.
    

    View Slide

24. 24 BUILD SOFTWARE TO TEST SOFTWARE
    Database
    Processes
    Software Testing at Scale
    is not
    Quality Assurance
    

    View Slide

25. 25 BUILD SOFTWARE TO TEST SOFTWARE
    Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS)
    

    View Slide

26. 26 BUILD SOFTWARE TO TEST SOFTWARE
    https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/
    

    View Slide

27. 27 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing in Information Service
    ● Accuracy
    ● Relevance
    ● Interpretability
    ● Accessibility
    http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf
    Data Quality
    Intrinsic
    Data Quality
    Contextual
    Data Quality
    Representational
    Data Quality
    Accessibility
    Data Quality
    Believability (1)
    Accuracy (4)
    Objectivity (8)
    Reputation (12)
    Value-added (2)
    Relevancy (3)
    Timeliness (9)
    Completeness (10)
    Appropriate amount of data (19)
    Interpretability (5)
    Ease of understanding (6)
    Representational consistency (13)
    Concise representation (17)
    Accessibility (7)
    Access security (18)
    A Conceptual Framework of Data Quality
    

    View Slide

28. 28 BUILD SOFTWARE TO TEST SOFTWARE
    

    View Slide

29. 29 BUILD SOFTWARE TO TEST SOFTWARE
    Three Things a Software Tester Can Learn
    ● 1
    ● 2
    ● 3
    

    View Slide

30. 30 BUILD SOFTWARE TO TEST SOFTWARE
    AI, Testing AI, AI Testing
    

    View Slide

31. 31 BUILD SOFTWARE TO TEST SOFTWARE
    AI, Testing AI, AI Testing
    

    View Slide

32. 32 BUILD SOFTWARE TO TEST SOFTWARE
    

    View Slide

33. 33 BUILD SOFTWARE TO TEST SOFTWARE
    33 BUILD SOFTWARE TO TEST SOFTWARE
    In a typical conventional computer system, the
    software is programmed by humans using an
    imperative language, which includes constructs
    such as if-then-else and loops. It is relatively
    easy for humans to understand how the system
    transforms inputs into outputs.
    

    View Slide

34. 34 BUILD SOFTWARE TO TEST SOFTWARE
    Challenges of Testing Self-Learning Systems
    ● Unexpected change
    ● Complex acceptance criteria
    ● Insufficient testing time
    ● Resource requirements
    ● Insufficient specifications of operational environment
    ● Complex test environment
    ● Undesirable behavior modifications
    

    View Slide

35. 35 BUILD SOFTWARE TO TEST SOFTWARE
    Data Quality Issues
    

    View Slide

36. 36 BUILD SOFTWARE TO TEST SOFTWARE
    Artificial Intelligence Techniques
    

    View Slide

37. 37 BUILD SOFTWARE TO TEST SOFTWARE
    Humans are Afraid of Artificial Intelligence
    

    View Slide

38. 38 BUILD SOFTWARE TO TEST SOFTWARE
    Test Oracles are Only Covered in AI Testing Syllabus
    Using Test Oracles in Automation
    Doug Hoffman (2006 Google Tech Talks)
    https://youtu.be/Yusd_ogkiaQ
    The Often Overlooked Test Oracle
    Doug Hoffman (2019 Association for Software Testing)
    https://youtu.be/hbJ1xpG7QHs
    http://www.kaner.com/pdfs/automation/
    Auto_Arch.pdf
    

    View Slide

39. 39 BUILD SOFTWARE TO TEST SOFTWARE
    Comparing ISTQB Syllabi
    Certifications Oracle Pages Date
    Foundation Level (CTFL) 2 69 2018-11-11
    AI Testing (CT-AI) 43 58 2021-10-01
    Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30
    Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09
    Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13
    Test Automation Engineer (CT-TAE) 3 60 2016-10-21
    Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30
    * Body of Knowledge contains additional 30 pages
    

    View Slide

40. 40 BUILD SOFTWARE TO TEST SOFTWARE
    Test Automation
    Manual Using Tools Mechanization Automation
    A tool is an object used to extend the ability of an individual to modify features of the surrounding environment
    Automation is the technology by which a process or procedure is performed with minimal human assistance
    

    View Slide

41. 41 BUILD SOFTWARE TO TEST SOFTWARE
    41 BUILD SOFTWARE TO TEST SOFTWARE
    NO
    AUTOMATION
    DRIVER
    ASSISTANCE
    PARTIAL
    AUTOMATION
    CONDITIONAL
    AUTOMATION
    HIGH
    AUTOMATION
    FULL
    AUTOMATION
    Steering and
    acceleration/
    deceleration
    Monitoring
    of driving
    environment
    Fallback when
    automation fails
    Automated
    Control:
    Human driver
    monitors the road
    Automated driving system
    monitors the road
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    0
    1
    2
    3
    4
    5
    Human driver Automated system
    Autonomy
    

    View Slide

42. 42 BUILD SOFTWARE TO TEST SOFTWARE
    Conclusion
    ● Unit Tests and CI/CD is Software
    ● TDD/BDD/ATDD is Software Development
    ● Software Testing and Software Development are Two Different Processes
    ● Testing in Agile Context is Still Testing
    ● Artificial Intelligence is Software
    ● Testing AI is Software Testing
    ● Understanding Autonomy is Necessary to Understand Automation in Testing
    ● Ideas from ISTQB AI and Agile Certifications Can Help You with Conventional Testing
    

    View Slide