Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ISTQB: AI and Agile Software Testing

ISTQB: AI and Agile Software Testing

Iosif Itkin, CEO and co-founder of Exactpro

Iosif reviewed how AI can be used to support software testing, as well as addressed the challenges associated with testing AI-Based systems. Iosif also highlighted the key software testers’ skills required to successfully contribute to large-scale Agile projects.

https://exactpro.com/events/external/qa-meetup-tbilisi-georgia?utm_source=Facebook&utm_medium=Organic&utm_campaign=Iosif-Itkin-16-06

QA Meetup. 18 June 2022

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Exactpro
PRO

June 20, 2022
Tweet

More Decks by Exactpro

Other Decks in Technology

Transcript

  1. 1 BUILD SOFTWARE TO TEST SOFTWARE
    BUILD SOFTWARE TO TEST SOFTWARE
    exactpro.com
    ISTQB: AI and Agile
    Software Testing
    Iosif Itkin
    18 JUNE 2022
    WYNDHAM GRAND TBILISI, BALLROOM
    Exactpro co-CEO and co-founder

    View Slide

  2. 2 BUILD SOFTWARE TO TEST SOFTWARE
    Contents
    ● ISTQB Portfolio of Certifications
    ● Comparing Foundation, AI and Agile Syllabi
    ● What Skills are Specific for Agile Testers
    ● What Challenges are Specific for AI Systems
    ● Why Study AI and Agile

    View Slide

  3. 3 BUILD SOFTWARE TO TEST SOFTWARE
    ISTQB Portfolio of Certifications

    View Slide

  4. 4 BUILD SOFTWARE TO TEST SOFTWARE
    ISTQB Portfolio of Certifications

    View Slide

  5. 5 BUILD SOFTWARE TO TEST SOFTWARE
    Comparing ISTQB Syllabi
    Certifications Oracle Pages Date
    Foundation Level (CTFL) 2 69 2018-11-11
    AI Testing (CT-AI) 43 58 2021-10-01
    Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30
    Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09
    Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13
    Test Automation Engineer (CT-TAE) 3 60 2016-10-21
    Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30
    * Body of Knowledge contains additional 30 pages

    View Slide

  6. 6 BUILD SOFTWARE TO TEST SOFTWARE
    Foundation Level Agile Tester (CTFL-AT)

    View Slide

  7. 7 BUILD SOFTWARE TO TEST SOFTWARE
    Advanced Level Syllabus Agile Technical Tester (ATT)

    View Slide

  8. 8 BUILD SOFTWARE TO TEST SOFTWARE

    View Slide

  9. 9 BUILD SOFTWARE TO TEST SOFTWARE
    9 BUILD SOFTWARE TO TEST SOFTWARE
    A tester on an Agile project will
    work differently than one
    working on a traditional project.

    View Slide

  10. 10 BUILD SOFTWARE TO TEST SOFTWARE
    2.3.1 Agile Tester Skills
    ● Be positive and solution-oriented with team members and stakeholders
    ● Display critical, quality-oriented, skeptical thinking about the product
    ● Actively acquire information from stakeholders (rather than relying entirely on written specifications)
    ● Accurately evaluate and report test results, test progress, and product quality
    ● Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and
    stakeholders
    ● Collaborate within the team, working in pairs with programmers and other team members
    ● Respond to change quickly, including changing, adding, or improving test cases
    ● Plan and organize their own work
    Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams

    View Slide

  11. 11 BUILD SOFTWARE TO TEST SOFTWARE
    2.3.2 The Role of a Tester in an Agile Team
    ● Understanding, implementing, and updating the test strategy
    ● Measuring and reporting test coverage across all applicable coverage dimensions
    ● Ensuring proper use of testing tools
    ● Configuring, using, and managing test environments and test data
    ● Reporting defects and working with the team to resolve them
    ● Coaching other team members in relevant aspects of testing
    ● Ensuring the appropriate testing tasks are scheduled during release and iteration planning
    ● Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of
    testability, consistency, and completeness
    ● Participating proactively in team retrospectives, suggesting and implementing improvements

    View Slide

  12. 12 BUILD SOFTWARE TO TEST SOFTWARE
    Agile is Context for Testing

    View Slide

  13. 13 BUILD SOFTWARE TO TEST SOFTWARE
    Test Driven Development

    View Slide

  14. 14 BUILD SOFTWARE TO TEST SOFTWARE
    Examples
    A/B Tests
    Story Tests (written first)
    UX (user experience) testing
    Prototypes (paper or wireframes)
    Simulations
    Exploratory Testing
    Workflows
    System Integration
    (business oriented)
    Usability Testing
    UAT (user acceptance testing)
    Unit Tests
    Component Tests (code level)
    Testing Connectivity
    Performance Testing
    Load Testing
    Security Testing
    Quality Attributes (...ilities)
    Business Facing
    Critique the Product
    Guide Development
    Q2
    Q1
    Q3
    Q4
    Technology Facing
    Agile testing quadrants, from (Gregory, Crispin 2014)
    Agile Testing Quadrants

    View Slide

  15. 15 BUILD SOFTWARE TO TEST SOFTWARE
    Test Driven Development
    Guinea Pig (“Sea” Pig)
    no relation no relation

    View Slide

  16. 16 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing Definition
    Software Testing is exploring software with the intent of finding bugs.
    Software Testing is an empirical technical investigation conducted to provide
    stakeholders with information about the quality of the product or service under test.
    Information
    Service

    View Slide

  17. 17 BUILD SOFTWARE TO TEST SOFTWARE
    All Testing is Exploratory Testing
    https://www.satisfice.com/blog/archives/1509

    View Slide

  18. 18 BUILD SOFTWARE TO TEST SOFTWARE
    Software Development and Testing
    Models
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development

    View Slide

  19. 19 BUILD SOFTWARE TO TEST SOFTWARE
    Development and Testing are Two Different Processes

    View Slide

  20. 20 BUILD SOFTWARE TO TEST SOFTWARE
    Software Definition
    Software is a collection of instructions and data that tell a computer how to work
    The computer (hardware) is able to execute the software. Each instruction causes
    the computer to carry out an operation - moving data, carrying out a computation,
    altering the control flow of instructions or manipulating hardware

    View Slide

  21. 21 BUILD SOFTWARE TO TEST SOFTWARE
    Examples
    A/B Tests
    Story Tests (written first)
    UX (user experience) testing
    Prototypes (paper or wireframes)
    Simulations
    Exploratory Testing
    Workflows
    System Integration
    (business oriented)
    Usability Testing
    UAT (user acceptance testing)
    Unit Tests
    Component Tests (code level)
    Testing Connectivity
    Performance Testing
    Load Testing
    Security Testing
    Quality Attributes (...ilities)
    Business Facing
    Critique the Product
    Guide Development
    Q2
    Q1
    Q3
    Q4
    Technology Facing
    Agile testing quadrants, from (Gregory, Crispin 2014)
    Agile Testing Quadrants
    Test
    Software
    Build
    Software

    View Slide

  22. 22 BUILD SOFTWARE TO TEST SOFTWARE
    Build Software to Test Software

    View Slide

  23. 23 BUILD SOFTWARE TO TEST SOFTWARE
    23 BUILD SOFTWARE TO TEST SOFTWARE
    The team should be relatively small;
    successful teams have been observed
    with as few as three people and as
    many as nine.

    View Slide

  24. 24 BUILD SOFTWARE TO TEST SOFTWARE
    Database
    Processes
    Software Testing at Scale
    is not
    Quality Assurance

    View Slide

  25. 25 BUILD SOFTWARE TO TEST SOFTWARE
    Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS)

    View Slide

  26. 26 BUILD SOFTWARE TO TEST SOFTWARE
    https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

    View Slide

  27. 27 BUILD SOFTWARE TO TEST SOFTWARE
    Software Testing in Information Service
    ● Accuracy
    ● Relevance
    ● Interpretability
    ● Accessibility
    http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf
    Data Quality
    Intrinsic
    Data Quality
    Contextual
    Data Quality
    Representational
    Data Quality
    Accessibility
    Data Quality
    Believability (1)
    Accuracy (4)
    Objectivity (8)
    Reputation (12)
    Value-added (2)
    Relevancy (3)
    Timeliness (9)
    Completeness (10)
    Appropriate amount of data (19)
    Interpretability (5)
    Ease of understanding (6)
    Representational consistency (13)
    Concise representation (17)
    Accessibility (7)
    Access security (18)
    A Conceptual Framework of Data Quality

    View Slide

  28. 28 BUILD SOFTWARE TO TEST SOFTWARE

    View Slide

  29. 29 BUILD SOFTWARE TO TEST SOFTWARE
    Three Things a Software Tester Can Learn
    ● 1
    ● 2
    ● 3

    View Slide

  30. 30 BUILD SOFTWARE TO TEST SOFTWARE
    AI, Testing AI, AI Testing

    View Slide

  31. 31 BUILD SOFTWARE TO TEST SOFTWARE
    AI, Testing AI, AI Testing

    View Slide

  32. 32 BUILD SOFTWARE TO TEST SOFTWARE

    View Slide

  33. 33 BUILD SOFTWARE TO TEST SOFTWARE
    33 BUILD SOFTWARE TO TEST SOFTWARE
    In a typical conventional computer system, the
    software is programmed by humans using an
    imperative language, which includes constructs
    such as if-then-else and loops. It is relatively
    easy for humans to understand how the system
    transforms inputs into outputs.

    View Slide

  34. 34 BUILD SOFTWARE TO TEST SOFTWARE
    Challenges of Testing Self-Learning Systems
    ● Unexpected change
    ● Complex acceptance criteria
    ● Insufficient testing time
    ● Resource requirements
    ● Insufficient specifications of operational environment
    ● Complex test environment
    ● Undesirable behavior modifications

    View Slide

  35. 35 BUILD SOFTWARE TO TEST SOFTWARE
    Data Quality Issues

    View Slide

  36. 36 BUILD SOFTWARE TO TEST SOFTWARE
    Artificial Intelligence Techniques

    View Slide

  37. 37 BUILD SOFTWARE TO TEST SOFTWARE
    Humans are Afraid of Artificial Intelligence

    View Slide

  38. 38 BUILD SOFTWARE TO TEST SOFTWARE
    Test Oracles are Only Covered in AI Testing Syllabus
    Using Test Oracles in Automation
    Doug Hoffman (2006 Google Tech Talks)
    https://youtu.be/Yusd_ogkiaQ
    The Often Overlooked Test Oracle
    Doug Hoffman (2019 Association for Software Testing)
    https://youtu.be/hbJ1xpG7QHs
    http://www.kaner.com/pdfs/automation/
    Auto_Arch.pdf

    View Slide

  39. 39 BUILD SOFTWARE TO TEST SOFTWARE
    Comparing ISTQB Syllabi
    Certifications Oracle Pages Date
    Foundation Level (CTFL) 2 69 2018-11-11
    AI Testing (CT-AI) 43 58 2021-10-01
    Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30
    Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09
    Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13
    Test Automation Engineer (CT-TAE) 3 60 2016-10-21
    Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30
    * Body of Knowledge contains additional 30 pages

    View Slide

  40. 40 BUILD SOFTWARE TO TEST SOFTWARE
    Test Automation
    Manual Using Tools Mechanization Automation
    A tool is an object used to extend the ability of an individual to modify features of the surrounding environment
    Automation is the technology by which a process or procedure is performed with minimal human assistance

    View Slide

  41. 41 BUILD SOFTWARE TO TEST SOFTWARE
    41 BUILD SOFTWARE TO TEST SOFTWARE
    NO
    AUTOMATION
    DRIVER
    ASSISTANCE
    PARTIAL
    AUTOMATION
    CONDITIONAL
    AUTOMATION
    HIGH
    AUTOMATION
    FULL
    AUTOMATION
    Steering and
    acceleration/
    deceleration
    Monitoring
    of driving
    environment
    Fallback when
    automation fails
    Automated
    Control:
    Human driver
    monitors the road
    Automated driving system
    monitors the road
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    SOME DRIVING
    MODES
    0
    1
    2
    3
    4
    5
    Human driver Automated system
    Autonomy

    View Slide

  42. 42 BUILD SOFTWARE TO TEST SOFTWARE
    Conclusion
    ● Unit Tests and CI/CD is Software
    ● TDD/BDD/ATDD is Software Development
    ● Software Testing and Software Development are Two Different Processes
    ● Testing in Agile Context is Still Testing
    ● Artificial Intelligence is Software
    ● Testing AI is Software Testing
    ● Understanding Autonomy is Necessary to Understand Automation in Testing
    ● Ideas from ISTQB AI and Agile Certifications Can Help You with Conventional Testing

    View Slide