Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ISTQB: AI and Agile Software Testing

ISTQB: AI and Agile Software Testing

Iosif Itkin, CEO and co-founder of Exactpro

Iosif reviewed how AI can be used to support software testing, as well as addressed the challenges associated with testing AI-Based systems. Iosif also highlighted the key software testers’ skills required to successfully contribute to large-scale Agile projects.

https://exactpro.com/events/external/qa-meetup-tbilisi-georgia?utm_source=Facebook&utm_medium=Organic&utm_campaign=Iosif-Itkin-16-06

QA Meetup. 18 June 2022

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Exactpro
PRO

June 20, 2022
Tweet

More Decks by Exactpro

Other Decks in Technology

Transcript

  1. 1 BUILD SOFTWARE TO TEST SOFTWARE BUILD SOFTWARE TO TEST

    SOFTWARE exactpro.com ISTQB: AI and Agile Software Testing Iosif Itkin 18 JUNE 2022 WYNDHAM GRAND TBILISI, BALLROOM Exactpro co-CEO and co-founder
  2. 2 BUILD SOFTWARE TO TEST SOFTWARE Contents • ISTQB Portfolio

    of Certifications • Comparing Foundation, AI and Agile Syllabi • What Skills are Specific for Agile Testers • What Challenges are Specific for AI Systems • Why Study AI and Agile
  3. 3 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certifications

  4. 4 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Portfolio of Certifications

  5. 5 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certifications

    Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages
  6. 6 BUILD SOFTWARE TO TEST SOFTWARE Foundation Level Agile Tester

    (CTFL-AT)
  7. 7 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Syllabus Agile

    Technical Tester (ATT)
  8. 8 BUILD SOFTWARE TO TEST SOFTWARE

  9. 9 BUILD SOFTWARE TO TEST SOFTWARE 9 BUILD SOFTWARE TO

    TEST SOFTWARE A tester on an Agile project will work differently than one working on a traditional project.
  10. 10 BUILD SOFTWARE TO TEST SOFTWARE 2.3.1 Agile Tester Skills

    • Be positive and solution-oriented with team members and stakeholders • Display critical, quality-oriented, skeptical thinking about the product • Actively acquire information from stakeholders (rather than relying entirely on written specifications) • Accurately evaluate and report test results, test progress, and product quality • Work effectively to define testable user stories, especially acceptance criteria, with customer representatives and stakeholders • Collaborate within the team, working in pairs with programmers and other team members • Respond to change quickly, including changing, adding, or improving test cases • Plan and organize their own work Continuous skills growth, including interpersonal skills growth, is essential for all testers, including those on Agile teams
  11. 11 BUILD SOFTWARE TO TEST SOFTWARE 2.3.2 The Role of

    a Tester in an Agile Team • Understanding, implementing, and updating the test strategy • Measuring and reporting test coverage across all applicable coverage dimensions • Ensuring proper use of testing tools • Configuring, using, and managing test environments and test data • Reporting defects and working with the team to resolve them • Coaching other team members in relevant aspects of testing • Ensuring the appropriate testing tasks are scheduled during release and iteration planning • Actively collaborating with developers and business stakeholders to clarify requirements, especially in terms of testability, consistency, and completeness • Participating proactively in team retrospectives, suggesting and implementing improvements
  12. 12 BUILD SOFTWARE TO TEST SOFTWARE Agile is Context for

    Testing
  13. 13 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development

  14. 14 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story

    Tests (written first) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workflows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants
  15. 15 BUILD SOFTWARE TO TEST SOFTWARE Test Driven Development Guinea

    Pig (“Sea” Pig) no relation no relation
  16. 16 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Definition Software

    Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test. Information Service
  17. 17 BUILD SOFTWARE TO TEST SOFTWARE All Testing is Exploratory

    Testing https://www.satisfice.com/blog/archives/1509
  18. 18 BUILD SOFTWARE TO TEST SOFTWARE Software Development and Testing

    Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development
  19. 19 BUILD SOFTWARE TO TEST SOFTWARE Development and Testing are

    Two Different Processes
  20. 20 BUILD SOFTWARE TO TEST SOFTWARE Software Definition Software is

    a collection of instructions and data that tell a computer how to work The computer (hardware) is able to execute the software. Each instruction causes the computer to carry out an operation - moving data, carrying out a computation, altering the control flow of instructions or manipulating hardware
  21. 21 BUILD SOFTWARE TO TEST SOFTWARE Examples A/B Tests Story

    Tests (written first) UX (user experience) testing Prototypes (paper or wireframes) Simulations Exploratory Testing Workflows System Integration (business oriented) Usability Testing UAT (user acceptance testing) Unit Tests Component Tests (code level) Testing Connectivity Performance Testing Load Testing Security Testing Quality Attributes (...ilities) Business Facing Critique the Product Guide Development Q2 Q1 Q3 Q4 Technology Facing Agile testing quadrants, from (Gregory, Crispin 2014) Agile Testing Quadrants Test Software Build Software
  22. 22 BUILD SOFTWARE TO TEST SOFTWARE Build Software to Test

    Software
  23. 23 BUILD SOFTWARE TO TEST SOFTWARE 23 BUILD SOFTWARE TO

    TEST SOFTWARE The team should be relatively small; successful teams have been observed with as few as three people and as many as nine.
  24. 24 BUILD SOFTWARE TO TEST SOFTWARE Database Processes Software Testing

    at Scale is not Quality Assurance
  25. 25 BUILD SOFTWARE TO TEST SOFTWARE Advanced Level Agile Test

    Leadership at Scale (CTAL-ATLaS)
  26. 26 BUILD SOFTWARE TO TEST SOFTWARE https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/

  27. 27 BUILD SOFTWARE TO TEST SOFTWARE Software Testing in Information

    Service • Accuracy • Relevance • Interpretability • Accessibility http://mitiq.mit.edu/Documents/Publications/TDQMpub/14_Beyond_Accuracy.pdf Data Quality Intrinsic Data Quality Contextual Data Quality Representational Data Quality Accessibility Data Quality Believability (1) Accuracy (4) Objectivity (8) Reputation (12) Value-added (2) Relevancy (3) Timeliness (9) Completeness (10) Appropriate amount of data (19) Interpretability (5) Ease of understanding (6) Representational consistency (13) Concise representation (17) Accessibility (7) Access security (18) A Conceptual Framework of Data Quality
  28. 28 BUILD SOFTWARE TO TEST SOFTWARE

  29. 29 BUILD SOFTWARE TO TEST SOFTWARE Three Things a Software

    Tester Can Learn • 1 • 2 • 3
  30. 30 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI

    Testing
  31. 31 BUILD SOFTWARE TO TEST SOFTWARE AI, Testing AI, AI

    Testing
  32. 32 BUILD SOFTWARE TO TEST SOFTWARE

  33. 33 BUILD SOFTWARE TO TEST SOFTWARE 33 BUILD SOFTWARE TO

    TEST SOFTWARE In a typical conventional computer system, the software is programmed by humans using an imperative language, which includes constructs such as if-then-else and loops. It is relatively easy for humans to understand how the system transforms inputs into outputs.
  34. 34 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing Self-Learning

    Systems • Unexpected change • Complex acceptance criteria • Insufficient testing time • Resource requirements • Insufficient specifications of operational environment • Complex test environment • Undesirable behavior modifications
  35. 35 BUILD SOFTWARE TO TEST SOFTWARE Data Quality Issues

  36. 36 BUILD SOFTWARE TO TEST SOFTWARE Artificial Intelligence Techniques

  37. 37 BUILD SOFTWARE TO TEST SOFTWARE Humans are Afraid of

    Artificial Intelligence
  38. 38 BUILD SOFTWARE TO TEST SOFTWARE Test Oracles are Only

    Covered in AI Testing Syllabus Using Test Oracles in Automation Doug Hoffman (2006 Google Tech Talks) https://youtu.be/Yusd_ogkiaQ The Often Overlooked Test Oracle Doug Hoffman (2019 Association for Software Testing) https://youtu.be/hbJ1xpG7QHs http://www.kaner.com/pdfs/automation/ Auto_Arch.pdf
  39. 39 BUILD SOFTWARE TO TEST SOFTWARE Comparing ISTQB Syllabi Certifications

    Oracle Pages Date Foundation Level (CTFL) 2 69 2018-11-11 AI Testing (CT-AI) 43 58 2021-10-01 Foundation Level Agile Tester (CTFL-AT) 4 30 2014-09-30 Advanced Level Syllabus Agile Technical Tester (ATT) 0 20 2019-12-09 Advanced Level Agile Test Leadership at Scale (CTAL-ATLaS) 0 6* 2022-05-13 Test Automation Engineer (CT-TAE) 3 60 2016-10-21 Advanced Level Test Analyst (CTAL-TA) 5 38 2021-06-30 * Body of Knowledge contains additional 30 pages
  40. 40 BUILD SOFTWARE TO TEST SOFTWARE Test Automation Manual Using

    Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance
  41. 41 BUILD SOFTWARE TO TEST SOFTWARE 41 BUILD SOFTWARE TO

    TEST SOFTWARE NO AUTOMATION DRIVER ASSISTANCE PARTIAL AUTOMATION CONDITIONAL AUTOMATION HIGH AUTOMATION FULL AUTOMATION Steering and acceleration/ deceleration Monitoring of driving environment Fallback when automation fails Automated Control: Human driver monitors the road Automated driving system monitors the road SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES SOME DRIVING MODES 0 1 2 3 4 5 Human driver Automated system Autonomy
  42. 42 BUILD SOFTWARE TO TEST SOFTWARE Conclusion • Unit Tests

    and CI/CD is Software • TDD/BDD/ATDD is Software Development • Software Testing and Software Development are Two Different Processes • Testing in Agile Context is Still Testing • Artificial Intelligence is Software • Testing AI is Software Testing • Understanding Autonomy is Necessary to Understand Automation in Testing • Ideas from ISTQB AI and Agile Certifications Can Help You with Conventional Testing