A4Q AI and Software Testing

A4Q AI and Software Testing

Iosif Itkin
co-CEO & co-founder, Exactpro

QA Meetup
26 February 2020, Saratov

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro Vimeo channel https://vimeo.com/exactpro
Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software
   exactpro.com
   Iosif Itkin
   co-CEO and co-founder
   Exactpro Systems
   25 February 2020
   

   View Slide

2. 2 Build Software to Test Software
   exactpro.com
   Introduction
   ● Started IT career 20 years ago as a software developer
   ● Exactpro co-CEO and co-founder
   ● Moved into performance and load testing in 2006
   ● Co-founded Exactpro on 9th September 2009
   ● EXTENT Software Testing and AI conference – 23 June 2020
   ● Working on th2.net next generation test tool
   ● ISTQB FL 2018, A4Q AI and Software Testing
   ● Build Software to Test Software
   

   View Slide

3. 3 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Contents
   • A few quick facts about Exactpro
   • A4Q AI and Software Testing
   • How is it different from the ISTQB Foundation Level?
   • What does an exam program consist of?
   • How difficult is it to prepare for it?
   • What training materials are there?
   • Which of these is applicable in practice?
   

   View Slide

4. 4 Build Software to Test Software
   exactpro.com
   EXACTPRO
   A few quick facts about Exactpro
   • A specialist firm focused on functional and non-functional testing of exchanges,
   clearing houses, depositories and other market infrastructures.
   • Incorporated in 2009 with 10 people, our
   company has experienced significant growth as
   satisfied clients require more services; now
   employing 570 specialists worldwide
   (130 in Saratov)
   • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018.
   Exactpro management buyout from LSEG in January 2018. Headquartered in
   the UK, with operations in the US and Eastern Europe.
   • We provide software testing services for mission critical
   technology that underpins global financial markets. Our clients
   are regulated by FCA, Bank of England and their counterparts
   from other countries.
   

   View Slide

5. 5 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Our clients are the leading exchange groups across the globe
   

   View Slide

6. 6 Build Software to Test Software
   exactpro.com
   EXACTPRO
   AI-based Systems in Finance
   Machine Learning in financial organizations:
   - already passed an initial development phase
   - the usage of live ML applications is about to
   dramatically increase over the next three years
   ● Market Surveillance Systems
   ● Conversational Assistants
   ● Algo Trading Systems
   ● Pricing Calculators
   ● Machine Readable News
   ● Insurance Claims
   https://www.bankofengland.co.uk/-
   /media/boe/files/report/2019/machine-learning-in-uk-
   financial-services.pdf
   

   View Slide

7. 7 Build Software to Test Software
   exactpro.com
   EXACTPRO
   Exactpro Partners and Cross Industry Work
   

   View Slide

8. 8 Build Software to Test Software
   exactpro.com
   Since 2012, 155 specialists have passed the ISTQB® Certified
   Tester Foundation Level certification exam in Exactpro
   

   View Slide

9. 9 Build Software to Test Software
   exactpro.com
   https://www.facebook.com/geostqb/
   National ISTQB Board in Georgia
   

   View Slide

10. 10 Build Software to Test Software
    exactpro.com
    EXACTPRO
    How this Syllabus is Organized
    There are three chapters with examinable content. The top-level heading for each
    chapter specifies the time for the chapter; timing is not provided below chapter level.
    For the A4Q AI and Software Testing Foundation training course, the syllabus
    requires a minimum of 17 hours, 10 minutes of instruction, distributed across the
    three chapters as follows:
    Chapter 1: AI and Software Testing Background 360 minutes
    Chapter 2: Testing Artificial Intelligence Systems 400 minutes
    Chapter 3: Using AI to Support Testing 270 minutes
    https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html
    

    View Slide

11. 11 Build Software to Test Software
    exactpro.com
    EXACTPRO
    How this Syllabus is Organized
    There are six chapters with examinable content. The top-level heading for each
    chapter specifies the time for the chapter; timing is not provided below chapter level.
    For accredited training courses, the syllabus requires a minimum of 16.75 hours of
    instruction, distributed across the six chapters as follows:
    Chapter 1: 175 minutes Fundamentals of Testing
    Chapter 2: 100 minutes Testing Throughout the Software Development Lifecycle
    Chapter 3: 135 minutes Static Testing
    Chapter 4: 330 minutes Test Techniques
    Chapter 5: 225 minutes Test Management
    Chapter 6: 40 minutes Tool Support for Testing
    

    View Slide

12. 12 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Similarities
    • 40 Multi-choice Questions
    • 26 Correct Answers to Pass
    • 60 Minutes (you can request 15 more minutes)
    • GASQ (160 EUR) or iSQI (Pearson VUE 175 EUR) in Russia
    • 40 Pages in AIST vs. 64 in ISTQB
    

    View Slide

13. 13 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Difference
    

    View Slide

14. 14 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Mock Exam
    https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/A4Q_AI-and-SW-Testing_MockExam_v1.0.pdf
    

    View Slide

15. 15 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Syllabus
    https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/AI_Software_Testing_Syllabus%20(1.0).pdf
    

    View Slide

16. 16 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Key Aspects of Artificial Intelligence
    1.1 What are Human Intelligence and Artificial Intelligence?
    1.2 History of AI
    1.3 Symbolic AI
    1.4 Sub-symbolic AI
    1.5 Some ML Algorithms in More Detail
    • Bayesian Belief Networks
    • Naïve Bayes classifier
    • Support Vector Machine Algorithm
    • K-means Algorithm
    • Artificial Neural Networks: Perceptron Learning Algorithm
    1.6 Applications and Limits of AI
    

    View Slide

17. 17 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.1 What are Human Intelligence and Artificial Intelligence?
    Types of Intelligence
    

    View Slide

18. 18 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.1 What are Human Intelligence and Artificial Intelligence?
    Turing Test
    https://www.pandorabots.com/mitsuku/
    

    View Slide

19. 19 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.2 History of AI
    

    View Slide

20. 20 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.3 Symbolic AI
    https://youtu.be/GWy4OajL-Dc
    

    View Slide

21. 21 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.3 Symbolic AI
    https://tmpaconf.org/ru/events/tmpa-school-2018/agenda-school-2018-ru
    

    View Slide

22. 22 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.3 Symbolic AI
    

    View Slide

23. 23 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.4 Sub-symbolic AI
    

    View Slide

24. 24 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.4 Sub-symbolic AI
    https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html
    

    View Slide

25. 25 Build Software to Test Software
    exactpro.com
    EXACTPRO
    

    View Slide

26. 26 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.5 Some ML Algorithms in More Detail
    https://www.coursera.org/learn/vvedenie-mashinnoe-obuchenie
    Краткая программа курса:
    Неделя 1. Введение. Примеры задач. Логические методы: решающие деревья и решающие леса.
    Неделя 2. Метрические методы классификации. Линейные методы, стохастический градиент.
    Неделя 3. Метод опорных векторов (SVM). Логистическая регрессия. Метрики качества классификации.
    Неделя 4. Линейная регрессия. Понижение размерности, метод главных компонент.
    Неделя 5. Композиции алгоритмов, градиентный бустинг. Нейронные сети.
    Неделя 6. Кластеризация и визуализация. Частичное обучение.
    Неделя 7. Прикладные задачи анализа данных: постановки и методы решения.
    

    View Slide

27. 27 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.5 Some ML Algorithms in More Detail
    Naïve Bayes classifier
    https://youtu.be/Q8l0Vip5YUw
    https://youtu.be/z5UQyCESW64
    Andrew Ng:
    

    View Slide

28. 28 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.6 Applications and Limits of AI
    Activities of Machine Learning
    https://the-modeling-agency.com/crisp-dm.pdf
    

    View Slide

29. 29 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.6 Applications and Limits of AI
    Possible Biases in AI Systems
    https://schedule.sxsw.com/2019/events/PP83596
    https://www.mckinsey.com/featured-insights/artificial-intelligence/tackling-
    bias-in-artificial-intelligence-and-in-humans
    

    View Slide

30. 30 Build Software to Test Software
    exactpro.com
    https://youtu.be/ki2fsH7Il8E
    29th January 2020
    

    View Slide

31. 31 Build Software to Test Software
    exactpro.com
    https://rbcs-us.com/site/assets/files/3793/a4q_ai_and_software_tester_foundation_course_outline_v0_9.pdf
    

    View Slide

32. 32 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.6 Applications and Limits of AI
    Possible Biases in AI Systems
    

    View Slide

33. 33 Build Software to Test Software
    exactpro.com
    EXACTPRO
    1.6 Applications and Limits of AI
    Ethical Issues in AI
    

    View Slide

34. 34 Build Software to Test Software
    exactpro.com
    EXACTPRO
    2.1 General Problems with Testing AI Systems
    Non-Deterministic Probabilistic Non-Testable Oracle Problem
    

    View Slide

35. 35 Build Software to Test Software
    exactpro.com
    Challenging Non-Functional Requirements
    Daily capacity - 100+ mln transactions
    Peak rates - 40k+ transactions per second
    Average round-trip latency - <100 microseconds
    Availability - 100%
    3000 trx 2.5 cm <1 mm
    

    View Slide

36. 36 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Build Software to Test Software
    Non-Deterministic Probabilistic Non-Testable Oracle Problem
    

    View Slide

37. 37 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Indirect Testing Methods
    Information
    extraction and
    Machine learning
    End-to-End
    Automated Test
    Library
    Whatever it
    takes!
    Test execution
    data and log
    analysis
    Passive Testing
    

    View Slide

38. 38 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Indirect Testing Methods
    Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp-
    content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf
    

    View Slide

39. 39 Build Software to Test Software
    exactpro.com
    EXACTPRO
    2.2 Machine Learning Model Training and Testing
    https://www.deeplearning.ai/
    • Training set — Which you run your learning algorithm on.
    • Development set — Which you use to tune parameters, select
    features, and make other decisions regarding the learning algorithm.
    Sometimes also called the hold-out cross validation set.
    • Test set — which you use to evaluate the performance of the algorithm,
    but not to make any decisions regarding what learning algorithm or
    parameters to use.
    

    View Slide

40. 40 Build Software to Test Software
    exactpro.com
    EXACTPRO
    2.3 AI Test Environments
    By Wbjimmyd - Own work, CC BY-SA 3.0,
    https://commons.wikimedia.org/w/index.php?c
    urid=30859962
    https://en.wikibooks.org/wiki/Artificial_Intelligence/AI_Agents_and_their_Environments
    

    View Slide

41. 41 Build Software to Test Software
    exactpro.com
    EXACTPRO
    AI Systems Quality Characteristics
    - Ability to learn: The capacity of the system to learn from use for the
    system itself, or data and events it is exposed to.
    - Ability to generalize: The ability of the system to apply to different
    and previously unseen scenarios.
    - Trustworthiness: The degree to which the system is trusted by
    stakeholders, for example a health diagnostic
    A4Q AI and Software Testing
    Foundation
    Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html
    Testing the
    Intelligence
    of your AI
    

    View Slide

42. 42 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Ability to Generalize:
    Scope of End-to-End and Negative Testing
    

    View Slide

43. 43 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Using AI to Support Testing
    3.1 AI in Testing
    3.2 Applying AI to Testing Tasks and Quality Management
    3.3 AI in Component Level Test Automation
    3.4 AI in Integration Level or System Level Test Automation
    3.5 AI-based Tool Support for Testing
    

    View Slide

44. 44 Build Software to Test Software
    exactpro.com
    EXACTPRO
    3.1 AI in Testing
    Oracle Problem
    

    View Slide

45. 45 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Tasks AI Can Be Applied To
    

    View Slide

46. 46 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Tasks AI Cannot Be Applied To
    

    View Slide

47. 47 Build Software to Test Software
    exactpro.com
    EXACTPRO
    Tasks AI Cannot Be Applied To
    https://www.istqb.org/documents/ISTQB_The_Vision_on_the_Future_of_Software_Testing_Final.pdf
    

    View Slide

48. 48 Build Software to Test Software
    exactpro.com
    Style-based GANs – Generating and Tuning
    Realistic Artificial Faces:
    https://www.lyrn.ai/2018/12/26/a-style-
    based-generator-architecture-for-generative-
    adversarialnetworks/
    This Person Does Not Exist:
    https://thispersondoesnotexist.com
    Using AI for Test Data Generation
    

    View Slide

49. 49 Build Software to Test Software
    exactpro.com
    Using AI for Test Data Generation
    

    View Slide

50. 50 Build Software to Test Software
    exactpro.com
    3.2 Applying AI to Testing Tasks and Quality Management
    BEFORE
    Extremely high memory consumption was observed
    in Prod system with release 1.2.3.4.5.6 installed
    during ITR and ITCH gateways crashed.
    AFTER
    Extremely high memory consumption was observed in Prod system
    with release 1.2.3.4.5.6 installed during ITR. All ITCH gateways
    consumed up to *** GB of RAM. They crashed, except for three pairs
    of ITCH gateways. Probably this issue can be related to issue
    #1234567. Backend logs, data files, core files, DB dumps are
    attached.
    Won’t Fix Pie Chart
    Won’t Fix
    Fix
    

    View Slide

51. 51 Build Software to Test Software
    exactpro.com
    Supervised Learning. Sailfish
    Machine
    Learning
    Feature engineering,
    dimensionality reduction,
    training & evaluation of the
    model
    Predictions and tips
    Cradle
    Data Warehousing
    Fetching raw data from tools
    under test. Processing it into
    datasets with Data Mart & BI
    support
    Test Execution and
    Analysis
    Predictions and tips for failures in
    order to leverage the smoothness
    and accuracy of the testing outputs
    analysis
    QA Analyst
    Fewer feedback loops, less rework, higher
    quality, faster turnaround, better alignment of
    roles on a project
    

    View Slide

52. 52 Build Software to Test Software
    exactpro.com
    https://unsplash.com/search/photos/san-francisco
    The First IEEE International Conference on Artificial
    Intelligence Testing (IEEE AITest 2019), 4-9 April
    2019, San Francisco East Bay, CA, USA.
    User-Assisted Log Analysis for Quality Control of
    Distributed Fintech Systems
    Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav
    Yavorskiy, Evgenii Tsymbalov, Andrey Novikov and
    Kirill Rudakov.
    IEEE International Conference on Artificial Intelligence Testing
    San Francisco East Bay, USA
    http://ieeeaitests.com/
    

    View Slide

53. 53 Build Software to Test Software
    exactpro.com
    3.3 AI in Component Level Test Automation
    

    View Slide

54. 54 Build Software to Test Software
    exactpro.com
    https://www.aitesting.org/
    https://medium.com/@jarbon
    

    View Slide

55. 55 Build Software to Test Software
    exactpro.com
    

    View Slide

56. 56 Build Software to Test Software
    exactpro.com
    EXACTPRO
    3.5 AI-based Tool Support for Testing
    • Configuration of the System
    • Return on Investment (ROI)
    • Effects on Existing Processes
    • Sensibility of Test Cases
    • Test Case Explosion
    • Maintainability
    • Severity of the Defects Found
    

    View Slide

57. 57 Build Software to Test Software
    exactpro.com
    EXACTPRO
    https://isqi.org/lp/ai-software-testing-certification/
    

    View Slide

58. 58 Build Software to Test Software
    exactpro.com
    13th – 16th April 2020 in Keble College, Oxford University, UK
    

    View Slide

59. 59 Build Software to Test Software
    exactpro.com
    https://www.youtube.com/c/exactprosystems
    

    View Slide