1 Build Software to Test Software exactpro.com Iosif Itkin co-CEO and co-founder Exactpro Systems 25 February 2020

2 Build Software to Test Software exactpro.com Introduction ● Started IT career 20 years ago as a software developer ● Exactpro co-CEO and co-founder ● Moved into performance and load testing in 2006 ● Co-founded Exactpro on 9th September 2009 ● EXTENT Software Testing and AI conference – 23 June 2020 ● Working on th2.net next generation test tool ● ISTQB FL 2018, A4Q AI and Software Testing ● Build Software to Test Software

3 Build Software to Test Software exactpro.com EXACTPRO Contents • A few quick facts about Exactpro • A4Q AI and Software Testing • How is it different from the ISTQB Foundation Level? • What does an exam program consist of? • How difficult is it to prepare for it? • What training materials are there? • Which of these is applicable in practice?

4 Build Software to Test Software exactpro.com EXACTPRO A few quick facts about Exactpro • A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories and other market infrastructures. • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 570 specialists worldwide (130 in Saratov) • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG in January 2018. Headquartered in the UK, with operations in the US and Eastern Europe. • We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries.

5 Build Software to Test Software exactpro.com EXACTPRO Our clients are the leading exchange groups across the globe

6 Build Software to Test Software exactpro.com EXACTPRO AI-based Systems in Finance Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years ● Market Surveillance Systems ● Conversational Assistants ● Algo Trading Systems ● Pricing Calculators ● Machine Readable News ● Insurance Claims https://www.bankofengland.co.uk/- /media/boe/files/report/2019/machine-learning-in-uk- financial-services.pdf

7 Build Software to Test Software exactpro.com EXACTPRO Exactpro Partners and Cross Industry Work

8 Build Software to Test Software exactpro.com Since 2012, 155 specialists have passed the ISTQB® Certified Tester Foundation Level certification exam in Exactpro

9 Build Software to Test Software exactpro.com https://www.facebook.com/geostqb/ National ISTQB Board in Georgia

10 Build Software to Test Software exactpro.com EXACTPRO How this Syllabus is Organized There are three chapters with examinable content. The top-level heading for each chapter specifies the time for the chapter; timing is not provided below chapter level. For the A4Q AI and Software Testing Foundation training course, the syllabus requires a minimum of 17 hours, 10 minutes of instruction, distributed across the three chapters as follows: Chapter 1: AI and Software Testing Background 360 minutes Chapter 2: Testing Artificial Intelligence Systems 400 minutes Chapter 3: Using AI to Support Testing 270 minutes https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html

11 Build Software to Test Software exactpro.com EXACTPRO How this Syllabus is Organized There are six chapters with examinable content. The top-level heading for each chapter specifies the time for the chapter; timing is not provided below chapter level. For accredited training courses, the syllabus requires a minimum of 16.75 hours of instruction, distributed across the six chapters as follows: Chapter 1: 175 minutes Fundamentals of Testing Chapter 2: 100 minutes Testing Throughout the Software Development Lifecycle Chapter 3: 135 minutes Static Testing Chapter 4: 330 minutes Test Techniques Chapter 5: 225 minutes Test Management Chapter 6: 40 minutes Tool Support for Testing

12 Build Software to Test Software exactpro.com EXACTPRO Similarities • 40 Multi-choice Questions • 26 Correct Answers to Pass • 60 Minutes (you can request 15 more minutes) • GASQ (160 EUR) or iSQI (Pearson VUE 175 EUR) in Russia • 40 Pages in AIST vs. 64 in ISTQB

13 Build Software to Test Software exactpro.com EXACTPRO Difference

14 Build Software to Test Software exactpro.com EXACTPRO Mock Exam https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/A4Q_AI-and-SW-Testing_MockExam_v1.0.pdf

15 Build Software to Test Software exactpro.com EXACTPRO Syllabus https://www.gasq.org/files/content/gasq/downloads/certification/A4Q%20AI%20&%20Software%20Testing/AI_Software_Testing_Syllabus%20(1.0).pdf

16 Build Software to Test Software exactpro.com EXACTPRO Key Aspects of Artificial Intelligence 1.1 What are Human Intelligence and Artificial Intelligence? 1.2 History of AI 1.3 Symbolic AI 1.4 Sub-symbolic AI 1.5 Some ML Algorithms in More Detail • Bayesian Belief Networks • Naïve Bayes classifier • Support Vector Machine Algorithm • K-means Algorithm • Artificial Neural Networks: Perceptron Learning Algorithm 1.6 Applications and Limits of AI

17 Build Software to Test Software exactpro.com EXACTPRO 1.1 What are Human Intelligence and Artificial Intelligence? Types of Intelligence

18 Build Software to Test Software exactpro.com EXACTPRO 1.1 What are Human Intelligence and Artificial Intelligence? Turing Test https://www.pandorabots.com/mitsuku/

19 Build Software to Test Software exactpro.com EXACTPRO 1.2 History of AI

20 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic AI https://youtu.be/GWy4OajL-Dc

21 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic AI https://tmpaconf.org/ru/events/tmpa-school-2018/agenda-school-2018-ru

22 Build Software to Test Software exactpro.com EXACTPRO 1.3 Symbolic AI

23 Build Software to Test Software exactpro.com EXACTPRO 1.4 Sub-symbolic AI

24 Build Software to Test Software exactpro.com EXACTPRO 1.4 Sub-symbolic AI https://www.immuniweb.com/blog/machine-learning-AI-deep-learning-terms.html

25 Build Software to Test Software exactpro.com EXACTPRO

26 Build Software to Test Software exactpro.com EXACTPRO 1.5 Some ML Algorithms in More Detail https://www.coursera.org/learn/vvedenie-mashinnoe-obuchenie Краткая программа курса: Неделя 1. Введение. Примеры задач. Логические методы: решающие деревья и решающие леса. Неделя 2. Метрические методы классификации. Линейные методы, стохастический градиент. Неделя 3. Метод опорных векторов (SVM). Логистическая регрессия. Метрики качества классификации. Неделя 4. Линейная регрессия. Понижение размерности, метод главных компонент. Неделя 5. Композиции алгоритмов, градиентный бустинг. Нейронные сети. Неделя 6. Кластеризация и визуализация. Частичное обучение. Неделя 7. Прикладные задачи анализа данных: постановки и методы решения.

27 Build Software to Test Software exactpro.com EXACTPRO 1.5 Some ML Algorithms in More Detail Naïve Bayes classifier https://youtu.be/Q8l0Vip5YUw https://youtu.be/z5UQyCESW64 Andrew Ng:

28 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications and Limits of AI Activities of Machine Learning https://the-modeling-agency.com/crisp-dm.pdf

29 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications and Limits of AI Possible Biases in AI Systems https://schedule.sxsw.com/2019/events/PP83596 https://www.mckinsey.com/featured-insights/artificial-intelligence/tackling- bias-in-artificial-intelligence-and-in-humans

30 Build Software to Test Software exactpro.com https://youtu.be/ki2fsH7Il8E 29th January 2020

31 Build Software to Test Software exactpro.com https://rbcs-us.com/site/assets/files/3793/a4q_ai_and_software_tester_foundation_course_outline_v0_9.pdf

32 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications and Limits of AI Possible Biases in AI Systems

33 Build Software to Test Software exactpro.com EXACTPRO 1.6 Applications and Limits of AI Ethical Issues in AI

34 Build Software to Test Software exactpro.com EXACTPRO 2.1 General Problems with Testing AI Systems Non-Deterministic Probabilistic Non-Testable Oracle Problem

35 Build Software to Test Software exactpro.com Challenging Non-Functional Requirements Daily capacity - 100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds Availability - 100% 3000 trx 2.5 cm <1 mm

36 Build Software to Test Software exactpro.com EXACTPRO Build Software to Test Software Non-Deterministic Probabilistic Non-Testable Oracle Problem

37 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing Methods Information extraction and Machine learning End-to-End Automated Test Library Whatever it takes! Test execution data and log analysis Passive Testing

38 Build Software to Test Software exactpro.com EXACTPRO Indirect Testing Methods Donaldson, A. (2018). “Metamorphic Testing for Non-testable Systems.” https://www.testandverification.com/wp- content/uploads/2018/VF_2018/Alastair_DonaldsonGraphicsFuzz_ICL.pdf

39 Build Software to Test Software exactpro.com EXACTPRO 2.2 Machine Learning Model Training and Testing https://www.deeplearning.ai/ • Training set — Which you run your learning algorithm on. • Development set — Which you use to tune parameters, select features, and make other decisions regarding the learning algorithm. Sometimes also called the hold-out cross validation set. • Test set — which you use to evaluate the performance of the algorithm, but not to make any decisions regarding what learning algorithm or parameters to use.

40 Build Software to Test Software exactpro.com EXACTPRO 2.3 AI Test Environments By Wbjimmyd - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?c urid=30859962 https://en.wikibooks.org/wiki/Artificial_Intelligence/AI_Agents_and_their_Environments

41 Build Software to Test Software exactpro.com EXACTPRO AI Systems Quality Characteristics - Ability to learn: The capacity of the system to learn from use for the system itself, or data and events it is exposed to. - Ability to generalize: The ability of the system to apply to different and previously unseen scenarios. - Trustworthiness: The degree to which the system is trusted by stakeholders, for example a health diagnostic A4Q AI and Software Testing Foundation Syllabus https://www.gasq.org/en/exam-modules/a4q-ai-and-software-testing.html Testing the Intelligence of your AI

42 Build Software to Test Software exactpro.com EXACTPRO Ability to Generalize: Scope of End-to-End and Negative Testing

43 Build Software to Test Software exactpro.com EXACTPRO Using AI to Support Testing 3.1 AI in Testing 3.2 Applying AI to Testing Tasks and Quality Management 3.3 AI in Component Level Test Automation 3.4 AI in Integration Level or System Level Test Automation 3.5 AI-based Tool Support for Testing

44 Build Software to Test Software exactpro.com EXACTPRO 3.1 AI in Testing Oracle Problem

45 Build Software to Test Software exactpro.com EXACTPRO Tasks AI Can Be Applied To

46 Build Software to Test Software exactpro.com EXACTPRO Tasks AI Cannot Be Applied To

47 Build Software to Test Software exactpro.com EXACTPRO Tasks AI Cannot Be Applied To https://www.istqb.org/documents/ISTQB_The_Vision_on_the_Future_of_Software_Testing_Final.pdf

48 Build Software to Test Software exactpro.com Style-based GANs – Generating and Tuning Realistic Artificial Faces: https://www.lyrn.ai/2018/12/26/a-style- based-generator-architecture-for-generative- adversarialnetworks/ This Person Does Not Exist: https://thispersondoesnotexist.com Using AI for Test Data Generation

49 Build Software to Test Software exactpro.com Using AI for Test Data Generation

50 Build Software to Test Software exactpro.com 3.2 Applying AI to Testing Tasks and Quality Management BEFORE Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR and ITCH gateways crashed. AFTER Extremely high memory consumption was observed in Prod system with release 1.2.3.4.5.6 installed during ITR. All ITCH gateways consumed up to *** GB of RAM. They crashed, except for three pairs of ITCH gateways. Probably this issue can be related to issue #1234567. Backend logs, data files, core files, DB dumps are attached. Won’t Fix Pie Chart Won’t Fix Fix

51 Build Software to Test Software exactpro.com Supervised Learning. Sailfish Machine Learning Feature engineering, dimensionality reduction, training & evaluation of the model Predictions and tips Cradle Data Warehousing Fetching raw data from tools under test. Processing it into datasets with Data Mart & BI support Test Execution and Analysis Predictions and tips for failures in order to leverage the smoothness and accuracy of the testing outputs analysis QA Analyst Fewer feedback loops, less rework, higher quality, faster turnaround, better alignment of roles on a project

52 Build Software to Test Software exactpro.com https://unsplash.com/search/photos/san-francisco The First IEEE International Conference on Artificial Intelligence Testing (IEEE AITest 2019), 4-9 April 2019, San Francisco East Bay, CA, USA. User-Assisted Log Analysis for Quality Control of Distributed Fintech Systems Iosif Itkin, Anna Gromova, Anton Sitnikov, Rostislav Yavorskiy, Evgenii Tsymbalov, Andrey Novikov and Kirill Rudakov. IEEE International Conference on Artificial Intelligence Testing San Francisco East Bay, USA http://ieeeaitests.com/

53 Build Software to Test Software exactpro.com 3.3 AI in Component Level Test Automation

54 Build Software to Test Software exactpro.com https://www.aitesting.org/ https://medium.com/@jarbon

55 Build Software to Test Software exactpro.com

56 Build Software to Test Software exactpro.com EXACTPRO 3.5 AI-based Tool Support for Testing • Configuration of the System • Return on Investment (ROI) • Effects on Existing Processes • Sensibility of Test Cases • Test Case Explosion • Maintainability • Severity of the Defects Found

57 Build Software to Test Software exactpro.com EXACTPRO https://isqi.org/lp/ai-software-testing-certification/

58 Build Software to Test Software exactpro.com 13th – 16th April 2020 in Keble College, Oxford University, UK

59 Build Software to Test Software exactpro.com https://www.youtube.com/c/exactprosystems