AI Development at Exactpro

Murad Mamedov, AI Researcher, Exactpro

Exactpro Webinar for Tbilisi State University | A Webinar on Software Testing and AI
16.06.2020

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. Build Software to Test Software exactpro.com AI Development at Exactpro Murad Mamedov, AI Researcher TSU Webinar, June 2020
2. 2 Build Software to Test Software exactpro.com Plan - Machine How we research AI applications in Software Testing and how we build AI Tool on top of this - Platform Test Harness platform of new generation. Its overview from AI perspective - People How we prepare Software Testers for new set of tasks from Data Science and Machine Learning area
3. 3 Build Software to Test Software exactpro.com Nostradamus Business Taks: To predict future states of the defects
4. 4 Build Software to Test Software exactpro.com Example of a Bug
5. 5 Build Software to Test Software exactpro.com AI Basics Fetch the Data Data Preprocessing Model Training Model Validation
6. 6 Build Software to Test Software exactpro.com Nostradamus High-Level Details: Business Taks: To predict future states of the defects Fetch the Data Data Preprocessing Model Training From bug-tracking systems Empty values transformation, TTR calculation, text normalization, stemming TF-IDF Vectorizer, SMOTE, Select Percentile, SVC
7. 7 Build Software to Test Software exactpro.com Example of Logs
8. 8 Build Software to Test Software exactpro.com WhaleShark High-Level Details: Business Taks: Simplify log analysis and automate the process of relations recognition Fetch the Data Data Preprocessing Model Training From back-end of the systems under test IDs transformation, text normalization TF-IDF Vectorizer, k-means, SVD, word2vec
9. 9 Build Software to Test Software exactpro.com Web Binary FIX Market Data Database How Sailfish Works
10. 10 Build Software to Test Software exactpro.com Sailfish Report send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (PASSED) [0.001s]: + Input Parameters - Verification: Message (PASSED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW NEW PASSED ... ... ... ... Passed Step of Test Case:
11. 11 Build Software to Test Software exactpro.com send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... Failure Example Failed Step of Test Case:
12. 12 Build Software to Test Software exactpro.com Decision Tree Example Trained Tree:
13. 13 Build Software to Test Software exactpro.com Narrow Range Example: Prediction Example
14. 14 Build Software to Test Software exactpro.com ML for Sailfish Business Taks: To make reports clear via discarding non-relevant messages High-Level Details: Business Taks: Automate the process of source and type recognition of the failures High-Level Details: Fetch the Data Data Preprocessing Model Training Marked up reports from test-library run in Sailfish Dimensionality reduction, Feature Importance, One-Hot Encoding Random Forest, SVN, Calibration Data Preprocessing Model Training Examples transformation, split by message type k-means, DB-scan, Agglomerative
15. 15 Build Software to Test Software exactpro.com Clustering Task
16. 16 Build Software to Test Software exactpro.com Organizational Workflow Computer Science Domain Expertise Math & Statistics Developers Testers DS/ML Engineers Data Analysis Model Training Validation & Verification Implementation
17. 17 Build Software to Test Software exactpro.com Organizational Workflow Computer Science Domain Expertise Math & Statistics Developers Testers DS/ML Engineers Data Analysis Model Training Validation & Verification Implementation
18. 18 Build Software to Test Software exactpro.com Organizational Workflow Computer Science Domain Expertise Math & Statistics Developers Testers DS/ML Engineers Data Analysis Model Training Validation & Verification Implementation
19. 19 Build Software to Test Software exactpro.com Publications
20. 20 Build Software to Test Software exactpro.com TH2 Schema
21. 21 Build Software to Test Software exactpro.com Data Science University Different areas of ML & DS: NLP, Clustering, Regression Analysis, etc Practice Driven Approach: Tasks are connected to Software Testing activities Team work: major part of the task should be done with colleagues Presenting: Each task includes reporting of what was done
22. 22 Build Software to Test Software exactpro.com Thank you! /m.m.ruslanovich /muradmamedov