AI Development at Exactpro

Murad Mamedov, AI Researcher, Exactpro

Exactpro Webinar for Tbilisi State University | A Webinar on Software Testing and AI
16.06.2020

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. Build Software to Test Software
   exactpro.com
   AI Development at Exactpro
   Murad Mamedov, AI Researcher
   TSU Webinar, June 2020
   

   View Slide

2. 2 Build Software to Test Software exactpro.com
   Plan
   - Machine
   How we research AI applications in Software Testing and how we build AI Tool on top of this
   - Platform
   Test Harness platform of new generation. Its overview from AI perspective
   - People
   How we prepare Software Testers for new set of tasks from Data Science and Machine Learning area
   

   View Slide

3. 3 Build Software to Test Software exactpro.com
   Nostradamus
   Business Taks: To predict future states of the defects
   

   View Slide

4. 4 Build Software to Test Software exactpro.com
   Example of a Bug
   

   View Slide

5. 5 Build Software to Test Software exactpro.com
   AI Basics
   Fetch the Data Data Preprocessing Model Training Model Validation
   

   View Slide

6. 6 Build Software to Test Software exactpro.com
   Nostradamus
   High-Level Details:
   Business Taks: To predict future states of the defects
   Fetch the Data Data Preprocessing Model Training
   From bug-tracking
   systems
   Empty values
   transformation, TTR
   calculation, text
   normalization, stemming
   TF-IDF Vectorizer,
   SMOTE, Select
   Percentile, SVC
   

   View Slide

7. 7 Build Software to Test Software exactpro.com
   Example of Logs
   

   View Slide

8. 8 Build Software to Test Software exactpro.com
   WhaleShark
   High-Level Details:
   Business Taks: Simplify log analysis and automate the process of relations recognition
   Fetch the Data Data Preprocessing Model Training
   From back-end of the
   systems under test
   IDs transformation, text
   normalization
   TF-IDF Vectorizer,
   k-means, SVD, word2vec
   

   View Slide

9. 9 Build Software to Test Software exactpro.com
   Web
   Binary
   FIX
   Market Data
   Database
   How Sailfish Works
   

   View Slide

10. 10 Build Software to Test Software exactpro.com
    Sailfish Report
    send NewOrderSingle (PASSED) [0.001s]
    - receive ExecutionReport (PASSED) [0.001s]:
    + Input Parameters
    - Verification: Message (PASSED):
    Field Expected Result Actual Result Status
    ... ... ... ...
    OrdType LIMIT LIMIT PASSED
    Status NEW NEW PASSED
    ... ... ... ...
    Passed Step of Test Case:
    

    View Slide

11. 11 Build Software to Test Software exactpro.com
    send NewOrderSingle (PASSED) [0.001s]
    - receive ExecutionReport (FAILED) [0.421s]:
    + Input Parameters
    + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED)
    + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED)
    + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED)
    - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED):
    Field Expected Result Actual Result Status
    ... ... ... ...
    OrdType LIMIT LIMIT PASSED
    Status NEW REJECTED FAILED
    ... ... ... ...
    Failure Example
    Failed Step of Test Case:
    

    View Slide

12. 12 Build Software to Test Software exactpro.com
    Decision Tree Example
    Trained Tree:
    

    View Slide

13. 13 Build Software to Test Software exactpro.com
    Narrow Range Example:
    Prediction Example
    

    View Slide

14. 14 Build Software to Test Software exactpro.com
    ML for Sailfish
    Business Taks: To make reports clear via
    discarding non-relevant messages
    High-Level Details:
    Business Taks: Automate the process of source
    and type recognition of the failures
    High-Level Details:
    Fetch the
    Data
    Data
    Preprocessing
    Model
    Training
    Marked up
    reports from
    test-library run
    in Sailfish
    Dimensionality
    reduction,
    Feature
    Importance,
    One-Hot
    Encoding
    Random
    Forest, SVN,
    Calibration
    Data
    Preprocessing
    Model
    Training
    Examples
    transformation,
    split by
    message type
    k-means,
    DB-scan,
    Agglomerative
    

    View Slide

15. 15 Build Software to Test Software exactpro.com
    Clustering Task
    

    View Slide

16. 16 Build Software to Test Software exactpro.com
    Organizational Workflow
    Computer
    Science
    Domain
    Expertise
    Math &
    Statistics
    Developers
    Testers
    DS/ML
    Engineers
    Data Analysis
    Model Training
    Validation &
    Verification
    Implementation
    

    View Slide

17. 17 Build Software to Test Software exactpro.com
    Organizational Workflow
    Computer
    Science
    Domain
    Expertise
    Math &
    Statistics
    Developers
    Testers
    DS/ML
    Engineers
    Data Analysis
    Model Training
    Validation &
    Verification
    Implementation
    

    View Slide

18. 18 Build Software to Test Software exactpro.com
    Organizational Workflow
    Computer
    Science
    Domain
    Expertise
    Math &
    Statistics
    Developers
    Testers
    DS/ML
    Engineers
    Data Analysis
    Model Training
    Validation &
    Verification
    Implementation
    

    View Slide

19. 19 Build Software to Test Software exactpro.com
    Publications
    

    View Slide

20. 20 Build Software to Test Software exactpro.com
    TH2 Schema
    

    View Slide

21. 21 Build Software to Test Software exactpro.com
    Data Science University
    Different areas of ML & DS: NLP, Clustering, Regression Analysis, etc
    Practice Driven Approach: Tasks are connected to Software Testing activities
    Team work: major part of the task should be done with colleagues
    Presenting: Each task includes reporting of what was done
    

    View Slide

22. 22 Build Software to Test Software exactpro.com
    Thank you!
    /m.m.ruslanovich
    /muradmamedov
    

    View Slide