Automated Analysis of Testing Reports using Machine Learning Techniques

Мурад Мамедов
Senior QA Analyst, Exactpro
SECR 2019

Automated testing of complex systems is a challenging task, not only from the testing library support perspective, but also from the test report analysis one. Each test run contains thousands of test cases and produces the same number of detailed reports which can provide us with knowledge about the behavior of the environment, the preconditions, the test-data setup, and, of course, about the defects. We propose automating defect report analytics as a way to improve software testing quality.

Transcript

1. Build Software to Test Software exactpro.com Automated Analysis of Testing Reports using Machine Learning Techniques Murad Mamedov, QA Analyst SECR, November 2019
2. 2 Build Software to Test Software exactpro.com automated tests continuous integration Hindsight on Automation
3. 3 Build Software to Test Software exactpro.com Hindsight on Automation CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
4. 4 Build Software to Test Software exactpro.com Hindsight on Automation CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
5. 5 Build Software to Test Software exactpro.com Hindsight on Automation CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
6. 6 Build Software to Test Software exactpro.com • human factor exclusion • knowledge management • standardization • labor costs reduction Usual QA Needs
7. 7 Build Software to Test Software exactpro.com • Failures Root Causes Explanation • Failures prioritization • Recommendations for test’s fix An Ideal Test Report
8. 8 Build Software to Test Software exactpro.com vs. Let’s Automate Decision Making
9. 9 Build Software to Test Software exactpro.com Who is Responsible? dev qa
10. 10 Build Software to Test Software exactpro.com How We Got the Need
11. 11 Build Software to Test Software exactpro.com • Test library: +60000 test-cases • 3-7 runs per sprint • 10-20k tests per run • Growing complexity of the software • Relentless integration of checks • Coverage Deepening How We Got the Need
12. 12 Build Software to Test Software exactpro.com • Detailed Analytics & Recommendations • Understanding of each failure • Failures Source Detection • Relevant Message Recognition Task Determination
13. 13 Build Software to Test Software exactpro.com • Detailed Analytics and Recommendations ⬆ Understanding of each failure ⬆ Failures Source Detection ⬆ Relevant Message Recognition Task Determination
14. 14 Build Software to Test Software exactpro.com Relevant message recognition send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (PASSED) [0.001s]: + Input Parameters - Verification: Message (PASSED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW NEW PASSED ... ... ... ... Passed Step of Test Case:
15. 15 Build Software to Test Software exactpro.com send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... Relevant message recognition Failed Step of Test Case:
16. 16 Build Software to Test Software exactpro.com Relevant message recognition Differences in Fields: message a message b
17. 17 Build Software to Test Software exactpro.com Relevant message recognition Trained Tree:
18. 18 Build Software to Test Software exactpro.com Relevant message recognition Feature Importance:
19. 19 Build Software to Test Software exactpro.com Narrow Range Example: Relevant message recognition
20. 20 Build Software to Test Software exactpro.com Calibration Output: before after percentage distribution Relevant message recognition
21. 21 Build Software to Test Software exactpro.com Key activities on data: • Data markup • Dimensionality reduction • Dataset Cleanup • Calibration Relevant message recognition
22. 22 Build Software to Test Software exactpro.com Metrics: Relevant message recognition
23. 23 Build Software to Test Software exactpro.com • Detailed Analytics & Recommendations • Understanding of each failure • Failures Source Recognition ✓ Relevant message recognition Next Level
24. 24 Build Software to Test Software exactpro.com Failures Source Detection Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
25. 25 Build Software to Test Software exactpro.com Failures Source Detection autotest environment Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
26. 26 Build Software to Test Software exactpro.com The same dataset but: • Only True-class examples • Examples into signatures • Split by message type Failures Source Detection
27. 27 Build Software to Test Software exactpro.com Failures Source Detection
28. 28 Build Software to Test Software exactpro.com Failures Source Detection
29. 29 Build Software to Test Software exactpro.com Failures Source Detection
30. 30 Build Software to Test Software exactpro.com Failures Source Detection
31. 31 Build Software to Test Software exactpro.com • Enlarge Dataset in semi-automated way • Reach 99% accuracy in failure recognition • Enhance Data Pipeline Further Work
32. 32 Build Software to Test Software exactpro.com Data Pipeline DS
33. 33 Build Software to Test Software exactpro.com Conclusion Buy order 100 lots of shares by $2000 TIF Side Price TimeInForce Size Price ... ... till the end of day
34. 34 Build Software to Test Software exactpro.com Thank you!