Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Analysis of Testing Reports using Mac...

SECR 2019
November 14, 2019

Automated Analysis of Testing Reports using Machine Learning Techniques

Мурад Мамедов
Senior QA Analyst, Exactpro
SECR 2019

Automated testing of complex systems is a challenging task, not only from the testing library support perspective, but also from the test report analysis one. Each test run contains thousands of test cases and produces the same number of detailed reports which can provide us with knowledge about the behavior of the environment, the preconditions, the test-data setup, and, of course, about the defects. We propose automating defect report analytics as a way to improve software testing quality.

SECR 2019

November 14, 2019
Tweet

More Decks by SECR 2019

Other Decks in Programming

Transcript

  1. Build Software to Test Software exactpro.com Automated Analysis of Testing

    Reports using Machine Learning Techniques Murad Mamedov, QA Analyst SECR, November 2019
  2. 3 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  3. 4 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  4. 5 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  5. 6 Build Software to Test Software exactpro.com • human factor

    exclusion • knowledge management • standardization • labor costs reduction Usual QA Needs
  6. 7 Build Software to Test Software exactpro.com • Failures Root

    Causes Explanation • Failures prioritization • Recommendations for test’s fix An Ideal Test Report
  7. 11 Build Software to Test Software exactpro.com • Test library:

    +60000 test-cases • 3-7 runs per sprint • 10-20k tests per run • Growing complexity of the software • Relentless integration of checks • Coverage Deepening How We Got the Need
  8. 12 Build Software to Test Software exactpro.com • Detailed Analytics

    & Recommendations • Understanding of each failure • Failures Source Detection • Relevant Message Recognition Task Determination
  9. 13 Build Software to Test Software exactpro.com • Detailed Analytics

    and Recommendations ⬆ Understanding of each failure ⬆ Failures Source Detection ⬆ Relevant Message Recognition Task Determination
  10. 14 Build Software to Test Software exactpro.com Relevant message recognition

    send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (PASSED) [0.001s]: + Input Parameters - Verification: Message (PASSED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW NEW PASSED ... ... ... ... Passed Step of Test Case:
  11. 15 Build Software to Test Software exactpro.com send NewOrderSingle (PASSED)

    [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... Relevant message recognition Failed Step of Test Case:
  12. 20 Build Software to Test Software exactpro.com Calibration Output: before

    after percentage distribution Relevant message recognition
  13. 21 Build Software to Test Software exactpro.com Key activities on

    data: • Data markup • Dimensionality reduction • Dataset Cleanup • Calibration Relevant message recognition
  14. 23 Build Software to Test Software exactpro.com • Detailed Analytics

    & Recommendations • Understanding of each failure • Failures Source Recognition ✓ Relevant message recognition Next Level
  15. 24 Build Software to Test Software exactpro.com Failures Source Detection

    Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
  16. 25 Build Software to Test Software exactpro.com Failures Source Detection

    autotest environment Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
  17. 26 Build Software to Test Software exactpro.com The same dataset

    but: • Only True-class examples • Examples into signatures • Split by message type Failures Source Detection
  18. 31 Build Software to Test Software exactpro.com • Enlarge Dataset

    in semi-automated way • Reach 99% accuracy in failure recognition • Enhance Data Pipeline Further Work
  19. 33 Build Software to Test Software exactpro.com Conclusion Buy order

    100 lots of shares by $2000 TIF Side Price TimeInForce Size Price ... ... till the end of day