Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automated Analysis of Testing Reports Using Machine Learning Techniques

Automated Analysis of Testing Reports Using Machine Learning Techniques

Murad Mamedov
Senior QA Analyst, Exactpro

SECR 2019
14-15 November 2019, Saint Petersburg

To learn more about Exactpro, visit our website https://exactpro.com/
EXTENT Conference website: https://extent.exactpro.com/
Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro Vimeo channel https://vimeo.com/exactproSubscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

5206c19df417b8876825b5561344c1a0?s=128

Exactpro
PRO

November 14, 2019
Tweet

Transcript

  1. Build Software to Test Software exactpro.com Automated Analysis of Testing

    Reports using Machine Learning Techniques Murad Mamedov, QA Analyst SECR, November 2019
  2. 2 Build Software to Test Software exactpro.com automated tests continuous

    integration Hindsight on Automation
  3. 3 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  4. 4 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  5. 5 Build Software to Test Software exactpro.com Hindsight on Automation

    CI tasks: • get source code from repository • build project • run test library • deploy project • send reports
  6. 6 Build Software to Test Software exactpro.com • human factor

    exclusion • knowledge management • standardization • labor costs reduction Usual QA Needs
  7. 7 Build Software to Test Software exactpro.com • Failures Root

    Causes Explanation • Failures prioritization • Recommendations for test’s fix An Ideal Test Report
  8. 8 Build Software to Test Software exactpro.com vs. Let’s Automate

    Decision Making
  9. 9 Build Software to Test Software exactpro.com Who is Responsible?

    dev qa
  10. 10 Build Software to Test Software exactpro.com How We Got

    the Need
  11. 11 Build Software to Test Software exactpro.com • Test library:

    +60000 test-cases • 3-7 runs per sprint • 10-20k tests per run • Growing complexity of the software • Relentless integration of checks • Coverage Deepening How We Got the Need
  12. 12 Build Software to Test Software exactpro.com • Detailed Analytics

    & Recommendations • Understanding of each failure • Failures Source Detection • Relevant Message Recognition Task Determination
  13. 13 Build Software to Test Software exactpro.com • Detailed Analytics

    and Recommendations ⬆ Understanding of each failure ⬆ Failures Source Detection ⬆ Relevant Message Recognition Task Determination
  14. 14 Build Software to Test Software exactpro.com Relevant message recognition

    send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (PASSED) [0.001s]: + Input Parameters - Verification: Message (PASSED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW NEW PASSED ... ... ... ... Passed Step of Test Case:
  15. 15 Build Software to Test Software exactpro.com send NewOrderSingle (PASSED)

    [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... Relevant message recognition Failed Step of Test Case:
  16. 16 Build Software to Test Software exactpro.com Relevant message recognition

    Differences in Fields: message a message b
  17. 17 Build Software to Test Software exactpro.com Relevant message recognition

    Trained Tree:
  18. 18 Build Software to Test Software exactpro.com Relevant message recognition

    Feature Importance:
  19. 19 Build Software to Test Software exactpro.com Narrow Range Example:

    Relevant message recognition
  20. 20 Build Software to Test Software exactpro.com Calibration Output: before

    after percentage distribution Relevant message recognition
  21. 21 Build Software to Test Software exactpro.com Key activities on

    data: • Data markup • Dimensionality reduction • Dataset Cleanup • Calibration Relevant message recognition
  22. 22 Build Software to Test Software exactpro.com Metrics: Relevant message

    recognition
  23. 23 Build Software to Test Software exactpro.com • Detailed Analytics

    & Recommendations • Understanding of each failure • Failures Source Recognition ✓ Relevant message recognition Next Level
  24. 24 Build Software to Test Software exactpro.com Failures Source Detection

    Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
  25. 25 Build Software to Test Software exactpro.com Failures Source Detection

    autotest environment Failures sources: • test’s code • test data • static reference data • dynamic reference data • SUT behavior
  26. 26 Build Software to Test Software exactpro.com The same dataset

    but: • Only True-class examples • Examples into signatures • Split by message type Failures Source Detection
  27. 27 Build Software to Test Software exactpro.com Failures Source Detection

  28. 28 Build Software to Test Software exactpro.com Failures Source Detection

  29. 29 Build Software to Test Software exactpro.com Failures Source Detection

  30. 30 Build Software to Test Software exactpro.com Failures Source Detection

  31. 31 Build Software to Test Software exactpro.com • Enlarge Dataset

    in semi-automated way • Reach 99% accuracy in failure recognition • Enhance Data Pipeline Further Work
  32. 32 Build Software to Test Software exactpro.com Data Pipeline DS

  33. 33 Build Software to Test Software exactpro.com Conclusion Buy order

    100 lots of shares by $2000 TIF Side Price TimeInForce Size Price ... ... till the end of day
  34. 34 Build Software to Test Software exactpro.com Thank you!