AI for Testing

Rostislav Yavorski
Head of Research, Exactpro

“In this lecture, we will provide an overview of using machine learning and AI tools in software testing. Particular applications include event clustering for system log analysis, recommendation systems for bug reporting, anomaly detection in non-functional testing, etc.”

QA Meetup. 25 April 2022

https://exactpro.com/events/external/qa-meetup-25-april?utm_source=speakerdeck&utm_medium=Refferer&utm_campaign=ai-for-testing

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup BUILD SOFTWARE TO TEST SOFTWARE exactpro.com AI for Testing Head of Research, Exactpro Rostislav Yavorski 25 APRIL 2022 ONLINE | HATCH WORKS, COLOMBO 1
2. 2 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Machine learning areas 2
3. 3 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Machine learning areas 3
4. 4 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Unsupervised Learning A machine learning technique in which the users do not need to supervise the model. It is a type of algorithm that learns patterns from untagged data: https://towardsdatascience.com/supervised-vs-unsupervised-learning-in-2-minutes-72dad148f242 4
5. 5 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Unsupervised Learning applications • Audience segmentation • Customer service personalization • Recommender Engines • Anomaly detection • Pattern recognition • Inventory management https://towardsdatascience.com/supervised-vs-unsupervised-learning-in-2-minutes-72dad148f242 5
6. 6 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Supervised Learning The algorithm uses labeled datasets to train how to classify data or predict outcomes accurately. It is a function that maps an input to an output based on example input-output pairs. https://www.guru99.com/supervised-vs-unsupervised-learning.html 6
7. 7 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup https://towardsdatascience.com/supervised-vs-unsupervised-learning-in-2-minutes-72dad148f242 7
8. 8 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Supervised Learning applications Healthcare and medical diagnosis Predicting stock price Weather forecasting Text categorization Spam detection Face recognition https://www.guru99.com/supervised-vs-unsupervised-learning.html 8
9. 9 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Reinforcement Learning Method based on rewarding desired behaviors and punishing undesired ones: https://vitalflux.com/reinforcement-learning-real-world-examples/ 9
10. 10 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Reinforcement Learning applications Self-driving cars (autonomous vehicles) Industry automation and learning-based robots Natural language processing for chatbot dialogue Dynamic treatment regimes in chronic disease https://vitalflux.com/reinforcement-learning-real-world-examples/ 10
11. 11 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 11
12. 12 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Simplest to apply 12
13. 13 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Requires data labeling 13
14. 14 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Most advanced 14
15. 15 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Applications to Software Testing 15
16. 16 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup #1. System anomaly detection An anomaly is something that differs from a norm: • a deviation • an exception • a rare occurrence or event that doesn’t fit into the pattern • anything that seems suspicious 16 https://jhui.github.io/2017/01/15/Machine-learning-anomaly/
17. 17 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 17
18. 18 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 18
19. 19 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Unsupervised learning for anomaly detection • A detection model is constructed using historical logs, which describe a variety of events of software systems. The model is used for: ◦ detecting various types of system behavior anomalies ◦ determining statistical load parameters • Another way is to extract semantic information of log events. The anomalies are detected from the contextual information in the log sequences based on the importance of different log events. 19
20. 20 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 20 System performance testing
21. 21 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup #2. AI for bug reporting Report readability assessment Duplication detection Automatic fault localization Automated bug assignment Determining bug severity Bug fixing time prediction 21 https://www.softwaretestinghelp.com/
22. 22 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup #3. AI for requirements text analysis Detect linguistic issues in requirements documents Evaluating the sufficiency of information and spec completeness Automated test cases generation from requirements Extracting modeling concepts and constructing UML models Transforming natural language format into formal specification 22 https://clockwise.software/blog/software-requirements-specification-document/
23. 23 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup #4. Tester’s virtual assistant What new errors appeared today? Are there any traces of recent bugs we fixed? Especially, related to the issue we saw yesterday? When was the last major change in logs structure and characteristics? What applications logged an abnormal number of errors today? What kind of anomaly was detected in recent logs? 23 https://www.freepik.com/
24. 24 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Thank You! 24