Machine Learning powered regression

How can we make a full regression of our platform before each of the releases, to be sure we have not damaged something unexpected when making our changes? Full regression of any web platform is a process which requires tremendous effort and is error-prone. It can be automated in a classical way, covered with test cases and scenarios, but it is extremely hard to cover the application in literal 100%. Doing the full regression manually is not a solution due to the effort needed and spending a few days of work doing so before each release would be an extreme waste of workforce. Our idea? Use a screenshot comparator, with a bit of data science flavour. First of all, we need to store a valid state of an application to assert if there have been any changes. As covering the whole application with test cases is time-consuming, we have thought about a different document covering the whole application - the sitemap. To ensure that we will use the proper version as a model, we should test the application manually, which will take some time, but we plan to do it only once. Afterwards, we can store the state of the application by making screenshots of every page. As we have a sitemap we can go through every subpage and make a screenshot of it displayed on different browsers. When we have a base we can compare to - every time we would like to make a regression, we can make new screenshots of the application, and compare those with the base ones and if we spot a difference, we know that something is happening. This non-standard approach should be the most efficient as the applications tend to change very often and managing and replacing the screenshots is the easiest way to keep the apps up to date.

Transcript

1. April 2019
2. Agenda: Problem Solutions Our suggested solution Some numbers Current work Machine learning The road ahead
3. Regression testing takes a lot of effort Problem
4. Regression testing takes a lot of effort Problem Manual testing Create automated testing suite Solutions
5. Automated snapshot testing with ML support Our solution
6. Some numbers Implemented for 3 live products Estimated 1 month of work to achieve 80% regression coverage even for complex applications. 1 QA supported by this tool is an equivalent of approximately 4 QA engineers.
7. E-commerce example: Language International ~50000 different landing pages 60 languages on site Compatible with multiple viewports and devices Result: 50000 pages * 60 languages * 10 devices = 30 000 000 pages to test before each release Some numbers
8. Some numbers: 30 000 000 Impossible to test manually. Several months to cover with tests by a team of QA engineers with classical approach. Maintenance/update of old tests is easily achieved by replacing screenshots almost immediately. In classical approach QA needs to change the source code of the tests. 10000 tests, 3000 tests failed, 100 files to update - inefficient to update, would take a week for QA team. We will take ½ day to validate and replace screenshots as a screenshot compares to 50 classic tests on average.
9. Current work - Web
10. Current work - Intelligent crawler
11. Current work - test execution Graphic UI API library - Java or PHP
12. USP vs the biggest competition Intelligent crawler to identify the dynamic part of the page and reach the different application states. Generate tests automatically for the most common cases.
13. USP vs the market By QAs and for QAs - optimized to boost the quality. Reach desired coverage in weeks instead of months. Build an efficient QA team - focus on what is important and automate the rest.
14. Machine learning - layout and item type recognition
15. Machine learning - layout and item type recognition
16. Machine learning - layout and item type recognition Google Vision AI
17. Machine learning - layout and item type recognition Tensorflow & Keras
18. Machine learning - object detection
19. Machine learning - mobile status interface ignore
20. Machine learning - mobile status interface ignore
21. The road ahead - automated Selenium scripts generation
22. The road ahead We still don’t have a modern UI for test cases and test suites management. Generate more sophisticated test data sets. Add user management in a more sophisticated way.
23. The road ahead Going open source
24. The road ahead Code preparation
25. The road ahead Future goals definition
26. Thank you!