Artificial Intelligence in Test Automation

https://www.meetup.com/KraQA-pl/events/248202478/?eventId=248202478

Transcript

1. Karol Przystalski, Jakub Blusiewicz Artificial Intelligence in Test Automation
2. Genesis We have experienced many times an issue where we need to jump in and develop many test cases in a short period of time. Using commonly known solutions was not enough fast and precise. The main goals that we had in mind before we started the development: → short development time, → precise and be able to handle even UX bugs/changes, → develop a solution that works well for new product and legacy ones.
3. Demo - before we start
4. Snapshot testing and A.I. Snapshot testing or UI snapshot testing is a type of automated tests where we check the difference between two images: expected and current. If there is a difference the test should fail.
5. Snapshot is an assertion itself Result of the comparison. Test failed.
6. To fail or not to fail? - pixels talking Power standing behind snapshot testing can be easily described on the image below. After comparing base screenshot and new one, result image shows us a difference (in our example test failed because “7” turned into “8”
7. Test failed - Let me fix myself ! Our solution allows user to fix tests without looking at the code. If we ensure that current version is correct, test will manage to fix itself by replacing base snapshot with the new one → Fixing is possible without having technical knowledge → Doesn’t require code changes → Tests running for example on jenkins can be fixed from any place, we don’t need local environment
8. A.I. - buzzword ?
9. A.I. Terms that are related to artificial intelligence and are/can/should be part of intelligent snapshot testing are the following: → pattern recognition, → computer vision, → data science, → reinforcement learning.
10. Machine learning Machine learning methods are/can be used in test automation, especially snapshot testing: → recurrent neural networks for data generation - avoid pesticide effect, → markov decision process/q-learning - generate new test cases, → convolution neural network - discover elements on a page/app and build test or prepare data based on that.
11. Advantages There are many advantages, but here are a few most important: → is able to test UI, → give the possibility to use machine learning methods to do test that were not possible or hard to develop before like image/video recognition, automated UX tests and many more, → easy maintenance.
12. Disadvantages Proposed solution has also some drawbacks like: → storage, → not everything can be test with proposed solution, → API tests? Not really, as many other type of tests.
13. Open source We will release the project as open source in the next weeks. Let us know if you are interested in contribution and leave your email to be informed when the project will be released as open source: https://goo.gl/forms/i4c98jwScj1ov0GG3 or send us an email to karol@codete.com or find us after the presentation.
14. Competitors In the meantime a similar project was developed - applitools. It doesn’t cover all features that we have on our list, but is the most similar to our solution. There are other solutions that are called snapshot testing tools like JEST, but most are related to a specific technology like React.
15. Further work → Rewrite it to Python → Cover not only web apps, but also native and mobile apps → Generate better reports → Introduce/enable more machine learning methods → Management tool
16. Any questions?