Reduce flaky test cases with Gradle retry plugin and Allure report

Elena Alejo (LINE Corporation, DSE Center SET)

LINE Developer Meetup #64 "Development and Test"
https://line.connpass.com/event/179721/

Transcript

1. Reduce flaky test cases with Gradle retry plugin & Allure report
2. The Team 2
3. SET team (Software Engineer in Test) 3 • We consider ourselves DEVELOPERS • Create testing tools for Developers based on their necessities • Experiment • Coach into Agile Testing I am a SET in LINE * Created by Avatar created by Avatars in pixels
4. Collaboration with Development Team LINE Shop 4
5. Bi-weekly meeting 5 • We do bi-weekly meeting with LINE Shop, is our time for improve testing at LINE Shop Goal Look for opportunities to improve testing Do experiments and demo them Create a backlog for the next two weeks, everybody is welcome to do any tasks Facilitate the creation of the test cases Testing good practices Out of scope SET team do all the testing At LINE, SET/Dev ratio is quite low, so we focus on provide Testing infrastructure to Developers, so they can make their own tests
6. One day in a bi-weekly meeting 6
7. 7 Sometimes Jenkins Job fails randomly It could be great if we can retry the test cases Next execution works without changing anything!
8. Issue: Flaky test cases makes build job execution unstable 8 By geek & poke
9. What are flaky test cases? 9
10. Flaky test cases • Give us different results under same conditions • Give us false negative, test fails consistently but code is ok Due • Test code quality is bad • Test dependencies with external system 10
11. Solution 1: Gradle Retry plugin 11
12. Gradle Retry plugin • Configure Retries adding Gradle retry plugin • You can configure the plugin to stop, if test is failing for more than x times 12 Retrying allow us to reduce the Noise of false negative test cases
13. Gradle Retry plugin • If a test fails, it will be retried up to 3 times Output: Builds percentage of failure drop We are building trust! 13
14. Next bi-weekly meeting 14
15. 15 Job Execution is green again! Yeap, but test Report is messy, repetitions makes harder to understand the report Also we did not Fix any issue, just put a band aid to the job execution! 4 retries Flaky test cases are still there!!
16. Solution 2: Allure Report 16
17. Allure is a test reporting tool “Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what have been tested in a neat web report form, but allows everyone participating in the development process to extract maximum of useful information from everyday execution of tests” 17
18. Next bi-weekly meeting WFH! 18 Zoom photos: “Every Type of Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/eve ry-type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
19. 19 Let me demo the Allure Report! Photos: Pixabay; Pexels
20. 20 Photos by: Christopher Pluta via Pixabay; Pexels
21. 21 Cool I like the colors! If a test case is retried it is not adding noise to the report I like it too! How I know if a test is Retried?, can I filter so I investigate if the test is flaky You cannot filter By retried, let’s customize our report!
22. Solution 2: Customize Allure Report In order to detect flaky test cases 22
23. Strategy Flaky test cases give us different results under same conditions Detect by Retrying 23
24. Next bi-weekly meeting WFH! 24 Inspired by “Every Type of Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/every- type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
25. 25 Let me demo the new Plugins Allure Report!
26. 26 Allure Retry Plugin Demo
27. 27 Allure Historic Retry Plugin Demo
28. 28 cool, so coverImageUrlOrBlank() is a flaky test case, because it failed and passed under same conditions Let’s mark it as flaky! But what happens if We access to an old test report data? As Jenkins deletes the build info every X Executions we will lose report data. Let’s fix it!
29. Solution 3: Store metrics In order to detect flaky test cases 29
30. Strategy Find trends in Job build metrics Detect by Creating reports in Kibana 30
31. Next bi-weekly meeting WFH! 31
32. 32 Let me demo the new ES Database
33. 33 ES Demo
34. 34 cool, so we are not going to lost data Anymore!! We can create a lot of metrics! I cannot wait to create more metrics! Finally!
35. 35 Conclusions
36. Conclusions (Bi-weekly meeting) Pros • It’s nice to have a space for chat about testing • Getting in touch with Developers allow us to know their necessities • Developers point of view is great for grow as SET • Collaboration with Dev helps us to understand its infrastructure 36
37. Conclusions (Bi-weekly meeting) Cons • Limitations, as we are not part of the development cycle we miss opportunities to improve testing • Difficult to get more developers time for testing 37
38. Conclusions (Allure) Pros • Open source project • Report’s UI is nice • Integration with several test engines (JUnit, Pytest…) • Able to integrate with Jenkins and other CI tools • Historic and metrics are a good starting point 38
39. Conclusions (Allure) Cons • A bit difficult to set up in a large project (once you understand how it works is easier) • Once you have more then 10.000 test cases, loading is slow • Historic works if you run test cases few times a day • Lack of documentations (up to date, examples...) • I have to read source code to be able to create customized plugins • Architecture is not best one (not easy to host it) • Community is big but not 100% supportive 39
40. It’s allure worth it? • It’s a report that can be used for multiple users • It’s a gateway to ES metrics • I can customize it easily 40 Allure can be used with most of LINE test projects! YES!
41. Questions! 41
42. Thanks! 42