Reduce flaky test cases with Gradle retry plugin and Allure report

Reduce flaky test cases with Gradle retry plugin and Allure report

Elena Alejo (LINE Corporation, DSE Center SET)

LINE Developer Meetup #64 "Development and Test"
https://line.connpass.com/event/179721/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers

July 08, 2020
Tweet

Transcript

  1. Reduce flaky test cases with Gradle retry plugin & Allure

    report
  2. The Team 2

  3. SET team (Software Engineer in Test) 3 • We consider

    ourselves DEVELOPERS • Create testing tools for Developers based on their necessities • Experiment • Coach into Agile Testing I am a SET in LINE * Created by Avatar created by Avatars in pixels
  4. Collaboration with Development Team LINE Shop 4

  5. Bi-weekly meeting 5 • We do bi-weekly meeting with LINE

    Shop, is our time for improve testing at LINE Shop Goal Look for opportunities to improve testing Do experiments and demo them Create a backlog for the next two weeks, everybody is welcome to do any tasks Facilitate the creation of the test cases Testing good practices Out of scope SET team do all the testing At LINE, SET/Dev ratio is quite low, so we focus on provide Testing infrastructure to Developers, so they can make their own tests
  6. One day in a bi-weekly meeting 6

  7. 7 Sometimes Jenkins Job fails randomly It could be great

    if we can retry the test cases Next execution works without changing anything!
  8. Issue: Flaky test cases makes build job execution unstable 8

    By geek & poke
  9. What are flaky test cases? 9

  10. Flaky test cases • Give us different results under same

    conditions • Give us false negative, test fails consistently but code is ok Due • Test code quality is bad • Test dependencies with external system 10
  11. Solution 1: Gradle Retry plugin 11

  12. Gradle Retry plugin • Configure Retries adding Gradle retry plugin

    • You can configure the plugin to stop, if test is failing for more than x times 12 Retrying allow us to reduce the Noise of false negative test cases
  13. Gradle Retry plugin • If a test fails, it will

    be retried up to 3 times Output: Builds percentage of failure drop We are building trust! 13
  14. Next bi-weekly meeting 14

  15. 15 Job Execution is green again! Yeap, but test Report

    is messy, repetitions makes harder to understand the report Also we did not Fix any issue, just put a band aid to the job execution! 4 retries Flaky test cases are still there!!
  16. Solution 2: Allure Report 16

  17. Allure is a test reporting tool “Allure Framework is a

    flexible lightweight multi-language test report tool that not only shows a very concise representation of what have been tested in a neat web report form, but allows everyone participating in the development process to extract maximum of useful information from everyday execution of tests” 17
  18. Next bi-weekly meeting WFH! 18 Zoom photos: “Every Type of

    Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/eve ry-type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
  19. 19 Let me demo the Allure Report! Photos: Pixabay; Pexels

  20. 20 Photos by: Christopher Pluta via Pixabay; Pexels

  21. 21 Cool I like the colors! If a test case

    is retried it is not adding noise to the report I like it too! How I know if a test is Retried?, can I filter so I investigate if the test is flaky You cannot filter By retried, let’s customize our report!
  22. Solution 2: Customize Allure Report In order to detect flaky

    test cases 22
  23. Strategy Flaky test cases give us different results under same

    conditions Detect by Retrying 23
  24. Next bi-weekly meeting WFH! 24 Inspired by “Every Type of

    Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/every- type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
  25. 25 Let me demo the new Plugins Allure Report!

  26. 26 Allure Retry Plugin Demo

  27. 27 Allure Historic Retry Plugin Demo

  28. 28 cool, so coverImageUrlOrBlank() is a flaky test case, because

    it failed and passed under same conditions Let’s mark it as flaky! But what happens if We access to an old test report data? As Jenkins deletes the build info every X Executions we will lose report data. Let’s fix it!
  29. Solution 3: Store metrics In order to detect flaky test

    cases 29
  30. Strategy Find trends in Job build metrics Detect by Creating

    reports in Kibana 30
  31. Next bi-weekly meeting WFH! 31

  32. 32 Let me demo the new ES Database

  33. 33 ES Demo

  34. 34 cool, so we are not going to lost data

    Anymore!! We can create a lot of metrics! I cannot wait to create more metrics! Finally!
  35. 35 Conclusions

  36. Conclusions (Bi-weekly meeting) Pros • It’s nice to have a

    space for chat about testing • Getting in touch with Developers allow us to know their necessities • Developers point of view is great for grow as SET • Collaboration with Dev helps us to understand its infrastructure 36
  37. Conclusions (Bi-weekly meeting) Cons • Limitations, as we are not

    part of the development cycle we miss opportunities to improve testing • Difficult to get more developers time for testing 37
  38. Conclusions (Allure) Pros • Open source project • Report’s UI

    is nice • Integration with several test engines (JUnit, Pytest…) • Able to integrate with Jenkins and other CI tools • Historic and metrics are a good starting point 38
  39. Conclusions (Allure) Cons • A bit difficult to set up

    in a large project (once you understand how it works is easier) • Once you have more then 10.000 test cases, loading is slow • Historic works if you run test cases few times a day • Lack of documentations (up to date, examples...) • I have to read source code to be able to create customized plugins • Architecture is not best one (not easy to host it) • Community is big but not 100% supportive 39
  40. It’s allure worth it? • It’s a report that can

    be used for multiple users • It’s a gateway to ES metrics • I can customize it easily 40 Allure can be used with most of LINE test projects! YES!
  41. Questions! 41

  42. Thanks! 42