An Empirical Investigation on the Readability of Manual and Generated Test Cases

Software testing is one of the most crucial tasks in the typical development process. Developers are usually required to write unit test cases for the code they implement. Since this is a time-consuming task, in last years many approaches and tools for automatic test case generation — such as EvoSuite — have been introduced. Nevertheless, developers have to maintain and evolve tests to sustain the changes in the source code; therefore, having readable test cases is important to ease such a process. However, it is still not clear whether developers make an effort in writing readable unit tests. Therefore, in this paper, we conduct an explorative study comparing the readability of manually written test cases with the classes they test. Moreover, we deepen such analysis looking at the readability of automatically generated test cases. Our results suggest that developers tend to neglect the readability of test cases and that automatically generated test cases are generally even less readable than manually written ones.

Transcript

1. An Empirical Investigation on the Readability of Manual and Generated Test Cases G. Grano, S. Scalabrino, H. C. Gall, R. Oliveto ICPC ERA 2018, 27-28 May, Gothenburg (Sweden) grano@ifi.uzh.ch giograno90
2. Unit Test First defensive line against bug introduction 2 — Giovanni Grano @ s.e.a.l.
3. Tests evolve with code 3 — Giovanni Grano @ s.e.a.l.
4. Time spent with tests 1 > 416 software engineers for over 13 years of recorded working time in the IDE > Developers spend 25% of their time engineering tests in their IDE > They look at tests when errors occur 1 Beller et.al - When, How, and Why Developers (Do Not) Test in Their IDEs 4 — Giovanni Grano @ s.e.a.l.
5. Same level of quality Same effort in writing good quality tests and production code > Changes are easier > Error comprehension 5 — Giovanni Grano @ s.e.a.l.
6. Are tests first-class citizens? 6 — Giovanni Grano @ s.e.a.l.
7. Quality ⊇ readability 7 — Giovanni Grano @ s.e.a.l.
8. Study Design > 3 Apache open-source projects > state-of-the-art readability model 2 > pairs of <test-production code> > Wilcoxon Test 2 Scalabrino et.al. - Improving Code Readability Models with Textual Features - ICPC 2016 8 — Giovanni Grano @ s.e.a.l.
9. Features > Structural Features (e.g. line length) > Visual Features (e.g. indentation) > Text-Based Code Readability Features (e.g. Comments and Identifiers Consistency) 9 — Giovanni Grano @ s.e.a.l.
10. RQ1 Are test cases and classes equally readable? 10 — Giovanni Grano @ s.e.a.l.
11. Test cases are less readable than the code they exercise
12. Well... we have tools! 12 — Giovanni Grano @ s.e.a.l.
13. RQ2 Are generated and manual tests equally readable? 13 — Giovanni Grano @ s.e.a.l.
14. Manually written tests are more readable!
15. Future work Why readability of test is lower? > good proxy for quality? > increase awareness > support test quality Can we improve the tools? 15 — Giovanni Grano @ s.e.a.l.