Test maintainability

Automatic tests are still considered as a necessary evil. Some developers still claim that clients don't pay for tests so we shouldn't write them. Others say that we shouldn't put much effort into tests, implementation is more important, right? Those developers tend to forget the overall cost of maintenance of their codebase. How many bugs you can find on an early stage of implementation just thanks to automatic tests? What's the cost of maintenance of tests written in a not clean way?

During this talk, I will go through some properties of well maintainable tests, present examples of tests hard to maintain and suggest a few ways to improve them. Presented properties and techniques might be valuable for engineers writing and refactoring tests. Finally, I will dive into the term testable design and discuss how to achieve it.

Transcript

1. TEST MAINTAINABILITY MICHAŁ KOWALCZYK
2. Why? What development techniques would you apply when developing… low latency embedded mobile 2
3. Why? In my project over 60% C++ code is in tests Testing is a second system Knowing that should I harness some special techniques of coding? 3
4. Me 6 years as C++ developer 5 years experience with Google Test Senior Software Engineer @ TomTom Michi - map visualisation engine Interested in cognitive sciences 4
5. Agenda Notions Properties of maintainable tests Complete Comprehensible Verifiable by inspection Easy to modify and extend Testable design 5
6. NOTIONS
7. Notions System under test Assertion, expectation Test case Test suite, fixture Loose coupling, dependency injection, mock 7
8. Notions Testing piramid End to end (few) Integration (some) Unit (tons) 8 e2e integration unit
9. Notions Testing piramid End to end (few) Integration (some) Unit (tons) 9 e2e integration unit technology.cap-hpi.com/blog/unit-testing-what-benefits-you-can-reap
10. Notions Testing piramid End to end (few) Integration (some) Unit (tons) 10 Unit (tons) e2e integration unit technology.cap-hpi.com/blog/unit-testing-what-benefits-you-can-reap
11. PROPERTIES OF MAINTAINABLE TESTS
12. Complete 12
13. Complete - edge cases EXPECT_EQ(1, sum(0, 1)); EXPECT_EQ(2, sum(1, 1)); EXPECT_EQ(3, sum(1, 2)); 13
14. Complete - edge cases EXPECT_EQ(1, sum(0, 1)); EXPECT_EQ(2, sum(1, 1)); EXPECT_EQ(3, sum(1, 2)); // sum(std::numeric_limits<int>::max(), 1)? 14
15. Complete - edge cases invalid inputs numeric type extremes empty sets 15
16. Complete - covered 16
17. Complete - covered /** * Returns a file stream to the given file. * * @throws std::invalid_argument * if filePath is incorrect. * @throws std::invalid_argument * if mode is not kRead or kWrite. */ std::fstream openFile(const char* filePath, int mode); 17
18. Complete - covered ASSERT_THROW(openFile(kInvalidPath, kValidMode), std::invalid_argument); ASSERT_THROW(openFile(kValidPath, kInvalidMode), std::invalid_argument); 18
19. Complete - covered const char* const kValidPath = nullptr; const char* const kInvalidPath = nullptr; const int kValidMode = kRead; const int kInvalidMode = kInvalid; ASSERT_THROW(openFile(kInvalidPath, kValidMode), std::invalid_argument); ASSERT_THROW(openFile(kValidPath, kInvalidMode), std::invalid_argument); 19
20. Complete - covered const char* const kValidPath = nullptr; const char* const kInvalidPath = nullptr; const int kValidMode = kRead; const int kInvalidMode = kInvalid; ASSERT_THROW(openFile(kInvalidPath, kValidMode), std::invalid_argument); ASSERT_THROW(openFile(kValidPath, kInvalidMode), std::invalid_argument); 20 const char* const kValidPath = nullptr; const char* const kInvalidPath = nullptr;
21. Complete - covered const char* const kValidPath = nullptr; const char* const kInvalidPath = nullptr; const int kValidMode = kRead; const int kInvalidMode = kInvalid; ASSERT_THROW(openFile(kInvalidPath, kValidMode), std::invalid_argument); ASSERT_THROW(openFile(kValidPath, kInvalidMode), std::invalid_argument); 21 ASSERT_THROW(openFile(kValidPath, kInvalidMode), std::invalid_argument);
22. std::fstream openFile(const char* filePath, int mode) { if (!isValidPath(filePath)) { throw std::invalid_argument("invalid file path"); } if (!isValidMode(mode)) { throw std::invalid_argument("invalid mode"); } // ... } 22
23. std::fstream openFile(const char* filePath, int mode) { if (!isValidPath(filePath)) { throw std::invalid_argument("invalid file path"); } if (!isValidMode(mode)) { throw std::invalid_argument("invalid mode"); } // ... } 23 throw std::invalid_argument("invalid mode");
24. Complete - covered Do my test even reach code that I want to test? Can I remove code and keep test pass? Can code coverage report help? Is writing tests for coverage good? How to write fail safe tests? dzone.com/articles/say-hello-to-your-test-coverage-comic
25. Comprehensible Any developer working on the project should easily understand what tests are testing www.pophangover.com/19748/i-have-no-idea-what-im-doing-in-15-pictures
26. Comprehensible Tests are scenarios (examples of API use) Time investment Reduce mental effort (no magic numbers, no implicit contracts) Ideally, keep things on one screen (reduce number of jumps) 26
27. Comprehensible - testing patterns 27
28. Arrange-Act-Assert Pattern (AAA) TEST(SuiteName, CaseName) { // arrange // act // assert } 28
29. Setup-Execute-Check-Teardown (Four Phase Test Pattern) TEST(SuiteName, CaseName) { // setup // execute // check // teardown } 29
30. Arrange-Expect-Act TEST(SuiteName, CaseName) { // arrange // expect // act } 30
31. Comprehensible - magic numbers EXPECT_EQ(3, container.get(5)); 31
32. Comprehensible - magic numbers int key(const int k) { return k; } int value(const int v) { return v } EXPECT_EQ(value(3), container.get(key(5))); 32
33. Comprehensible - naming Ideally, name of failed test case should point where the bug is In most cases, good name should consist of Action (push_back) Preconditions (capacityExhausted) Postconditions (increaseCapacity, reallocate, copyAll, moveAll, …) Consistent scheme (e.g. given_when_then) 33
34. Comprehensible - naming Can you tell what these tests are testing? InserterTest.lifetime StringTest.streaming_invalid CarTest.stop 34
35. Comprehensible - differentiable 35
36. double pow2_v1(double val) { return val * val; } double pow2_v2(double val) { return std::pow(val, 2.0); } double hypotenuse(double side1, double side2) { return std::sqrt(pow2_v1(side1) + pow2_v2(side2)); } 36
37. Comprehensible - differentiable Emphasise what differs a test case from others If there is extra value in a redundant test case, explain intent well Naming a test doStuff_complexCase is not way to go Fuzz testing? 37
38. Verifiable by inspection Test is not tested so it should be verifiable by inspection Quick analysis of a test case should give confidence that it is correct twitter.com/jxxf/status/974715055479644162
39. Verifiable by inspection - minimalistic Long test case Long test suite If I cannot write a short test case… Does my code follows single responsibility principle? Can I move some processing to other classes? Can I use mocks so I don’t have to create the world? 39
40. Verifiable by inspection - one flow TEST(GreatClassTests, Test1) { helper(true); // ... } TEST(GreatClassTests, Test2) { helper(false); // ... } 40
41. void helper(bool flag) { preStuff(); if (flag) { trueStuff(); } else { falseStuff(); } postStuff(); } 41
42. void helperTrue() { preStuff(); trueStuff(); postStuff(); } void helperFalse() { preStuff(); falseStuff(); postStuff(); } 42
43. TEST(GreatClassTests, Test1) { helperTrue(); // ... } TEST(GreatClassTests, Test2) { helperFalse(); // ... } 43
44. Verifiable by inspection - one flow Avoid control flow statements Some code duplicity is fine Cyclometric complexity Implementation Test 44
45. Easy to modify and extend 45
46. class StringCollectionAndCaseShifter { public: // StringCollection void add(const std::string& value); std::vector<std::string>& values(); public: // CaseShifter void uppercase(); void lowercase(); }; 46
47. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 47
48. class StringCollectionAndCaseShifter { public: // StringCollection void add(const std::string& value); std::vector<std::string>& values(); public: // CaseShifter void uppercase(); void lowercase(); }; 48
49. class StringCollectionAndCaseShifter { public: // StringCollection void add(const std::string& value); std::vector<std::string>& values(); public: // CaseShifter void uppercase(); void lowercase(); }; 49 public: // StringCollection void add(const std::string& value); std::vector<std::string>& values();
50. class StringCollection { public: void add(const std::string& value); std::vector<std::string>& values(); }; 50
51. class StringCollectionAndCaseShifter { public: // StringCollection void add(const std::string& value); std::vector<std::string>& values(); public: // CaseShifter void uppercase(); void lowercase(); }; 51 public: // CaseShifter void uppercase(); void lowercase();
52. class CaseShifter { public: /** * Collection lifetime must be not shorter * than CaseShifter lifetime. */ CaseShifter(StringCollection& collection); void uppercase(); void lowercase(); }; 52
53. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 53
54. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 54 StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty());
55. TEST(StringCollectionAndCaseShifterTest, emptyCollection_valuesReturnEmptySet) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); } 55
56. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 56 StringCollectionAndCaseShifter s; s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values());
57. TEST(StringCollectionAndCaseShifterTest, oneElementAdded_valuesReturnOneElement) { StringCollectionAndCaseShifter s; s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); } 57
58. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 58 StringCollectionAndCaseShifter s; s.add("one"); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values());
59. TEST(StringCollectionAndCaseShifterTest, twoElementsAdded_valuesReturnTwoElements) { StringCollectionAndCaseShifter s; s.add("one"); s.add("two"); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 59
60. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 60 StringCollectionAndCaseShifter s; s.add("one"); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values());
61. TEST(StringCollectionAndCaseShifterTest, lowercaseElements_uppercase_uppercaseElements) { StringCollectionAndCaseShifter s; s.add("one"); s.add("two"); s.uppercase(); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); } 61
62. TEST(StringCollectionAndCaseShifterTest, masterpiece) { StringCollectionAndCaseShifter s; EXPECT_TRUE(s.values().empty()); s.add("one"); EXPECT_THAT(ElementsAre("one"), s.values()); s.uppercase(); EXPECT_THAT(ElementsAre("ONE"), s.values()); s.add("TWO"); EXPECT_THAT(ElementsAre("ONE", "TWO"), s.values()); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 62 StringCollectionAndCaseShifter s; s.add("one"); s.add("TWO"); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values());
63. TEST(StringCollectionAndCaseShifterTest, uppercaseElements_lowercase_lowercaseElements) { StringCollectionAndCaseShifter s; s.add("ONE"); s.add("TWO"); s.lowercase(); EXPECT_THAT(ElementsAre("one", "two"), s.values()); } 63
64. Easy to modify and extend StringCollectionTests emptyCollection_valuesReturnEmptySet oneElementAdded_valuesReturnOneElement twoElementsAdded_valuesReturnTwoElements CaseShifterTests lowercaseElements_uppercase_uppercaseElements uppercaseElements_lowercase_lowercaseElements 64
65. Easy to modify and extend Tests shouldn't be an obstacle in refactoring code Reduce number of expectations/assertions (add test cases instead) Don’t add unnecessary code to tests Keep high granularity of test cases Write a test as an example for a test to be added (copy-paste) 65
66. Easy to modify and extend - fixtures Use interface segregation principle in a fixture Keep high granularity of fixtures so they are not an obstacle in refactoring Complex setup and teardown hinder adding a new test case 66
67. Properties of maintainable tests summary Complete Comprehensible Verifiable by inspection Easy to modify and extend Conveying intent is the key! 67
68. TESTABLE DESIGN
69. Testable design Design hard to test Create the world No observation means Methods for testing purpose only Inheriting implementation class in tests Numerous mutable methods (stateful) 69
70. Testable design Design easy to test Stateless Single responsibility principle Dependency injection TDD 70
71. Learn more XUnitTest patterns
 xunitpatterns.com All your tests are terrible
 https://www.youtube.com/watch?v=u5senBJUkPc
72. Thanks! @michal_k_k