The Cartography of Testing: CodeMash 2015

Developers writing tests is now very very common. But testing remains a big, confusing, and controversial topic. This talk attempts to take a fresh look at testing, and try to get at a deeper more nuanced understanding of it. So even if you do TDD every day, hopefully you’ll come away with a refreshed and invigorated perspective.

Transcript

1. THE CARTOGRAPHY OF TESTING BY KEVIN BERRIDGE From Matrix Pointe Software
2. THE CARTOGRAPHY OF TESTING BY KEVIN BERRIDGE From Matrix Pointe Software
3. CONTENTIOUS ARGUMENTS
4. EASY TO GET LOST
5. MAP OUT THE TESTING SPACE
6. WHY WRITE TESTS? HOW DO WE WRITE “GOOD” TESTS?
7. WHY WRITE TESTS?
8. BECAUSE.
9. Uncle Bob Martin
10. REASONS
11. –Pragmatic Programmer “It ends up being cheaper in the long run.”
12. –Practical Object-Oriented Design In Ruby “The true purpose of testing, just like the true purpose of design, is to reduce costs.”
13. TESTING REDUCES COSTS?
14. I DON’T HAVE TIME TO TEST
15. COST == TIME
16. TESTING REDUCES COSTS?
17. TESTING REDUCES TIME?
18. No Tests 0 5000 10000 15000 20000 Production Test
19. 2x Test LOC 0 5000 10000 15000 20000 Production Test
20. –Practical Object-Oriented Design In Ruby “Fixing bugs early always lowers costs.”
21. –Code Complete “It is cheaper to build high- quality software than it is to build and fix low-quality software.”
22. –Software Test Automation “Run more often. Runs tests that are difficult or impossible manually. Better use of resources. Consistency & repeatability.”
23. –Practical Object-Oriented Design In Ruby “Tests provide the only reliable documentation of design.”
24. –Test-Driven Development “Be sure to factor debugging, integrating, and explaining time into your metrics, though.”
25. –Code Complete “The debugging job is easier.”
26. –Refactoring “These days I hardly ever spend more than a few minutes debugging.”
27. –Software Test Automation “The true value of automated tests is often in providing confidence rather than finding defects.”
28. wk1 wk2 wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11 wk12 PRODUCTIVITY
29. wk1 wk2 wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11 wk12 PRODUCTIVITY
30. SOFTWARE ROT
31. –@ChrisOldwood, Twitter “Not refactoring after making a code change is like putting an empty cereal box back in the cupboard.”
32. REFACTORING
33. BIG CHANGES
34. SMALL CHANGES
36. SOFTWARE ROT
37. WHAT DOES THIS HAVE TO DO WITH TESTING?
38. FEAR OF CHANGING CODE SHOE HORNING IN CHANGES SOFTWARE ROT
39. –Refactoring “If you want to refactor, the essential precondition is having solid tests.”
40. –Refactoring “Whenever I do a refactoring, the first step is always the same. I need to build a solid set of tests for that section of code. The tests are essential because even though I follow refactorings structured to avoid most of the opportunities for introducing bugs, I'm still human and still make mistakes. Thus I need solid tests.”
41. –Working Effectively With Legacy Code “To me, legacy code is simply code without tests.”
42. Martin Fowler
43. IF YOU DON’T HAVE TESTS, YOU WON’T CHANGE THE CODE
44. –Clean Code “It is unit tests that keep our code flexible, maintainable, and reusable." "If you have tests, you do not fear making changes to the code!”
45. –Refactoring “The tests give you the security to change the program later.”
46. –Clean Code “Code without tests is bad code. It doesn't matter how well written it is; it doesn't matter how pretty or object-oriented or well- encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them, we really don't know if our code is getting better or worse.”
47. –Growing Object-Oriented Software “We need to keep the code as simple as possible, so it's easier to understand and modify. …Simplicity takes effort, so we constantly refactor our code as we work with it. …The test suites in the feedback loops protect us against our own mistakes as we improve (and therefore change) the code.”
48. –Practical Object-Oriented Design In Ruby “The art of writing changeable code requires the ability to write high-value tests. Tests give you confidence to refactor constantly.”
49. WHY WRITE TESTS?
50. TESTS ARE THE BEST AND CHEAPEST TOOL WE HAVE TO MANAGE FEAR AND REPLACE IT WITH CONFIDENCE
51. WRITE “GOOD” TESTS SO YOU CAN (AND WILL) REFACTOR SO YOUR CODE DOESN’T ROT
52. –Test-Driven Development “My goal is to feel better about a project after a year than I did in the starry-eyed beginning, and TDD helps me achieve this.”
53. HOW TO WRITE “GOOD” TESTS?
54. PREFER “SMALLER” TESTS
55. MORE THAN ONE KIND OF TEST
56. UNIT INTEGRATION SYSTEM ACCEPTANCE VALIDATION/ VERIFICATION PERFORMANCE RESOURCE-EXHAUSTION USABILITY FUNCTIONAL DEVELOPER BLACK BOX GLASS BOX ISOLATED CHARACTERIZATION END-TO-END
58. SYSTEM ACCEPTANCE FUNCTIONAL BLACK BOX END-TO-END
59. END-TO-END
60. INTEGRATION DEVELOPER GLASS BOX ISOLATED CHARACTERIZATION UNIT
61. UNIT
62. Unit End-To-End
63. T O O O D S UI
64. T O O O D S UI
65. T O I V D S UI
66. PREFER “SMALLER” TESTS
67. FAST VS. SLOW
68. FEEDBACK
69. –Working Effectively With Legacy Code “Do you want your feedback in a minute or overnight?”
70. –Working Effectively With Legacy Code “Unit testing is one of the most important components in legacy code work. System-level regression tests are great, but small, localized tests are invaluable. They can give you feedback as you develop and allow you to refactor with much more safety.”
71. –Working Effectively With Legacy Code “The feedback we get from [testing] is very useful. It pays to [test] at a finer- grained level.”
72. COMPLEXITY
73. T O O O
74. –Growing Object-Oriented Software “If we test at too large a grain, the combinatorial explosion of trying all the possible paths through the code will bring development to a halt.”
75. –Working Effectively With Legacy Code “There are a few problems with large tests: Error localization, Execution time, Coverage”
76. http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/
77. 310 59,049
78. T O O O
79. T O I V
80. –Working Effectively With Legacy Code “There are a few problems with large tests: Error localization, Execution time, Coverage”
81. USE “LARGER” TESTS APPROPRIATELY
82. Acceptance Integration Unit 10% 20% 70% http://jamescrisp.org/2011/05/30/automated-testing-and-the-test-pyramid/
84. THE GOOS APPROACH
85. –Growing Object-Oriented Software “We start work on a new feature by writing failing acceptance tests that demonstrate that the system does not yet have the feature we're about to write and track our progress towards completion of the feature.”
86. END-TO-END: PROVE THE EXISTENCE OF A FEATURE NOT THE CORRECTNESS
87. “INTEGRATION” TESTS
88. –Mike Bland, http://martinfowler.com/articles/testing-culture.html “You should not be catching bugs at the integration or system levels that could have been caught at the unit level.”
89. THE EXTRACT CLASS REFACTORING
90. T O
91. T O O
92. T T O O
93. T T O O
94. T T O O
95. PREVENT TEST ROT
96. –Clean Code “Test code is just as important as production code.”
97. –Sandi Metz “I examine the original test and morph it from an integration test into an actual unit test.”
98. T T O O
99. TEST DOUBLES MOCKS STUBS FAKES SUBSTITUTES
100. ALTERNATE REALITY
101. bit.ly/kwb-oop
102. –Growing Object-Oriented Software “…investigate why the test is hard to write and refactor the code to improve its structure. We call this 'listening to the tests.'”
103. RESPOND TO THE TESTS
104. –Practical Object-Oriented Design In Ruby “Tests are the canary in the coal mine; when the design is bad, testing is hard.”
105. –Growing Object-Oriented Software “We've found that the qualities that make an object easy to test also make our code responsive to change.”
106. DEPENDENCY
107. –Working Effectively With Legacy Code “Dependency is one of the most critical problems in software development. Much legacy code work involves breaking dependencies so that change can be easier.”
108. –Test-Driven Development “Dependency is the key problem in software development at all scales.”
109. –Working Effectively With Legacy Code “When classes depend directly on things that are hard to use in a test, they are hard to modify and hard to work with.”
110. –Practical Object-Oriented Design In Ruby “Dealing with objects as if they are only and exactly the messages to which they respond lets you design a changeable application, and it is your understanding of the importance of this perspective that allows you to create tests that provide maximum benefit at minimum cost.”
111. TEST BEHAVIOR NOT IMPLEMENTATION
112. –Kent Beck “Separate interface from implementation thinking. I have a tendency to pollute API design decisions with implementation speculation.”
113. T O C Q
114. T I O C Q
115. T I O C Q
116. T O I C Q
117. –Practical Object-Oriented Design in Ruby “Some outgoing messages have no side effects… Outgoing messages like this are known as queries and they need not be tested by the sending object.”
118. –Practical Object-Oriented Design in Ruby “Many outgoing messages do have side effects… These messages are commands and it is the responsibility of the sending object to prove that they are properly sent.”
119. WRITE CLEAR TESTS
120. POORLY WRITTEN TESTS ARE A HINDRANCE
121. –Growing Object-Oriented Software “For TDD to be sustainable, the tests must do more than verify the behavior of the code; they must also express that behavior clearly -- they must be readable.”
122. –Clean Code “The dirtier the tests, the harder they are to change… So the tests become viewed as an ever- increasing liability.”
123. –Test-Driven Development “"Test methods should be easy to read.”
124. –Test-Driven Development “The first criterion for your tests is confidence. …The second criterion is communication.”
125. –Growing Object-Oriented Software “We want our test code to read like a declarative description of what is being testing.”
126. –Growing Object-Oriented Software “We want to make sure the tests pull their weight by making them expressive, so that we can tell what's important when we read them and when they fail”
127. MAKE EVERY SYMBOL IN THE TEST ESSENTIAL
128. –Clean Code “…the technique of building a domain-specific language for your tests”
129. CARE ABOUT THE FAILURE MESSAGE
130. –Software Test Automation “Test cases must be designed with debugging in mind by asking 'What would I want to know when this test fails.’”
131. –Growing Object-Oriented Software “When we get the 'right' failure, we check that the diagnostics are helpful.”
132. WHY WRITE TESTS?
133. TESTING REDUCES COSTS
134. SOFTWARE ROT
135. HOW TO WRITE “GOOD” TESTS?
136. PREFER “SMALLER” TESTS
137. USE “LARGER” TESTS APPROPRIATELY
138. PREVENT TEST ROT
139. RESPOND TO THE TESTS
140. TEST BEHAVIOR NOT IMPLEMENTATION
141. WRITE CLEAR TESTS
142. MAKE EVERY SYMBOL IN THE TEST ESSENTIAL
143. CARE ABOUT THE FAILURE MESSAGE
145. –Kevin Berridge, @kberridge “Thanks.”