The Cartography of Testing: CodeMash 2015

Developers writing tests is now very very common. But testing remains a big, confusing, and controversial topic. This talk attempts to take a fresh look at testing, and try to get at a deeper more nuanced understanding of it. So even if you do TDD every day, hopefully you’ll come away with a refreshed and invigorated perspective.

Transcript

1. THE
   CARTOGRAPHY
   OF
   TESTING
   BY KEVIN BERRIDGE
   From Matrix Pointe Software
   

   View Slide

2. THE
   CARTOGRAPHY
   OF
   TESTING
   BY KEVIN BERRIDGE
   From Matrix Pointe Software
   

   View Slide

3. CONTENTIOUS
   ARGUMENTS
   

   View Slide

4. EASY TO
   GET LOST
   

   View Slide

5. MAP OUT THE
   TESTING SPACE
   

   View Slide

6. WHY WRITE TESTS?
   HOW DO WE WRITE
   “GOOD” TESTS?
   

   View Slide

7. WHY WRITE TESTS?
   

   View Slide

8. BECAUSE.
   

   View Slide

9. Uncle Bob Martin
   

   View Slide

10. REASONS
    

    View Slide

11. –Pragmatic Programmer
    “It ends up being cheaper
    in the long run.”
    

    View Slide

12. –Practical Object-Oriented Design In
    Ruby
    “The true purpose of testing,
    just like the true purpose of
    design, is to reduce costs.”
    

    View Slide

13. TESTING
    REDUCES
    COSTS?
    

    View Slide

14. I DON’T HAVE TIME TO TEST
    

    View Slide

15. COST == TIME
    

    View Slide

16. TESTING
    REDUCES
    COSTS?
    

    View Slide

17. TESTING
    REDUCES
    TIME?
    

    View Slide

18. No Tests
    0
    5000
    10000
    15000
    20000
    Production Test
    

    View Slide

19. 2x Test LOC
    0
    5000
    10000
    15000
    20000
    Production Test
    

    View Slide

20. –Practical Object-Oriented Design In
    Ruby
    “Fixing bugs early always
    lowers costs.”
    

    View Slide

21. –Code Complete
    “It is cheaper to build high-
    quality software than it is to
    build and fix low-quality
    software.”
    

    View Slide

22. –Software Test Automation
    “Run more often.
    Runs tests that are difficult or
    impossible manually.
    Better use of resources.
    Consistency & repeatability.”
    

    View Slide

23. –Practical Object-Oriented Design In
    Ruby
    “Tests provide the only
    reliable documentation of
    design.”
    

    View Slide

24. –Test-Driven Development
    “Be sure to factor
    debugging, integrating, and
    explaining time into your
    metrics, though.”
    

    View Slide

25. –Code Complete
    “The debugging job is
    easier.”
    

    View Slide

26. –Refactoring
    “These days I hardly ever
    spend more than a few
    minutes debugging.”
    

    View Slide

27. –Software Test Automation
    “The true value of
    automated tests is often in
    providing confidence rather
    than finding defects.”
    

    View Slide

28. wk1 wk2 wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11 wk12
    PRODUCTIVITY
    

    View Slide

29. wk1 wk2 wk3 wk4 wk5 wk6 wk7 wk8 wk9 wk10 wk11 wk12
    PRODUCTIVITY
    

    View Slide

30. SOFTWARE
    ROT
    

    View Slide

31. –@ChrisOldwood, Twitter
    “Not refactoring after making a
    code change is like putting an
    empty cereal box back in the
    cupboard.”
    

    View Slide

32. REFACTORING
    

    View Slide

33. BIG
    CHANGES
    

    View Slide

34. SMALL
    CHANGES
    

    View Slide

35. View Slide

36. SOFTWARE
    ROT
    

    View Slide

37. WHAT DOES THIS HAVE
    TO DO WITH TESTING?
    

    View Slide

38. FEAR OF
    CHANGING
    CODE
    SHOE HORNING
    IN CHANGES
    SOFTWARE
    ROT
    

    View Slide

39. –Refactoring
    “If you want to refactor, the
    essential precondition is
    having solid tests.”
    

    View Slide

40. –Refactoring
    “Whenever I do a refactoring, the first step is
    always the same. I need to build a solid set of
    tests for that section of code. The tests are
    essential because even though I follow
    refactorings structured to avoid most of the
    opportunities for introducing bugs, I'm still human
    and still make mistakes. Thus I need solid tests.”
    

    View Slide

41. –Working Effectively With Legacy Code
    “To me, legacy code is
    simply code without tests.”
    

    View Slide

42. Martin Fowler
    

    View Slide

43. IF YOU DON’T HAVE TESTS,
    YOU WON’T CHANGE THE
    CODE
    

    View Slide

44. –Clean Code
    “It is unit tests that keep our
    code flexible, maintainable,
    and reusable."
    "If you have tests, you do
    not fear making changes to
    the code!”
    

    View Slide

45. –Refactoring
    “The tests give you the
    security to change the
    program later.”
    

    View Slide

46. –Clean Code
    “Code without tests is bad code. It doesn't
    matter how well written it is; it doesn't matter
    how pretty or object-oriented or well-
    encapsulated it is. With tests, we can change
    the behavior of our code quickly and
    verifiably. Without them, we really don't know
    if our code is getting better or worse.”
    

    View Slide

47. –Growing Object-Oriented Software
    “We need to keep the code as simple as
    possible, so it's easier to understand and
    modify. …Simplicity takes effort, so we
    constantly refactor our code as we work with
    it. …The test suites in the feedback loops
    protect us against our own mistakes as we
    improve (and therefore change) the code.”
    

    View Slide

48. –Practical Object-Oriented Design In
    Ruby
    “The art of writing changeable
    code requires the ability to write
    high-value tests. Tests give you
    confidence to refactor
    constantly.”
    

    View Slide

49. WHY WRITE TESTS?
    

    View Slide

50. TESTS ARE THE BEST AND
    CHEAPEST TOOL WE HAVE TO
    MANAGE FEAR AND REPLACE IT
    WITH CONFIDENCE
    

    View Slide

51. WRITE
    “GOOD” TESTS
    SO YOU CAN
    (AND WILL)
    REFACTOR
    SO YOUR
    CODE DOESN’T
    ROT
    

    View Slide

52. –Test-Driven Development
    “My goal is to feel better about a project after
    a year than I did in the starry-eyed
    beginning, and TDD helps me achieve this.”
    

    View Slide

53. HOW TO WRITE
    “GOOD” TESTS?
    

    View Slide

54. PREFER
    “SMALLER”
    TESTS
    

    View Slide

55. MORE THAN ONE
    KIND OF TEST
    

    View Slide

56. UNIT
    INTEGRATION
    SYSTEM
    ACCEPTANCE
    VALIDATION/
    VERIFICATION
    PERFORMANCE
    RESOURCE-EXHAUSTION
    USABILITY
    FUNCTIONAL
    DEVELOPER
    BLACK BOX
    GLASS BOX
    ISOLATED
    CHARACTERIZATION
    END-TO-END
    

    View Slide

57. View Slide

58. SYSTEM
    ACCEPTANCE
    FUNCTIONAL
    BLACK BOX
    END-TO-END
    

    View Slide

59. END-TO-END
    

    View Slide

60. INTEGRATION
    DEVELOPER
    GLASS BOX
    ISOLATED
    CHARACTERIZATION
    UNIT
    

    View Slide

61. UNIT
    

    View Slide

62. Unit End-To-End
    

    View Slide

63. T
    O
    O
    O
    D
    S
    UI
    

    View Slide

64. T
    O
    O
    O
    D
    S
    UI
    

    View Slide

65. T
    O
    I
    V
    D
    S
    UI
    

    View Slide

66. PREFER
    “SMALLER”
    TESTS
    

    View Slide

67. FAST VS. SLOW
    

    View Slide

68. FEEDBACK
    

    View Slide

69. –Working Effectively With Legacy Code
    “Do you want your feedback
    in a minute or overnight?”
    

    View Slide

70. –Working Effectively With Legacy Code
    “Unit testing is one of the most important components
    in legacy code work. System-level regression tests are
    great, but small, localized tests are invaluable. They
    can give you feedback as you develop and allow you
    to refactor with much more safety.”
    

    View Slide

71. –Working Effectively With Legacy Code
    “The feedback we get from
    [testing] is very useful. It
    pays to [test] at a finer-
    grained level.”
    

    View Slide

72. COMPLEXITY
    

    View Slide

73. T
    O
    O
    O
    

    View Slide

74. –Growing Object-Oriented Software
    “If we test at too large a grain,
    the combinatorial explosion of
    trying all the possible paths
    through the code will bring
    development to a halt.”
    

    View Slide

75. –Working Effectively With Legacy Code
    “There are a few problems with
    large tests: Error localization,
    Execution time, Coverage”
    

    View Slide

76. http://blog.thecodewhisperer.com/2010/10/16/integrated-tests-are-a-scam/
    

    View Slide

77. 310
    59,049
    

    View Slide

78. T
    O
    O
    O
    

    View Slide

79. T
    O
    I
    V
    

    View Slide

80. –Working Effectively With Legacy Code
    “There are a few problems with
    large tests: Error localization,
    Execution time, Coverage”
    

    View Slide

81. USE “LARGER”
    TESTS
    APPROPRIATELY
    

    View Slide

82. Acceptance
    Integration
    Unit
    10%
    20%
    70%
    http://jamescrisp.org/2011/05/30/automated-testing-and-the-test-pyramid/
    

    View Slide

83. View Slide

84. THE GOOS
    APPROACH
    

    View Slide

85. –Growing Object-Oriented Software
    “We start work on a new feature by
    writing failing acceptance tests that
    demonstrate that the system does not
    yet have the feature we're about to
    write and track our progress towards
    completion of the feature.”
    

    View Slide

86. END-TO-END:
    PROVE THE
    EXISTENCE OF A
    FEATURE NOT THE
    CORRECTNESS
    

    View Slide

87. “INTEGRATION”
    TESTS
    

    View Slide

88. –Mike Bland, http://martinfowler.com/articles/testing-culture.html
    “You should not be catching bugs
    at the integration or system levels
    that could have been caught at
    the unit level.”
    

    View Slide

89. THE EXTRACT CLASS
    REFACTORING
    

    View Slide

90. T O
    

    View Slide

91. T O O
    

    View Slide

92. T T
    O O
    

    View Slide

93. T T
    O O
    

    View Slide

94. T T
    O O
    

    View Slide

95. PREVENT
    TEST
    ROT
    

    View Slide

96. –Clean Code
    “Test code is just as
    important as production
    code.”
    

    View Slide

97. –Sandi Metz
    “I examine the original test and
    morph it from an integration test
    into an actual unit test.”
    

    View Slide

98. T T
    O O
    

    View Slide

99. TEST DOUBLES
    MOCKS
    STUBS
    FAKES
    SUBSTITUTES
    

    View Slide

100. ALTERNATE REALITY
     

     View Slide

101. bit.ly/kwb-oop
     

     View Slide

102. –Growing Object-Oriented Software
     “…investigate why the test is hard to
     write and refactor the code to improve
     its structure. We call this 'listening to
     the tests.'”
     

     View Slide

103. RESPOND
     TO THE
     TESTS
     

     View Slide

104. –Practical Object-Oriented Design In
     Ruby
     “Tests are the canary in the coal mine;
     when the design is bad, testing is
     hard.”
     

     View Slide

105. –Growing Object-Oriented Software
     “We've found that the qualities that
     make an object easy to test also make
     our code responsive to change.”
     

     View Slide

106. DEPENDENCY
     

     View Slide

107. –Working Effectively With Legacy Code
     “Dependency is one of the most critical
     problems in software development.
     Much legacy code work involves
     breaking dependencies so that change
     can be easier.”
     

     View Slide

108. –Test-Driven Development
     “Dependency is the key problem in
     software development at all scales.”
     

     View Slide

109. –Working Effectively With Legacy Code
     “When classes depend directly
     on things that are hard to use in
     a test, they are hard to modify
     and hard to work with.”
     

     View Slide

110. –Practical Object-Oriented Design In
     Ruby
     “Dealing with objects as if they are only
     and exactly the messages to which they
     respond lets you design a changeable
     application, and it is your understanding
     of the importance of this perspective that
     allows you to create tests that provide
     maximum benefit at minimum cost.”
     

     View Slide

111. TEST BEHAVIOR
     NOT
     IMPLEMENTATION
     

     View Slide

112. –Kent Beck
     “Separate interface from
     implementation thinking. I have
     a tendency to pollute API
     design decisions with
     implementation speculation.”
     

     View Slide

113. T O
     C
     Q
     

     View Slide

114. T I O
     C
     Q
     

     View Slide

115. T I O
     C
     Q
     

     View Slide

116. T O
     I
     C
     Q
     

     View Slide

117. –Practical Object-Oriented Design in
     Ruby
     “Some outgoing messages have
     no side effects… Outgoing
     messages like this are known as
     queries and they need not be
     tested by the sending object.”
     

     View Slide

118. –Practical Object-Oriented Design in
     Ruby
     “Many outgoing messages do have
     side effects… These messages are
     commands and it is the
     responsibility of the sending object
     to prove that they are properly sent.”
     

     View Slide

119. WRITE
     CLEAR
     TESTS
     

     View Slide

120. POORLY WRITTEN TESTS
     ARE A HINDRANCE
     

     View Slide

121. –Growing Object-Oriented Software
     “For TDD to be sustainable, the
     tests must do more than verify the
     behavior of the code; they must
     also express that behavior clearly --
     they must be readable.”
     

     View Slide

122. –Clean Code
     “The dirtier the tests, the harder
     they are to change… So the tests
     become viewed as an ever-
     increasing liability.”
     

     View Slide

123. –Test-Driven Development
     “"Test methods should be
     easy to read.”
     

     View Slide

124. –Test-Driven Development
     “The first criterion for your tests
     is confidence. …The second
     criterion is communication.”
     

     View Slide

125. –Growing Object-Oriented Software
     “We want our test code to read
     like a declarative description of
     what is being testing.”
     

     View Slide

126. –Growing Object-Oriented Software
     “We want to make sure the tests
     pull their weight by making them
     expressive, so that we can tell
     what's important when we read
     them and when they fail”
     

     View Slide

127. MAKE EVERY
     SYMBOL IN THE
     TEST ESSENTIAL
     

     View Slide

128. –Clean Code
     “…the technique of building
     a domain-specific language
     for your tests”
     

     View Slide

129. CARE ABOUT
     THE FAILURE
     MESSAGE
     

     View Slide

130. –Software Test Automation
     “Test cases must be designed
     with debugging in mind by
     asking 'What would I want to
     know when this test fails.’”
     

     View Slide

131. –Growing Object-Oriented Software
     “When we get the 'right'
     failure, we check that the
     diagnostics are helpful.”
     

     View Slide

132. WHY WRITE TESTS?
     

     View Slide

133. TESTING
     REDUCES
     COSTS
     

     View Slide

134. SOFTWARE
     ROT
     

     View Slide

135. HOW TO WRITE
     “GOOD” TESTS?
     

     View Slide

136. PREFER
     “SMALLER”
     TESTS
     

     View Slide

137. USE “LARGER”
     TESTS
     APPROPRIATELY
     

     View Slide

138. PREVENT
     TEST
     ROT
     

     View Slide

139. RESPOND
     TO THE
     TESTS
     

     View Slide

140. TEST BEHAVIOR
     NOT
     IMPLEMENTATION
     

     View Slide

141. WRITE
     CLEAR
     TESTS
     

     View Slide

142. MAKE EVERY
     SYMBOL IN THE
     TEST ESSENTIAL
     

     View Slide

143. CARE ABOUT
     THE FAILURE
     MESSAGE
     

     View Slide

144. View Slide

145. –Kevin Berridge, @kberridge
     “Thanks.”
     

     View Slide