reality is expensive: a better way of thinking about mock objects

Videos of this talk:
Rocky Mountain Ruby '12 — http://www.youtube.com/watch?v=TU3glG08BJI
Øredev 2012 — http://vimeo.com/53276460
SCNA 2012 — http://vimeo.com/54045166

This talk is an introduction to the different ways people use test doubles (be they mocks, fakes, stubs, or spies). Our goal will be to establish a more sophisticated means of communicating on the topic. This talk discusses our need to understand that reality is a variable under our control, the pros & cons of the different testing strategies, and the smells of test double abuse.

Transcript

2. imagine: testing landing gear
3. if our tests prove that our code works
4. then "fake" means "cheat," right?
5. we're all cheaters
6. selenium
7. selenium A user who moves the mouse & keyboard exactly how you tell it to.
8. selenium A user who moves the mouse & keyboard exactly how you tell it to. Over and over.
9. selenium A user who moves the mouse & keyboard exactly how you tell it to. Over and over. Every time.
10. test data
11. test data Runner cleans the database on every test run.
12. test data Runner cleans the database on every test run. Got years of data migrations?
13. test data Runner cleans the database on every test run. Got years of data migrations? No problem!
14. stamina
15. stamina Runtime is always started fresh for tests.
16. stamina Runtime is always started fresh for tests. Subtle memory leak?
17. stamina Runtime is always started fresh for tests. Subtle memory leak? Tests won't mind!
18. web scale
19. web scale Tests run against a single-process runner or server.
20. web scale Tests run against a single-process runner or server. Race conditions?
21. web scale Tests run against a single-process runner or server. Race conditions? No deadlocks here!
22. reality is expensive
23. why not budget it? reality is expensive
24. so, instead of asking
25. is this test too fake?
26. perhaps, we should ask
27. how much reality does this test need?
28. unknown fakeness we talked about
29. known fakeness? but what about
30. 1. identify fake things we already consciously do in our tests know thy fakeness
31. 2. understand those fake things well enough to appraise their costs & benefits know thy fakeness
32. 3. write more valuable tests by strategically deciding how much reality to give them know thy fakeness
33. oft-overheard at conferences
34. "Mocks are Good!" oft-overheard at conferences
35. "Mocks are Good!" Bad! oft-overheard at conferences
36. "Only Mock What You Own!" oft-overheard at conferences
37. "Only Mock What You Own!" external systems! oft-overheard at conferences
38. "Mock everything!" oft-overheard at conferences
39. "Mock everything!" avoid over-mocking! oft-overheard at conferences
40. I'd some
41. nuance I'd some
42. rigor I'd some
43. thoughtfulness I'd some
44. types of tests
45. Dog App Tester Dog App end-to-end test
46. DogFeederTest DogRepo Database DogFeeder integration test BoneRepo
47. DogFeederTest DogRepo Fake Database DogFeeder "unit" test BoneRepo
48. DogFeederTest Fake DogRepo DogFeeder isolation test Fake BoneRepo
49. mock The popular term for a fake object that takes the place of a real object when a test is executed. n. \ˈmäk, ˈmȯk\
50. test double The precise term for a fake object that takes the place of a real object when a test is executed. n. \ˈtest də-bəl\
51. think "stunt double"
52. our silly catch-all term test double
53. an alternative implementation to stand-in for something you depend on fake test double
54. replies to certain messages with responses that help you write tests fake test double stub
55. ensures certain messages are received (and explodes upon receiving any unexpected messages) fake test double stub mock
56. stealthily records all interactions, allowing you to make assertions about them later fake test double stub mock spy
57. how should we use them?
58. hold your fake horses!
59. before the
60. why should we use them?
61. why do we test at all?
62. ACCEPTANCE prove the app works as promised
63. ACCEPTANCE prove the app works as promised
64. SPECIFICATION examples of how the code behaves
65. SPECIFICATION examples of how the code behaves
66. REGRESSION prevent bugs from coming back
67. REGRESSION prevent bugs from coming back
68. DESIGN shape code by listening to tests
69. DESIGN shape code by listening to tests
70. CHARACTERIZE safely improve legacy code
71. CHARACTERIZE safely improve legacy code
72. this test provide?" ask, "what value will
73. value-oriented test strategies
74. prove it works
75. prove it works minimize fakeness
76. prove it works only write end-to-end & integration tests minimize fakeness
77. value cost prove it works
78. value cost prove it works • confidence that passing equals working
79. value cost prove it works • confidence that passing equals working • fewer tests → less effort spent writing tests
80. value cost prove it works • confidence that passing equals working • fewer tests → less effort spent writing tests • tests can't provide many design cues
81. value cost prove it works • confidence that passing equals working • fewer tests → less effort spent writing tests • tests can't provide many design cues • slow feedback
82. value cost prove it works • confidence that passing equals working • fewer tests → less effort spent writing tests • tests can't provide many design cues • slow feedback • high coverage is infeasible
83. passage of time build duration
84. passage of time build duration what we expect
85. passage of time build duration
86. passage of time build duration what actually happens once you account for growth of the system
87. mocking boundaries
88. mocking boundaries fake remote systems, never your app's code
89. mocking boundaries fake remote systems, never your app's code primarily write unit & end-to-end tests
90. value cost mocking boundaries
91. value cost mocking boundaries • each object is exercised by many tests
92. value cost mocking boundaries • each object is exercised by many tests • practical tests without deep knowledge of test doubles
93. value cost mocking boundaries • each object is exercised by many tests • practical tests without deep knowledge of test doubles • tests give less feedback about interactions
94. value cost mocking boundaries • each object is exercised by many tests • practical tests without deep knowledge of test doubles • tests give less feedback about interactions • one change → many test fixes
95. value cost mocking boundaries • each object is exercised by many tests • practical tests without deep knowledge of test doubles • tests give less feedback about interactions • one change → many test fixes • end-to-end is extra redundant
96. case-by-case
97. case-by-case cater your testing approach to the needs of each situation
98. value cost case-by-case
99. value cost case-by-case • developers free to choose best approach
100. value cost case-by-case • developers free to choose best approach • your team is likely already doing it!
101. value cost case-by-case • developers free to choose best approach • your team is likely already doing it! • test doubles easily abused
102. value cost case-by-case • developers free to choose best approach • your team is likely already doing it! • test doubles easily abused • no strategy → murky test value
103. value cost case-by-case • developers free to choose best approach • your team is likely already doing it! • test doubles easily abused • no strategy → murky test value • "wanna mock that?" timesink
104. the GOOS way
105. the GOOS way isolation tests to drive design
106. the GOOS way isolation tests to drive design end-to-end tests to prove it works
107. value cost the GOOS way
108. value cost the GOOS way • TDD gives rich feedback about interactions
109. value cost the GOOS way • TDD gives rich feedback about interactions • consistent, fast, complete
110. value cost the GOOS way • TDD gives rich feedback about interactions • consistent, fast, complete • easy to extract from app to libs
111. value cost the GOOS way • TDD gives rich feedback about interactions • consistent, fast, complete • easy to extract from app to libs • requires discipline & practice
112. value cost the GOOS way • TDD gives rich feedback about interactions • consistent, fast, complete • easy to extract from app to libs • requires discipline & practice • awkward "in- between" tests
113. value cost the GOOS way • TDD gives rich feedback about interactions • consistent, fast, complete • easy to extract from app to libs • requires discipline & practice • awkward "in- between" tests • frameworks dislike isolation
114. my favorite strategy
115. 1. agree on literally any clear strategy my favorite strategy
116. 1. agree on literally any clear strategy 2. follow that strategy consistently my favorite strategy
117. test double smells
118. smell # 1 test doubles are used in integration tests
119. perhaps test integrity was compromised to address painful setup or assertions
120. suggestion respond to pain by changing your code, not your tests
121. smell #2 test replaces methods on the subject-under-test with test doubles
122. perhaps subject has too many responsibilities
123. suggestion extract a new object as a collaborator of the subject
124. smell #3 your objects extend framework classes that you don't own
125. perhaps inherited behavior inhibits good test isolation
126. suggestion avoid such extension or forgo isolation testing of those classes
127. smell #4 test contains lots of code just to configure test doubles
128. perhaps the unit's number of collaborators or their collective surface area is too large
129. suggestion reduce the count of methods on which the subject depends
130. smell #5 test doubles are used to stand-in for 3rd party APIs
131. perhaps test pain isn't actionable, because design of 3rd party code can't be changed
132. suggestion wrap third-party libraries in small abstractions you own
133. blame the code before the test
134. blame the code before the test double ^
135. www.testdouble.com @andrewvida @jasonkarns @dmosher @joelhelbling @searls @kbaribeau @timwingfield @toddkaufman