467 tests, 0 failures, 0 confidence

Your tests failed. Again. And they didn't just fail, they failed you. There was no bug. There was no regression. You changed something trivial -- an implementation detail -- and your app still works but your tests are broken. They don't have your back. What purpose do they serve?

In this talk we'll evaluate the unit tests from an open source codebase. We will cover the roles a test plays in the lifecycle of a project, and explore guidelines that let us assess their worth, and determine which tests should live, which must die, and which are missing.

Transcript

1. ................................. ................................. ................................. ................................. ............................ 467 tests, 0 failures, 0 confidence Katrina Owen @kytrinyx JumpstartLab
2. Pointe •◀ .
3. ......F...
4. F F F F F F F F F
5. test results not implementation
6. results
7. command query separation . . . . . . . . . . . . . . . . . . ✂
8. returns a value, has no side effects query
9. has side effects, returns nil command
10. . . . . . . . . . . . . . . . . . ✂ command query
11. query @waiter.menu
12. command @bill.add(steak)
13. both ... it’s complicated
14. responsibility and collaboration
15. mock expectation
16. mock stub . . . . . . . . . . . . . . . . . ✂ assertion dumb value
17. who is responsible?
18. sender :message receiver outgoing incoming
19. incoming assert the state of the system
20. outgoing assert collaboration
21. rules of thumb
22. Query Incoming ?
23. Query Incoming assert return value
24. Query Command Incoming assert return value ?
25. Query Command Incoming assert return value assert side effects
26. Query Command Incoming Outgoing assert return value assert side effects ?
27. Query Command Incoming Outgoing assert return value assert side effects do not assert
28. Query Command Incoming Outgoing assert return value assert side effects do not assert ?
29. Query Command Incoming Outgoing assert return value assert side effects do not assert assert message sent
30. Query Command Incoming Private Outgoing assert return value assert side effects ? ? do not assert assert message sent
31. Query Command Incoming Private Outgoing assert return value assert side effects do not assert do not assert do not assert assert message sent
32. Query Command Incoming Private Outgoing assert return value assert side effects do not assert do not assert do not assert assert message sent
33. 1incoming query assert return value
34. 1incoming query assert return value assert_equal "...", @waiter.menu
35. assert direct, local side effects 2 incoming command
36. assert direct, local side effects 2 incoming command bill.add(steak)
37. assert direct, local side effects 2 bill.add(steak) assert_equal 32.90, bill.total incoming command
38. 3outgoing query do not assert
39. 3outgoing query do not assert chef.todays_special
40. assert message send 4 outgoing command
41. assert message send kitchen.expect(:order_up) 4 outgoing command
42. assert message send kitchen.expect(:order_up) waiter.i_would_like(:steak) 4 outgoing command
43. assert message send kitchen.expect(:order_up) waiter.i_would_like(:steak) kitchen.verify 4 outgoing command
44. assert message send 4 outgoing command kitchen.should_receive(:order_up) waiter.i_would_like(:steak)
45. 5private message do not assert
46. 5private message do not assert note_to_self(stuff)
47. unit test
48. unit test return values +
49. unit test return values + side effects +
50. unit test return values + side effects + critical interations +
51. unit test return values + side effects + critical interations + implementation details -
52. meet practice theory
53. github.com/it-agile codersdojo_client
54. @kata.generate 'ruby.rspec', 'prime'
55. code kata small coding exercise
56. @kata.generate 'ruby.rspec', 'prime'
57. @kata.generate 'ruby.rspec', 'prime'
58. @kata.generate 'ruby.rspec', 'prime'
59. @kata.generate 'ruby.rspec', 'prime'
60. class Prime def foo 'fixme' end end
61. class Prime def foo 'fixme' end end describe Prime do let(:prime) { Prime.new } its(:foo) { should == 'bar' } end
63. limit work in progress
64. Query Command Incoming Private Outgoing ✓
65. incoming command assert direct, local side effects
66. describe Scaffolder do it "generates the files and directories" it "replaces placeholders" end
67. describe Scaffolder do before (:each) do @shell = mock @shell.should_receive(:shell_extension).any_number_of_times.and_return "cmd" @shell.should_receive(:remove_command_name).any_number_of_times.and_return "del" @shell.should_receive(:path_separator).any_number_of_times.and_return ";" @shell.should_receive(:dir_separator).any_number_of_times.and_return "/" Scaffolder.should_receive(:current_file).any_number_of_times.and_return("aDir/lib/aFile.rb") @scaffolder = Scaffolder.new @shell end it "generates the files and directories" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and_return ["a", "b"] @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/b", "." @shell.should_receive(:file?).with("b").and_return fals @scaffolder.scaffold "a.template", 'myKata' end it "replaces placeholders" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and_return ["a", "README", "run-once.sh", "run-endless.sh"] @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/README", "." @shell.should_receive(:file?).with("README").and_return true @shell.should_receive(:read_file).with("README").and_return '%rm% %kata_file%\nb.%sh%\nc%:%d' @shell.should_receive(:write_file).with "README", 'del myKata\nb.cmd\nc;d' @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-once.sh", "." @shell.should_receive(:file?).with("run-once.sh").and_return true @shell.should_receive(:read_file).with("run-once.sh").and_return "%rm% a\nb.%sh%\nc%:%d" @shell.should_receive(:write_file).with "run-once.sh", "del a\nb.cmd\nc;d" @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-endless.sh", "." @shell.should_receive(:file?).with("run-endless.sh").and_return true @shell.should_receive(:read_file).with("run-endless.sh").and_return "%rm% a\nb.%sh%\nc%:%d" @shell.should_receive(:write_file).with "run-endless.sh", "del a\nb.cmd\nc;d" @scaffolder.scaffold "a.template", 'myKata' end end
68. describe Scaffolder do it "generates the files and directories" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/b", "." @shell.should_receive(:file?).with("b").and_return false @scaffolder.generate "a.template", 'myKata' end end
69. describe Scaffolder do it "replaces placeholders" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/README", "." @shell.should_receive(:file?).with("README").and_return true @shell.should_receive(:read_file).with("README").and_return '%rm% %kata_file%\ @shell.should_receive(:write_file).with "README", 'del myKata\nb.cmd\nc;d' @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-once.sh", "." @shell.should_receive(:file?).with("run-once.sh").and_return true @shell.should_receive(:read_file).with("run-once.sh").and_return "%rm% a\nb.%s @shell.should_receive(:write_file).with "run-once.sh", "del a\nb.cmd\nc;d" @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-endless.sh", @shell.should_receive(:file?).with("run-endless.sh").and_return true @shell.should_receive(:read_file).with("run-endless.sh").and_return "%rm% a\nb @shell.should_receive(:write_file).with "run-endless.sh", "del a\nb.cmd\nc;d" @scaffolder.generate "a.template", 'myKata' end end
70. describe Scaffolder do before(:each) do @shell = mock @shell.should_receive(:shell_extension).any_number_of_times.and_return "cmd" @shell.should_receive(:remove_command_name).any_number_of_times.and_return "de @shell.should_receive(:path_separator).any_number_of_times.and_return ";" @shell.should_receive(:dir_separator).any_number_of_times.and_return "/" Scaffolder.should_receive(:current_file).any_number_of_times.and_return("aDir/ @scaffolder = Scaffolder.new @shell end end
71. ZOM G ......F...
72. <3
73. test all the things
74. 30 assertions
75. 30 assertions @shell.should_receive(:x)
76. 30 assertions @shell.should_receive(:x) .with(y)
77. 30 assertions @shell.should_receive(:x) .with(y) .and_return z
78. ......... ......... ......... .........
79. unit tests 30 assertions 20 assertions 10 assertions 0 assertions
80. unit tests testing what, exactly?
81. @kata.generate README prime.rb run.sh ... ?
82. @kata.generate README prime.rb run.sh ...
83. magic happens here
84. @shell.should_receive(:real_dir_entries).with("aDir/ @shell.should_receive(:cp_r).with "aDir/templates/a. @shell.should_receive(:file?).with("a").and_return f @shell.should_receive(:cp_r).with "aDir/templates/a. @shell.should_receive(:file?).with("README").and_ret @shell.should_receive(:read_file).with("README").and @shell.should_receive(:write_file).with "README", 'd @shell.should_receive(:cp_r).with "aDir/templates/a. @shell.should_receive(:file?).with("run-once.sh").an @shell.should_receive(:read_file).with("run-once.sh" @shell.should_receive(:write_file).with "run-once.sh @shell.should_receive(:cp_r).with "aDir/templates/a. @shell.should_receive(:file?).with("run-endless.sh") @shell.should_receive(:read_file).with("run-endless. @shell.should_receive(:write_file).with "run-endless
85. delegation vs micro-management
86. unit tests have conceded defeat
87. describe Scaffolder do before (:each) do @shell = mock @shell.should_receive(:shell_extension).any_number_of_times.and_return "cmd" @shell.should_receive(:remove_command_name).any_number_of_times.and_return "del" @shell.should_receive(:path_separator).any_number_of_times.and_return ";" @shell.should_receive(:dir_separator).any_number_of_times.and_return "/" Scaffolder.should_receive(:current_file).any_number_of_times.and_return("aDir/lib/aFile.rb") @scaffolder = Scaffolder.new @shell end it "generates the files and directories" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and_return ["a", "b"] @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/b", "." @shell.should_receive(:file?).with("b").and_return false @scaffolder.scaffold "a.template", 'myKata' end it "replaces placeholders" do @shell.should_receive(:real_dir_entries).with("aDir/templates/a.template").and_return ["a", "README", "run-once.sh", "run-endless.sh"] @shell.should_receive(:cp_r).with "aDir/templates/a.template/a", "." @shell.should_receive(:file?).with("a").and_return false @shell.should_receive(:cp_r).with "aDir/templates/a.template/README", "." @shell.should_receive(:file?).with("README").and_return true @shell.should_receive(:read_file).with("README").and_return '%rm% %kata_file%\nb.%sh%\nc%:%d' @shell.should_receive(:write_file).with "README", 'del myKata\nb.cmd\nc;d' @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-once.sh", "." @shell.should_receive(:file?).with("run-once.sh").and_return true @shell.should_receive(:read_file).with("run-once.sh").and_return "%rm% a\nb.%sh%\nc%:%d" @shell.should_receive(:write_file).with "run-once.sh", "del a\nb.cmd\nc;d" @shell.should_receive(:cp_r).with "aDir/templates/a.template/run-endless.sh", "." @shell.should_receive(:file?).with("run-endless.sh").and_return true @shell.should_receive(:read_file).with("run-endless.sh").and_return "%rm% a\nb.%sh%\nc%:%d" @shell.should_receive(:write_file).with "run-endless.sh", "del a\nb.cmd\nc;d" @scaffolder.scaffold "a.template", 'myKata' end end
88. unit tests for the generated kata files
89. integration tests for the generated kata files
90. characterization
91. characterization run the code
92. characterization run the code make assertions about whatever happens
93. describe Scaffolder do end
94. describe Scaffolder do it "works" do scaffolder.generate('ruby.rspec', 'prime') end end
95. describe Scaffolder do let(:scaffolder) { } it "works" do scaffolder.generate('ruby.rspec', 'prime') end end
96. describe Scaffolder do let(:scaffolder) { Scaffolder.new } it "works" do scaffolder.generate('ruby.rspec', 'prime') end end
97. describe Scaffolder do let(:scaffolder) { Scaffolder.new(shell) } it "works" do scaffolder.generate('ruby.rspec', 'prime') end end
98. describe Scaffolder do let(:shell) { ShellWrapper.new } let(:scaffolder) { Scaffolder.new(shell) } it "works" do scaffolder.scaffold('ruby.rspec', 'prime') end end
100. $ git status
101. $ git status # Untracked files: # #!.meta #!README #!prime.rb #!run-endless.sh #!run-once.sh
102. describe Scaffolder do let(:files) do [ '.meta', 'prime.rb', 'README', 'run-endless.sh', 'run-once.sh' ] end it "works"... end
103. describe Scaffolder do it "works" do scaffolder.scaffold('ruby.rspec', 'prime') expect(File.read('.meta')).to eq('...') expect(File.read('README')).to eq('...') # etc end end
104. describe Scaffolder do after(:each) do files.each do |file| FileUtils.rm_f(file) end end it "works" do # ... end end
105. Finished in 0.00632 seconds 1 example, 0 failures .
106. I’m no genius and so can you
107. reality vs fantasy
108. red flag time to refactor
109. the secret fate of a characterization test
110. confidence to refactor
111. decouple template source and kata destination
112. simplify the algorithm
113. spot the abstractions
114. a KataFile KataFile = Struct.new(:name, :contents)
115. kata a Translator template
116. a KataStore ./templates/ruby.rspec/* [kata_file1, kata_file2, ...] [kata_file1, kata_file2, ...] /tmp/*
117. or something like that
118. theory vs practice
119. clarity and chaos
120. Query Command Incoming Private Outgoing assert return value assert side effects do not assert do not assert do not assert assert message sent
121. F F F F F F F thank you <3 <3 <3 ......... c
122. Practical Object Oriented Design in Ruby Sandi Metz The Magic Tricks of Testing Sandi Metz Working Effectively with Legacy Code Michael Feathers CodersDojo Client FreeBSD License, (c) The CodersDojo Team Limited Red Joseph Wilk Fonts & Colors Damon Davison