Can you trust your tests?

A talk I gave at Sardines.rb (Ruby meetup) in Lisbon, Portugal.
It's about test code quality, coverage and mutation testing.

1. CAN YOU TRUST YOUR TESTS? Tadas Ščerbinskas 2016 sardines.rb Lisbon, Portugal
2. Vaidas Pilkauskas (@liucijus) • Vilnius JUG co-founder • Vilnius Scala leader • CodeRetreat facilitator Thanks to co-author
3. AGENDA • Test quality • Coverage • Mutation testing in theory • Mutation testing in practice
4. TEST QUALITY Readable Focused Concise Well named
5. PROD VS. TEST CODE QUALITY Code has bugs. Tests are code. Tests have bugs.
7. CODE COVERAGE
8. TYPES OF CODE COVERAGE • Lines • Branches • Instructions • Cyclomatic Complexity • Methods • & more
9. LINES expect(name).to eq 'Tadas'
10. LINES expect(name).to eq 'Tadas' def name 'Tadas' end
11. LINES expect(name).to eq 'Tadas' def name 'Tadas' end 100% coverage
12. LINES/BRANCHES expect(name).to eq 'Tadas' def name(initials = false) initials ? 'TS' : 'Tadas' end
13. LINES/BRANCHES expect(name).to eq 'Tadas' def name(initials = false) initials ? 'TS' : 'Tadas' end still 100% line coverage
14. LINES/BRANCHES expect(name).to eq 'Tadas' expect(name(true)).to eq 'TS' def name(initials = false) initials ? 'TS' : 'Tadas' end 100% line & branch coverage
15. COVERAGE TOOL FOR RUBY
18. CAN YOU TRUST 100% COVERAGE? • Code coverage can only show what is not tested • Can be gamed • Still a great tool
19. MUTATION TESTING
20. def ten?(number) number == 10 end WHAT EXACTLY IS A MUTATION?
21. def ten?(number) number == 10 end WHAT EXACTLY IS A MUTATION? def ten?(number) number != 10 end def ten?(number) true end def ten?(number) nil end
22. TERMINOLOGY Applying a mutation to some code creates a mutant. If all tests pass - mutant has survived. If a test fails - mutant is killed.
23. FAILING IS THE NEW PASSING
24. MUTATION TOOL FOR RUBY
25. • Active project • Good reporting • Only works with rspec • minitest support comming soon
26. USAGE > mutant --include lib \ --require virtus \ --use rspec \ Virtus::Attribute#type
27. MUTATIONS • Literal / primitive and compound statement deletion • Argument deletion / rename • Swap Unary operator • Conditionals See all at: https://github.com/mbj/mutant/tree/master/meta
28. HOW DOES IT WORK?
29. HOW DOES IT WORK? String#gsub
30. HOW DOES IT WORK? String#gsub
31. Parser::CurrentRuby.parse('2 + 3') => s(:send, s(:int, 2), :+, s(:int, 3))
32. EXAMPLE #1
33. RSpec.describe 'Max.of'
34. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } end
35. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✘ end module Max def self.of(*numbers) end end
36. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ end module Max def self.of(*numbers) 1 end end
37. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✘ end module Max def self.of(*numbers) 1 end end
38. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ end module Max def self.of(*numbers) numbers.first end end
39. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✘ end module Max def self.of(*numbers) numbers.first end end
40. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
41. LET'S CHECK COVERAGE
43. NOW LET'S RUN MUTANT
44. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
45. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if true current_max = num end end current_max end end Mutation
46. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) numbers.last end end
47. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✘ end module Max def self.of(*numbers) numbers.last end end
48. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
49. BABY STEPS MATTER
50. EXAMPLE #2 Real-world this time
51. Mutant configuration: Matcher: #<Mutant::Matcher::Config match_expressions: [ProductAvailability#calendar_schedule]> Integration: Mutant::Integration::Rspec Jobs: 4 Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 158 Results: 158 Kills: 152 Alive: 6 Runtime: 1003.04s Killtime: 2303.94s Overhead: -56.46% Mutations/s: 0.16 Coverage: 96.20%
52. def calendar_schedule(from:, to:) - if (to < from) - return [] - end + self def calendar_schedule(from:, to:) if (to < from) - return [] + self end
53. - schedule = (from_date..to_date).map do |event_date| + schedule = (from_date...to_date).map do |event_date|
54. Mutant configuration: Matcher: #<Mutant::Matcher::Config match_expressions: [ProductAvailability#products_ids_without_try_on]> Integration: Mutant::Integration::Rspec Jobs: 4 Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 185 Results: 185 Kills: 173 Alive: 12 Runtime: 853.86s Killtime: -69319153.99s Overhead: -100.00% Mutations/s: 0.22 Coverage: 93.51%
55. - if (date.nil? || (date > 2.years.from_now)) + if (self.nil? || (date > 2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || date.eql?(2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 1.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 3.years.from_now))
56. - available_items = items_from_db. pluck(:id, :block_periods, :real_stock) + available_items = items_from_db. pluck(:id, :real_stock) - ids_with_try_on = availability.values.flatten.uniq + ids_with_try_on = availability.values.flatten
57. WHAT IF MUTANT SURVIVES • Add additional tests • Simplify your code • TDD - minimal amount of code to pass the test
58. CHALLENGES • High computation cost - slow • Equivalent mutants - false negatives • Infinite loops
59. TEST SELECTION > mutant --include lib \ --require virtus \ --use rspec \ Virtus::Attribute#type Longest RSpec example group descriptions’ prefix match
60. EQUIVALENT MUTATIONS i = 10 while i != 0 do_something i -= 1 end i = 10 while i > 0 do_something i -= 1 end
61. INFINITE RUNTIME while expression do_something end while true do_something end
62. USE TIMEOUTS config.around(:each) do |example| Timeout.timeout(5, &example) end
63. DISADVANTAGES • Can slow down your TDD rhythm • May be very noisy • Doesn't work with meta-programming
64. ADVANTAGES • Influences code style • Helps spot dead features • Puts emphasis on minimal code to pass the test • Encourages to write isolated unit tests
65. USAGE SCENARIOS • Just run sometimes • Continuous integration • TDD with mutation testing only on new changes • Add mutation testing to your legacy project, but do not fail a build - produce warning report
66. SUMMARY Code coverage highlights code that is not tested. It shows which code you have executed in tests.
67. SUMMARY Mutation testing highlights code that is tested. It shows which code you have asserted in tests.
68. WHO WAS THIS GUY? Tadas Ščerbinskas tadas@chic-by-choice.com @tadassce
69. WHO WAS THIS GUY? Tadas Ščerbinskas tadas@chic-by-choice.com @tadassce
70. JOIN US! Come say hi here or tadas@chic-by-choice.com!
71. OBRIGADO Tadas Ščerbinskas tadas@chic-by-choice.com @tadassce