Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Can you trust your tests?

Tadas Sce
September 27, 2016

Can you trust your tests?

A talk I gave at Sardines.rb (Ruby meetup) in Lisbon, Portugal.
It's about test code quality, coverage and mutation testing.

Tadas Sce

September 27, 2016
Tweet

More Decks by Tadas Sce

Other Decks in Technology

Transcript

  1. Vaidas Pilkauskas (@liucijus) • Vilnius JUG co-founder • Vilnius Scala

    leader • CodeRetreat facilitator Thanks to co-author
  2. AGENDA • Test quality • Coverage • Mutation testing in

    theory • Mutation testing in practice
  3. TYPES OF CODE COVERAGE • Lines • Branches • Instructions

    • Cyclomatic Complexity • Methods • & more
  4. LINES/BRANCHES expect(name).to eq 'Tadas' expect(name(true)).to eq 'TS' def name(initials =

    false) initials ? 'TS' : 'Tadas' end 100% line & branch coverage
  5. CAN YOU TRUST 100% COVERAGE? • Code coverage can only

    show what is not tested • Can be gamed • Still a great tool
  6. def ten?(number) number == 10 end WHAT EXACTLY IS A

    MUTATION? def ten?(number) number != 10 end def ten?(number) true end def ten?(number) nil end
  7. TERMINOLOGY Applying a mutation to some code creates a mutant.

    If all tests pass - mutant has survived. If a test fails - mutant is killed.
  8. • Active project • Good reporting • Only works with

    rspec • minitest support comming soon
  9. MUTATIONS • Literal / primitive and compound statement deletion •

    Argument deletion / rename • Swap Unary operator • Conditionals See all at: https://github.com/mbj/mutant/tree/master/meta
  10. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✘

    end module Max def self.of(*numbers) end end
  11. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    end module Max def self.of(*numbers) 1 end end
  12. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✘ end module Max def self.of(*numbers) 1 end end
  13. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ end module Max def self.of(*numbers) numbers.first end end
  14. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✘ end module Max def self.of(*numbers) numbers.first end end
  15. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
  16. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
  17. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if true current_max = num end end current_max end end Mutation
  18. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) numbers.last end end
  19. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✘ end module Max def self.of(*numbers) numbers.last end end
  20. RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔

    it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end
  21. Mutant configuration: Matcher: #<Mutant::Matcher::Config match_expressions: [ProductAvailability#calendar_schedule]> Integration: Mutant::Integration::Rspec Jobs: 4

    Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 158 Results: 158 Kills: 152 Alive: 6 Runtime: 1003.04s Killtime: 2303.94s Overhead: -56.46% Mutations/s: 0.16 Coverage: 96.20%
  22. def calendar_schedule(from:, to:) - if (to < from) - return

    [] - end + self def calendar_schedule(from:, to:) if (to < from) - return [] + self end
  23. Mutant configuration: Matcher: #<Mutant::Matcher::Config match_expressions: [ProductAvailability#products_ids_without_try_on]> Integration: Mutant::Integration::Rspec Jobs: 4

    Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 185 Results: 185 Kills: 173 Alive: 12 Runtime: 853.86s Killtime: -69319153.99s Overhead: -100.00% Mutations/s: 0.22 Coverage: 93.51%
  24. - if (date.nil? || (date > 2.years.from_now)) + if (self.nil?

    || (date > 2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || date.eql?(2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 1.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 3.years.from_now))
  25. - available_items = items_from_db. pluck(:id, :block_periods, :real_stock) + available_items =

    items_from_db. pluck(:id, :real_stock) - ids_with_try_on = availability.values.flatten.uniq + ids_with_try_on = availability.values.flatten
  26. WHAT IF MUTANT SURVIVES • Add additional tests • Simplify

    your code • TDD - minimal amount of code to pass the test
  27. TEST SELECTION > mutant --include lib \ --require virtus \

    --use rspec \ Virtus::Attribute#type Longest RSpec example group descriptions’ prefix match
  28. EQUIVALENT MUTATIONS i = 10 while i != 0 do_something

    i -= 1 end i = 10 while i > 0 do_something i -= 1 end
  29. DISADVANTAGES • Can slow down your TDD rhythm • May

    be very noisy • Doesn't work with meta-programming
  30. ADVANTAGES • Influences code style • Helps spot dead features

    • Puts emphasis on minimal code to pass the test • Encourages to write isolated unit tests
  31. USAGE SCENARIOS • Just run sometimes • Continuous integration •

    TDD with mutation testing only on new changes • Add mutation testing to your legacy project, but do not fail a build - produce warning report
  32. SUMMARY Code coverage highlights code that is not tested. It

    shows which code you have executed in tests.