Slide 1

Slide 1 text

CAN YOU TRUST YOUR TESTS? Tadas Ščerbinskas 2016 sardines.rb Lisbon, Portugal

Slide 2

Slide 2 text

Vaidas Pilkauskas (@liucijus) • Vilnius JUG co-founder • Vilnius Scala leader • CodeRetreat facilitator Thanks to co-author

Slide 3

Slide 3 text

AGENDA • Test quality • Coverage • Mutation testing in theory • Mutation testing in practice

Slide 4

Slide 4 text

TEST QUALITY Readable Focused Concise Well named

Slide 5

Slide 5 text

PROD VS. TEST CODE QUALITY Code has bugs. Tests are code. Tests have bugs.

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

CODE COVERAGE

Slide 8

Slide 8 text

TYPES OF CODE COVERAGE • Lines • Branches • Instructions • Cyclomatic Complexity • Methods • & more

Slide 9

Slide 9 text

LINES expect(name).to eq 'Tadas'

Slide 10

Slide 10 text

LINES expect(name).to eq 'Tadas' def name 'Tadas' end

Slide 11

Slide 11 text

LINES expect(name).to eq 'Tadas' def name 'Tadas' end 100% coverage

Slide 12

Slide 12 text

LINES/BRANCHES expect(name).to eq 'Tadas' def name(initials = false) initials ? 'TS' : 'Tadas' end

Slide 13

Slide 13 text

LINES/BRANCHES expect(name).to eq 'Tadas' def name(initials = false) initials ? 'TS' : 'Tadas' end still 100% line coverage

Slide 14

Slide 14 text

LINES/BRANCHES expect(name).to eq 'Tadas' expect(name(true)).to eq 'TS' def name(initials = false) initials ? 'TS' : 'Tadas' end 100% line & branch coverage

Slide 15

Slide 15 text

COVERAGE TOOL FOR RUBY

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

CAN YOU TRUST 100% COVERAGE? • Code coverage can only show what is not tested • Can be gamed • Still a great tool

Slide 19

Slide 19 text

MUTATION TESTING

Slide 20

Slide 20 text

def ten?(number) number == 10 end WHAT EXACTLY IS A MUTATION?

Slide 21

Slide 21 text

def ten?(number) number == 10 end WHAT EXACTLY IS A MUTATION? def ten?(number) number != 10 end def ten?(number) true end def ten?(number) nil end

Slide 22

Slide 22 text

TERMINOLOGY Applying a mutation to some code creates a mutant. If all tests pass - mutant has survived. If a test fails - mutant is killed.

Slide 23

Slide 23 text

FAILING IS THE NEW PASSING

Slide 24

Slide 24 text

MUTATION TOOL FOR RUBY

Slide 25

Slide 25 text

• Active project • Good reporting • Only works with rspec • minitest support comming soon

Slide 26

Slide 26 text

USAGE > mutant --include lib \ --require virtus \ --use rspec \ Virtus::Attribute#type

Slide 27

Slide 27 text

MUTATIONS • Literal / primitive and compound statement deletion • Argument deletion / rename • Swap Unary operator • Conditionals See all at: https://github.com/mbj/mutant/tree/master/meta

Slide 28

Slide 28 text

HOW DOES IT WORK?

Slide 29

Slide 29 text

HOW DOES IT WORK? String#gsub

Slide 30

Slide 30 text

HOW DOES IT WORK? String#gsub

Slide 31

Slide 31 text

Parser::CurrentRuby.parse('2 + 3') => s(:send, s(:int, 2), :+, s(:int, 3))

Slide 32

Slide 32 text

EXAMPLE #1

Slide 33

Slide 33 text

RSpec.describe 'Max.of'

Slide 34

Slide 34 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } end

Slide 35

Slide 35 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✘ end module Max def self.of(*numbers) end end

Slide 36

Slide 36 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ end module Max def self.of(*numbers) 1 end end

Slide 37

Slide 37 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✘ end module Max def self.of(*numbers) 1 end end

Slide 38

Slide 38 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ end module Max def self.of(*numbers) numbers.first end end

Slide 39

Slide 39 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✘ end module Max def self.of(*numbers) numbers.first end end

Slide 40

Slide 40 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end

Slide 41

Slide 41 text

LET'S CHECK COVERAGE

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

NOW LET'S RUN MUTANT

Slide 44

Slide 44 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end

Slide 45

Slide 45 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if true current_max = num end end current_max end end Mutation

Slide 46

Slide 46 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ end module Max def self.of(*numbers) numbers.last end end

Slide 47

Slide 47 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✘ end module Max def self.of(*numbers) numbers.last end end

Slide 48

Slide 48 text

RSpec.describe 'Max.of' do it { expect(Max.of(1)).to eq 1 } ✔ it { expect(Max.of(2)).to eq 2 } ✔ it { expect(Max.of(4,5)).to eq 5 } ✔ it { expect(Max.of(7,6)).to eq 7 } ✔ end module Max def self.of(*numbers) current_max = numbers.first numbers.each do |num| if num > current_max current_max = num end end current_max end end

Slide 49

Slide 49 text

BABY STEPS MATTER

Slide 50

Slide 50 text

EXAMPLE #2 Real-world this time

Slide 51

Slide 51 text

Mutant configuration: Matcher: # Integration: Mutant::Integration::Rspec Jobs: 4 Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 158 Results: 158 Kills: 152 Alive: 6 Runtime: 1003.04s Killtime: 2303.94s Overhead: -56.46% Mutations/s: 0.16 Coverage: 96.20%

Slide 52

Slide 52 text

def calendar_schedule(from:, to:) - if (to < from) - return [] - end + self def calendar_schedule(from:, to:) if (to < from) - return [] + self end

Slide 53

Slide 53 text

- schedule = (from_date..to_date).map do |event_date| + schedule = (from_date...to_date).map do |event_date|

Slide 54

Slide 54 text

Mutant configuration: Matcher: # Integration: Mutant::Integration::Rspec Jobs: 4 Includes: [] Requires: ["./config/environment"] Subjects: 1 Mutations: 185 Results: 185 Kills: 173 Alive: 12 Runtime: 853.86s Killtime: -69319153.99s Overhead: -100.00% Mutations/s: 0.22 Coverage: 93.51%

Slide 55

Slide 55 text

- if (date.nil? || (date > 2.years.from_now)) + if (self.nil? || (date > 2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || date.eql?(2.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 1.years.from_now)) - if (date.nil? || (date > 2.years.from_now)) + if (date.nil? || (date > 3.years.from_now))

Slide 56

Slide 56 text

- available_items = items_from_db. pluck(:id, :block_periods, :real_stock) + available_items = items_from_db. pluck(:id, :real_stock) - ids_with_try_on = availability.values.flatten.uniq + ids_with_try_on = availability.values.flatten

Slide 57

Slide 57 text

WHAT IF MUTANT SURVIVES • Add additional tests • Simplify your code • TDD - minimal amount of code to pass the test

Slide 58

Slide 58 text

CHALLENGES • High computation cost - slow • Equivalent mutants - false negatives • Infinite loops

Slide 59

Slide 59 text

TEST SELECTION > mutant --include lib \ --require virtus \ --use rspec \ Virtus::Attribute#type Longest RSpec example group descriptions’ prefix match

Slide 60

Slide 60 text

EQUIVALENT MUTATIONS i = 10 while i != 0 do_something i -= 1 end i = 10 while i > 0 do_something i -= 1 end

Slide 61

Slide 61 text

INFINITE RUNTIME while expression do_something end while true do_something end

Slide 62

Slide 62 text

USE TIMEOUTS config.around(:each) do |example| Timeout.timeout(5, &example) end

Slide 63

Slide 63 text

DISADVANTAGES • Can slow down your TDD rhythm • May be very noisy • Doesn't work with meta-programming

Slide 64

Slide 64 text

ADVANTAGES • Influences code style • Helps spot dead features • Puts emphasis on minimal code to pass the test • Encourages to write isolated unit tests

Slide 65

Slide 65 text

USAGE SCENARIOS • Just run sometimes • Continuous integration • TDD with mutation testing only on new changes • Add mutation testing to your legacy project, but do not fail a build - produce warning report

Slide 66

Slide 66 text

SUMMARY Code coverage highlights code that is not tested. It shows which code you have executed in tests.

Slide 67

Slide 67 text

SUMMARY Mutation testing highlights code that is tested. It shows which code you have asserted in tests.

Slide 68

Slide 68 text

WHO WAS THIS GUY? Tadas Ščerbinskas [email protected] @tadassce

Slide 69

Slide 69 text

WHO WAS THIS GUY? Tadas Ščerbinskas [email protected] @tadassce

Slide 70

Slide 70 text

JOIN US! Come say hi here or [email protected]!

Slide 71

Slide 71 text

OBRIGADO Tadas Ščerbinskas [email protected] @tadassce