Effective Testing or how to sleep when you are on call

Light talk on how to power up your testing with mutations

Transcript

1. EFFECTIVE TESTING
   or how to sleep when you are on call
   

   View Slide

2. FAST
   WHAT MAKES A TEST A GOOD TEST?
   
   

   View Slide

3. WHAT MAKES A TEST A GOOD TEST?
   “Sensitivity means that, of all
   mistakes that actually happen
   when writing your system, a high
   percentage are caught by the test
   suite, and a high percentage of
   those are caught early in the run.”
   RELIABILITY
   

   View Slide

4. View Slide

5. class Group
   def accessible?(user)
   return true if public?
   owner_or_admin(user) || is_member(user)
   end
   spec
   describe “#accessible?" do
   let(:user) { create(:user) }
   context "when the group is private" do
   let(:owner) { create(:user) }
   let(:group) { create(:group, :private, user: owner.id) }
   context "when the user is a member" do
   before do
   group.group_members << create(:group_member,
   user: user)
   end
   it { expect(group.accessible?(user)).to be true }
   end
   end
   end
   

   View Slide

6. Code coverage is checking that your code has been called
   somewhere, but what does it really says about the quality of
   your tests?
   

   View Slide

7. View Slide

8. ➤ The goal of mutation tests is to make your tests fail.
   ➤ Mutations are going to slightly change your code based on a
   set of defined operators. For each discrete change, the tests
   for that code are going to be run.
   ➤ Value
   ➤ Decision
   ➤ Statement
   ➤ If your tests kill all the mutants, then your code is notably
   more safe than just with line coverage
   MUTATION TESTING
   

   View Slide

9. ➤ https://github.com/mbj/mutant*
   ➤ Supports RSpec and minitest
   * there might be some cases when the mutants are all killed but still your code is not 100% covered
   TOOLS
   

   View Slide

10. MUTANTS
    group.rb
    def accessible(user)
    return true if public?
    owner_or_admin(user) || is_member(user)
    end
    

    View Slide

11. MUTANT OPERATORS
    Source https://github.com/mbj/mutant/blob/master/meta/if.rb
    

    View Slide

12. MUTANT OPERATORS
    Source https://github.com/mbj/mutant/blob/master/lib/mutant/mutator/node/
    if.rb
    

    View Slide

13. MUTANT OPERATORS
    Source https://github.com/mbj/mutant/blob/master/lib/mutant/mutator/
    node.rb#L102
    

    View Slide

14. MUTANT OPERATORS
    Source https://github.com/mbj/mutant/blob/master/lib/mutant/mutator/node/
    if.rb
    

    View Slide

15. WHAT DOES MUTANT TELL US ABOUT OUR 100% CODE COVERAGE?
    

    View Slide

16. describe “#accessible?" do
    let(:user) { create(:user) }
    context "when the group is private" do
    let(:owner) { create(:user) }
    let(:group) { create(:group, :private, user: owner.id) }
    context "when the user is an admin of the website or the owner of the group" do
    it “returns true" do
    admin = create(:user, :admin)
    expect(group.accessible?(admin)).to be true
    expect(group.accessible?(owner)).to be true
    end
    end
    context "when the user is a member" do
    it “returns true" do
    user = create(:user)
    group.group_members << create(:group_member, user: user)
    expect(group.accessible?(user)).to be true
    end
    end
    context "when the user is not a member" do
    it { expect(group.accessible?(create(:user)).to be false }
    end
    end
    context "when the group is public" do
    it “returns true" do
    expect(group.accessible?(create(:user)).to be true
    end
    end
    end
    

    View Slide

17. group.rb
    def accessible(user)
    return true if public?
    owner_or_admin(user) || is_member(user)
    end
    MUTANTS
    

    View Slide

18. DOWNSIDES
    ➤ cRuby/MRI - Ruby 2.5.x-2.6
    ➤ As you can imagine, mutation tests can be extremely time
    consuming, so it's almost impossible to run them on the
    whole codebase with the same frequency you HAVE TO run
    your usual tests
    ➤ If you deviate from the MVC pattern, it’s a tad more difficult
    to make mutant run all the appropriate tests for the class you
    want to test
    

    View Slide

19. SOLUTIONS
    ➤ mutant allows you to narrow the mutations down to a single
    method -> run mutant after you finished writing the tests for
    your method to ensure that your code is fully covered
    ➤ run mutant continuously just to cover the most sensitive part
    of your application
    ➤ the —since REVISION option will make mutant mutate only
    the code that has been modified since the specified point, thus
    cutting down execution time and scope
    ➤ add a top level namespace that eventually allows mutant to
    match example groups with: Foo::Bar#baz, Foo::Bar,
    Foo
    

    View Slide

20. ➤ http://www.sitepoint.com/mutation-testing-mutant/
    ➤ http://solnic.eu/2013/01/23/mutation-testing-with-
    mutant.html
    ➤ http://en.wikipedia.org/wiki/Mutation_testing
    ➤ https://www.youtube.com/watch?v=WccaOMuf01Y
    ➤ (RailsConf 2014 keynote - live coding around 25 minutes in)
    ➤ https://www.youtube.com/watch?v=cLh3yMKEqHc (5
    minutes introduction to mutation testing)
    

    View Slide

21. Special thanks to @mbj for having reviewed this presentation
    

    View Slide

22. @KFMolli on twitter
    nirnaeth on github and Speaker Deck
    monica.giambitto@protonmail.ch
    THANKS!
    @KFMOLLI
    GITHUB.COM/NIRNAETH
    SPEAKERDECK.COM/NIRNAETH
    MONICA.GIAMBITTO@PM.ME
    

    View Slide