Mutation Testing - Pixels Camp 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

Transcript

1. 1
   An intro to Mutation
   Testing - or why
   coverage sucks
   Pedro Rijo
   Pixels Camp
   March 22nd, 2019
   

   View Slide

2. Self-introduction
   1
   Motivation
   2
   What is mutation testing
   4
   PIT - Java mutation testing framework
   5
   AGENDA
   2
   MT at Feedzai
   6
   Code Coverage
   3
   Final Remarks
   7
   

   View Slide

3. Who?
   3
   

   View Slide

4. 4
   @pedrorijo91
   https://pedrorijo.com/
   Background
   

   View Slide

5. 5
   2009 - 2014
   2014 - 2015
   2015 - 2017
   Background
   Since 2017
   

   View Slide

6. 6
   ● We catch the bad guys on
   e-commerce fraud
   ● State of the art AI and Machine
   Learning
   About Feedzai
   ● Real time processing
   ● Used by biggest banks, payment
   processors, and retailers across the
   globe
   

   View Slide

7. 7
   Feedzai Customers
   

   View Slide

8. 8
   About Feedzai
   

   View Slide

9. 9
   About Feedzai
   

   View Slide

10. 10
    About Feedzai
    

    View Slide

11. Ensuring high quality
    11
    

    View Slide

12. ● Bugs may cause
    ● (Continuous) automated testing is fundamental
    12
    High Quality Code
    

    View Slide

13. ● All our code has Unit Tests (BE and FE)
    ● Many Integration tests (selenium)
    ● Many System Tests (docker)
    ● Failure Model (http://bit.ly/feedzai-failure-model)
    13
    High Quality Code
    

    View Slide

14. ● Most run before merging code
    ● Few longer builds are run periodically
    14
    High Quality Code
    

    View Slide

15. 15
    High Quality Code
    

    View Slide

16. 16
    ● How do I safely refactor my tests?
    ● How do I know if I can trust a test suite I inherited?
    ● How do I ensure my team is writing effective tests?
    ● How do I know I have enough tests to refactor my code safely?
    High Quality Code
    

    View Slide

17. 17
    Not even TDD can answer these questions
    High Quality Code
    

    View Slide

18. Code Coverage
    18
    

    View Slide

19. 19
    Code Coverage is a measurement of the percentage of code lines
    executed during the test suite.
    

    View Slide

20. 20
    ● Line coverage
    ● Statement coverage
    ● Branch coverage
    ● Method coverage
    Code Coverage
    ● Data coverage
    ● Path coverage
    ● Modified condition coverage
    ● etc
    https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
    

    View Slide

21. 21
    Code Coverage
    

    View Slide

22. 22
    100% CC but
    what if X = 10?
    Code Coverage
    

    View Slide

23. 23
    100% CC but
    no asserts
    Code Coverage
    

    View Slide

24. What is Mutation Testing (MT)?
    24
    

    View Slide

25. 25
    Mutation Testing
    

    View Slide

26. 26
    ● MT is a way to check the strength of our tests
    ● CC is not reliable as we have seen
    ● 1971 by Richard Lipton
    ● Basic idea: introduce small bugs and check if test suite finds those bugs
    ○ Netflix Chaos Monkey anyone?
    Mutation Testing
    

    View Slide

27. 27
    public boolean isOldEnough(int age) {
    return age >= 18;
    }
    public boolean isOldEnough(int age) {
    return age > 18;
    }
    Mutation Testing
    

    View Slide

28. 28
    ● MT simulates easy bugs - competent programmer hypothesis
    According to this hypothesis, programmers write programs that are almost perfect. The competent
    programmer hypothesis says that program faults are syntactically small and can be corrected with a few
    keystrokes.
    ● MT should be included in the development lifecycle (should be run before
    commit/push/PR)
    Mutation Testing
    

    View Slide

29. Basic MT concepts
    29
    

    View Slide

30. 30
    ● Mutation Operators/mutators
    ● Mutants
    ● Killed mutation
    ● Survived mutation
    ● Equivalent Mutations
    Mutation Testing - Basic Concepts
    

    View Slide

31. 31
    Mutation Testing - Equivalent Mutations
    

    View Slide

32. 32
    Mutation Testing - Equivalent Mutations
    

    View Slide

33. 33
    ● Mutations in dead/useless code
    ● Mutations that affect only performance
    ● Mutations that can’t be triggered due to logic elsewhere in the program
    ● Mutations that alter only internal state
    Mutation Testing - Equivalent Mutations
    

    View Slide

34. 34
    Mutation Testing - Examples
    

    View Slide

35. 35
    Mutation Testing - Examples
    

    View Slide

36. 36
    Mutation Testing - Examples
    

    View Slide

37. 37
    Mutation Testing - Examples
    

    View Slide

38. PIT - Java Mutation Testing Framework
    38
    

    View Slide

39. 39
    PIT - Java Mutation Testing Framework
    

    View Slide

40. 40
    ● Easy to use
    ● Maven support: http://pitest.org/quickstart/maven/
    ● Very efficient
    ● multi module support: https://github.com/STAMP-project/pitmp-maven-plugin
    PIT - Java Mutation Testing Framework
    

    View Slide

41. 41
    PIT - Java Mutation Testing Framework
    

    View Slide

42. 42
    PIT - Java Mutation Testing Framework
    $ mvn org.pitest:pitest-maven:mutationCoverage
    $ mvn eu.stamp-project:pitmp-maven-plugin:run
    

    View Slide

43. 43
    ● Report under /target/pit-reports/YYYYMMDDHHmm
    PIT - Java Mutation Testing Framework
    

    View Slide

44. View Slide

45. 45
    ● MT score threshold to fail builds
    ● skip methods (toString, equals, hashcode)
    ● skip full classes
    ● skip test classes
    PIT - Java Mutation Testing Framework
    

    View Slide

46. Adding MT to Feedzai codebase
    46
    

    View Slide

47. 47
    ● PoC using smaller repos
    ○ OpenML
    ■ https://github.com/feedzai/feedzai-openml/pull/33
    ■ https://github.com/feedzai/feedzai-openml/pull/34
    ● internal libs
    Using MT at Feedzai
    

    View Slide

48. 48
    Using MT at Feedzai
    

    View Slide

49. 49
    ● Redundant logic in code
    ● Missing test cases
    ○ boundary values
    ○ null checks
    ○ etc
    Using MT at Feedzai
    

    View Slide

50. 50
    Using MT at Feedzai
    

    View Slide

51. 51
    Using MT at Feedzai
    

    View Slide

52. 52
    Using MT at Feedzai
    

    View Slide

53. 53
    Using MT at Feedzai
    

    View Slide

54. 54
    Can we add MT
    everywhere?
    Using MT at Feedzai
    

    View Slide

55. 55
    ● Common complaints with MT
    ○ Equivalent mutations
    ○ MT is slow
    ■ compile code and run tests
    ■ PIT is very efficient (bytecode manipulation and coverage info)
    ● Doesn’t work with mocks
    ○ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit
    Using MT at Feedzai
    

    View Slide

56. 56
    ● Replace full method logic by “nullable” statement
    ● Much less mutants
    ● good preliminary analysis
    ● https://github.com/STAMP-project/pitest-descartes
    Extreme Mutation
    

    View Slide

57. 57
    Extreme Mutation
    

    View Slide

58. 58
    Conclusion
    ● Keeping code base bug-free is hard
    ● We need to know our tests strength
    ● Code coverage has flaws
    ● Mutation Testing is a better metric
    ● MT is computational heavy...
    ● ...but seems the right path :)
    

    View Slide

59. 59
    ● https://pedrorijo.com/blog/intro-mutation/
    ● https://github.com/mbj/mutant for Ruby
    ● http://stryker-mutator.io/ for Javascript, Scala, and C#
    ● https://github.com/sugakandrey/scalamu for Scala
    Resources
    

    View Slide

60. 60
    @pedrorijo91 https://pedrorijo.com/
    Questions?
    

    View Slide