Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mutation Testing - Pixels Camp 2019

Mutation Testing - Pixels Camp 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

Pedro Rijo

March 22, 2019
Tweet

More Decks by Pedro Rijo

Other Decks in Programming

Transcript

  1. 1
    An intro to Mutation
    Testing - or why
    coverage sucks
    Pedro Rijo
    Pixels Camp
    March 22nd, 2019

    View full-size slide

  2. Self-introduction
    1
    Motivation
    2
    What is mutation testing
    4
    PIT - Java mutation testing framework
    5
    AGENDA
    2
    MT at Feedzai
    6
    Code Coverage
    3
    Final Remarks
    7

    View full-size slide

  3. 4
    @pedrorijo91
    https://pedrorijo.com/
    Background

    View full-size slide

  4. 5
    2009 - 2014
    2014 - 2015
    2015 - 2017
    Background
    Since 2017

    View full-size slide

  5. 6
    ● We catch the bad guys on
    e-commerce fraud
    ● State of the art AI and Machine
    Learning
    About Feedzai
    ● Real time processing
    ● Used by biggest banks, payment
    processors, and retailers across the
    globe

    View full-size slide

  6. 7
    Feedzai Customers

    View full-size slide

  7. 8
    About Feedzai

    View full-size slide

  8. 9
    About Feedzai

    View full-size slide

  9. 10
    About Feedzai

    View full-size slide

  10. Ensuring high quality
    11

    View full-size slide

  11. ● Bugs may cause
    ● (Continuous) automated testing is fundamental
    12
    High Quality Code

    View full-size slide

  12. ● All our code has Unit Tests (BE and FE)
    ● Many Integration tests (selenium)
    ● Many System Tests (docker)
    ● Failure Model (http://bit.ly/feedzai-failure-model)
    13
    High Quality Code

    View full-size slide

  13. ● Most run before merging code
    ● Few longer builds are run periodically
    14
    High Quality Code

    View full-size slide

  14. 15
    High Quality Code

    View full-size slide

  15. 16
    ● How do I safely refactor my tests?
    ● How do I know if I can trust a test suite I inherited?
    ● How do I ensure my team is writing effective tests?
    ● How do I know I have enough tests to refactor my code safely?
    High Quality Code

    View full-size slide

  16. 17
    Not even TDD can answer these questions
    High Quality Code

    View full-size slide

  17. Code Coverage
    18

    View full-size slide

  18. 19
    Code Coverage is a measurement of the percentage of code lines
    executed during the test suite.

    View full-size slide

  19. 20
    ● Line coverage
    ● Statement coverage
    ● Branch coverage
    ● Method coverage
    Code Coverage
    ● Data coverage
    ● Path coverage
    ● Modified condition coverage
    ● etc
    https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria

    View full-size slide

  20. 21
    Code Coverage

    View full-size slide

  21. 22
    100% CC but
    what if X = 10?
    Code Coverage

    View full-size slide

  22. 23
    100% CC but
    no asserts
    Code Coverage

    View full-size slide

  23. What is Mutation Testing (MT)?
    24

    View full-size slide

  24. 25
    Mutation Testing

    View full-size slide

  25. 26
    ● MT is a way to check the strength of our tests
    ● CC is not reliable as we have seen
    ● 1971 by Richard Lipton
    ● Basic idea: introduce small bugs and check if test suite finds those bugs
    ○ Netflix Chaos Monkey anyone?
    Mutation Testing

    View full-size slide

  26. 27
    public boolean isOldEnough(int age) {
    return age >= 18;
    }
    public boolean isOldEnough(int age) {
    return age > 18;
    }
    Mutation Testing

    View full-size slide

  27. 28
    ● MT simulates easy bugs - competent programmer hypothesis
    According to this hypothesis, programmers write programs that are almost perfect. The competent
    programmer hypothesis says that program faults are syntactically small and can be corrected with a few
    keystrokes.
    ● MT should be included in the development lifecycle (should be run before
    commit/push/PR)
    Mutation Testing

    View full-size slide

  28. Basic MT concepts
    29

    View full-size slide

  29. 30
    ● Mutation Operators/mutators
    ● Mutants
    ● Killed mutation
    ● Survived mutation
    ● Equivalent Mutations
    Mutation Testing - Basic Concepts

    View full-size slide

  30. 31
    Mutation Testing - Equivalent Mutations

    View full-size slide

  31. 32
    Mutation Testing - Equivalent Mutations

    View full-size slide

  32. 33
    ● Mutations in dead/useless code
    ● Mutations that affect only performance
    ● Mutations that can’t be triggered due to logic elsewhere in the program
    ● Mutations that alter only internal state
    Mutation Testing - Equivalent Mutations

    View full-size slide

  33. 34
    Mutation Testing - Examples

    View full-size slide

  34. 35
    Mutation Testing - Examples

    View full-size slide

  35. 36
    Mutation Testing - Examples

    View full-size slide

  36. 37
    Mutation Testing - Examples

    View full-size slide

  37. PIT - Java Mutation Testing Framework
    38

    View full-size slide

  38. 39
    PIT - Java Mutation Testing Framework

    View full-size slide

  39. 40
    ● Easy to use
    ● Maven support: http://pitest.org/quickstart/maven/
    ● Very efficient
    ● multi module support: https://github.com/STAMP-project/pitmp-maven-plugin
    PIT - Java Mutation Testing Framework

    View full-size slide

  40. 41
    PIT - Java Mutation Testing Framework

    View full-size slide

  41. 42
    PIT - Java Mutation Testing Framework
    $ mvn org.pitest:pitest-maven:mutationCoverage
    $ mvn eu.stamp-project:pitmp-maven-plugin:run

    View full-size slide

  42. 43
    ● Report under /target/pit-reports/YYYYMMDDHHmm
    PIT - Java Mutation Testing Framework

    View full-size slide

  43. 45
    ● MT score threshold to fail builds
    ● skip methods (toString, equals, hashcode)
    ● skip full classes
    ● skip test classes
    PIT - Java Mutation Testing Framework

    View full-size slide

  44. Adding MT to Feedzai codebase
    46

    View full-size slide

  45. 47
    ● PoC using smaller repos
    ○ OpenML
    ■ https://github.com/feedzai/feedzai-openml/pull/33
    ■ https://github.com/feedzai/feedzai-openml/pull/34
    ● internal libs
    Using MT at Feedzai

    View full-size slide

  46. 48
    Using MT at Feedzai

    View full-size slide

  47. 49
    ● Redundant logic in code
    ● Missing test cases
    ○ boundary values
    ○ null checks
    ○ etc
    Using MT at Feedzai

    View full-size slide

  48. 50
    Using MT at Feedzai

    View full-size slide

  49. 51
    Using MT at Feedzai

    View full-size slide

  50. 52
    Using MT at Feedzai

    View full-size slide

  51. 53
    Using MT at Feedzai

    View full-size slide

  52. 54
    Can we add MT
    everywhere?
    Using MT at Feedzai

    View full-size slide

  53. 55
    ● Common complaints with MT
    ○ Equivalent mutations
    ○ MT is slow
    ■ compile code and run tests
    ■ PIT is very efficient (bytecode manipulation and coverage info)
    ● Doesn’t work with mocks
    ○ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit
    Using MT at Feedzai

    View full-size slide

  54. 56
    ● Replace full method logic by “nullable” statement
    ● Much less mutants
    ● good preliminary analysis
    ● https://github.com/STAMP-project/pitest-descartes
    Extreme Mutation

    View full-size slide

  55. 57
    Extreme Mutation

    View full-size slide

  56. 58
    Conclusion
    ● Keeping code base bug-free is hard
    ● We need to know our tests strength
    ● Code coverage has flaws
    ● Mutation Testing is a better metric
    ● MT is computational heavy...
    ● ...but seems the right path :)

    View full-size slide

  57. 59
    ● https://pedrorijo.com/blog/intro-mutation/
    ● https://github.com/mbj/mutant for Ruby
    ● http://stryker-mutator.io/ for Javascript, Scala, and C#
    ● https://github.com/sugakandrey/scalamu for Scala
    Resources

    View full-size slide

  58. 60
    @pedrorijo91 https://pedrorijo.com/
    Questions?

    View full-size slide