Mutation Testing - Pixels Camp 2019

Mutation Testing - Pixels Camp 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

A2640a0b7fe187c9b9c36af6bf1fe985?s=128

Pedro Rijo

March 22, 2019
Tweet

Transcript

  1. 1.

    1 An intro to Mutation Testing - or why coverage

    sucks Pedro Rijo Pixels Camp March 22nd, 2019
  2. 2.

    Self-introduction 1 Motivation 2 What is mutation testing 4 PIT

    - Java mutation testing framework 5 AGENDA 2 MT at Feedzai 6 Code Coverage 3 Final Remarks 7
  3. 3.
  4. 5.
  5. 6.

    6 • We catch the bad guys on e-commerce fraud

    • State of the art AI and Machine Learning About Feedzai • Real time processing • Used by biggest banks, payment processors, and retailers across the globe
  6. 13.

    • All our code has Unit Tests (BE and FE)

    • Many Integration tests (selenium) • Many System Tests (docker) • Failure Model (http://bit.ly/feedzai-failure-model) 13 High Quality Code
  7. 14.

    • Most run before merging code • Few longer builds

    are run periodically 14 High Quality Code
  8. 16.

    16 • How do I safely refactor my tests? •

    How do I know if I can trust a test suite I inherited? • How do I ensure my team is writing effective tests? • How do I know I have enough tests to refactor my code safely? High Quality Code
  9. 19.

    19 Code Coverage is a measurement of the percentage of

    code lines executed during the test suite.
  10. 20.

    20 • Line coverage • Statement coverage • Branch coverage

    • Method coverage Code Coverage • Data coverage • Path coverage • Modified condition coverage • etc https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
  11. 26.

    26 • MT is a way to check the strength

    of our tests • CC is not reliable as we have seen • 1971 by Richard Lipton • Basic idea: introduce small bugs and check if test suite finds those bugs ◦ Netflix Chaos Monkey anyone? Mutation Testing
  12. 27.

    27 public boolean isOldEnough(int age) { return age >= 18;

    } public boolean isOldEnough(int age) { return age > 18; } Mutation Testing
  13. 28.

    28 • MT simulates easy bugs - competent programmer hypothesis

    According to this hypothesis, programmers write programs that are almost perfect. The competent programmer hypothesis says that program faults are syntactically small and can be corrected with a few keystrokes. • MT should be included in the development lifecycle (should be run before commit/push/PR) Mutation Testing
  14. 30.

    30 • Mutation Operators/mutators • Mutants • Killed mutation •

    Survived mutation • Equivalent Mutations Mutation Testing - Basic Concepts
  15. 33.

    33 • Mutations in dead/useless code • Mutations that affect

    only performance • Mutations that can’t be triggered due to logic elsewhere in the program • Mutations that alter only internal state Mutation Testing - Equivalent Mutations
  16. 40.

    40 • Easy to use • Maven support: http://pitest.org/quickstart/maven/ •

    Very efficient • multi module support: https://github.com/STAMP-project/pitmp-maven-plugin PIT - Java Mutation Testing Framework
  17. 44.

    44

  18. 45.

    45 • MT score threshold to fail builds • skip

    methods (toString, equals, hashcode) • skip full classes • skip test classes PIT - Java Mutation Testing Framework
  19. 47.

    47 • PoC using smaller repos ◦ OpenML ▪ https://github.com/feedzai/feedzai-openml/pull/33

    ▪ https://github.com/feedzai/feedzai-openml/pull/34 • internal libs Using MT at Feedzai
  20. 49.

    49 • Redundant logic in code • Missing test cases

    ◦ boundary values ◦ null checks ◦ etc Using MT at Feedzai
  21. 55.

    55 • Common complaints with MT ◦ Equivalent mutations ◦

    MT is slow ▪ compile code and run tests ▪ PIT is very efficient (bytecode manipulation and coverage info) • Doesn’t work with mocks ◦ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit Using MT at Feedzai
  22. 56.

    56 • Replace full method logic by “nullable” statement •

    Much less mutants • good preliminary analysis • https://github.com/STAMP-project/pitest-descartes Extreme Mutation
  23. 58.

    58 Conclusion • Keeping code base bug-free is hard •

    We need to know our tests strength • Code coverage has flaws • Mutation Testing is a better metric • MT is computational heavy... • ...but seems the right path :)
  24. 59.

    59 • https://pedrorijo.com/blog/intro-mutation/ • https://github.com/mbj/mutant for Ruby • http://stryker-mutator.io/ for

    Javascript, Scala, and C# • https://github.com/sugakandrey/scalamu for Scala Resources