Mutation Testing - Pixels Camp 2019

Mutation Testing - Pixels Camp 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

A2640a0b7fe187c9b9c36af6bf1fe985?s=128

Pedro Rijo

March 22, 2019
Tweet

Transcript

  1. 1 An intro to Mutation Testing - or why coverage

    sucks Pedro Rijo Pixels Camp March 22nd, 2019
  2. Self-introduction 1 Motivation 2 What is mutation testing 4 PIT

    - Java mutation testing framework 5 AGENDA 2 MT at Feedzai 6 Code Coverage 3 Final Remarks 7
  3. Who? 3

  4. 4 @pedrorijo91 https://pedrorijo.com/ Background

  5. 5 2009 - 2014 2014 - 2015 2015 - 2017

    Background Since 2017
  6. 6 • We catch the bad guys on e-commerce fraud

    • State of the art AI and Machine Learning About Feedzai • Real time processing • Used by biggest banks, payment processors, and retailers across the globe
  7. 7 Feedzai Customers

  8. 8 About Feedzai

  9. 9 About Feedzai

  10. 10 About Feedzai

  11. Ensuring high quality 11

  12. • Bugs may cause • (Continuous) automated testing is fundamental

    12 High Quality Code
  13. • All our code has Unit Tests (BE and FE)

    • Many Integration tests (selenium) • Many System Tests (docker) • Failure Model (http://bit.ly/feedzai-failure-model) 13 High Quality Code
  14. • Most run before merging code • Few longer builds

    are run periodically 14 High Quality Code
  15. 15 High Quality Code

  16. 16 • How do I safely refactor my tests? •

    How do I know if I can trust a test suite I inherited? • How do I ensure my team is writing effective tests? • How do I know I have enough tests to refactor my code safely? High Quality Code
  17. 17 Not even TDD can answer these questions High Quality

    Code
  18. Code Coverage 18

  19. 19 Code Coverage is a measurement of the percentage of

    code lines executed during the test suite.
  20. 20 • Line coverage • Statement coverage • Branch coverage

    • Method coverage Code Coverage • Data coverage • Path coverage • Modified condition coverage • etc https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
  21. 21 Code Coverage

  22. 22 100% CC but what if X = 10? Code

    Coverage
  23. 23 100% CC but no asserts Code Coverage

  24. What is Mutation Testing (MT)? 24

  25. 25 Mutation Testing

  26. 26 • MT is a way to check the strength

    of our tests • CC is not reliable as we have seen • 1971 by Richard Lipton • Basic idea: introduce small bugs and check if test suite finds those bugs ◦ Netflix Chaos Monkey anyone? Mutation Testing
  27. 27 public boolean isOldEnough(int age) { return age >= 18;

    } public boolean isOldEnough(int age) { return age > 18; } Mutation Testing
  28. 28 • MT simulates easy bugs - competent programmer hypothesis

    According to this hypothesis, programmers write programs that are almost perfect. The competent programmer hypothesis says that program faults are syntactically small and can be corrected with a few keystrokes. • MT should be included in the development lifecycle (should be run before commit/push/PR) Mutation Testing
  29. Basic MT concepts 29

  30. 30 • Mutation Operators/mutators • Mutants • Killed mutation •

    Survived mutation • Equivalent Mutations Mutation Testing - Basic Concepts
  31. 31 Mutation Testing - Equivalent Mutations

  32. 32 Mutation Testing - Equivalent Mutations

  33. 33 • Mutations in dead/useless code • Mutations that affect

    only performance • Mutations that can’t be triggered due to logic elsewhere in the program • Mutations that alter only internal state Mutation Testing - Equivalent Mutations
  34. 34 Mutation Testing - Examples

  35. 35 Mutation Testing - Examples

  36. 36 Mutation Testing - Examples

  37. 37 Mutation Testing - Examples

  38. PIT - Java Mutation Testing Framework 38

  39. 39 PIT - Java Mutation Testing Framework

  40. 40 • Easy to use • Maven support: http://pitest.org/quickstart/maven/ •

    Very efficient • multi module support: https://github.com/STAMP-project/pitmp-maven-plugin PIT - Java Mutation Testing Framework
  41. 41 PIT - Java Mutation Testing Framework

  42. 42 PIT - Java Mutation Testing Framework $ mvn org.pitest:pitest-maven:mutationCoverage

    $ mvn eu.stamp-project:pitmp-maven-plugin:run
  43. 43 • Report under <project>/target/pit-reports/YYYYMMDDHHmm PIT - Java Mutation Testing

    Framework
  44. 44

  45. 45 • MT score threshold to fail builds • skip

    methods (toString, equals, hashcode) • skip full classes • skip test classes PIT - Java Mutation Testing Framework
  46. Adding MT to Feedzai codebase 46

  47. 47 • PoC using smaller repos ◦ OpenML ▪ https://github.com/feedzai/feedzai-openml/pull/33

    ▪ https://github.com/feedzai/feedzai-openml/pull/34 • internal libs Using MT at Feedzai
  48. 48 Using MT at Feedzai

  49. 49 • Redundant logic in code • Missing test cases

    ◦ boundary values ◦ null checks ◦ etc Using MT at Feedzai
  50. 50 Using MT at Feedzai

  51. 51 Using MT at Feedzai

  52. 52 Using MT at Feedzai

  53. 53 Using MT at Feedzai

  54. 54 Can we add MT everywhere? Using MT at Feedzai

  55. 55 • Common complaints with MT ◦ Equivalent mutations ◦

    MT is slow ▪ compile code and run tests ▪ PIT is very efficient (bytecode manipulation and coverage info) • Doesn’t work with mocks ◦ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit Using MT at Feedzai
  56. 56 • Replace full method logic by “nullable” statement •

    Much less mutants • good preliminary analysis • https://github.com/STAMP-project/pitest-descartes Extreme Mutation
  57. 57 Extreme Mutation

  58. 58 Conclusion • Keeping code base bug-free is hard •

    We need to know our tests strength • Code coverage has flaws • Mutation Testing is a better metric • MT is computational heavy... • ...but seems the right path :)
  59. 59 • https://pedrorijo.com/blog/intro-mutation/ • https://github.com/mbj/mutant for Ruby • http://stryker-mutator.io/ for

    Javascript, Scala, and C# • https://github.com/sugakandrey/scalamu for Scala Resources
  60. 60 @pedrorijo91 https://pedrorijo.com/ Questions?