Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mutation Testing - Quality Talks, May 2019

Mutation Testing - Quality Talks, May 2019

Mutation testing has been evolving into a real candidate to become the de facto metric for assessing the quality of a test suite, defying the throne that has been occupied by code coverage until now.

The concept of mutation testing is quite simple: mutations are introduced on the codebase and the test suite is run against the mutated code. If your test suite is strong, then it should catch the mutation, by having at least one test failing.

In this talk I plan to show how coverage by itself doesn't guarantee quality on the test suite, and how mutation testing can effectively find bugs on your code and tests.

A2640a0b7fe187c9b9c36af6bf1fe985?s=128

Pedro Rijo

May 23, 2019
Tweet

Transcript

  1. 1 An intro to Mutation Testing - or why coverage

    sucks Pedro Rijo Quality Talks May 23rd, 2019
  2. Who? 2

  3. 3 @pedrorijo91 https://pedrorijo.com/ Background

  4. 4 2009 - 2014 2014 - 2015 2015 - 2017

    Background Since 2017
  5. 5 • We catch the bad guys on e-commerce fraud

    • State of the art AI and Machine Learning About Feedzai • Real time processing • Used by biggest banks, payment processors, and retailers across the globe
  6. 6 Feedzai Customers

  7. 7 About Feedzai

  8. Ensuring high quality 8

  9. • Bugs may cause • (Continuous) automated testing is fundamental

    9 High Quality Code
  10. 10 About Feedzai

  11. • All our code has Unit Tests (BE and FE)

    • Many Integration tests (selenium) • Many System Tests (docker) • Failure Model (http://bit.ly/feedzai-failure-model) 11 High Quality Code
  12. 12 High Quality Code

  13. Code Coverage 13

  14. 14 Code Coverage is a measurement of the percentage of

    code lines executed during the test suite.
  15. 15 • Line coverage • Statement coverage • Branch coverage

    • Method coverage Code Coverage • Data coverage • Path coverage • Modified condition coverage • etc https://en.wikipedia.org/wiki/Code_coverage#Coverage_criteria
  16. 16 Code Coverage

  17. 17 100% CC but what if X = 10? Code

    Coverage
  18. 18 100% CC but no asserts Code Coverage

  19. What is Mutation Testing (MT)? 19

  20. 20 • MT is a way to check the strength

    of our tests • CC is not reliable as we have seen • 1971 by Richard Lipton • Basic idea: introduce small bugs and check if test suite finds those bugs ◦ Netflix Chaos Monkey anyone? Mutation Testing
  21. 21 public boolean isOldEnough(int age) { return age >= 18;

    } public boolean isOldEnough(int age) { return age > 18; } Mutation Testing
  22. 22 • MT simulates easy bugs - competent programmer hypothesis

    According to this hypothesis, programmers write programs that are almost perfect. The competent programmer hypothesis says that program faults are syntactically small and can be corrected with a few keystrokes. • MT should be included in the development lifecycle (should be run before commit/push/PR) Mutation Testing
  23. Basic MT concepts 23

  24. 24 • Mutation Operators/mutators • Mutants • Killed mutation •

    Survived mutation • Equivalent Mutations Mutation Testing - Basic Concepts
  25. 25 Mutation Testing - Equivalent Mutations

  26. 26 Mutation Testing - Equivalent Mutations

  27. 27 • Mutations in dead/useless code • Mutations that affect

    only performance • Mutations that can’t be triggered due to logic elsewhere in the program • Mutations that alter only internal state Mutation Testing - Equivalent Mutations
  28. 28 Mutation Testing - Examples

  29. 29 Mutation Testing - Examples

  30. 30 Mutation Testing - Examples

  31. 31 Mutation Testing - Examples

  32. PIT - Java Mutation Testing Framework 32

  33. 33 PIT - Java Mutation Testing Framework

  34. 34 • Easy to use • Maven support: http://pitest.org/quickstart/maven/ •

    Very efficient • multi module support: https://github.com/STAMP-project/pitmp-maven-plugin • Generates mutants and runs test suite automatically PIT - Java Mutation Testing Framework
  35. 35 PIT - Java Mutation Testing Framework

  36. 36 PIT - Java Mutation Testing Framework $ mvn org.pitest:pitest-maven:mutationCoverage

    $ mvn eu.stamp-project:pitmp-maven-plugin:run
  37. 37 • Report under <project>/target/pit-reports/YYYYMMDDHHmm PIT - Java Mutation Testing

    Framework
  38. 38

  39. 39 • MT score threshold to fail builds • skip

    methods (toString, equals, hashcode) • skip full classes • skip test classes PIT - Java Mutation Testing Framework
  40. Adding MT to Feedzai codebase 40

  41. 41 • PoC using smaller repos ◦ OpenML ▪ https://github.com/feedzai/feedzai-openml/pull/33

    ▪ https://github.com/feedzai/feedzai-openml/pull/34 • internal libs Using MT at Feedzai
  42. 42 Using MT at Feedzai

  43. 43 • Redundant logic in code • Missing test cases

    ◦ boundary values ◦ null checks ◦ etc Using MT at Feedzai
  44. 44 Using MT at Feedzai

  45. 45 Using MT at Feedzai

  46. 46 Using MT at Feedzai

  47. 47 Using MT at Feedzai

  48. 48 Can we add MT everywhere? Using MT at Feedzai

  49. 49 • Common complaints with MT ◦ Equivalent mutations ◦

    MT is slow ▪ compile code and run tests ▪ PIT is very efficient (bytecode manipulation and coverage info) ▪ incremental analysis • Doesn’t work with mocks ◦ PIT supports JMock, EasyMock, Mockito, PowerMock and JMockit Using MT at Feedzai
  50. 50 • Replace full method logic by “nullable” statement •

    Much less mutants • good preliminary analysis • https://github.com/STAMP-project/pitest-descartes Extreme Mutation
  51. 51 Extreme Mutation

  52. 52 Conclusion • Keeping code base bug-free is hard •

    We need to know our tests strength • Code coverage has flaws • Mutation Testing is a better metric • MT is computational heavy... • ...but seems the right path :)
  53. 53 • https://pedrorijo.com/blog/intro-mutation/ • https://github.com/mbj/mutant for Ruby • http://stryker-mutator.io/ for

    Javascript, Scala, and C# • https://github.com/sugakandrey/scalamu for Scala Resources
  54. 54 WE ARE HIRING ! http://bit.ly/feedzai-tech-blog https://careers.feedzai.com/

  55. 55 @pedrorijo91 https://pedrorijo.com/ Questions?