Mutation testing in continuous delivery pipelines

Mutation testing in continuous delivery pipelines

Mutation testing is a method of evaluating and improving the quality of our test suites. It introduces subtle bugs to source code of our projects and checks that our tests catch those synthetic faults. High computational cost of this technique have prevented it from being widely used in our standard programming toolchain.

Can we try to sneak it in through the backdoor of our continuous delivery pipelines? How much of it can we automate? How can our version control system help us in the process?

Let's try to answer those questions together.

As seen at Continuous Lifecycle 2017 in Mannheim, Germany.

Ae7a42fb716793697b1d222f3cc753b8?s=128

Jan Stępień

November 16, 2017
Tweet

Transcript

  1. Mutation testing with Jan Stępień in continuous delivery pipelines janstepien.com

    @janstepien
  2. @innoQ @cljmuc

  3. None
  4. None
  5. None
  6. We write tests

  7. We write tests

  8. Who is testing our tests? @janstepien is not

  9. Mutation testing

  10. Competent programmer hypothesis

  11. Coupling effect hypothesis

  12. Mutants which
 don’t get killed
 become survivors

  13. None
  14. Off the shelf …and many more Mutant
 for Ruby PIT


    for Java Stryker
 for JavaScript
  15. Mutation testing for Clojure github.com/jstepien/mutant servus! Mutant.

  16. 1. Read all source files in a directory 2. Generate

    mutants from your code 3. Run the test suite for every mutant 4. Report all survivors
  17. Generating mutants

  18. x and y x or y x < 1 x

    <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { a + b + 1 }
  19. x and y x or y x < 1 x

    <= 1 if (a) b else c if (a) c else b f (a, b) { a + b } f (a, b) { null }
  20. Recompiling the code

  21. Running tests

  22. Reporting survivors

  23. Let’s talk about problems

  24. It’s slow Do fewer, do faster,
 or do smarter. —

    Offutt and Untch, 2001
  25. 1. Select what to mutate 2. Select your mutation operators

    3. Sample your mutants Do fewer
  26. Do faster 1. Don’t restart the virtual machine 2. Run

    tests in parallel
  27. 1. Reorder the test suite 2. Execute only relevant tests

    Do smarter
  28. Mutate continuously @janstepien

  29. git diff master~..master

  30. Introduce gradually @janstepien

  31. (…) our 3D engine has a lot of unit tests


    trying to cover as many features as possible over more than 100k lines of code. — Llorens Marti Garcia, IMVU, 2015
  32. We find mutation testing to be highly valuable, and I

    hope this idea can help other engineers deliver solid, well-tested code. — Llorens Marti Garcia, IMVU, 2015
  33. Mutation testing Yours continuously, Jan Stępień in continuous delivery pipelines

    jan.stepien@innoq.com @janstepien