Mutation Testing in Elixir

Mutation Testing in Elixir

Mutation testing is a technique whereby you test the quality of your tests. Elixir is a language providing great tools for AST manipulation. Mutation testing in Elixir can be a match made in heaven. But is it? In this talk I cover some topics needed to understand how one can go about developing a mutation testing library in Elixir, and how I made my proof-of-concept mutation testing library: exavier. Join me in mutating code and failing tests for fun and (zero) profit!

5cbc409f0a53f2ece26de41651799ae0?s=128

Daniel Serrano

November 27, 2019
Tweet

Transcript

  1. mutation testing elixir in Daniel Serrano 2019

  2. outline • what is mutation testing? • exavier • elixir

    AST • exunit • mutators • biggest drawback • rough edges • future
  3. mutation testing who watches the watchmen?

  4. mutation testing who tests the tests?

  5. mutation testing ✨ mutation testing ✨

  6. DeMillo, Lipton and Sayward 1978 mutation testing

  7. 1. write green tests 2. modify code under test 3.

    check tests now fail 1. 2. 3. code test code test code test mutation testing
  8. • killed mutant test fails after mutation (expected, good) code

    mutation testing test
  9. • surviving mutant test stays green after mutation (unexpected, bad)

    code test mutation testing
  10. • threshold mutation testing cover threshold percentage of killed mutants

    below which mutation testing fails (e.g., 80%) mutation testing
  11. • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  12. how tho?

  13. elixir AST

  14. elixir AST * + 3 1 2 abstract syntax tree

    tree representation of the abstract syntactic structure of the source code (1+2) * 3
  15. elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  16. elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  17. elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  18. elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  19. elixir AST * + 3 1 2 {:*, [line: 1],

    [ {:+, [line: 1], [ 1, 2 ]}, 3 ]} operator metadata arguments
  20. elixir AST {atom, keyword, list} operator metadata arguments

  21. • ast.ninja • Kernel.quote/2 demo • Code.string_to_quoted/2 • Code.compile_quoted/2

  22. elixir AST Arjan Scherpenisse – AST ninja

  23. • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  24. exunit

  25. exunit • default elixir testing framework • mix task mix

    test
  26. exunit

  27. exunit

  28. • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  29. exunit • allows definition of custom formatters • provides callbacks

    for events •:test_started •:test_finished • etc. • e.g., junit_formatter
  30. exunit

  31. • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  32. mutators • used by pitest (JVM) and mutant (Ruby) •

    e.g., following pitest, AOR1 (Arithmetic Operator Replacement Mutator #1) original mutated + - - + * / / * % *
  33. mutators

  34. mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The

    remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
  35. mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The

    remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
  36. mutators

  37. we’re ready

  38. exavier • Run code line coverage analysis for each module,

    sequentially • Mutate the code according to each available mutator For each module, in parallel: For each mutator, sequentially: 1. Mutate code with given mutator 2. Run tests (now against mutated code) 3. Report (% mutants survived vs. killed)
  39. exavier •mix exavier.test

  40. biggest drawback • currently mutating a single module all at

    once and running all tests against mutated module problem?
  41. biggest drawback • skews number of survived mutants, e.g.:

  42. rough edges • Macros • might be hard to provide

    useful output • may have to track macro expansion (if possible) • Edge cases, e.g.: • division by zero (Issue #7) • function references as division (Issue #2)
  43. future • get back to work on exavier • hopefully

    with the help of the community ⚠ first order of business: fix “biggest drawback”
  44. the end thank you questions? dnlserrano.dev