Mutation Testing in Elixir

Mutation Testing in Elixir

Mutation testing is a technique whereby you test the quality of your tests. Elixir is a language providing great tools for AST manipulation. Mutation testing in Elixir can be a match made in heaven. But is it? In this talk I cover some topics needed to understand how one can go about developing a mutation testing library in Elixir, and how I made my proof-of-concept mutation testing library: exavier. Join me in mutating code and failing tests for fun and (zero) profit!

5cbc409f0a53f2ece26de41651799ae0?s=128

Daniel Serrano

November 27, 2019
Tweet

Transcript

  1. 2.

    outline • what is mutation testing? • exavier • elixir

    AST • exunit • mutators • biggest drawback • rough edges • future
  2. 7.

    1. write green tests 2. modify code under test 3.

    check tests now fail 1. 2. 3. code test code test code test mutation testing
  3. 10.

    • threshold mutation testing cover threshold percentage of killed mutants

    below which mutation testing fails (e.g., 80%) mutation testing
  4. 11.

    • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  5. 12.
  6. 14.

    elixir AST * + 3 1 2 abstract syntax tree

    tree representation of the abstract syntactic structure of the source code (1+2) * 3
  7. 15.

    elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  8. 16.

    elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  9. 17.

    elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  10. 18.

    elixir AST * + 3 1 2 (1+2) * 3

    {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
  11. 19.

    elixir AST * + 3 1 2 {:*, [line: 1],

    [ {:+, [line: 1], [ 1, 2 ]}, 3 ]} operator metadata arguments
  12. 23.

    • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  13. 24.
  14. 26.
  15. 27.
  16. 28.

    • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  17. 29.

    exunit • allows definition of custom formatters • provides callbacks

    for events •:test_started •:test_finished • etc. • e.g., junit_formatter
  18. 30.
  19. 31.

    • elixir mutation testing library • goal • install exavier

    library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
  20. 32.

    mutators • used by pitest (JVM) and mutant (Ruby) •

    e.g., following pitest, AOR1 (Arithmetic Operator Replacement Mutator #1) original mutated + - - + * / / * % *
  21. 33.
  22. 34.

    mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The

    remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
  23. 35.

    mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The

    remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
  24. 36.
  25. 38.

    exavier • Run code line coverage analysis for each module,

    sequentially • Mutate the code according to each available mutator For each module, in parallel: For each mutator, sequentially: 1. Mutate code with given mutator 2. Run tests (now against mutated code) 3. Report (% mutants survived vs. killed)
  26. 40.

    biggest drawback • currently mutating a single module all at

    once and running all tests against mutated module problem?
  27. 42.

    rough edges • Macros • might be hard to provide

    useful output • may have to track macro expansion (if possible) • Edge cases, e.g.: • division by zero (Issue #7) • function references as division (Issue #2)
  28. 43.

    future • get back to work on exavier • hopefully

    with the help of the community ⚠ first order of business: fix “biggest drawback”