Mutation Testing in Elixir

Mutation testing is a technique whereby you test the quality of your tests. Elixir is a language providing great tools for AST manipulation. Mutation testing in Elixir can be a match made in heaven. But is it? In this talk I cover some topics needed to understand how one can go about developing a mutation testing library in Elixir, and how I made my proof-of-concept mutation testing library: exavier. Join me in mutating code and failing tests for fun and (zero) profit!

Transcript

1. mutation testing elixir in Daniel Serrano 2019
2. outline • what is mutation testing? • exavier • elixir AST • exunit • mutators • biggest drawback • rough edges • future
3. mutation testing who watches the watchmen?
4. mutation testing who tests the tests?
5. mutation testing ✨ mutation testing ✨
6. DeMillo, Lipton and Sayward 1978 mutation testing
7. 1. write green tests 2. modify code under test 3. check tests now fail 1. 2. 3. code test code test code test mutation testing
8. • killed mutant test fails after mutation (expected, good) code mutation testing test
9. • surviving mutant test stays green after mutation (unexpected, bad) code test mutation testing
10. • threshold mutation testing cover threshold percentage of killed mutants below which mutation testing fails (e.g., 80%) mutation testing
11. • elixir mutation testing library • goal • install exavier library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
12. how tho?
13. elixir AST
14. elixir AST * + 3 1 2 abstract syntax tree tree representation of the abstract syntactic structure of the source code (1+2) * 3
15. elixir AST * + 3 1 2 (1+2) * 3 {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
16. elixir AST * + 3 1 2 (1+2) * 3 {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
17. elixir AST * + 3 1 2 (1+2) * 3 {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
18. elixir AST * + 3 1 2 (1+2) * 3 {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]}
19. elixir AST * + 3 1 2 {:*, [line: 1], [ {:+, [line: 1], [ 1, 2 ]}, 3 ]} operator metadata arguments
20. elixir AST {atom, keyword, list} operator metadata arguments
21. • ast.ninja • Kernel.quote/2 demo • Code.string_to_quoted/2 • Code.compile_quoted/2
22. elixir AST Arjan Scherpenisse – AST ninja
23. • elixir mutation testing library • goal • install exavier library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
24. exunit
25. exunit • default elixir testing framework • mix task mix test
26. exunit
27. exunit
28. • elixir mutation testing library • goal • install exavier library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
29. exunit • allows definition of custom formatters • provides callbacks for events •:test_started •:test_finished • etc. • e.g., junit_formatter
30. exunit
31. • elixir mutation testing library • goal • install exavier library • run mix exavier.test • mutate code • run tests • show mutation testing report exavier
32. mutators • used by pitest (JVM) and mutant (Ruby) • e.g., following pitest, AOR1 (Arithmetic Operator Replacement Mutator #1) original mutated + - - + * / / * % *
33. mutators
34. mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
35. mutators • e.g., following pitest, REMOVE_CONDITIONALS (Remove Conditionals Mutator) “The remove conditionals mutator will remove all conditionals statements such that the guarded statements always execute” original mutated
36. mutators
37. we’re ready
38. exavier • Run code line coverage analysis for each module, sequentially • Mutate the code according to each available mutator For each module, in parallel: For each mutator, sequentially: 1. Mutate code with given mutator 2. Run tests (now against mutated code) 3. Report (% mutants survived vs. killed)
39. exavier •mix exavier.test
40. biggest drawback • currently mutating a single module all at once and running all tests against mutated module problem?
41. biggest drawback • skews number of survived mutants, e.g.:
42. rough edges • Macros • might be hard to provide useful output • may have to track macro expansion (if possible) • Edge cases, e.g.: • division by zero (Issue #7) • function references as division (Issue #2)
43. future • get back to work on exavier • hopefully with the help of the community ⚠ first order of business: fix “biggest drawback”
44. the end thank you questions? dnlserrano.dev