Mutation Testing in Elixir

Mutation testing is a technique whereby you test the quality of your tests. Elixir is a language providing great tools for AST manipulation. Mutation testing in Elixir can be a match made in heaven. But is it? In this talk I cover some topics needed to understand how one can go about developing a mutation testing library in Elixir, and how I made my proof-of-concept mutation testing library: exavier. Join me in mutating code and failing tests for fun and (zero) profit!

Transcript

1. mutation
   testing
   elixir
   in
   Daniel Serrano
   2019
   

   View Slide

2. outline
   • what is mutation testing?
   • exavier
   • elixir AST
   • exunit
   • mutators
   • biggest drawback
   • rough edges
   • future
   

   View Slide

3. mutation testing
   who watches the watchmen?
   

   View Slide

4. mutation testing
   who tests the tests?
   

   View Slide

5. mutation testing
   ✨ mutation testing ✨
   

   View Slide

6. DeMillo, Lipton and Sayward
   1978
   mutation testing
   

   View Slide

7. 1. write green tests
   2. modify code under test
   3. check tests now fail
   1. 2. 3.
   code test code test code test
   mutation testing
   

   View Slide

8. • killed mutant
   test fails after mutation
   (expected, good)
   code
   mutation testing
   test
   

   View Slide

9. • surviving mutant
   test stays green after mutation (unexpected,
   bad)
   code test
   mutation testing
   

   View Slide

10. • threshold
    mutation testing cover threshold
    percentage of killed mutants below which mutation
    testing fails (e.g., 80%)
    mutation testing
    

    View Slide

11. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier
    

    View Slide

12. how tho?
    

    View Slide

13. elixir
    AST
    

    View Slide

14. elixir AST
    *
    + 3
    1 2
    abstract syntax tree
    tree representation of the
    abstract syntactic structure
    of the source code
    (1+2) * 3
    

    View Slide

15. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    

    View Slide

16. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    

    View Slide

17. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    

    View Slide

18. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    

    View Slide

19. elixir AST
    *
    + 3
    1 2
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    operator metadata
    arguments
    

    View Slide

20. elixir AST
    {atom, keyword, list}
    operator metadata arguments
    

    View Slide

21. • ast.ninja
    • Kernel.quote/2
    demo
    • Code.string_to_quoted/2
    • Code.compile_quoted/2
    

    View Slide

22. elixir AST
    Arjan Scherpenisse – AST ninja
    

    View Slide

23. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier
    

    View Slide

24. exunit
    

    View Slide

25. exunit
    • default elixir testing framework
    • mix task mix test
    

    View Slide

26. exunit
    

    View Slide

27. exunit
    

    View Slide

28. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier
    

    View Slide

29. exunit
    • allows definition of custom formatters
    • provides callbacks for events
    •:test_started
    •:test_finished
    • etc.
    • e.g., junit_formatter
    

    View Slide

30. exunit
    

    View Slide

31. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier
    

    View Slide

32. mutators
    • used by pitest (JVM) and mutant (Ruby)
    • e.g., following pitest, AOR1
    (Arithmetic Operator Replacement Mutator #1)
    original mutated
    + -
    - +
    * /
    / *
    % *
    

    View Slide

33. mutators
    

    View Slide

34. mutators
    • e.g., following pitest, REMOVE_CONDITIONALS
    (Remove Conditionals Mutator)
    “The remove conditionals mutator will remove all
    conditionals statements such that the guarded statements
    always execute”
    original mutated
    

    View Slide

35. mutators
    • e.g., following pitest, REMOVE_CONDITIONALS
    (Remove Conditionals Mutator)
    “The remove conditionals mutator will remove all
    conditionals statements such that the guarded statements
    always execute”
    original mutated
    

    View Slide

36. mutators
    

    View Slide

37. we’re ready
    

    View Slide

38. exavier
    • Run code line coverage analysis for each module,
    sequentially
    • Mutate the code according to each available mutator
    For each module, in parallel:
    For each mutator, sequentially:
    1. Mutate code with given mutator
    2. Run tests (now against mutated code)
    3. Report (% mutants survived vs. killed)
    

    View Slide

39. exavier
    •mix exavier.test
    

    View Slide

40. biggest drawback
    • currently mutating a single module all at once and
    running all tests against mutated module
    problem?
    

    View Slide

41. biggest drawback
    • skews number of survived mutants, e.g.:
    

    View Slide

42. rough edges
    • Macros
    • might be hard to provide useful output
    • may have to track macro expansion (if possible)
    • Edge cases, e.g.:
    • division by zero (Issue #7)
    • function references as division (Issue #2)
    

    View Slide

43. future
    • get back to work on exavier
    • hopefully with the help of the community
    ⚠
    first order of business: fix “biggest drawback”
    

    View Slide

44. the end
    thank you
    questions?
    dnlserrano.dev
    

    View Slide