Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mutation Testing in Elixir

Mutation Testing in Elixir

Mutation testing is a technique whereby you test the quality of your tests. Elixir is a language providing great tools for AST manipulation. Mutation testing in Elixir can be a match made in heaven. But is it? In this talk I cover some topics needed to understand how one can go about developing a mutation testing library in Elixir, and how I made my proof-of-concept mutation testing library: exavier. Join me in mutating code and failing tests for fun and (zero) profit!

Daniel Serrano

November 27, 2019
Tweet

More Decks by Daniel Serrano

Other Decks in Programming

Transcript

  1. mutation
    testing
    elixir
    in
    Daniel Serrano
    2019

    View Slide

  2. outline
    • what is mutation testing?
    • exavier
    • elixir AST
    • exunit
    • mutators
    • biggest drawback
    • rough edges
    • future

    View Slide

  3. mutation testing
    who watches the watchmen?

    View Slide

  4. mutation testing
    who tests the tests?

    View Slide

  5. mutation testing
    ✨ mutation testing ✨

    View Slide

  6. DeMillo, Lipton and Sayward
    1978
    mutation testing

    View Slide

  7. 1. write green tests
    2. modify code under test
    3. check tests now fail
    1. 2. 3.
    code test code test code test
    mutation testing

    View Slide

  8. • killed mutant
    test fails after mutation
    (expected, good)
    code
    mutation testing
    test

    View Slide

  9. • surviving mutant
    test stays green after mutation (unexpected,
    bad)
    code test
    mutation testing

    View Slide

  10. • threshold
    mutation testing cover threshold
    percentage of killed mutants below which mutation
    testing fails (e.g., 80%)
    mutation testing

    View Slide

  11. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier

    View Slide

  12. how tho?

    View Slide

  13. elixir
    AST

    View Slide

  14. elixir AST
    *
    + 3
    1 2
    abstract syntax tree
    tree representation of the
    abstract syntactic structure
    of the source code
    (1+2) * 3

    View Slide

  15. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}

    View Slide

  16. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}

    View Slide

  17. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}

    View Slide

  18. elixir AST
    *
    + 3
    1 2
    (1+2) * 3
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}

    View Slide

  19. elixir AST
    *
    + 3
    1 2
    {:*, [line: 1], [
    {:+, [line: 1], [
    1,
    2
    ]},
    3
    ]}
    operator metadata
    arguments

    View Slide

  20. elixir AST
    {atom, keyword, list}
    operator metadata arguments

    View Slide

  21. • ast.ninja
    • Kernel.quote/2
    demo
    • Code.string_to_quoted/2
    • Code.compile_quoted/2

    View Slide

  22. elixir AST
    Arjan Scherpenisse – AST ninja

    View Slide

  23. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier

    View Slide

  24. exunit

    View Slide

  25. exunit
    • default elixir testing framework
    • mix task mix test

    View Slide

  26. exunit

    View Slide

  27. exunit

    View Slide

  28. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier

    View Slide

  29. exunit
    • allows definition of custom formatters
    • provides callbacks for events
    •:test_started
    •:test_finished
    • etc.
    • e.g., junit_formatter

    View Slide

  30. exunit

    View Slide

  31. • elixir mutation testing library
    • goal
    • install exavier library
    • run mix exavier.test
    • mutate code
    • run tests
    • show mutation testing report
    exavier

    View Slide

  32. mutators
    • used by pitest (JVM) and mutant (Ruby)
    • e.g., following pitest, AOR1
    (Arithmetic Operator Replacement Mutator #1)
    original mutated
    + -
    - +
    * /
    / *
    % *

    View Slide

  33. mutators

    View Slide

  34. mutators
    • e.g., following pitest, REMOVE_CONDITIONALS
    (Remove Conditionals Mutator)
    “The remove conditionals mutator will remove all
    conditionals statements such that the guarded statements
    always execute”
    original mutated

    View Slide

  35. mutators
    • e.g., following pitest, REMOVE_CONDITIONALS
    (Remove Conditionals Mutator)
    “The remove conditionals mutator will remove all
    conditionals statements such that the guarded statements
    always execute”
    original mutated

    View Slide

  36. mutators

    View Slide

  37. we’re ready

    View Slide

  38. exavier
    • Run code line coverage analysis for each module,
    sequentially
    • Mutate the code according to each available mutator
    For each module, in parallel:
    For each mutator, sequentially:
    1. Mutate code with given mutator
    2. Run tests (now against mutated code)
    3. Report (% mutants survived vs. killed)

    View Slide

  39. exavier
    •mix exavier.test

    View Slide

  40. biggest drawback
    • currently mutating a single module all at once and
    running all tests against mutated module
    problem?

    View Slide

  41. biggest drawback
    • skews number of survived mutants, e.g.:

    View Slide

  42. rough edges
    • Macros
    • might be hard to provide useful output
    • may have to track macro expansion (if possible)
    • Edge cases, e.g.:
    • division by zero (Issue #7)
    • function references as division (Issue #2)

    View Slide

  43. future
    • get back to work on exavier
    • hopefully with the help of the community

    first order of business: fix “biggest drawback”

    View Slide

  44. the end
    thank you
    questions?
    dnlserrano.dev

    View Slide