@sixty_north Cosmic Ray: Mutation Testing for Python Who watches the watchers? 1 Austin Bingham @austin_bingham Wednesday, May 13, 15 

What is mutation testing? 2 Wednesday, May 13, 15 

What is mutation testing? 2 Apply small changes to code Wednesday, May 13, 15 

What is mutation testing? 2 Apply small changes to code Run test suite Wednesday, May 13, 15 

What is mutation testing? 2 Apply small changes to code Run test suite Test suite fails – KILLED Wednesday, May 13, 15 

What is mutation testing? 2 Apply small changes to code Run test suite Test suite fails – KILLED Test suite error – INCOMPETENT Wednesday, May 13, 15 

What is mutation testing? 2 Apply small changes to code Run test suite Test suite fails – KILLED Test suite error – INCOMPETENT Test suite passes – SURVIVED!!! Wednesday, May 13, 15 

What does mutation testing tell us? 3 Wednesday, May 13, 15 

What does mutation testing tell us? 3 Killed Tests properly detected the mutation. Wednesday, May 13, 15 

What does mutation testing tell us? 3 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Wednesday, May 13, 15 

What does mutation testing tell us? 3 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Wednesday, May 13, 15 

What does mutation testing tell us? 3 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Tests are inadequate for detecting defects in necessary code either Wednesday, May 13, 15 

What does mutation testing tell us? 3 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Tests are inadequate for detecting defects in necessary code either Mutated code is extraneous or Wednesday, May 13, 15 

4 Wednesday, May 13, 15 

Examples of mutations 5 Wednesday, May 13, 15 

Examples of mutations 5 Replace relational operator x > 1 x < 1 Wednesday, May 13, 15 

Examples of mutations 5 Replace relational operator x > 1 x < 1 break/continue replacement break continue Wednesday, May 13, 15 

Examples of mutations 5 • AOD - arithmetic operator deletion • AOR - arithmetic operator replacement • ASR - assignment operator replacement • BCR - break continue replacement • COD - conditional operator deletion • COI - conditional operator insertion • CRP - constant replacement • DDL - decorator deletion • EHD - exception handler deletion • EXS - exception swallowing • IHD - hiding variable deletion • IOD - overriding method deletion • IOP - overridden method calling position change • LCR - logical connector replacement • LOD - logical operator deletion • LOR - logical operator replacement • ROR - relational operator replacement • SCD - super calling deletion • SCI - super calling insert • SIR - slice index remove Replace relational operator x > 1 x < 1 break/continue replacement break continue Wednesday, May 13, 15 

Limitations of mutation testing 6 Wednesday, May 13, 15 

Limitations of mutation testing 6 slow Wednesday, May 13, 15 

Limitations of mutation testing 6 slow infinite loops Wednesday, May 13, 15 

Limitations of mutation testing 6 slow infinite loops equivalent mutants Wednesday, May 13, 15 

https://github.com/sixty-north/cosmic-ray/ Cosmic Ray: Python mutation testing 7 Wednesday, May 13, 15 

Abstract syntax tree manipulation ast ‣ Identify nodes which can be manipulated ‣ Modify copies of ASTs using ast.NodeTransformer 8 Wednesday, May 13, 15 

Arrange for mutated ASTs to be imported importlib ‣ importlib.abc.MetaPathFinder ‣ sys.meta_path ‣ Loaders ‣ importlib.machinery.ModuleSpec 9 import turtle Wednesday, May 13, 15 

Mutant isolation with multiple processes multiprocessing 10 cosmic-ray Process 1 Mutant 1 Process 2 Mutant 2 Process 3 Mutant 3 ‣ Avoid cross-mutant interference ‣ Significantly simplifies spawning processes Wednesday, May 13, 15 

Test discovery unittest 11 Wednesday, May 13, 15 

Actor model implementation in Python pykka 12 Wednesday, May 13, 15 

Event loop to drive actors asyncio 13 Wednesday, May 13, 15 

14 Demo Wednesday, May 13, 15 

15 Thank you! @sixty_north Austin Bingham @austin_bingham Wednesday, May 13, 15