Mutation Testing in Python

ACCU 2016 presentation slides

1. @sixty_north Mutation Testing in Python Theory and Practice 1 Austin Bingham @austin_bingham
8. Agenda 1. Introduction to the theory of 
 mutation testing 2. Overview of practical difficulties 3. Cosmic Ray: mutation testing 
 for Python 4. Demo 5. Questions 3
9. 4 Mutation Testing
10. 5 “Mutation testing is conceptually quite simple. Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived. The quality of your tests can be gauged from the percentage of mutations killed.” - pitest.org
11. What is mutation testing? 6
12. What is mutation testing? 6 Code under test + test suite
13. What is mutation testing? 6 Code under test + test suite Introduce single change to code under test
14. What is mutation testing? 6 Code under test + test suite Introduce single change to code under test Run test suite
15. What is mutation testing? 6 Code under test + test suite Introduce single change to code under test Run test suite Ideally, all changes will result in test failures
16. A nested loop of mutation and testing Basic algorithm for operator in mutation-operators: for site in operator.sites(code): operator.mutate(site) run_tests() 7
17. What does mutation testing tell us? 8
18. What does mutation testing tell us? 8 Killed Tests properly detected the mutation.
19. What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed.
20. What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant!
21. What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Tests are inadequate for detecting defects in necessary code either
22. What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Tests are inadequate for detecting defects in necessary code either Mutated code is extraneous or
24. 10 Goals of Mutation Testing
25. Do my tests meaningfully cover my code's functionality Goal #1: Coverage analysis Is a line executed? versus Is functionality verified? 11
26. Survivors can indicate code which is no longer necessary Goal #2: Detect unnecessary code 12
27. 13 Types of Mutations
28. Examples of mutations 14
29. Examples of mutations 14 Replace relational operator x > 1 x < 1
30. Examples of mutations 14 Replace relational operator x > 1 x < 1 break/continue replacement break continue
31. Examples of mutations 14 • AOD - arithmetic operator deletion • AOR - arithmetic operator replacement • ASR - assignment operator replacement • BCR - break continue replacement • COD - conditional operator deletion • COI - conditional operator insertion • CRP - constant replacement • DDL - decorator deletion • EHD - exception handler deletion • EXS - exception swallowing • IHD - hiding variable deletion • IOD - overriding method deletion • IOP - overridden method calling position change • LCR - logical connector replacement • LOD - logical operator deletion • LOR - logical operator replacement • ROR - relational operator replacement • SCD - super calling deletion • SCI - super calling insert • SIR - slice index remove Replace relational operator x > 1 x < 1 break/continue replacement break continue
32. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf
33. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement
 0 ! 4

34. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement
 0 ! 4
 ‣ Constant for scalar variable replacement
 some_func(x) ! some_func(42)

35. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement
 0 ! 4
 ‣ Constant for scalar variable replacement
 some_func(x) ! some_func(42)
 ‣ Arithmetic operator replacement
 x + y ! x * y
36. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement
 0 ! 4
 ‣ Constant for scalar variable replacement
 some_func(x) ! some_func(42)
 ‣ Arithmetic operator replacement
 x + y ! x * y ‣ Relational operator replacement
 x < y ! x <= y

37. Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement
 0 ! 4
 ‣ Constant for scalar variable replacement
 some_func(x) ! some_func(42)
 ‣ Arithmetic operator replacement
 x + y ! x * y ‣ Relational operator replacement
 x < y ! x <= y
 ‣ Unary operator insertion
 int x = 1 ! int x = -1
38. Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf
39. Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Changing an access modifier
 public int x ! private int x

40. Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Changing an access modifier
 public int x ! private int x
 ‣ Remove overloading method
 int foo() {} ! int foo() {}

41. Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Changing an access modifier
 public int x ! private int x
 ‣ Remove overloading method
 int foo() {} ! int foo() {}
 ‣ Change base class order
 class X(A, B) ! class X(B, A)
42. Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/ifi/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Changing an access modifier
 public int x ! private int x
 ‣ Remove overloading method
 int foo() {} ! int foo() {}
 ‣ Change base class order
 class X(A, B) ! class X(B, A) ‣ Change parameter order (?)
 foo(a, b) ! foo(b, a)
43. Mutations which only make sense for (some) functional languages Functional mutations 17 Duc Le, Mohammad Amin Alipour, Rahul Gopinath, Alex Groce - http://web.engr.oregonstate.edu/~alipourm/pub/fp_mutation.pdf
44. Mutations which only make sense for (some) functional languages Functional mutations 17 Duc Le, Mohammad Amin Alipour, Rahul Gopinath, Alex Groce - http://web.engr.oregonstate.edu/~alipourm/pub/fp_mutation.pdf ‣ Change order of pattern matching
 take 0 _ = []
 take _ [] = []
 take n (x:xs) = x : take (n-1) xs
 ↓
 take _ [] = []
 take 0 _ = []
 take n (x:xs) = x : take’(n-1) xs
45. 18 Complexities of Mutation Testing
46. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0)
47. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do?
48. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible!
49. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible! ‣ After baselining:
50. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible! ‣ After baselining: • only run tests on modified code
51. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible! ‣ After baselining: • only run tests on modified code • only mutate modified code
52. Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible! ‣ After baselining: • only run tests on modified code • only mutate modified code ‣ Speed up test suite
53. Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20
54. Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20
55. Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20 "Good luck with that." Alan Turing (apocryphal)
56. Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20 "Good luck with that." Alan Turing (apocryphal)
57. Some mutants have no detectable differences in functionality Complexity #3: Equivalent mutants 21 def consume(iterator, n): """Advance the iterator n-steps ahead. If n is none, consume entirely.""" # Use functions that consume iterators at C speed. if n is None: # feed the entire iterator into a zero-length deque collections.deque(iterator, maxlen=0) else: # advance to the empty slice starting at position n next(islice(iterator, n, n), None)
58. Some mutants have no detectable differences in functionality Complexity #3: Equivalent mutants 21 def consume(iterator, n): """Advance the iterator n-steps ahead. If n is none, consume entirely.""" # Use functions that consume iterators at C speed. if n is None: # feed the entire iterator into a zero-length deque collections.deque(iterator, maxlen=0) else: # advance to the empty slice starting at position n next(islice(iterator, n, n), None)
59. Some mutants have no detectable differences in functionality Complexity #3: Equivalent mutants 22 if __name__ == '__main__': run()
60. 23 Cosmic Ray: Mutation Testing for Python github.com/sixty-north/cosmic-ray
61. What do we need to do to make this work? Implementation challenge 24
62. What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 24
63. What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 24
64. What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 3. Run a test suite against each mutant. 24
65. What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 3. Run a test suite against each mutant. 24 While also dealing with the complexities!
66. 25 Operators
67. 1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26
68. 1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites
69. 1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites 1 - 2 Job #2: Perform mutations on request
70. 1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites 1 - 2 Job #2: Perform mutations on request - Not a job - Decide when to perform mutations
71. Operator cores take action when a potential mutation site is detected Operator cores 27 operator core site detected
72. Operator cores take action when a potential mutation site is detected Operator cores 27 operator core site detected Current cores 1. Counting: counts number
 of mutations 2. Mutating: requests mutation
 at correct time
73. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3)
74. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use…
75. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use… ‣ Generating ASTs from 
 Python source code
76. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use… ‣ Generating ASTs from 
 Python source code ‣ Walking/transforming ASTs
77. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use… ‣ Generating ASTs from 
 Python source code ‣ Walking/transforming ASTs ‣ Manipulating AST nodes 
 cleanly
78. Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use… ‣ Generating ASTs from 
 Python source code ‣ Walking/transforming ASTs ‣ Manipulating AST nodes 
 cleanly Plus we use compile() to transform ASTs into code objects at runtime
79. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore ReplaceConstant
80. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() ReplaceConstant
81. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() ReplaceConstant
82. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() 3. visit_mutation_site() ReplaceConstant
83. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() 3. visit_mutation_site() 4. visit_mutation_site() ReplaceConstant
84. The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() 3. visit_mutation_site() 4. visit_mutation_site() 5. mutate() ReplaceConstant
85. Converts unary-sub to unary-add Example operator: Reverse unary subtraction 30 class ReverseUnarySub(Operator): def visit_UnaryOp(self, node): if isinstance(node.op, ast.USub): return self.visit_mutation_site(node) else: return node def mutate(self, node): node.op = ast.UAdd() return node
86. Operators summary 31
87. Operators summary ‣ Use ast to transform source code into abstract syntax trees. 31
88. Operators summary ‣ Use ast to transform source code into abstract syntax trees. ‣ Implement operators which are able to detect mutation sites and perform mutations. 31
89. Operators summary ‣ Use ast to transform source code into abstract syntax trees. ‣ Implement operators which are able to detect mutation sites and perform mutations. ‣ Use different cores to control exactly what the operators are doing. 31
90. 32 Installing modules
91. Python provides a sophisticated system for performing module imports Module management: overview 33
92. Python provides a sophisticated system for performing module imports Module management: overview finders Responsible for producing loaders when they recognize a module name 33
93. Python provides a sophisticated system for performing module imports Module management: overview finders Responsible for producing loaders when they recognize a module name 33 loaders Responsible for populating module namespaces on import
94. Python provides a sophisticated system for performing module imports Module management: overview finders Responsible for producing loaders when they recognize a module name 33 loaders Responsible for populating module namespaces on import sys.meta_path A list of finders which are queried in order with module names when import is executed
95. Cosmic Ray implements a custom finder Module management: Finder ‣ The finder associates module names with ASTs ‣ It produces loaders for those modules which are under mutation 34
96. Cosmic Ray implements a custom finder Module management: Finder 35 class ASTFinder(MetaPathFinder): def __init__(self, fullname, ast): self._fullname = fullname self._ast = ast def find_spec(self, fullname, path, target=None): if fullname == self._fullname: return ModuleSpec(fullname, ASTLoader(self._ast, fullname)) else: return None
97. Cosmic Ray implements a custom loader Module management: Loader ‣ The loader compiles its AST in the namespace of a new module object 36
98. Cosmic Ray implements a custom loader Module management: Loader 37 class ASTLoader: def __init__(self, ast, name): self._ast = ast self._name = name def exec_module(self, mod): exec(compile(self._ast, self._name, 'exec'), mod.__dict__)
99. Module installation summary 38
100. Module installation summary ‣ Use MutatingCore to generate mutated ASTs 38
101. Module installation summary ‣ Use MutatingCore to generate mutated ASTs ‣ Use compile() to produce code objects from mutated ASTs 38
102. Module installation summary ‣ Use MutatingCore to generate mutated ASTs ‣ Use compile() to produce code objects from mutated ASTs ‣ Use finders, loaders, and sys.meta_path to advertise and install these mutated modules 38
103. 39 Figuring out what
 to mutate
104. This seems like the natural boundary for mutation testing in the Python universe Cosmic Ray operates on a package ‣ The user specifies a single package for mutation ‣ Cosmic Ray scans the package for all of its modules ‣ There are limitations to the kinds of modules it can mutate ‣ It is possible to exclude modules which should not be mutated 40
105. Sub-packages and modules are discovered automatically Finding modules 41 find_modules.py def find_modules(name): module_names = [name] while module_names: module_name = module_names.pop() try: module = importlib.import_module(module_name) yield module if hasattr(module, '__path__'): for _, name, _ in pkgutil.iter_modules(module.__path__): module_names.append('{}.{}'.format(module_name, name)) except Exception: # pylint:disable=broad-except LOG.exception('Unable to import %s', module_name)
106. An interesting problem! Counting potential mutants 42 1 + 2 * 3 1 - 2 * 3 1 + 2 / 3 2 * 3 1 + 16 * 3 ?
107. 43 Running tests
108. Encapsulate the differences between various testing systems Test runners 44 TestRunner UnittestRunner def _run()
109. Encapsulate the differences between various testing systems Test runners 44 TestRunner UnittestRunner def _run() test directory
110. Testing overview 45
111. Testing overview ‣ Figure out what to mutate 45
112. Testing overview ‣ Figure out what to mutate ‣ Create a mutant 45
113. Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant 45
114. Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant ‣ Tell TestRunner to run the tests 45
115. Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant ‣ Tell TestRunner to run the tests 45 In a separate process
116. There is no perfect strategy for detecting them Dealing with incompetent mutants 46 Image by o5com - https://www.flickr.com/photos/o5com/5488964999 Absolute timeout or Based on a baseline
117. 47 The rest of the tech
118. Test runners and operators are provided by dynamically discovered modules Test system and operator plugins ‣ Using OpenStack's stevedore plugin system ‣ Plugins can come from external packages 48 cosmic_ray py.test my_package unittest my_test_system Number Replacer plugins MyOperator
119. Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . celery task queue celery worker
120. Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 1. Task added to queue cosmic-ray exec celery task queue celery worker
121. Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 2. Task sent to worker 1. Task added to queue cosmic-ray exec celery task queue celery worker
122. Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 2. Task sent to worker cosmic-ray worker 3. Worker started in new process 1. Task added to queue cosmic-ray exec celery task queue celery worker
123. Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 2. Task sent to worker cosmic-ray worker 3. Worker started in new process 1. Task added to queue cosmic-ray exec celery task queue celery worker celeryproject.org
124. Use an embedded database to keep track of work and results Staging of work 50
125. Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done 50
126. Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results 50
127. Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results ‣ Allows interruption and resumption of runs 50
128. Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results ‣ Allows interruption and resumption of runs ‣ Natural place for results 50
129. Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results ‣ Allows interruption and resumption of runs ‣ Natural place for results 50 github.com/ msiemens/
130. Describe command-line syntax in comment strings…like magic! docopt: command-line interface description language 51 """usage: cosmic-ray counts [options] [--exclude-modules=P ...] <top-module> Count the number of tests that would be run for a given testing configuration. This is mostly useful for estimating run times and keeping track of testing statistics. options: --no-local-import Allow importing module from the current directory --test-runner=R Test-runner plugin to use [default: unittest] --exclude-modules=P Pattern of module names to exclude from mutation """ $ cosmic-ray —no-local-import —exclude-modules=“.*.test” foo
131. Describe command-line syntax in comment strings…like magic! docopt: command-line interface description language 51 """usage: cosmic-ray counts [options] [--exclude-modules=P ...] <top-module> Count the number of tests that would be run for a given testing configuration. This is mostly useful for estimating run times and keeping track of testing statistics. options: --no-local-import Allow importing module from the current directory --test-runner=R Test-runner plugin to use [default: unittest] --exclude-modules=P Pattern of module names to exclude from mutation """ $ cosmic-ray —no-local-import —exclude-modules=“.*.test” foo docopt.org
132. 52 Remaining work
133. There’s plenty left to do if you’re interested! Remaining work 53 github.com/sixty-north/cosmic-ray/issues
134. There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts
 github.com/sixty-north/cosmic-ray/issues
135. There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts
 ‣Exceptions and processing instructions
 github.com/sixty-north/cosmic-ray/issues
136. There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts
 ‣Exceptions and processing instructions
 ‣Support for more kinds of modules
 github.com/sixty-north/cosmic-ray/issues
137. There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts
 ‣Exceptions and processing instructions
 ‣Support for more kinds of modules
 ‣Integration with coverage testing github.com/sixty-north/cosmic-ray/issues
138. 54 Demo
139. 55 Thank you! @sixty_north Austin Bingham @austin_bingham
140. 55 Thank you! @sixty_north Austin Bingham @austin_bingham