Mutation Testing in Python

Slide 1

Slide 1 text

@sixty_north Mutation Testing in Python Theory and Practice 1 Austin Bingham @austin_bingham

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Agenda 1. Introduction to the theory of   mutation testing 2. Overview of practical diﬃculties 3. Cosmic Ray: mutation testing   for Python 4. Demo 5. Questions 3

Slide 9

Slide 9 text

4 Mutation Testing

Slide 10

Slide 10 text

5 “Mutation testing is conceptually quite simple. Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived. The quality of your tests can be gauged from the percentage of mutations killed.” - pitest.org

Slide 11

Slide 11 text

What is mutation testing? 6

Slide 12

Slide 12 text

What is mutation testing? 6 Code under test + test suite

Slide 13

Slide 13 text

What is mutation testing? 6 Code under test + test suite Introduce single change to code under test

Slide 14

Slide 14 text

What is mutation testing? 6 Code under test + test suite Introduce single change to code under test Run test suite

Slide 15

Slide 15 text

What is mutation testing? 6 Code under test + test suite Introduce single change to code under test Run test suite Ideally, all changes will result in test failures

Slide 16

Slide 16 text

A nested loop of mutation and testing Basic algorithm for operator in mutation-operators: for site in operator.sites(code): operator.mutate(site) run_tests() 7

Slide 17

Slide 17 text

What does mutation testing tell us? 8

Slide 18

Slide 18 text

What does mutation testing tell us? 8 Killed Tests properly detected the mutation.

Slide 19

Slide 19 text

What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed.

Slide 20

Slide 20 text

What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant!

Slide 21

Slide 21 text

What does mutation testing tell us? 8 Killed Tests properly detected the mutation. Incompetent Mutation produced code which is inherently flawed. Survived Tests failed to detect the mutant! Tests are inadequate for detecting defects in necessary code either

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

10 Goals of Mutation Testing

Slide 25

Slide 25 text

Do my tests meaningfully cover my code's functionality Goal #1: Coverage analysis Is a line executed? versus Is functionality veriﬁed? 11

Slide 26

Slide 26 text

Survivors can indicate code which is no longer necessary Goal #2: Detect unnecessary code 12

Slide 27

Slide 27 text

13 Types of Mutations

Slide 28

Slide 28 text

Examples of mutations 14

Slide 29

Slide 29 text

Examples of mutations 14 Replace relational operator x > 1 x < 1

Slide 30

Slide 30 text

Examples of mutations 14 Replace relational operator x > 1 x < 1 break/continue replacement break continue

Slide 31

Slide 31 text

Examples of mutations 14 • AOD - arithmetic operator deletion • AOR - arithmetic operator replacement • ASR - assignment operator replacement • BCR - break continue replacement • COD - conditional operator deletion • COI - conditional operator insertion • CRP - constant replacement • DDL - decorator deletion • EHD - exception handler deletion • EXS - exception swallowing • IHD - hiding variable deletion • IOD - overriding method deletion • IOP - overridden method calling position change • LCR - logical connector replacement • LOD - logical operator deletion • LOR - logical operator replacement • ROR - relational operator replacement • SCD - super calling deletion • SCI - super calling insert • SIR - slice index remove Replace relational operator x > 1 x < 1 break/continue replacement break continue

Slide 32

Slide 32 text

Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/iﬁ/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Some mutations are very widely applicable Language-agnostic mutations 15 Lionel Brand - http://www.uio.no/studier/emner/matnat/iﬁ/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Constant replacement  0 ! 4  ‣ Constant for scalar variable replacement  some_func(x) ! some_func(42)  ‣ Arithmetic operator replacement  x + y ! x * y

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/iﬁ/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Mutations which only make sense for (some) OO languages Object-oriented mutations 16 Lionel Brand - http://www.uio.no/studier/emner/matnat/iﬁ/INF4290/v10/undervisningsmateriale/INF4290-Mutest.pdf ‣ Changing an access modiﬁer  public int x ! private int x  ‣ Remove overloading method  int foo() {} ! int foo() {} 

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Mutations which only make sense for (some) functional languages Functional mutations 17 Duc Le, Mohammad Amin Alipour, Rahul Gopinath, Alex Groce - http://web.engr.oregonstate.edu/~alipourm/pub/fp_mutation.pdf ‣ Change order of pattern matching  take 0 _ = []  take _ [] = []  take n (x:xs) = x : take (n-1) xs  ↓  take _ [] = []  take 0 _ = []  take n (x:xs) = x : take’(n-1) xs

Slide 45

Slide 45 text

18 Complexities of Mutation Testing

Slide 46

Slide 46 text

Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0)

Slide 47

Slide 47 text

Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do?

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Long test suites, large code bases, and many operators can add up Complexity #1: It takes a loooooooong time 19 Image credit: John Mainstone (CC BY-SA 3.0) What to do? ‣ Parallelize as much as possible! ‣ After baselining:

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20

Slide 54

Slide 54 text

Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20

Slide 55

Slide 55 text

Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20 "Good luck with that." Alan Turing (apocryphal)

Slide 56

Slide 56 text

Some incompetent mutants are harder to detect that others Complexity #2: Incompetence detection 20 "Good luck with that." Alan Turing (apocryphal)

Slide 57

Slide 57 text

Some mutants have no detectable differences in functionality Complexity #3: Equivalent mutants 21 def consume(iterator, n): """Advance the iterator n-steps ahead. If n is none, consume entirely.""" # Use functions that consume iterators at C speed. if n is None: # feed the entire iterator into a zero-length deque collections.deque(iterator, maxlen=0) else: # advance to the empty slice starting at position n next(islice(iterator, n, n), None)

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Some mutants have no detectable differences in functionality Complexity #3: Equivalent mutants 22 if __name__ == '__main__': run()

Slide 60

Slide 60 text

23 Cosmic Ray: Mutation Testing for Python github.com/sixty-north/cosmic-ray

Slide 61

Slide 61 text

What do we need to do to make this work? Implementation challenge 24

Slide 62

Slide 62 text

What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 24

Slide 63

Slide 63 text

What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 24

Slide 64

Slide 64 text

What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 3. Run a test suite against each mutant. 24

Slide 65

Slide 65 text

What do we need to do to make this work? Implementation challenge 1. Determine which mutations to make. 2. Make those mutations one at a time. 3. Run a test suite against each mutant. 24 While also dealing with the complexities!

Slide 66

Slide 66 text

25 Operators

Slide 67

Slide 67 text

1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26

Slide 68

Slide 68 text

1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites

Slide 69

Slide 69 text

1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites 1 - 2 Job #2: Perform mutations on request

Slide 70

Slide 70 text

1 + 2 Operators sit at the center of Cosmic Ray’s…well…operations Core concept: Operators 26 Job #1: Identify potential mutation sites 1 - 2 Job #2: Perform mutations on request - Not a job - Decide when to perform mutations

Slide 71

Slide 71 text

Operator cores take action when a potential mutation site is detected Operator cores 27 operator core site detected

Slide 72

Slide 72 text

Operator cores take action when a potential mutation site is detected Operator cores 27 operator core site detected Current cores 1. Counting: counts number  of mutations 2. Mutating: requests mutation  at correct time

Slide 73

Slide 73 text

Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3)

Slide 74

Slide 74 text

Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use…

Slide 75

Slide 75 text

Slide 76

Slide 76 text

Slide 77

Slide 77 text

Abstract syntax trees: the basis for Cosmic Ray’s mutation operators Python’s standard ast module 28 1 + 2 * 3 add num(1) mul num(2) num(3) ast elements we use… ‣ Generating ASTs from   Python source code ‣ Walking/transforming ASTs ‣ Manipulating AST nodes   cleanly

Slide 78

Slide 78 text

Slide 79

Slide 79 text

The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore ReplaceConstant

Slide 80

Slide 80 text

The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() ReplaceConstant

Slide 81

Slide 81 text

The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() ReplaceConstant

Slide 82

Slide 82 text

The operator base class, subclasses, and cores all do a little dance Operators: putting it all together 29 ast.NodeTransformer Operator MutatingCore 1. visit() 2. visit_Num() 3. visit_mutation_site() ReplaceConstant

Slide 83

Slide 83 text

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Converts unary-sub to unary-add Example operator: Reverse unary subtraction 30 class ReverseUnarySub(Operator): def visit_UnaryOp(self, node): if isinstance(node.op, ast.USub): return self.visit_mutation_site(node) else: return node def mutate(self, node): node.op = ast.UAdd() return node

Slide 86

Slide 86 text

Operators summary 31

Slide 87

Slide 87 text

Operators summary ‣ Use ast to transform source code into abstract syntax trees. 31

Slide 88

Slide 88 text

Operators summary ‣ Use ast to transform source code into abstract syntax trees. ‣ Implement operators which are able to detect mutation sites and perform mutations. 31

Slide 89

Slide 89 text

Operators summary ‣ Use ast to transform source code into abstract syntax trees. ‣ Implement operators which are able to detect mutation sites and perform mutations. ‣ Use diﬀerent cores to control exactly what the operators are doing. 31

Slide 90

Slide 90 text

32 Installing modules

Slide 91

Slide 91 text

Python provides a sophisticated system for performing module imports Module management: overview 33

Slide 92

Slide 92 text

Python provides a sophisticated system for performing module imports Module management: overview finders Responsible for producing loaders when they recognize a module name 33

Slide 93

Slide 93 text

Python provides a sophisticated system for performing module imports Module management: overview finders Responsible for producing loaders when they recognize a module name 33 loaders Responsible for populating module namespaces on import

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Cosmic Ray implements a custom ﬁnder Module management: Finder ‣ The ﬁnder associates module names with ASTs ‣ It produces loaders for those modules which are under mutation 34

Slide 96

Slide 96 text

Cosmic Ray implements a custom ﬁnder Module management: Finder 35 class ASTFinder(MetaPathFinder): def __init__(self, fullname, ast): self._fullname = fullname self._ast = ast def find_spec(self, fullname, path, target=None): if fullname == self._fullname: return ModuleSpec(fullname, ASTLoader(self._ast, fullname)) else: return None

Slide 97

Slide 97 text

Cosmic Ray implements a custom loader Module management: Loader ‣ The loader compiles its AST in the namespace of a new module object 36

Slide 98

Slide 98 text

Cosmic Ray implements a custom loader Module management: Loader 37 class ASTLoader: def __init__(self, ast, name): self._ast = ast self._name = name def exec_module(self, mod): exec(compile(self._ast, self._name, 'exec'), mod.__dict__)

Slide 99

Slide 99 text

Module installation summary 38

Slide 100

Slide 100 text

Module installation summary ‣ Use MutatingCore to generate mutated ASTs 38

Slide 101

Slide 101 text

Module installation summary ‣ Use MutatingCore to generate mutated ASTs ‣ Use compile() to produce code objects from mutated ASTs 38

Slide 102

Slide 102 text

Module installation summary ‣ Use MutatingCore to generate mutated ASTs ‣ Use compile() to produce code objects from mutated ASTs ‣ Use ﬁnders, loaders, and sys.meta_path to advertise and install these mutated modules 38

Slide 103

Slide 103 text

39 Figuring out what  to mutate

Slide 104

Slide 104 text

This seems like the natural boundary for mutation testing in the Python universe Cosmic Ray operates on a package ‣ The user speciﬁes a single package for mutation ‣ Cosmic Ray scans the package for all of its modules ‣ There are limitations to the kinds of modules it can mutate ‣ It is possible to exclude modules which should not be mutated 40

Slide 105

Slide 105 text

Sub-packages and modules are discovered automatically Finding modules 41 find_modules.py def find_modules(name): module_names = [name] while module_names: module_name = module_names.pop() try: module = importlib.import_module(module_name) yield module if hasattr(module, '__path__'): for _, name, _ in pkgutil.iter_modules(module.__path__): module_names.append('{}.{}'.format(module_name, name)) except Exception: # pylint:disable=broad-except LOG.exception('Unable to import %s', module_name)

Slide 106

Slide 106 text

An interesting problem! Counting potential mutants 42 1 + 2 * 3 1 - 2 * 3 1 + 2 / 3 2 * 3 1 + 16 * 3 ?

Slide 107

Slide 107 text

43 Running tests

Slide 108

Slide 108 text

Encapsulate the differences between various testing systems Test runners 44 TestRunner UnittestRunner def _run()

Slide 109

Slide 109 text

Encapsulate the differences between various testing systems Test runners 44 TestRunner UnittestRunner def _run() test directory

Slide 110

Slide 110 text

Testing overview 45

Slide 111

Slide 111 text

Testing overview ‣ Figure out what to mutate 45

Slide 112

Slide 112 text

Testing overview ‣ Figure out what to mutate ‣ Create a mutant 45

Slide 113

Slide 113 text

Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant 45

Slide 114

Slide 114 text

Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant ‣ Tell TestRunner to run the tests 45

Slide 115

Slide 115 text

Testing overview ‣ Figure out what to mutate ‣ Create a mutant ‣ Install the mutant ‣ Tell TestRunner to run the tests 45 In a separate process

Slide 116

Slide 116 text

There is no perfect strategy for detecting them Dealing with incompetent mutants 46 Image by o5com - https://www.ﬂickr.com/photos/o5com/5488964999 Absolute timeout or Based on a baseline

Slide 117

Slide 117 text

47 The rest of the tech

Slide 118

Slide 118 text

Test runners and operators are provided by dynamically discovered modules Test system and operator plugins ‣ Using OpenStack's stevedore plugin system ‣ Plugins can come from external packages 48 cosmic_ray py.test my_package unittest my_test_system Number Replacer plugins MyOperator

Slide 119

Slide 119 text

Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . celery task queue celery worker

Slide 120

Slide 120 text

Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 1. Task added to queue cosmic-ray exec celery task queue celery worker

Slide 121

Slide 121 text

Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 2. Task sent to worker 1. Task added to queue cosmic-ray exec celery task queue celery worker

Slide 122

Slide 122 text

Used to distribute tasks to more than one machine elery: distributed task queue 49 celery worker . . . 2. Task sent to worker cosmic-ray worker 3. Worker started in new process 1. Task added to queue cosmic-ray exec celery task queue celery worker

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Use an embedded database to keep track of work and results Staging of work 50

Slide 125

Slide 125 text

Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done 50

Slide 126

Slide 126 text

Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results 50

Slide 127

Slide 127 text

Use an embedded database to keep track of work and results Staging of work ‣ Use CountingCore to determine work-to-be-done ‣ Only schedule work items that don’t have results ‣ Allows interruption and resumption of runs 50

Slide 128

Slide 128 text

Slide 129

Slide 129 text

Slide 130

Slide 130 text

Describe command-line syntax in comment strings…like magic! docopt: command-line interface description language 51 """usage: cosmic-ray counts [options] [--exclude-modules=P ...] Count the number of tests that would be run for a given testing configuration. This is mostly useful for estimating run times and keeping track of testing statistics. options: --no-local-import Allow importing module from the current directory --test-runner=R Test-runner plugin to use [default: unittest] --exclude-modules=P Pattern of module names to exclude from mutation """ $ cosmic-ray —no-local-import —exclude-modules=“.*.test” foo

Slide 131

Slide 131 text

Slide 132

Slide 132 text

52 Remaining work

Slide 133

Slide 133 text

There’s plenty left to do if you’re interested! Remaining work 53 github.com/sixty-north/cosmic-ray/issues

Slide 134

Slide 134 text

There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts  github.com/sixty-north/cosmic-ray/issues

Slide 135

Slide 135 text

There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts  ‣Exceptions and processing instructions  github.com/sixty-north/cosmic-ray/issues

Slide 136

Slide 136 text

There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts  ‣Exceptions and processing instructions  ‣Support for more kinds of modules  github.com/sixty-north/cosmic-ray/issues

Slide 137

Slide 137 text

There’s plenty left to do if you’re interested! Remaining work 53 ‣Properly implementing timeouts  ‣Exceptions and processing instructions  ‣Support for more kinds of modules  ‣Integration with coverage testing github.com/sixty-north/cosmic-ray/issues