2 This Talk Comparison with differential testing and metamorphic testing Broad vision that might inspire future research Illustrative Examples Intramorphic Testing as a general white-box methodology to the test oracle problem
3 Motivating Example Is this program correct? def bubble_sort(arr): length = len(arr) for i in range(length): for j in range(0, length - i - 1): if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[i]
4 Motivating Example def bubble_sort(arr): length = len(arr) for i in range(length): for j in range(0, length - i - 1): if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[i] Is this program correct?
12 Intramorphic Testing I P P' O'' O Intramorphic Testing Modified program Oracle for each other We propose Intramorphic Testing as a general white- box methodology to tackle the test oracle problem
13 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other General methodologies whose concrete realizations require domain-specific insights
14 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other
15 Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other Background: General Methodologies
17 Background: Differential Testing [3, 1, 2] bubble_sort merge_sort [1, 2, 3] insertion_sort [1, 2, 3] [1, 2, 3] Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Differential testing is a black-box technique that works well when systems implement the same behavior
24 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Metamorphic Testing I I' P O O' Derived follow-up input Oracle for each other I P P' O'' O Intramorphic Testing Modified program Oracle for each other
29 Background: Metamorphic Testing The original tech report illustrated the technique using small examples and inspired the proposed intramorphic testing paper and its content
31 Problem: No Whitebox Technique Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other ?
32 Intramorphic Testing I P P' O'' O Intramorphic Testing Modified program Oracle for each other We propose Intramorphic Testing as a white-box methodology aiming to complement differential testing and metamorphic testing
34 Intramorphic Testing Replacing a specific component in a system might have an anticipated effect on the overall system C1 Cn O P Cn-1 P' = P[Ci '/Ci ] Ci '/Ci C1 Ci ' Cn O' Cn-1 Ci Intramorphic transformation Test oracles for each other
35 Intramorphic Testing Implementing a reverse sorting function and reversing either function’s output should yield the same result as the original sorting function
41 Example 1: AST Printing a 3 2 + * class Constant: def __init__(self, value): self.value = value class Variable: def __init__(self, name): self.name = name class Operation: def __init__(self, operator, left, right): self.operator = operator self.left = left self.right = right tree = Operation('*', Operation('+', Variable('a'), Constant(3)), Constant(2))
43 Example 1: AST Printing a 3 2 + * class Operation: # ... def as_string(self): left = self.left.as_string() right = self.right.as_string() if self.operator == '*': if isinstance(self.left, Operation) and self.left.operator == '+': left = '(' + left + ')' if isinstance(self.right, Operation) and self.right.operator == '+': right = '(' + right + ')' return '%s %s %s' % (left, self.operator, right)
44 Example 1: AST Printing Provide alternative (simpler) print implementations, and compare that they include the same operands and operators Infix: (a + 3) * 2 Prefix: * + a 3 2 Postfix: a 3 + 2 * a 3 2 + *
46 Example 1: AST Printing (a + 3) * 2 * + a 3 2 a 3 + 2 * Strip parentheses a + 3 * 2 * + a 3 2 a 3 + 2 * sort * + 2 3 a * + 2 3 a * + 2 3 a Oracles For each other
47 Example 2: Monte Carlo Simulation def get_pi_approximation(): inside = 0 for _ in range(1000000): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/1000000 return pi
49 Example 2: Monte Carlo Simulation def get_pi_approximation(n): inside = 0 for _ in range(n): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/n return pi def get_pi_approximation(): inside = 0 for _ in range(1000000): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/1000000 return pi Add a parameter n
51 Example 3: Knapsack Problem Given a knapsack with a given capacity, the goal is to pack items to maximize the value of items in the knapsack. Capacity: 5 kg Value: 10, weight: 0.1 kg Value: 1500, weight: 2.2 kg
56 Discussion: Domain • Compilers • Database systems • SMT solvers • … Similar to differential testing and metamorphic testing, domain-specific insights will be necessary to realize effective approaches
60 Discussion: Practicality Developers How should I maintain multiple versions? How can the IDE support me? … We hope that future research will propose approaches to lower the cost of using intramorphic testing
mrigger
schulta
hiroki13
joisino
yuyay
yoshipon
tathi
kuri8ive
smatsumori
gpeyre
inoichan
yasumuusan
isayan
jensimmons
hursman
mongodb
keathley
chriscoyier
akmur
kastner
wjessup
zakiwarfel
shpigford
62gerente
chrisshort