2 This Talk Comparison with differential testing and metamorphic testing Broad vision that might inspire future research Illustrative Examples Intramorphic Testing as a general white-box methodology to the test oracle problem
3 Motivating Example Is this program correct? def bubble_sort(arr): length = len(arr) for i in range(length): for j in range(0, length - i - 1): if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[i]
4 Motivating Example def bubble_sort(arr): length = len(arr) for i in range(length): for j in range(0, length - i - 1): if arr[j] > arr[j+1]: arr[j], arr[j+1] = arr[j+1], arr[i] Is this program correct?
12 Intramorphic Testing I P P' O'' O Intramorphic Testing Modified program Oracle for each other We propose Intramorphic Testing as a general white- box methodology to tackle the test oracle problem
13 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other General methodologies whose concrete realizations require domain-specific insights
14 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other
15 Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other Background: General Methodologies
17 Background: Differential Testing [3, 1, 2] bubble_sort merge_sort [1, 2, 3] insertion_sort [1, 2, 3] [1, 2, 3] Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Differential testing is a black-box technique that works well when systems implement the same behavior
24 Background: General Methodologies Differential Testing I P1 P2 O1 P3 O2 O3 Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Metamorphic Testing I I' P O O' Derived follow-up input Oracle for each other I P P' O'' O Intramorphic Testing Modified program Oracle for each other
29 Background: Metamorphic Testing The original tech report illustrated the technique using small examples and inspired the proposed intramorphic testing paper and its content
31 Problem: No Whitebox Technique Differential Testing I P1 P2 O1 P3 O2 O3 Metamorphic Testing I I' P O O' I P P' O'' O Intramorphic Testing Oracle for each other Interchangeable programs: P1 (I) = P2 (I) = P3 (I) Derived follow-up input Oracle for each other Modified program Oracle for each other ?
32 Intramorphic Testing I P P' O'' O Intramorphic Testing Modified program Oracle for each other We propose Intramorphic Testing as a white-box methodology aiming to complement differential testing and metamorphic testing
34 Intramorphic Testing Replacing a specific component in a system might have an anticipated effect on the overall system C1 Cn O P Cn-1 P' = P[Ci '/Ci ] Ci '/Ci C1 Ci ' Cn O' Cn-1 Ci Intramorphic transformation Test oracles for each other
35 Intramorphic Testing Implementing a reverse sorting function and reversing either function’s output should yield the same result as the original sorting function
41 Example 1: AST Printing a 3 2 + * class Constant: def __init__(self, value): self.value = value class Variable: def __init__(self, name): self.name = name class Operation: def __init__(self, operator, left, right): self.operator = operator self.left = left self.right = right tree = Operation('*', Operation('+', Variable('a'), Constant(3)), Constant(2))
43 Example 1: AST Printing a 3 2 + * class Operation: # ... def as_string(self): left = self.left.as_string() right = self.right.as_string() if self.operator == '*': if isinstance(self.left, Operation) and self.left.operator == '+': left = '(' + left + ')' if isinstance(self.right, Operation) and self.right.operator == '+': right = '(' + right + ')' return '%s %s %s' % (left, self.operator, right)
44 Example 1: AST Printing Provide alternative (simpler) print implementations, and compare that they include the same operands and operators Infix: (a + 3) * 2 Prefix: * + a 3 2 Postfix: a 3 + 2 * a 3 2 + *
46 Example 1: AST Printing (a + 3) * 2 * + a 3 2 a 3 + 2 * Strip parentheses a + 3 * 2 * + a 3 2 a 3 + 2 * sort * + 2 3 a * + 2 3 a * + 2 3 a Oracles For each other
47 Example 2: Monte Carlo Simulation def get_pi_approximation(): inside = 0 for _ in range(1000000): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/1000000 return pi
49 Example 2: Monte Carlo Simulation def get_pi_approximation(n): inside = 0 for _ in range(n): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/n return pi def get_pi_approximation(): inside = 0 for _ in range(1000000): x = random.random() y = random.random() if x**2+y**2 <= 1: inside += 1 pi = 4*inside/1000000 return pi Add a parameter n
51 Example 3: Knapsack Problem Given a knapsack with a given capacity, the goal is to pack items to maximize the value of items in the knapsack. Capacity: 5 kg Value: 10, weight: 0.1 kg Value: 1500, weight: 2.2 kg
56 Discussion: Domain • Compilers • Database systems • SMT solvers • … Similar to differential testing and metamorphic testing, domain-specific insights will be necessary to realize effective approaches
60 Discussion: Practicality Developers How should I maintain multiple versions? How can the IDE support me? … We hope that future research will propose approaches to lower the cost of using intramorphic testing
mrigger
wide_vsix
upura
ido_kara_deru
tomoshige_n
o_hasegawa
yumakoizumi
koharite
seiichiinoue
kuribayashi4
mertcooking
kei18
kitachan_black
jasonvnalue
aki_iinuma
speakerdeck
michaelherold
brianwarren
reverentgeek
erikaheidi
holman
cherdarchuk
bkeepers
chriscoyier
colly