Skeletal Program Enumeration for Rigorous Compiler Testing

Slide 1

Slide 1 text

Presented by Liang Gong Skeletal Program Enumeration for Rigorous Compiler Testing Qirun Zhang, Chengnian Sun, Zhendong Su Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley.

Slide 2

Slide 2 text

Motivation Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. • Test generation for compilers • Existing works: • Program generation (Csmith) • Generate test from scratch • Program mutation (EMI) • Randomly remove statements Random  Opportunistic Favor large/complex programs 2

Slide 3

Slide 3 text

Program Enumeration Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. • Observation • Bug-triggering programs are small • ~30 LOC (Sun et al. ISSTA’16) • Program mutation • mutations of statements • Change variable names • Search space is bounded  enumeration • Confirmed Bugs: • GCC/Clang (217),CompCert (29), Scala (42) 3

Slide 4

Slide 4 text

Change Identifiers Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. mutate identifiers can trigger optimizations 4 Original Test Mutated Test • Constant propagation • Dead code elimination

Slide 5

Slide 5 text

Motivating Example Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 5 Mutated program triggers a bug related to alias in GCC.

Slide 6

Slide 6 text

Motivating Example Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 6 Mutated program triggers a bug related to constant folding. GCC crash

Slide 7

Slide 7 text

Program Enumeration Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. Goal: Enumerate all identifier mutations • Step-1: Convert an existing test into skeleton 7 Test Skeleton

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Program Enumeration Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. Goal: Enumerate all identifier mutations • Step-2: fill in the holes in a skeleton • Formulate as a set partition problem (well-known) 12 Generated Test Skeleton Generated Test 2 1 3 4 5 6 { {1,3,4,5}, {2, 6} } { {1,3,4,5}, {2, 6} } 64 (naive)  31 (set partition) 

Slide 13

Slide 13 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 13 The number of ways to partition n elements into k subsets

Slide 14

Slide 14 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 14 The number of ways to partition n elements into k subsets

Slide 15

Slide 15 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 15

Slide 16

Slide 16 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 16 Partition n+1 elements into k subsets

Slide 17

Slide 17 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 17 Partition n+1 elements into k subsets • The selected element is partitioned to a set s, where | s | = 1

Slide 18

Slide 18 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 18 Partition n+1 elements into k subsets • The selected element is partitioned to a set s, where | s | = 1 • The selected element is partitioned to a set s, where | s | > 1

Slide 19

Slide 19 text

Stirling Partition Number Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 19 • Now we can get all partitions recursively: function par(n, k) { ... // handle base cases return union( add(n, par(n-1, k)), par(n-1, k-1)); }

Slide 20

Slide 20 text

Time Complexity Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 20 • A skeleton with n holes and k variables. Olver et al. NIST Handbook of Mathematical Functions.

Slide 21

Slide 21 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. Each hole in a skeleton can be filled with a different set of variables. 21 Global scope hole local scope

Slide 22

Slide 22 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. Naïve approach: 22

Slide 23

Slide 23 text

Set Partition (with Scopes) Naïve approach: X 23

Slide 24

Slide 24 text

Set Partition (with Scopes) Naïve approach does not completely enumerate X 24

Slide 25

Slide 25 text

Set Partition (with Scopes) Naïve approach does not completely enumerate X 25

Slide 26

Slide 26 text

Set Partition (with Scopes) Naïve approach does not completely enumerate X Locally: = { {3}, {4} } 26

Slide 27

Slide 27 text

Set Partition (with Scopes) Naïve approach does not completely enumerate X Locally: = Globally: ≠ { {3}, {4} } { {1,2,3}, {4}, {5} } ≠ { {1,2,4}, {3}, {5} } 27

Slide 28

Slide 28 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. promote local holes to be global holes 28

Slide 29

Slide 29 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 29 , { 3 }, { 4 }, { 3, 4 } promote local holes to be global holes

Slide 30

Slide 30 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 30 , { 3 }, { 4 }, { 3, 4 } X promote local holes to be global holes

Slide 31

Slide 31 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 31 , { 3 }, { 4 }, { 3, 4 } X X { 3 } promote local holes to be global holes

Slide 32

Slide 32 text

Set Partition (with Scopes) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 32 , { 3 }, { 4 }, { 3, 4 } X X { 3 } X { 4 } { 3, 4 } promote local holes to be global holes

Slide 33

Slide 33 text

Evaluation: size reduction • 94 orders of reduction • Still too large in practice Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 33

Slide 34

Slide 34 text

Evaluation: size reduction • 94 orders of reduction • Still too large in practice • Throw away tests with > 10K variants Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 34

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Evaluation: size reduction • 94 orders of reduction • Still too large in practice • Throw away tests with > 10K variants • Retained 90% of the original tests Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 36

Slide 37

Slide 37 text

Evaluation: size reduction • Throw away tests with > 10K variants • Retained 90% of the original tests Characteristics of tests after filtering Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 37

Slide 38

Slide 38 text

Evaluation: coverage GCC: ~30% Clang: ~20% GCC Clang (PM-X: EMI deletes X statements) Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 38

Slide 39

Slide 39 text

Evaluation: bugs Liang Gong, Electric Engineering & Computer Science, University of California, Berkeley. 39