Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Finding 1,500 bugs in the SMT Solvers Z3 and CVC5

Finding 1,500 bugs in the SMT Solvers Z3 and CVC5

Satisfiability Modulo Theory (SMT) solvers are essential tools for many advances in programming languages and formal methods. Examples include symbolic execution engines, software model checkers, and program verifiers. SMT solvers' robustness is crucial---Soundness bugs SMT solvers can invalidate client applications' results and lead to disasters in safety-critical or security-critical domains. We ran a large-scale testing campaign to put Z3 and CVC4/5's robustness to the test: As of today, We have found 1,500+ unique bugs in Z3 and CVC4--1,100+ have already been fixed by the developers and 400+ are critical soundness bugs. This talk covers the techniques used and lessons learned from this campaign. Specifically, we will discuss: (1) Semantic Fusion, a general, metamorphic testing methodology, (2) Type-Aware Mutation, a highly effective testing technique for testing SMT solvers, and (3) outline open challenges in making SMT solvers (and other formal methods tools) more correct and faster.

Dominik Winterer

June 24, 2022
Tweet

Other Decks in Research

Transcript

  1. Finding 1,500+ Bugs in the SMT Solver
    s

    Z3 and CVC5
    Dominik Wintere
    r

    ETH Zurich


    Talk at Paderborn Universit
    y

    Jun 21st, 2022
    @DominikWinterer
    wintered.github.io

    View Slide

  2. SMT Problem
    φ : x > 0 ∧ x < 0

    View Slide

  3. SMT Problem
    UNSAT
    φ : x > 0 ∧ x < 0

    View Slide

  4. SMT Problem
    φ : x > 0 ∧ x < 1

    View Slide

  5. SMT Problem
    SAT
    φ : x > 0 ∧ x < 1

    View Slide

  6. SMT Problem
    SAT
    φ : x > 0 ∧ x < 1
    x = 0.5

    View Slide

  7. SMT Solver
    SMT Solvers
    φ : x > 0 ∧ x < 1

    View Slide

  8. SAT
    SMT Solver
    SMT Solvers
    φ : x > 0 ∧ x < 1
    8

    View Slide

  9. SMT Background Theories
    9
    ArrayEx
    FixedSizeBitVectors
    Core
    FloatingPoints
    Ints
    Reals
    Real_Ints
    Strings

    View Slide

  10. SMT Solvers: Important Software Foundations
    SMT Solver
    Formal
    veri
    fi
    cation
    Symbolic
    Execution
    Access Policy
    Analysis
    Security Safety

    10

    View Slide

  11. SMT Solvers: Important Software Foundations
    Boogie
    11
    ALT Ergo …

    Security Safety

    View Slide

  12. SMT Solver UNSAT
    Bug in an SMT Solver
    φ : x > 0 ∧ x < 1
    12

    View Slide

  13. SMT Solver UNSAT
    Bug in an SMT Solver
    φ : x > 0 ∧ x < 1
    13
    x = 0.5

    View Slide

  14. SMT Solvers should be robust!
    SMT Solver
    Access Policy
    Analysis
    Symbolic
    Execution
    Formal
    Veri
    fi
    cation
    Security Security

    14

    View Slide

  15. SMT Solvers should be robust!
    SMT Solver
    Symbolic
    Execution
    Access Policy
    nalysis
    Security
    Safety

    Formal
    Veri
    fi
    cation
    15

    View Slide

  16. Ensuring the Robustness of SMT Solvers
    16
    Proo
    f

    Certi
    fi
    cates
    Formally
    veri
    fi
    e
    d

    SMT solver
    Automated

    Testing

    View Slide

  17. Ensuring the Robustness of SMT Solvers
    17
    Proo
    f

    Certi
    fi
    cates
    Formally
    veri
    fi
    e
    d

    SMT solver
    Automated

    Testing

    View Slide

  18. Project Yin-Yang for SMT Solver Testing
    [Summary: 1,560 (total) / 1,061 (
    fi
    xed)]


    [Z3 bugs: 1,147 (total) / 779 (
    fi
    xed)]

    [CVC4 bugs: 413 (total) / 282 (
    fi
    xed)]


    [Bugs in default mode (Z3): 680 (total) / 479 (
    fi
    xed)]

    [Bugs in default mode (CVC4): 204 (total) / 147 (
    fi
    xed)]


    [Soundness bugs (Z3): 375 (total) / 228 (
    fi
    xed)]

    [Soundness bugs (CVC4): 71 (total) / 60 (
    fi
    xed)]

    July 2019 -

    View Slide

  19. Roadmap
    1. Introductio
    n

    2. Testing Technique
    s

    a) Semantic Fusio
    n

    b) Type-aware Operator Mutatio
    n

    c) Generative Type-Aware Operator Mutation


    3. Current and Future Directions
    19

    View Slide

  20. SMT-LIB language
    • Standard format for modern SMT solver
    s

    • LISP-dialec
    t

    • Declarative: no re-assignments of
    variable
    s

    • SMT-LIB is quite expressive
    20
    Variable Declarations
    Constraints
    Solver Query

    View Slide

  21. SMT-LIB language
    φ : x > 0 ∧ x < 1
    21

    View Slide

  22. SMT-LIB language
    φ : x > 0 ∧ x < 1 (declare-fun x () Real
    )

    (assert (and (> x 0)(< x 1))
    )

    (check-sat
    )

    22

    View Slide

  23. SMT-LIB language
    φ : x > 0 ∧ x < 1 (declare-fun x () Real
    )

    (assert (and (> x 0)(< x 1))
    )

    (check-sat
    )

    23

    View Slide

  24. SMT-LIB language
    φ : x > 0 ∧ x < 1 (declare-fun x () Real
    )

    (assert (and (> x 0)(< x 1))
    )

    (check-sat
    )

    24

    View Slide

  25. SMT-LIB language
    φ : x > 0 ∧ x < 1 (declare-fun x () Real
    )

    (assert (and (> x 0)(< x 1))
    )

    (check-sat
    )

    25

    View Slide

  26. Roadmap
    1. Introductio
    n

    2. Testing Technique
    s

    a) Semantic Fusio
    n

    b) Type-aware Operator Mutatio
    n

    c) Generative Type-Aware Operator Mutation


    3. Open Problems
    26

    View Slide

  27. Semantic Fusion
    φ1
    φ2
    27

    View Slide

  28. Semantic Fusion
    φconcat
    28

    View Slide

  29. Semantic Fusion
    φfused
    29

    View Slide

  30. Semantic Fusion
    φ1
    = x > 0 ∧ x > 1
    φ2
    = y < 0 ∧ y < 1
    SAT
    SAT
    30

    View Slide

  31. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    31

    View Slide

  32. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    32

    View Slide

  33. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    x = 2 y = − 2
    33

    View Slide

  34. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z
    34

    View Slide

  35. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    35

    View Slide

  36. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y Fusion Function
    36

    View Slide

  37. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    37

    View Slide

  38. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    x = z − y y = z − x
    38

    View Slide

  39. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    x = z − y y = z − x
    Inversion Functions
    39

    View Slide

  40. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    x = z − y y = z − x
    40

    View Slide

  41. Semantic Fusion
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φ1
    φ2
    φconcat
    = SAT
    z = x + y
    x = z − y y = z − x
    41

    View Slide

  42. Semantic Fusion
    (x > 0 ∧ (z − y) > 1) ∧ ((z − x) < 0 ∧ y < 1)
    z = x + y
    x = z − y y = z − x
    φ1
    φ2
    φfused
    =
    42

    View Slide

  43. Semantic Fusion
    (x > 0 ∧ (z − y) > 1) ∧ ((z − x) < 0 ∧ y < 1)
    z = x + y
    x = z − y y = z − x
    φfused
    = SAT
    43

    View Slide

  44. Semantic Fusion
    (x > 0 ∧ (z − y) > 1) ∧ ((z − x) < 0 ∧ y < 1)
    z = x + y
    x = z − y y = z − x
    φfused
    = SAT
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φconcat
    =
    x = 2 y = − 2
    44

    View Slide

  45. Semantic Fusion
    (x > 0 ∧ (z − y) > 1) ∧ ((z − x) < 0 ∧ y < 1)
    z = x + y
    x = z − y y = z − x
    φfused
    = SAT
    (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1)
    φconcat
    =
    x = 2 y = − 2
    z = x + y = 0
    45

    View Slide

  46. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w)
    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    SAT
    SAT
    46

    View Slide

  47. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w)
    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    47

    View Slide

  48. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    48

    View Slide

  49. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    49

    View Slide

  50. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    50

    View Slide

  51. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= x (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= y (- 1))))
    z = x * y
    51

    View Slide

  52. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= (div z y) (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= (div z x) (- 1))))
    z = x * y
    52

    View Slide

  53. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= (div z y) (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= (div z x) (- 1))))
    SAT
    53

    View Slide

  54. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= (div z y) (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= (div z x) (- 1))))
    SAT
    $ cvc4 example.smt
    2

    unsat
    54

    View Slide

  55. Semantic Fusion: an Example
    (declare-fun x () Int
    )

    (declare-fun w () Bool
    )

    (declare-fun y () Int
    )

    (declare-fun v () Bool
    )

    (declare-fun z () Int
    )

    (assert (= (div z y) (- 1))
    )

    (assert (= w (= x (- 1)))
    )

    (assert w
    )

    (assert (= v (not (= y (- 1))))
    )

    (assert (ite v false (= (div z x) (- 1))))
    SAT
    https://github.com/CVC4/CVC4/issues/3413
    $ cvc4 example.smt
    2

    unsat
    55

    View Slide

  56. Semantic Fusion
    φ2
    = y < 0 ∧ y > 1
    UNSAT
    UNSAT
    φ1
    = x > 1 ∧ x < 0
    56

    View Slide

  57. Semantic Fusion
    (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1)
    φ1
    φ2
    57

    View Slide

  58. Semantic Fusion
    (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1)
    φ1
    φ2
    φconcat
    = UNSAT
    58

    View Slide

  59. Semantic Fusion
    (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1)
    φ1
    φ2
    φconcat
    =
    z
    UNSAT
    59

    View Slide

  60. Semantic Fusion
    φ1
    φ2
    φconcat
    =
    z = x + y
    UNSAT
    (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1)
    60

    View Slide

  61. Semantic Fusion
    φ1
    φ2
    φconcat
    =
    z = x + y
    x = z − y y = z − x
    UNSAT
    (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1)
    61

    View Slide

  62. Semantic Fusion
    (x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1)
    z = x + y
    x = z − y y = z − x
    φ1
    φ2
    φfused
    =
    62

    View Slide

  63. Semantic Fusion
    φfused
    = SAT
    (x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1)
    63

    View Slide

  64. Semantic Fusion
    φfused
    = SAT
    x = 2 y = 2
    z = 0
    (x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1)
    64

    View Slide

  65. Semantic Fusion
    φfused
    = ((x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1)) ∧ z = x + y
    Fusion Constraint
    65

    View Slide

  66. Semantic Fusion
    φfused
    = ((x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1)) ∧ z = x + y
    UNSAT
    66

    View Slide

  67. Semantic Fusion: an Example
    (declare-fun x () Real)


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x))))
    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real)


    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0)))
    UNSAT
    UNSAT
    67

    View Slide

  68. Semantic Fusion: an Example
    (declare-fun x () Real)


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x))))
    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real)


    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0)))
    68

    View Slide

  69. Semantic Fusion: an Example
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0))))
    69

    View Slide

  70. (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0))))
    Semantic Fusion: an Example
    70

    View Slide

  71. (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0))))
    Semantic Fusion: an Example
    71

    View Slide

  72. Semantic Fusion: an Example
    z = x * y
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 x) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< y v) (>= w v)


    (< (/ w v ) 0) (> y 0))))
    72

    View Slide

  73. Semantic Fusion: an Example
    z = x * y
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 (/ z y)) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< (/ z x) v) (>= w v)


    (< (/ w v ) 0) (> (/ z x) 0))))
    (assert (= z (* x y))
    )

    (assert (= x (/ z y))
    )

    (assert (= y (/ z x)))
    73

    View Slide

  74. Semantic Fusion: an Example
    UNSAT
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 (/ z y)) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< (/ z x) v) (>= w v)


    (< (/ w v ) 0) (> (/ z x) 0)))
    )

    (assert (= z (* x y))
    )

    (assert (= x (/ z y))
    )

    (assert (= y (/ z x)))
    74

    View Slide

  75. Semantic Fusion: an Example
    UNSAT
    % z3 example.smt
    2

    sat
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 (/ z y)) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< (/ z x) v) (>= w v)


    (< (/ w v ) 0) (> (/ z x) 0)))
    )

    (assert (= z (* x y))
    )

    (assert (= x (/ z y))
    )

    (assert (= y (/ z x)))
    75

    View Slide

  76. Semantic Fusion: an Example
    UNSAT
    https://github.com/Z3Prover/z3/issues/2391
    % z3 example.smt
    2

    sat
    (declare-fun x () Real
    )

    (declare-fun y () Real)


    (declare-fun w () Real)


    (declare-fun v () Real
    )

    (declare-fun z () Real
    )

    (assert (or


    (assert (not (= (+ (+ 1.0 (/ z y)) 6.0)


    (+ 7.0 x)))
    )

    (assert (and (< (/ z x) v) (>= w v)


    (< (/ w v ) 0) (> (/ z x) 0)))
    )

    (assert (= z (* x y))
    )

    (assert (= x (/ z y))
    )

    (assert (= y (/ z x)))
    76

    View Slide

  77. Empirical Evaluation
    77

    View Slide

  78. Empirical Evaluation
    • Tool YinYang, our realization of Semantic Fusion
    78

    View Slide

  79. Empirical Evaluation
    • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019)
    79

    View Slide

  80. Empirical Evaluation
    • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019)

    • Bug reduction with C-Reduce
    80

    View Slide

  81. Empirical Evaluation
    • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019)

    • Bug reduction with C-Reduce

    • Bug reports on issue trackers of Z3 and CVC4
    81

    View Slide

  82. How many bugs can YinYang find?
    82

    View Slide

  83. How many bugs can YinYang find?
    83

    View Slide

  84. How many bugs can YinYang find?
    84

    View Slide

  85. How many bugs can YinYang find?
    85

    View Slide

  86. How many bugs can YinYang find?
    86

    View Slide

  87. How many bugs can YinYang find?
    87

    View Slide

  88. Significance of the bug finding results
    88

    View Slide

  89. Significance of the bug finding results
    89

    View Slide

  90. Significance of the bug finding results
    90

    View Slide

  91. Significance of the bug finding results
    YinYang found 24 in Z3, 5 in CVC4 in 4 months
    91

    View Slide

  92. Significance of the bug finding results
    92

    View Slide

  93. Significance of the bug finding results
    93

    View Slide

  94. Significance of the bug finding results
    3.5+ years
    old
    94

    View Slide

  95. Significance of the bug finding results
    1.5+ years
    old
    95

    View Slide

  96. Significance of the bug finding results
    YinYang found longstanding soundness bugs
    1.5+ years
    old
    3.5+ years
    old
    96

    View Slide

  97. Is Semantic Fusion necessary?
    97

    View Slide

  98. Is Semantic Fusion necessary?
    𝜑
    1
    ConcatFuzz
    𝜑
    2
    SAT SAT

    𝜑
    1
    𝜑
    2

    UNSAT UNSAT
    98

    View Slide

  99. Is Semantic Fusion necessary?
    𝜑
    1
    ConcatFuzz can only retrigger 5/50 bugs
    ConcatFuzz
    𝜑
    2
    SAT SAT

    𝜑
    1
    𝜑
    2

    UNSAT UNSAT
    99

    View Slide

  100. Is Semantic Fusion necessary?
    100

    View Slide

  101. Is Semantic Fusion necessary?
    101

    View Slide

  102. Is Semantic Fusion necessary?
    102

    View Slide

  103. Is Semantic Fusion necessary?
    2800 LoC
    480 LoC
    103

    View Slide

  104. Is Semantic Fusion necessary?
    YinYang consistently achieves higher coverage
    2800 LoC
    480 LoC
    104

    View Slide

  105. Z3 #2376
    % cat formula.smt2
    (declare-fun a () Real)
    (declare-fun b () Real)
    (declare-fun c () Real)
    (declare-fun d () Real)
    (declare-fun j () Real)
    (declare-fun e () Real)
    (assert (not (exists ((f Real))
    (=> (and (< (/ 0 0) c) (< (/ 0 (* 2.0 b))
    d))(= (= 0.0 a) (not (=> (<= f a) (<= e
    j))))))))
    (check-sat)
    % cvc4 formula.smt2
    unsat


    %z3 formula.smt2
    sat
    105

    View Slide

  106. Z3 #2376
    % cat formula.smt2
    (declare-fun a () Real)
    (declare-fun b () Real)
    (declare-fun c () Real)
    (declare-fun d () Real)
    (declare-fun j () Real)
    (declare-fun e () Real)
    (assert (not (exists ((f Real))
    (=> (and (< (/ 0 0) c) (< (/ 0 (* 2.0 b))
    d))(= (= 0.0 a) (not (=> (<= f a) (<= e
    j))))))))
    (check-sat)
    % cvc4 formula.smt2
    unsat


    %z3 formula.smt2
    sat
    106

    View Slide

  107. CVC4 #3412
    % cat formula.smt2
    (declare-fun a () Int)
    (declare-fun b () Int)
    (assert (= (div a b) (- 1)))
    (check-sat)
    % z3 formula.smt2
    sat


    % cvc4 formula.smt2
    unsat


    107

    View Slide

  108. CVC4 #3412
    % cat formula.smt2
    (declare-fun a () Int)
    (declare-fun b () Int)
    (assert (= (div a b) (- 1)))
    (check-sat)
    % z3 formula.smt2
    sat


    % cvc4 formula.smt2
    unsat


    108

    View Slide

  109. CVC4 #3412
    % cat formula.smt2
    (declare-fun a () Int)
    (declare-fun b () Int)
    (assert (= (div a b) (- 1)))
    (check-sat)
    % z3 formula.smt2
    sat


    % cvc4 formula.smt2
    unsat


    109

    View Slide

  110. Z3 #4153
    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (assert (=
    (str.++ (str.substr "1" 0 (str.len a))
    "0") b))
    (assert (< (str.to.int b) 0))
    (check-sat
    )

    % z3-4.8-7 formula.smt2
    unsat
    % z3 formula.smt2
    sa
    t

    110

    View Slide

  111. Z3 #4153
    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (assert (=
    (str.++ (str.substr "1" 0 (str.len a))
    "0") b))
    (assert (< (str.to.int b) 0))
    (check-sat
    )

    % z3-4.8-7 formula.smt2
    unsat
    % z3 formula.smt2
    sa
    t

    111

    View Slide

  112. CVC4 #3217
    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (declare-fun c () String)
    (declare-fun d () String)
    (assert
    (or (not (= (str.suffixof "B"
    (str.replace "A" b "B"))
    (= ( str.substr a 0 (str.len b)) "A")))
    (not (= (not (= c "A")) (str.suffixof "A"
    (str.replace "A" c "B"))))))
    (assert (= a (str.++ (str.++ b "") d)))
    (check-sat)


    % z3 formula.smt2
    unsat
    % cvc4 formula.smt2
    sa
    t

    112

    View Slide

  113. CVC4 #3217
    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (declare-fun c () String)
    (declare-fun d () String)
    (assert
    (or (not (= (str.suffixof "B"
    (str.replace "A" b "B"))
    (= ( str.substr a 0 (str.len b)) "A")))
    (not (= (not (= c "A")) (str.suffixof "A"
    (str.replace "A" c "B"))))))
    (assert (= a (str.++ (str.++ b "") d)))
    (check-sat)


    % z3 formula.smt2
    unsat
    % cvc4 formula.smt2
    sa
    t

    113

    View Slide

  114. Z3 #2618 & CVC4 #3357
    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (declare-fun c () String)
    (assert (str.in.re c
    (re.* (re.union (str.to.re "aa")
    (str.to.re "")))))
    (assert (= 0 (str.to.int
    (str.replace a b (str.at a
    (str.len a))))))
    (assert (= a (str.++ b c)))
    (check-sat)


    % cvc4 formula.smt2
    unsat
    % z3 formula.smt2
    sa
    t

    114

    View Slide

  115. Z3 #2618 & CVC4 #3357
    % z3 unreduced.smt2
    sa
    t

    % cvc4 unreduced.smt2
    sa
    t

    % cat formula.smt2
    (declare-fun a () String)
    (declare-fun b () String)
    (declare-fun c () String)
    (assert (str.in.re c
    (re.* (re.union (str.to.re "aa")
    (str.to.re "")))))
    (assert (= 0 (str.to.int
    (str.replace a b (str.at a
    (str.len a))))))
    (assert (= a (str.++ b c)))
    (check-sat)


    % cvc4 formula.smt2
    unsat
    % z3 formula.smt2
    sa
    t

    Z3 and CVC4 are both unsound
    on the unreduced test!
    115

    View Slide

  116. Roadmap
    1. Introductio
    n

    2. Testing Technique
    s

    a) Semantic Fusio
    n

    b) Type-aware Operator Mutatio
    n

    c) Generative Type-Aware Operator Mutation


    3. Current and Future Directions
    116

    View Slide

  117. Type-aware operator mutation
    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    117

    View Slide

  118. Type-aware operator mutation
    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    118

    View Slide

  119. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    119

    View Slide

  120. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    Int Int Int Int
    120

    View Slide

  121. (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    Int Int Int Int
    =

    >

    <

    >
    =

    <
    =

    Type-aware operator mutation
    Operator candidates:
    121
    121

    View Slide

  122. (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    Int Int Int Int
    Type-aware operator mutation
    Operator candidates:
    =

    >

    <

    >
    =

    <
    =

    122

    View Slide

  123. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    123

    View Slide

  124. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > bug.smt2
    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cvc4 bug.smt
    2

    unsa
    t

    $ z3 bug.smt
    2

    sat
    124

    View Slide

  125. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > bug.smt2
    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cvc4 bug.smt
    2

    unsa
    t

    $ z3 bug.smt
    2

    sat ✘

    125

    View Slide

  126. Type-aware operator mutation
    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (= (* 2 b) a)))
    )

    (check-sat
    )

    $ cat > bug.smt2
    $ cat > formula.smt
    2

    (assert (forall ((a Int)
    )

    (exist ((b Int)
    )

    (distinct (* 2 b) a)))
    )

    (check-sat
    )

    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cvc4 bug.smt
    2

    unsa
    t

    $ z3 bug.smt
    2

    sat
    https://github.com/Z3Prover/z3/issues/3973


    126

    View Slide

  127. Type-aware operator mutation chain
    127

    View Slide

  128. Type-aware operator mutation chain
    (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    128

    View Slide

  129. (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    Type-aware operator mutation chain
    129

    View Slide

  130. (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    Type-aware operator mutation chain
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (* a a ) 1))
    (check-sat)
    130

    View Slide

  131. (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    Type-aware operator mutation chain
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (* a a ) 1))
    (check-sat)
    131

    View Slide

  132. (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    Type-aware operator mutation chain
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (* a a ) 1))
    (check-sat)
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)
    132

    View Slide

  133. (declare-fun a () Real
    )

    (assert (> (/ (* 2 a) a) (* a a ) 1))


    (check-sat)
    Type-aware operator mutation chain
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (* a a ) 1))
    (check-sat)
    (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)

    133

    View Slide

  134. (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)
    Type-aware operator mutation chain
    134

    View Slide

  135. (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)
    Type-aware operator mutation chain
    $ cat > bug.smt2
    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat
    135

    View Slide

  136. (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)
    Type-aware operator mutation chain
    $ cat > bug.smt2
    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat ✘

    https://github.com/Z3Prover/z3/issues/2715
    136

    View Slide

  137. (declare-fun a () Real
    )

    (assert (= (/ (* 2 a) a) (/ a a ) 1))
    (check-sat)
    Type-aware operator mutation chain
    $ cat > bug.smt2
    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat ✘

    https://github.com/Z3Prover/z3/issues/2715
    137

    View Slide

  138. Empirical Evaluation
    138

    View Slide

  139. Empirical Evaluation
    • Tool: OpFuzz

    • Bug hunting: Sep 2019 - Oct 2020

    • Testing targets: Z3 and CVC4

    • Seeds: SMT-LIB benchmarks (300k+ formulas)
    139

    View Slide

  140. Empirical Evaluation
    • Tool: OpFuzz

    • Bug hunting: Sep 2019 - Oct 2020

    • Testing targets: Z3 and CVC4

    • Seeds: SMT-LIB benchmarks (300k+ formulas)
    140

    View Slide

  141. Empirical Evaluation
    • Tool: OpFuzz

    • Bug hunting: Sep 2019 - Oct 2020

    • Testing targets: Z3 and CVC4

    • Seeds: SMT-LIB benchmarks (300k+ formulas)
    141

    View Slide

  142. Empirical Evaluation
    • Tool: OpFuzz

    • Bug hunting: Sep 2019 - Oct 2020

    • Testing targets: Z3 and CVC4

    • Seeds: SMT-LIB benchmarks (300k+ formulas)
    142

    View Slide

  143. Bug Findings
    Status Z3 CVC4 Total
    Reported 811 281 1,092
    Con
    fi
    rmed 578 241 819
    Fixed 521 164 685
    Duplicate 85 18 106
    Bug status as of 30 Oct 2020
    143

    View Slide

  144. Bug Findings
    Status Z3 CVC4 Total
    Reported 811 281 1,092
    Con
    fi
    rmed 578 241 819
    Fixed 521 164 685
    Duplicate 85 18 106
    Bug status as of 30 Oct 2020
    144

    View Slide

  145. Bug Findings
    Status Z3 CVC4 Total
    Reported 811 281 1,092
    Con
    fi
    rmed 578 241 819
    Fixed 521 164 685
    Duplicate 85 18 106
    Bug status as of 30 Oct 2020
    145

    View Slide

  146. Bug Findings
    Status Z3 CVC4 Total
    Reported 811 281 1,092
    Con
    fi
    rmed 578 241 819
    Fixed 521 164 685
    Duplicate 85 18 106
    Bug status as of 30 Oct 2020
    146

    View Slide

  147. Bug Findings
    Status Z3 CVC4 Total
    Soundness 157 27 184
    Invalid model 83 19 102
    Crash 316 185 501
    Others 22 10 32
    Types of con
    fi
    rmed bugs
    147

    View Slide

  148. Bug Findings
    Status Z3 CVC4 Total
    Soundness 157 27 184
    Invalid model 83 19 102
    Crash 316 185 501
    Others 22 10 32
    Types of con
    fi
    rmed bugs
    148

    View Slide

  149. Bug Findings
    Status Z3 CVC4 Total
    Soundness 157 27 184
    Invalid model 83 19 102
    Crash 316 185 501
    Others 22 10 32
    Types of con
    fi
    rmed bugs
    149

    View Slide

  150. Bug Findings
    Logic Z3 CVC4 Total
    Soundness 157 27 184
    Invalid model 83 19 102
    Crash 316 185 501
    Others 22 10 32
    Types of con
    fi
    rmed bugs
    150

    View Slide

  151. Bug Findings
    Number of options for triggering the bugs
    #Options Z3 CVC4 Total
    Default 388 101 489
    1 109 67 176
    2 45 31 76
    >= 3 36 42 78
    151

    View Slide

  152. Bug Findings
    Number of options for triggering the bugs
    #Options Z3 CVC4 Total
    Default 388 101 489
    1 109 67 176
    2 45 31 76
    >= 3 36 42 78
    152

    View Slide

  153. Bug Findings
    Number of options for triggering the bugs
    #Options Z3 CVC4 Total
    Default 388 101 489
    1 109 67 176
    2 45 31 76
    >= 3 36 42 78
    153

    View Slide

  154. Bugs in Z3 Bugs in CVC4
    Approach soundness all soundness all
    StringFuzz 0 0 - -
    BanditFuzz 0 0 - -
    Bugariu et al. 1 3 0 0
    YinYang 24 36 5 8
    STORM 17 21 0 0
    OpFuzz 114 316 11 185
    Comparison to Previous Approaches
    Con
    fi
    rmed bugs in the default modes of the solvers
    154

    View Slide

  155. Comparison to Previous Approaches
    Con
    fi
    rmed bugs in the default modes of the solvers
    Bugs in Z3 Bugs in CVC4
    Approach soundness all soundness all
    StringFuzz 0 0 - -
    BanditFuzz 0 0 - -
    Bugariu et al. 1 3 0 0
    YinYang 24 36 5 8
    STORM 17 21 0 0
    OpFuzz 114 316 11 185
    155

    View Slide

  156. Comparison to Previous Approaches
    OpFuzz found many more soundness bugs in Z3 and
    CVC4’s default modes than all previous approaches
    Bugs in Z3 Bugs in CVC4
    Approach soundness all soundness all
    StringFuzz 0 0 - -
    BanditFuzz 0 0 - -
    Bugariu et al. 1 3 0 0
    YinYang 24 36 5 8
    STORM 17 21 0 0
    OpFuzz 114 316 11 185
    156

    View Slide

  157. Z3 Soundness Bug #2832
    $ cat bug.smt
    2

    (declare-const a (_ BitVec 8)
    )

    (declare-const b (_ BitVec 8))


    (declare-const c (_ BitVec 8))
    (assert (= (bvxnor a b c
    )

    (bvxnor (bvxnor a b) c)))


    (check-sat
    )

    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat ✘

    https://github.com/Z3Prover/z3/issues/2832
    157

    View Slide

  158. Z3 Soundness Bug #2832
    $ cat bug.smt
    2

    (declare-const a (_ BitVec 8)
    )

    (declare-const b (_ BitVec 8))


    (declare-const c (_ BitVec 8))
    (assert (= (bvxnor a b c)
    (bvxnor (bvxnor a b) c)))


    (check-sat
    )

    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat ✘

    https://github.com/Z3Prover/z3/issues/2832
    158

    View Slide

  159. Z3 Soundness Bug #2832
    (bvxnor true true true) = (not (bvxor true true true)
    )

    (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true)))


    (bvnxor true true) = (not (bvxor false true))
    )

    true false 




    (bvxnor a b c) (not (bvxor a b c))

    159

    View Slide

  160. Z3 Soundness Bug #2832
    (bvxnor true true true) = (not (bvxor true true true)
    )

    (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true)))


    (bvnxor true true) = (not (bvxor false true))
    )

    true false 




    (bvxnor a b c) (not (bvxor a b c))

    160

    View Slide

  161. Z3 Soundness Bug #2832
    (bvxnor true true true) = (not (bvxor true true true)
    )

    (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true))


    (bvnxor true true) = (not (bvxor false true))
    )

    true false 




    (bvxnor a b c) (not (bvxor a b c))

    161

    View Slide

  162. Z3 Soundness Bug #2832
    (bvxnor true true true) = (not (bvxor true true true)
    )

    (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true))


    (bvnxor true true) = (not (bvxor false true)
    )

    true false 




    (bvxnor a b c) (not (bvxor a b c))

    162

    View Slide

  163. Z3 Soundness Bug #2832
    (bvxnor true true true) = (not (bvxor true true true)
    )

    (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true))


    (bvnxor true true) = (not (bvxor false true)
    )

    true false 




    (bvxnor a b c) (not (bvxor a b c))

    163

    View Slide

  164. Z3 Soundness Bug #2832
    $ cat bug.smt
    2

    (declare-const a (_ BitVec 8)
    )

    (declare-const b (_ BitVec 8))


    (declare-const c (_ BitVec 8))
    (assert (= (bvxnor a b c)
    (bvxnor (bvxnor a b) c)))


    (check-sat
    )

    $ cvc4 bug.smt
    2

    sa
    t

    $ z3 bug.smt
    2

    unsat ✘

    https://github.com/Z3Prover/z3/issues/2832
    Root Cause: Unsound rewriter
    164

    View Slide

  165. Z3 Soundness Bug #2830
    $ cat bug.smt
    2

    (declare-fun a () Int
    )

    (declare-fun b (Int) Bool
    )

    (assert (b 0))


    (push
    )

    (assert (distinct true (= a 0) (not (b 0))))


    (check-sat
    )

    $ cvc4 bug.smt
    2

    unsa
    t

    $ z3 bug.smt
    2

    sat ✘

    https://github.com/Z3Prover/z3/issues/2830
    165

    View Slide

  166. Z3 Soundness Bug #2830
    $ cat bug.smt
    2

    (declare-fun a () Int
    )

    (declare-fun b (Int) Bool
    )

    (assert (b 0))


    (push
    )

    (assert (distinct true (= a 0) (not (b 0))))


    (check-sat
    )

    $ cvc4 bug.smt
    2

    unsa
    t

    $ z3 bug.smt
    2

    sat ✘

    https://github.com/Z3Prover/z3/issues/2830
    Root Cause: Last argument
    omitted in distinct operator
    166

    View Slide

  167. CVC4 Soundness Bug #3497
    $ cat bug.smt
    2

    (declare-fun x () String)


    (declare-fun y () String)


    (assert (= (str.indexof x y 1)


    (str.len x)))


    (assert (str.contains x y))


    (check-sat
    )

    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat ✘

    https://github.com/CVC4/CVC4/issues/3497
    167

    View Slide

  168. CVC4 Soundness Bug #3497
    $ cat bug.smt
    2

    (declare-fun x () String)


    (declare-fun y () String)


    (assert (= (str.indexof x y 1)


    (str.len x)))


    (assert (str.contains x y))


    (check-sat
    )

    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat ✘

    https://github.com/CVC4/CVC4/issues/3497
    168

    View Slide

  169. CVC4 Soundness Bug #3497
    $ cat bug.smt
    2

    (declare-fun x () String)


    (declare-fun y () String)


    (assert (= (str.indexof x y 1)


    (str.len x)))


    (assert (str.contains x y))


    (check-sat
    )

    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat ✘

    https://github.com/CVC4/CVC4/issues/3497
    169

    View Slide

  170. CVC4 Soundness Bug #4469
    $ cat bug.smt
    2

    (set-logic QF_AUFBVLIA
    )

    (declare-fun a () Int)


    (declare-fun b (Int) Int)


    (assert (distinct (b a)


    (b (b a))))


    (check-sat
    )

    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat ✘

    https://github.com/CVC4/CVC4/issues/4469
    170

    View Slide

  171. CVC4 Soundness Bug #4469
    $ cat bug.smt
    2

    (set-logic QF_AUFBVLIA
    )

    (declare-fun a () Int)


    (declare-fun b (Int) Int)


    (assert (distinct (b a)


    (b (b a))))


    (check-sat
    )

    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat ✘

    https://github.com/CVC4/CVC4/issues/4469
    171

    View Slide

  172. CVC4 Soundness Bug #3475
    $ cat bug.smt
    2

    (set-logic ALL)

    (declare-fun x () Real
    )

    (assert (< x 0)
    )

    (assert (not (= (/ (sqrt x) (sqrt x)) x))
    )

    (check-sat
    )



    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat

    https://github.com/CVC4/CVC4/issues/3475


    172

    View Slide

  173. Roadmap
    1. Introductio
    n

    2. Testing Technique
    s

    a) Semantic Fusio
    n

    b) Type-aware Operator Mutatio
    n

    c) Generative Type-Aware Operator Mutation


    3. Current and Future Directions
    173

    View Slide

  174. Generative Type-Aware Mutation
    • Idea: Extend OpFuzz

    • Expressions

    • Support Generation: Control the size of Formula

    • Formula can grow & shrink
    174

    View Slide

  175. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    Generative Type-Aware Mutation
    175
    Generative Type-Aware Mutation

    View Slide

  176. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    Bool: (> (- (str.to_int (str.++ x x))) 0)
    )

    Int : 0, (- (str.to_int (str.++ x x))) 0),


    String : (str.++ x x),
    x

    176
    Generative Type-Aware Mutation

    View Slide

  177. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    177
    Generative Type-Aware Mutation

    View Slide

  178. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    String
    178
    Generative Type-Aware Mutation

    View Slide

  179. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    String
    Bool: (> (- (str.to_int (str.++ x x))) 0)
    )

    Int : 0, (- (str.to_int (str.++ x x))) 0),


    String : (str.++ x x),
    x

    179
    Generative Type-Aware Mutation

    View Slide

  180. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    String
    Bool: (> (- (str.to_int (str.++ x x))) 0)
    )

    Int : 0, (- (str.to_int (str.++ x x))) 0),


    String : (str.++ x x),
    x

    Operator candidates:

    str.from_int
    str.replace
    str.++
    ...
    180
    Generative Type-Aware Mutation

    View Slide

  181. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ str.from_int x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    String
    Bool: (> (- (str.to_int (str.++ x x))) 0)
    )

    Int : 0,, (- (str.to_int (str.++ x x))) 0),


    String : (str.++ x x),
    x

    (str.from_int Int String)
    String
    181
    Generative Type-Aware Mutation

    View Slide

  182. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat mutant.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ (str.from_int 0) x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    Sat
    182
    Generative Type-Aware Mutation

    View Slide

  183. $ cat formula.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ x x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    sat
    $ cat mutant.smt
    2

    (declare-fun x () String)


    (assert (> (- (str.to_int


    (str.++ (str.from_int 0) x))) 0))


    (check-sat)


    $ cvc4 formula.smt
    2

    sa
    t

    $ z3 formula.smt
    2

    unsat
    183
    Generative Type-Aware Mutation

    View Slide

  184. More Bugs
    Status Z3 CVC4 Total
    Reported 177 60 237
    Con
    fi
    rmed 135 54 189
    Fixed 132 44 176
    Duplicate 9 5 14
    Bug status as of 30 Sep 2021
    184

    View Slide

  185. Roadmap
    1. Introductio
    n

    2. Testing Technique
    s

    a) Semantic Fusio
    n

    b) Type-aware Operator Mutatio
    n

    c) Generative Type-Aware Operator Mutation


    3. Current and Future Directions
    185

    View Slide

  186. Current and Future Directions
    Yin-Yang [PLDI ’20]
    OpFuzz [OOPSLA ’20]
    TypeFuzz [OOPSLA ’21]
    Correctness
    Robust & Fast SMT
    Solvers
    Performance
    186

    View Slide

  187. Current and Future Directions
    Yin-Yang [PLDI ’20]
    OpFuzz [OOPSLA ’20]
    TypeFuzz [OOPSLA ’21]
    Correctness
    Robust & Fast SMT
    Solvers
    Performance
    187

    View Slide

  188. “Testing can only show the presence of errors,
    not their absence.” 

    - Edsger W. Dijkstra
    188

    View Slide

  189. Bounded Guarantees for SMT Solvers
    Software
    Bug triggers
    Guarantees
    Bound
    Grammar
    Fuzzer
    189

    View Slide

  190. Current and Future Directions
    Yin-Yang [PLDI ’20]
    OpFuzz [OOPSLA ’20]
    TypeFuzz [OOPSLA ’21]
    Correctness
    Robust & Fast SMT
    Solvers
    Performance
    190

    View Slide

  191. SMT Solver Flags
    191

    View Slide

  192. SMT Solver Flags
    Z3 has 300+
    fl
    ags, all of which
    could in
    fl
    uence its performance
    192

    View Slide

  193. Problem Statement
    Given:

    Solver

    Flags
    Benchmarks
    Wanted:

    Understand whether Flags are pair wisely independent
    193

    View Slide

  194. Summary @DominikWinterer
    wintered.github.io
    https://testsmt.github.io/
    194

    View Slide

  195. Zhendong Su
    Prof@ETH Zurich
    Chengyu Zhang
    Postdoc@ETH Zurich
    Collaborators
    Jiwon Park
    Intern@ETH Zurich
    ➜ Ph.D. student@UC Berkeley
    195

    View Slide

  196. Sponsors
    196

    View Slide

  197. Bonus Slides

    View Slide

  198. CVC4 Soundness Bug #3475
    $ cat bug.smt
    2

    (set-logic ALL)

    (declare-fun x () Real
    )

    (assert (< x 0)
    )

    (assert (not (= (/ (sqrt x) (sqrt x)) x))
    )

    (check-sat
    )



    $ z3 bug.smt
    2

    sa
    t

    $ cvc4 bug.smt
    2

    unsat

    https://github.com/CVC4/CVC4/issues/3475


    Formula is satisfiable


    negative x = -1


    (/ sqrt(-1) sqrt(-1)) = 1
    198

    View Slide

  199. CVC4 Soundness Bug #3475
    Simplification:

    (sqrt x) = choice real y s.t. x*x = y


    199

    View Slide

  200. CVC4 Soundness Bug #3475
    Simplification:

    (sqrt x) = choice real y s.t. x*x = y


    Problem: Inadmissible for negative y (since no real x exists)

    200

    View Slide