Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Finding 1,500 bugs in the SMT Solvers Z3 and CVC5

Finding 1,500 bugs in the SMT Solvers Z3 and CVC5

Satisfiability Modulo Theory (SMT) solvers are essential tools for many advances in programming languages and formal methods. Examples include symbolic execution engines, software model checkers, and program verifiers. SMT solvers' robustness is crucial---Soundness bugs SMT solvers can invalidate client applications' results and lead to disasters in safety-critical or security-critical domains. We ran a large-scale testing campaign to put Z3 and CVC4/5's robustness to the test: As of today, We have found 1,500+ unique bugs in Z3 and CVC4--1,100+ have already been fixed by the developers and 400+ are critical soundness bugs. This talk covers the techniques used and lessons learned from this campaign. Specifically, we will discuss: (1) Semantic Fusion, a general, metamorphic testing methodology, (2) Type-Aware Mutation, a highly effective testing technique for testing SMT solvers, and (3) outline open challenges in making SMT solvers (and other formal methods tools) more correct and faster.

Dominik Winterer

June 24, 2022
Tweet

Other Decks in Research

Transcript

  1. Finding 1,500+ Bugs in the SMT Solver s Z3 and

    CVC5 Dominik Wintere r ETH Zurich
 
 Talk at Paderborn Universit y Jun 21st, 2022 @DominikWinterer wintered.github.io
  2. SMT Problem φ : x > 0 ∧ x <

    0
  3. SMT Problem UNSAT φ : x > 0 ∧ x

    < 0
  4. SMT Problem φ : x > 0 ∧ x <

    1
  5. SMT Problem SAT φ : x > 0 ∧ x

    < 1
  6. SMT Problem SAT φ : x > 0 ∧ x

    < 1 x = 0.5
  7. SMT Solver SMT Solvers φ : x > 0 ∧

    x < 1
  8. SAT SMT Solver SMT Solvers φ : x > 0

    ∧ x < 1 8
  9. SMT Background Theories 9 ArrayEx FixedSizeBitVectors Core FloatingPoints Ints Reals

    Real_Ints Strings
  10. SMT Solvers: Important Software Foundations SMT Solver Formal veri fi

    cation Symbolic Execution Access Policy Analysis Security Safety … 10
  11. SMT Solvers: Important Software Foundations Boogie 11 ALT Ergo …

    … Security Safety …
  12. SMT Solver UNSAT Bug in an SMT Solver φ :

    x > 0 ∧ x < 1 12
  13. SMT Solver UNSAT Bug in an SMT Solver φ :

    x > 0 ∧ x < 1 13 x = 0.5
  14. SMT Solvers should be robust! SMT Solver Access Policy Analysis

    Symbolic Execution Formal Veri fi cation Security Security … 14
  15. SMT Solvers should be robust! SMT Solver Symbolic Execution Access

    Policy nalysis Security Safety … Formal Veri fi cation 15
  16. Ensuring the Robustness of SMT Solvers 16 Proo f Certi

    fi cates Formally veri fi e d SMT solver Automated
 Testing
  17. Ensuring the Robustness of SMT Solvers 17 Proo f Certi

    fi cates Formally veri fi e d SMT solver Automated
 Testing
  18. Project Yin-Yang for SMT Solver Testing [Summary: 1,560 (total) /

    1,061 ( fi xed)]
 
 [Z3 bugs: 1,147 (total) / 779 ( fi xed)]
 [CVC4 bugs: 413 (total) / 282 ( fi xed)]
 
 [Bugs in default mode (Z3): 680 (total) / 479 ( fi xed)]
 [Bugs in default mode (CVC4): 204 (total) / 147 ( fi xed)]
 
 [Soundness bugs (Z3): 375 (total) / 228 ( fi xed)]
 [Soundness bugs (CVC4): 71 (total) / 60 ( fi xed)]
 July 2019 -
  19. Roadmap 1. Introductio n 2. Testing Technique s a) Semantic

    Fusio n b) Type-aware Operator Mutatio n c) Generative Type-Aware Operator Mutation 3. Current and Future Directions 19
  20. SMT-LIB language • Standard format for modern SMT solver s

    • LISP-dialec t • Declarative: no re-assignments of variable s • SMT-LIB is quite expressive 20 Variable Declarations Constraints Solver Query
  21. SMT-LIB language φ : x > 0 ∧ x <

    1 21
  22. SMT-LIB language φ : x > 0 ∧ x <

    1 (declare-fun x () Real ) (assert (and (> x 0)(< x 1)) ) (check-sat ) 22
  23. SMT-LIB language φ : x > 0 ∧ x <

    1 (declare-fun x () Real ) (assert (and (> x 0)(< x 1)) ) (check-sat ) 23
  24. SMT-LIB language φ : x > 0 ∧ x <

    1 (declare-fun x () Real ) (assert (and (> x 0)(< x 1)) ) (check-sat ) 24
  25. SMT-LIB language φ : x > 0 ∧ x <

    1 (declare-fun x () Real ) (assert (and (> x 0)(< x 1)) ) (check-sat ) 25
  26. Roadmap 1. Introductio n 2. Testing Technique s a) Semantic

    Fusio n b) Type-aware Operator Mutatio n c) Generative Type-Aware Operator Mutation 3. Open Problems 26
  27. Semantic Fusion φ1 φ2 27

  28. Semantic Fusion φconcat 28

  29. Semantic Fusion φfused 29

  30. Semantic Fusion φ1 = x > 0 ∧ x >

    1 φ2 = y < 0 ∧ y < 1 SAT SAT 30
  31. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 31
  32. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT 32
  33. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT x = 2 y = − 2 33
  34. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z 34
  35. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y 35
  36. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y Fusion Function 36
  37. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y 37
  38. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y x = z − y y = z − x 38
  39. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y x = z − y y = z − x Inversion Functions 39
  40. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y x = z − y y = z − x 40
  41. Semantic Fusion (x > 0 ∧ x > 1) ∧

    (y < 0 ∧ y < 1) φ1 φ2 φconcat = SAT z = x + y x = z − y y = z − x 41
  42. Semantic Fusion (x > 0 ∧ (z − y) >

    1) ∧ ((z − x) < 0 ∧ y < 1) z = x + y x = z − y y = z − x φ1 φ2 φfused = 42
  43. Semantic Fusion (x > 0 ∧ (z − y) >

    1) ∧ ((z − x) < 0 ∧ y < 1) z = x + y x = z − y y = z − x φfused = SAT 43
  44. Semantic Fusion (x > 0 ∧ (z − y) >

    1) ∧ ((z − x) < 0 ∧ y < 1) z = x + y x = z − y y = z − x φfused = SAT (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1) φconcat = x = 2 y = − 2 44
  45. Semantic Fusion (x > 0 ∧ (z − y) >

    1) ∧ ((z − x) < 0 ∧ y < 1) z = x + y x = z − y y = z − x φfused = SAT (x > 0 ∧ x > 1) ∧ (y < 0 ∧ y < 1) φconcat = x = 2 y = − 2 z = x + y = 0 45
  46. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w) (declare-fun y () Int ) (declare-fun v () Bool ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) SAT SAT 46
  47. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w) (declare-fun y () Int ) (declare-fun v () Bool ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) 47
  48. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) 48
  49. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) 49
  50. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) 50
  51. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= x (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= y (- 1)))) z = x * y 51
  52. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= (div z y) (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= (div z x) (- 1)))) z = x * y 52
  53. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= (div z y) (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= (div z x) (- 1)))) SAT 53
  54. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= (div z y) (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= (div z x) (- 1)))) SAT $ cvc4 example.smt 2 unsat 54
  55. Semantic Fusion: an Example (declare-fun x () Int ) (declare-fun

    w () Bool ) (declare-fun y () Int ) (declare-fun v () Bool ) (declare-fun z () Int ) (assert (= (div z y) (- 1)) ) (assert (= w (= x (- 1))) ) (assert w ) (assert (= v (not (= y (- 1)))) ) (assert (ite v false (= (div z x) (- 1)))) SAT https://github.com/CVC4/CVC4/issues/3413 $ cvc4 example.smt 2 unsat 55
  56. Semantic Fusion φ2 = y < 0 ∧ y >

    1 UNSAT UNSAT φ1 = x > 1 ∧ x < 0 56
  57. Semantic Fusion (x > 1 ∧ x < 0) ∨

    (y < 0 ∧ y > 1) φ1 φ2 57
  58. Semantic Fusion (x > 1 ∧ x < 0) ∨

    (y < 0 ∧ y > 1) φ1 φ2 φconcat = UNSAT 58
  59. Semantic Fusion (x > 1 ∧ x < 0) ∨

    (y < 0 ∧ y > 1) φ1 φ2 φconcat = z UNSAT 59
  60. Semantic Fusion φ1 φ2 φconcat = z = x +

    y UNSAT (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1) 60
  61. Semantic Fusion φ1 φ2 φconcat = z = x +

    y x = z − y y = z − x UNSAT (x > 1 ∧ x < 0) ∨ (y < 0 ∧ y > 1) 61
  62. Semantic Fusion (x > 1 ∧ (z − y) <

    0) ∨ ((z − x) < 0 ∧ y > 1) z = x + y x = z − y y = z − x φ1 φ2 φfused = 62
  63. Semantic Fusion φfused = SAT (x > 1 ∧ (z

    − y) < 0) ∨ ((z − x) < 0 ∧ y > 1) 63
  64. Semantic Fusion φfused = SAT x = 2 y =

    2 z = 0 (x > 1 ∧ (z − y) < 0) ∨ ((z − x) < 0 ∧ y > 1) 64
  65. Semantic Fusion φfused = ((x > 1 ∧ (z −

    y) < 0) ∨ ((z − x) < 0 ∧ y > 1)) ∧ z = x + y Fusion Constraint 65
  66. Semantic Fusion φfused = ((x > 1 ∧ (z −

    y) < 0) ∨ ((z − x) < 0 ∧ y > 1)) ∧ z = x + y UNSAT 66
  67. Semantic Fusion: an Example (declare-fun x () Real) (assert (not

    (= (+ (+ 1.0 x) 6.0) (+ 7.0 x)))) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0))) UNSAT UNSAT 67
  68. Semantic Fusion: an Example (declare-fun x () Real) (assert (not

    (= (+ (+ 1.0 x) 6.0) (+ 7.0 x)))) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0))) 68
  69. Semantic Fusion: an Example (declare-fun x () Real ) (declare-fun

    y () Real) (declare-fun w () Real) (declare-fun v () Real ) (assert (or (assert (not (= (+ (+ 1.0 x) 6.0) (+ 7.0 x))) ) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0)))) 69
  70. (declare-fun x () Real ) (declare-fun y () Real) (declare-fun

    w () Real) (declare-fun v () Real ) (assert (or (assert (not (= (+ (+ 1.0 x) 6.0) (+ 7.0 x))) ) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0)))) Semantic Fusion: an Example 70
  71. (declare-fun x () Real ) (declare-fun y () Real) (declare-fun

    w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 x) 6.0) (+ 7.0 x))) ) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0)))) Semantic Fusion: an Example 71
  72. Semantic Fusion: an Example z = x * y (declare-fun

    x () Real ) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 x) 6.0) (+ 7.0 x))) ) (assert (and (< y v) (>= w v) (< (/ w v ) 0) (> y 0)))) 72
  73. Semantic Fusion: an Example z = x * y (declare-fun

    x () Real ) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 (/ z y)) 6.0) (+ 7.0 x))) ) (assert (and (< (/ z x) v) (>= w v) (< (/ w v ) 0) (> (/ z x) 0)))) (assert (= z (* x y)) ) (assert (= x (/ z y)) ) (assert (= y (/ z x))) 73
  74. Semantic Fusion: an Example UNSAT (declare-fun x () Real )

    (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 (/ z y)) 6.0) (+ 7.0 x))) ) (assert (and (< (/ z x) v) (>= w v) (< (/ w v ) 0) (> (/ z x) 0))) ) (assert (= z (* x y)) ) (assert (= x (/ z y)) ) (assert (= y (/ z x))) 74
  75. Semantic Fusion: an Example UNSAT % z3 example.smt 2 sat

    (declare-fun x () Real ) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 (/ z y)) 6.0) (+ 7.0 x))) ) (assert (and (< (/ z x) v) (>= w v) (< (/ w v ) 0) (> (/ z x) 0))) ) (assert (= z (* x y)) ) (assert (= x (/ z y)) ) (assert (= y (/ z x))) 75
  76. Semantic Fusion: an Example UNSAT https://github.com/Z3Prover/z3/issues/2391 % z3 example.smt 2

    sat (declare-fun x () Real ) (declare-fun y () Real) (declare-fun w () Real) (declare-fun v () Real ) (declare-fun z () Real ) (assert (or (assert (not (= (+ (+ 1.0 (/ z y)) 6.0) (+ 7.0 x))) ) (assert (and (< (/ z x) v) (>= w v) (< (/ w v ) 0) (> (/ z x) 0))) ) (assert (= z (* x y)) ) (assert (= x (/ z y)) ) (assert (= y (/ z x))) 76
  77. Empirical Evaluation 77

  78. Empirical Evaluation • Tool YinYang, our realization of Semantic Fusion

    78
  79. Empirical Evaluation • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019) 79
  80. Empirical Evaluation • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019) • Bug reduction with C-Reduce 80
  81. Empirical Evaluation • Tool YinYang, our realization of Semantic Fusion

    • Bug hunting with YinYang (July-October 2019) • Bug reduction with C-Reduce • Bug reports on issue trackers of Z3 and CVC4 81
  82. How many bugs can YinYang find? 82

  83. How many bugs can YinYang find? 83

  84. How many bugs can YinYang find? 84

  85. How many bugs can YinYang find? 85

  86. How many bugs can YinYang find? 86

  87. How many bugs can YinYang find? 87

  88. Significance of the bug finding results 88

  89. Significance of the bug finding results 89

  90. Significance of the bug finding results 90

  91. Significance of the bug finding results YinYang found 24 in

    Z3, 5 in CVC4 in 4 months 91
  92. Significance of the bug finding results 92

  93. Significance of the bug finding results 93

  94. Significance of the bug finding results 3.5+ years old 94

  95. Significance of the bug finding results 1.5+ years old 95

  96. Significance of the bug finding results YinYang found longstanding soundness

    bugs 1.5+ years old 3.5+ years old 96
  97. Is Semantic Fusion necessary? 97

  98. Is Semantic Fusion necessary? 𝜑 1 ConcatFuzz 𝜑 2 SAT

    SAT ∧ 𝜑 1 𝜑 2 ∨ UNSAT UNSAT 98
  99. Is Semantic Fusion necessary? 𝜑 1 ConcatFuzz can only retrigger

    5/50 bugs ConcatFuzz 𝜑 2 SAT SAT ∧ 𝜑 1 𝜑 2 ∨ UNSAT UNSAT 99
  100. Is Semantic Fusion necessary? 100

  101. Is Semantic Fusion necessary? 101

  102. Is Semantic Fusion necessary? 102

  103. Is Semantic Fusion necessary? 2800 LoC 480 LoC 103

  104. Is Semantic Fusion necessary? YinYang consistently achieves higher coverage 2800

    LoC 480 LoC 104
  105. Z3 #2376 % cat formula.smt2 (declare-fun a () Real) (declare-fun

    b () Real) (declare-fun c () Real) (declare-fun d () Real) (declare-fun j () Real) (declare-fun e () Real) (assert (not (exists ((f Real)) (=> (and (< (/ 0 0) c) (< (/ 0 (* 2.0 b)) d))(= (= 0.0 a) (not (=> (<= f a) (<= e j)))))))) (check-sat) % cvc4 formula.smt2 unsat %z3 formula.smt2 sat 105
  106. Z3 #2376 % cat formula.smt2 (declare-fun a () Real) (declare-fun

    b () Real) (declare-fun c () Real) (declare-fun d () Real) (declare-fun j () Real) (declare-fun e () Real) (assert (not (exists ((f Real)) (=> (and (< (/ 0 0) c) (< (/ 0 (* 2.0 b)) d))(= (= 0.0 a) (not (=> (<= f a) (<= e j)))))))) (check-sat) % cvc4 formula.smt2 unsat %z3 formula.smt2 sat 106
  107. CVC4 #3412 % cat formula.smt2 (declare-fun a () Int) (declare-fun

    b () Int) (assert (= (div a b) (- 1))) (check-sat) % z3 formula.smt2 sat % cvc4 formula.smt2 unsat 107
  108. CVC4 #3412 % cat formula.smt2 (declare-fun a () Int) (declare-fun

    b () Int) (assert (= (div a b) (- 1))) (check-sat) % z3 formula.smt2 sat % cvc4 formula.smt2 unsat 108
  109. CVC4 #3412 % cat formula.smt2 (declare-fun a () Int) (declare-fun

    b () Int) (assert (= (div a b) (- 1))) (check-sat) % z3 formula.smt2 sat % cvc4 formula.smt2 unsat 109
  110. Z3 #4153 % cat formula.smt2 (declare-fun a () String) (declare-fun

    b () String) (assert (= (str.++ (str.substr "1" 0 (str.len a)) "0") b)) (assert (< (str.to.int b) 0)) (check-sat ) % z3-4.8-7 formula.smt2 unsat % z3 formula.smt2 sa t 110
  111. Z3 #4153 % cat formula.smt2 (declare-fun a () String) (declare-fun

    b () String) (assert (= (str.++ (str.substr "1" 0 (str.len a)) "0") b)) (assert (< (str.to.int b) 0)) (check-sat ) % z3-4.8-7 formula.smt2 unsat % z3 formula.smt2 sa t 111
  112. CVC4 #3217 % cat formula.smt2 (declare-fun a () String) (declare-fun

    b () String) (declare-fun c () String) (declare-fun d () String) (assert (or (not (= (str.suffixof "B" (str.replace "A" b "B")) (= ( str.substr a 0 (str.len b)) "A"))) (not (= (not (= c "A")) (str.suffixof "A" (str.replace "A" c "B")))))) (assert (= a (str.++ (str.++ b "") d))) (check-sat) % z3 formula.smt2 unsat % cvc4 formula.smt2 sa t 112
  113. CVC4 #3217 % cat formula.smt2 (declare-fun a () String) (declare-fun

    b () String) (declare-fun c () String) (declare-fun d () String) (assert (or (not (= (str.suffixof "B" (str.replace "A" b "B")) (= ( str.substr a 0 (str.len b)) "A"))) (not (= (not (= c "A")) (str.suffixof "A" (str.replace "A" c "B")))))) (assert (= a (str.++ (str.++ b "") d))) (check-sat) % z3 formula.smt2 unsat % cvc4 formula.smt2 sa t 113
  114. Z3 #2618 & CVC4 #3357 % cat formula.smt2 (declare-fun a

    () String) (declare-fun b () String) (declare-fun c () String) (assert (str.in.re c (re.* (re.union (str.to.re "aa") (str.to.re ""))))) (assert (= 0 (str.to.int (str.replace a b (str.at a (str.len a)))))) (assert (= a (str.++ b c))) (check-sat) % cvc4 formula.smt2 unsat % z3 formula.smt2 sa t 114
  115. Z3 #2618 & CVC4 #3357 % z3 unreduced.smt2 sa t

    % cvc4 unreduced.smt2 sa t % cat formula.smt2 (declare-fun a () String) (declare-fun b () String) (declare-fun c () String) (assert (str.in.re c (re.* (re.union (str.to.re "aa") (str.to.re ""))))) (assert (= 0 (str.to.int (str.replace a b (str.at a (str.len a)))))) (assert (= a (str.++ b c))) (check-sat) % cvc4 formula.smt2 unsat % z3 formula.smt2 sa t Z3 and CVC4 are both unsound on the unreduced test! 115
  116. Roadmap 1. Introductio n 2. Testing Technique s a) Semantic

    Fusio n b) Type-aware Operator Mutatio n c) Generative Type-Aware Operator Mutation 3. Current and Future Directions 116
  117. Type-aware operator mutation $ cat > formula.smt 2 (assert (forall

    ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat 117
  118. Type-aware operator mutation $ cat > formula.smt 2 (assert (forall

    ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat 118
  119. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat 119
  120. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat Int Int Int Int 120
  121. (assert (forall ((a Int) ) (exist ((b Int) ) (=

    (* 2 b) a))) ) (check-sat ) $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat Int Int Int Int = > < > = < = Type-aware operator mutation Operator candidates: 121 121
  122. (assert (forall ((a Int) ) (exist ((b Int) ) (=

    (* 2 b) a))) ) (check-sat ) $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat Int Int Int Int Type-aware operator mutation Operator candidates: = > < > = < = 122
  123. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat 123
  124. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > bug.smt2 $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cvc4 bug.smt 2 unsa t $ z3 bug.smt 2 sat 124
  125. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > bug.smt2 $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cvc4 bug.smt 2 unsa t $ z3 bug.smt 2 sat ✘ ✔ 125
  126. Type-aware operator mutation (assert (forall ((a Int) ) (exist ((b

    Int) ) (= (* 2 b) a))) ) (check-sat ) $ cat > bug.smt2 $ cat > formula.smt 2 (assert (forall ((a Int) ) (exist ((b Int) ) (distinct (* 2 b) a))) ) (check-sat ) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cvc4 bug.smt 2 unsa t $ z3 bug.smt 2 sat https://github.com/Z3Prover/z3/issues/3973 ✔ ✘ 126
  127. Type-aware operator mutation chain 127

  128. Type-aware operator mutation chain (declare-fun a () Real ) (assert

    (> (/ (* 2 a) a) (* a a ) 1)) (check-sat) 128
  129. (declare-fun a () Real ) (assert (> (/ (* 2

    a) a) (* a a ) 1)) (check-sat) Type-aware operator mutation chain 129
  130. (declare-fun a () Real ) (assert (> (/ (* 2

    a) a) (* a a ) 1)) (check-sat) Type-aware operator mutation chain (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (* a a ) 1)) (check-sat) 130
  131. (declare-fun a () Real ) (assert (> (/ (* 2

    a) a) (* a a ) 1)) (check-sat) Type-aware operator mutation chain (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (* a a ) 1)) (check-sat) 131
  132. (declare-fun a () Real ) (assert (> (/ (* 2

    a) a) (* a a ) 1)) (check-sat) Type-aware operator mutation chain (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (* a a ) 1)) (check-sat) (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (/ a a ) 1)) (check-sat) 132
  133. (declare-fun a () Real ) (assert (> (/ (* 2

    a) a) (* a a ) 1)) (check-sat) Type-aware operator mutation chain (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (* a a ) 1)) (check-sat) (declare-fun a () Real ) (assert (= (/ (* 2 a) a) (/ a a ) 1)) (check-sat) … 133
  134. (declare-fun a () Real ) (assert (= (/ (* 2

    a) a) (/ a a ) 1)) (check-sat) Type-aware operator mutation chain 134
  135. (declare-fun a () Real ) (assert (= (/ (* 2

    a) a) (/ a a ) 1)) (check-sat) Type-aware operator mutation chain $ cat > bug.smt2 $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat 135
  136. (declare-fun a () Real ) (assert (= (/ (* 2

    a) a) (/ a a ) 1)) (check-sat) Type-aware operator mutation chain $ cat > bug.smt2 $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat ✘ ✔ https://github.com/Z3Prover/z3/issues/2715 136
  137. (declare-fun a () Real ) (assert (= (/ (* 2

    a) a) (/ a a ) 1)) (check-sat) Type-aware operator mutation chain $ cat > bug.smt2 $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat ✘ ✔ https://github.com/Z3Prover/z3/issues/2715 137
  138. Empirical Evaluation 138

  139. Empirical Evaluation • Tool: OpFuzz • Bug hunting: Sep 2019

    - Oct 2020 • Testing targets: Z3 and CVC4 • Seeds: SMT-LIB benchmarks (300k+ formulas) 139
  140. Empirical Evaluation • Tool: OpFuzz • Bug hunting: Sep 2019

    - Oct 2020 • Testing targets: Z3 and CVC4 • Seeds: SMT-LIB benchmarks (300k+ formulas) 140
  141. Empirical Evaluation • Tool: OpFuzz • Bug hunting: Sep 2019

    - Oct 2020 • Testing targets: Z3 and CVC4 • Seeds: SMT-LIB benchmarks (300k+ formulas) 141
  142. Empirical Evaluation • Tool: OpFuzz • Bug hunting: Sep 2019

    - Oct 2020 • Testing targets: Z3 and CVC4 • Seeds: SMT-LIB benchmarks (300k+ formulas) 142
  143. Bug Findings Status Z3 CVC4 Total Reported 811 281 1,092

    Con fi rmed 578 241 819 Fixed 521 164 685 Duplicate 85 18 106 Bug status as of 30 Oct 2020 143
  144. Bug Findings Status Z3 CVC4 Total Reported 811 281 1,092

    Con fi rmed 578 241 819 Fixed 521 164 685 Duplicate 85 18 106 Bug status as of 30 Oct 2020 144
  145. Bug Findings Status Z3 CVC4 Total Reported 811 281 1,092

    Con fi rmed 578 241 819 Fixed 521 164 685 Duplicate 85 18 106 Bug status as of 30 Oct 2020 145
  146. Bug Findings Status Z3 CVC4 Total Reported 811 281 1,092

    Con fi rmed 578 241 819 Fixed 521 164 685 Duplicate 85 18 106 Bug status as of 30 Oct 2020 146
  147. Bug Findings Status Z3 CVC4 Total Soundness 157 27 184

    Invalid model 83 19 102 Crash 316 185 501 Others 22 10 32 Types of con fi rmed bugs 147
  148. Bug Findings Status Z3 CVC4 Total Soundness 157 27 184

    Invalid model 83 19 102 Crash 316 185 501 Others 22 10 32 Types of con fi rmed bugs 148
  149. Bug Findings Status Z3 CVC4 Total Soundness 157 27 184

    Invalid model 83 19 102 Crash 316 185 501 Others 22 10 32 Types of con fi rmed bugs 149
  150. Bug Findings Logic Z3 CVC4 Total Soundness 157 27 184

    Invalid model 83 19 102 Crash 316 185 501 Others 22 10 32 Types of con fi rmed bugs 150
  151. Bug Findings Number of options for triggering the bugs #Options

    Z3 CVC4 Total Default 388 101 489 1 109 67 176 2 45 31 76 >= 3 36 42 78 151
  152. Bug Findings Number of options for triggering the bugs #Options

    Z3 CVC4 Total Default 388 101 489 1 109 67 176 2 45 31 76 >= 3 36 42 78 152
  153. Bug Findings Number of options for triggering the bugs #Options

    Z3 CVC4 Total Default 388 101 489 1 109 67 176 2 45 31 76 >= 3 36 42 78 153
  154. Bugs in Z3 Bugs in CVC4 Approach soundness all soundness

    all StringFuzz 0 0 - - BanditFuzz 0 0 - - Bugariu et al. 1 3 0 0 YinYang 24 36 5 8 STORM 17 21 0 0 OpFuzz 114 316 11 185 Comparison to Previous Approaches Con fi rmed bugs in the default modes of the solvers 154
  155. Comparison to Previous Approaches Con fi rmed bugs in the

    default modes of the solvers Bugs in Z3 Bugs in CVC4 Approach soundness all soundness all StringFuzz 0 0 - - BanditFuzz 0 0 - - Bugariu et al. 1 3 0 0 YinYang 24 36 5 8 STORM 17 21 0 0 OpFuzz 114 316 11 185 155
  156. Comparison to Previous Approaches OpFuzz found many more soundness bugs

    in Z3 and CVC4’s default modes than all previous approaches Bugs in Z3 Bugs in CVC4 Approach soundness all soundness all StringFuzz 0 0 - - BanditFuzz 0 0 - - Bugariu et al. 1 3 0 0 YinYang 24 36 5 8 STORM 17 21 0 0 OpFuzz 114 316 11 185 156
  157. Z3 Soundness Bug #2832 $ cat bug.smt 2 (declare-const a

    (_ BitVec 8) ) (declare-const b (_ BitVec 8)) (declare-const c (_ BitVec 8)) (assert (= (bvxnor a b c ) (bvxnor (bvxnor a b) c))) (check-sat ) $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat ✘ ✔ https://github.com/Z3Prover/z3/issues/2832 157
  158. Z3 Soundness Bug #2832 $ cat bug.smt 2 (declare-const a

    (_ BitVec 8) ) (declare-const b (_ BitVec 8)) (declare-const c (_ BitVec 8)) (assert (= (bvxnor a b c) (bvxnor (bvxnor a b) c))) (check-sat ) $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat ✘ ✔ https://github.com/Z3Prover/z3/issues/2832 158
  159. Z3 Soundness Bug #2832 (bvxnor true true true) = (not

    (bvxor true true true) ) (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true))) (bvnxor true true) = (not (bvxor false true)) ) true false 
 ≠ (bvxnor a b c) (not (bvxor a b c)) ≠ 159
  160. Z3 Soundness Bug #2832 (bvxnor true true true) = (not

    (bvxor true true true) ) (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true))) (bvnxor true true) = (not (bvxor false true)) ) true false 
 ≠ (bvxnor a b c) (not (bvxor a b c)) ≠ 160
  161. Z3 Soundness Bug #2832 (bvxnor true true true) = (not

    (bvxor true true true) ) (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true)) (bvnxor true true) = (not (bvxor false true)) ) true false 
 ≠ (bvxnor a b c) (not (bvxor a b c)) ≠ 161
  162. Z3 Soundness Bug #2832 (bvxnor true true true) = (not

    (bvxor true true true) ) (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true)) (bvnxor true true) = (not (bvxor false true) ) true false 
 ≠ (bvxnor a b c) (not (bvxor a b c)) ≠ 162
  163. Z3 Soundness Bug #2832 (bvxnor true true true) = (not

    (bvxor true true true) ) (bvnxor (bvnxor true true) true)) = (not (bvxor (bvxor true true) true)) (bvnxor true true) = (not (bvxor false true) ) true false 
 ≠ (bvxnor a b c) (not (bvxor a b c)) ≠ 163
  164. Z3 Soundness Bug #2832 $ cat bug.smt 2 (declare-const a

    (_ BitVec 8) ) (declare-const b (_ BitVec 8)) (declare-const c (_ BitVec 8)) (assert (= (bvxnor a b c) (bvxnor (bvxnor a b) c))) (check-sat ) $ cvc4 bug.smt 2 sa t $ z3 bug.smt 2 unsat ✘ ✔ https://github.com/Z3Prover/z3/issues/2832 Root Cause: Unsound rewriter 164
  165. Z3 Soundness Bug #2830 $ cat bug.smt 2 (declare-fun a

    () Int ) (declare-fun b (Int) Bool ) (assert (b 0)) (push ) (assert (distinct true (= a 0) (not (b 0)))) (check-sat ) $ cvc4 bug.smt 2 unsa t $ z3 bug.smt 2 sat ✘ ✔ https://github.com/Z3Prover/z3/issues/2830 165
  166. Z3 Soundness Bug #2830 $ cat bug.smt 2 (declare-fun a

    () Int ) (declare-fun b (Int) Bool ) (assert (b 0)) (push ) (assert (distinct true (= a 0) (not (b 0)))) (check-sat ) $ cvc4 bug.smt 2 unsa t $ z3 bug.smt 2 sat ✘ ✔ https://github.com/Z3Prover/z3/issues/2830 Root Cause: Last argument omitted in distinct operator 166
  167. CVC4 Soundness Bug #3497 $ cat bug.smt 2 (declare-fun x

    () String) (declare-fun y () String) (assert (= (str.indexof x y 1) (str.len x))) (assert (str.contains x y)) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat ✘ ✔ https://github.com/CVC4/CVC4/issues/3497 167
  168. CVC4 Soundness Bug #3497 $ cat bug.smt 2 (declare-fun x

    () String) (declare-fun y () String) (assert (= (str.indexof x y 1) (str.len x))) (assert (str.contains x y)) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat ✘ ✔ https://github.com/CVC4/CVC4/issues/3497 168
  169. CVC4 Soundness Bug #3497 $ cat bug.smt 2 (declare-fun x

    () String) (declare-fun y () String) (assert (= (str.indexof x y 1) (str.len x))) (assert (str.contains x y)) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat ✘ ✔ https://github.com/CVC4/CVC4/issues/3497 169
  170. CVC4 Soundness Bug #4469 $ cat bug.smt 2 (set-logic QF_AUFBVLIA

    ) (declare-fun a () Int) (declare-fun b (Int) Int) (assert (distinct (b a) (b (b a)))) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat ✘ ✔ https://github.com/CVC4/CVC4/issues/4469 170
  171. CVC4 Soundness Bug #4469 $ cat bug.smt 2 (set-logic QF_AUFBVLIA

    ) (declare-fun a () Int) (declare-fun b (Int) Int) (assert (distinct (b a) (b (b a)))) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat ✘ ✔ https://github.com/CVC4/CVC4/issues/4469 171
  172. CVC4 Soundness Bug #3475 $ cat bug.smt 2 (set-logic ALL)


    (declare-fun x () Real ) (assert (< x 0) ) (assert (not (= (/ (sqrt x) (sqrt x)) x)) ) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat
 https://github.com/CVC4/CVC4/issues/3475 ✘ ✔ 172
  173. Roadmap 1. Introductio n 2. Testing Technique s a) Semantic

    Fusio n b) Type-aware Operator Mutatio n c) Generative Type-Aware Operator Mutation 3. Current and Future Directions 173
  174. Generative Type-Aware Mutation • Idea: Extend OpFuzz • Expressions •

    Support Generation: Control the size of Formula • Formula can grow & shrink 174
  175. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat Generative Type-Aware Mutation 175 Generative Type-Aware Mutation
  176. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat Bool: (> (- (str.to_int (str.++ x x))) 0) ) Int : 0, (- (str.to_int (str.++ x x))) 0), … String : (str.++ x x), x 176 Generative Type-Aware Mutation
  177. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat formula.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat 177 Generative Type-Aware Mutation
  178. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat formula.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat String 178 Generative Type-Aware Mutation
  179. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat formula.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat String Bool: (> (- (str.to_int (str.++ x x))) 0) ) Int : 0, (- (str.to_int (str.++ x x))) 0), … String : (str.++ x x), x 179 Generative Type-Aware Mutation
  180. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat formula.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat String Bool: (> (- (str.to_int (str.++ x x))) 0) ) Int : 0, (- (str.to_int (str.++ x x))) 0), … String : (str.++ x x), x Operator candidates: str.from_int str.replace str.++ ... 180 Generative Type-Aware Mutation
  181. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat formula.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ str.from_int x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat String Bool: (> (- (str.to_int (str.++ x x))) 0) ) Int : 0,, (- (str.to_int (str.++ x x))) 0), … String : (str.++ x x), x (str.from_int Int String) String 181 Generative Type-Aware Mutation
  182. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat mutant.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ (str.from_int 0) x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 Sat 182 Generative Type-Aware Mutation
  183. $ cat formula.smt 2 (declare-fun x () String) (assert (>

    (- (str.to_int (str.++ x x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 sat $ cat mutant.smt 2 (declare-fun x () String) (assert (> (- (str.to_int (str.++ (str.from_int 0) x))) 0)) (check-sat) $ cvc4 formula.smt 2 sa t $ z3 formula.smt 2 unsat 183 Generative Type-Aware Mutation
  184. More Bugs Status Z3 CVC4 Total Reported 177 60 237

    Con fi rmed 135 54 189 Fixed 132 44 176 Duplicate 9 5 14 Bug status as of 30 Sep 2021 184
  185. Roadmap 1. Introductio n 2. Testing Technique s a) Semantic

    Fusio n b) Type-aware Operator Mutatio n c) Generative Type-Aware Operator Mutation 3. Current and Future Directions 185
  186. Current and Future Directions Yin-Yang [PLDI ’20] OpFuzz [OOPSLA ’20]

    TypeFuzz [OOPSLA ’21] Correctness Robust & Fast SMT Solvers Performance 186
  187. Current and Future Directions Yin-Yang [PLDI ’20] OpFuzz [OOPSLA ’20]

    TypeFuzz [OOPSLA ’21] Correctness Robust & Fast SMT Solvers Performance 187
  188. “Testing can only show the presence of errors, not their

    absence.” 
 - Edsger W. Dijkstra 188
  189. Bounded Guarantees for SMT Solvers Software Bug triggers Guarantees Bound

    Grammar Fuzzer 189
  190. Current and Future Directions Yin-Yang [PLDI ’20] OpFuzz [OOPSLA ’20]

    TypeFuzz [OOPSLA ’21] Correctness Robust & Fast SMT Solvers Performance 190
  191. SMT Solver Flags 191

  192. SMT Solver Flags Z3 has 300+ fl ags, all of

    which could in fl uence its performance 192
  193. Problem Statement Given: Solver
 Flags Benchmarks Wanted: Understand whether Flags

    are pair wisely independent 193
  194. Summary @DominikWinterer wintered.github.io https://testsmt.github.io/ 194

  195. Zhendong Su Prof@ETH Zurich Chengyu Zhang Postdoc@ETH Zurich Collaborators Jiwon

    Park Intern@ETH Zurich ➜ Ph.D. student@UC Berkeley 195
  196. Sponsors 196

  197. Bonus Slides

  198. CVC4 Soundness Bug #3475 $ cat bug.smt 2 (set-logic ALL)


    (declare-fun x () Real ) (assert (< x 0) ) (assert (not (= (/ (sqrt x) (sqrt x)) x)) ) (check-sat ) $ z3 bug.smt 2 sa t $ cvc4 bug.smt 2 unsat
 https://github.com/CVC4/CVC4/issues/3475 ✘ ✔ Formula is satisfiable negative x = -1 (/ sqrt(-1) sqrt(-1)) = 1 198
  199. CVC4 Soundness Bug #3475 Simplification: 
 (sqrt x) = choice

    real y s.t. x*x = y 199
  200. CVC4 Soundness Bug #3475 Simplification: 
 (sqrt x) = choice

    real y s.t. x*x = y Problem: Inadmissible for negative y (since no real x exists) 200