A Late Treatment of C Precondition in Dynamic Symbolic Execution Testing Tools (RV 2013)

A Late Treatment of C Precondition
in Dynamic Symbolic Execution Testing Tools
Mickaël Delahaye1 Nikolai Kosmatov2
1 2
RV 2013
September 27, 2013
M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 1 / 20

View Slide

Introduction Context
Automated test generation
• Testing is hard: automation is the way to go
• Test input generation: no (smart) oracle
• But how?
• Black box vs. white box
White box
• Classic goal: code coverage
• Path-oriented methods
• Dynamic Symbolic Execution (DSE)
• Combine dynamic and symbolic executions
• Dynamic: on the ﬂy program exploration, use some concrete values
(to speed things up or stay in supported theories)
• Symbolic: guiding the exploration to new areas of the execution tree

View Slide

DSE process
as implemented in PathCrawler
f is the C function under test
and π the current partial path in f

View Slide

DSE process
(A1
) init., π := ε

View Slide

DSE process
(A1
) init., π := ε
(A2
) symb. exec. π in f

View Slide

DSE process
(A1
) init., π := ε
(A2
) symb. exec. π in f (A3
) generate test t
(A5
) compute next π
ok
fail

View Slide

DSE process
(A1
) init., π := ε
(A2
) generate test t
(A5
) compute next π (A4
) execute f on t
ok
fail ok
fail

View Slide

DSE process
(A1
) init., π := ε
(A2
) generate test t
(A5
) execute f on t
ok
fail ok
fail
ok

View Slide

DSE process
(A1
) init., π := ε
(A2
) generate test t
(A5
) execute f on t
ﬁnish
ok
fail ok
fail
ok
no more paths in f

View Slide

Test precondition
• A means to define the input space to test
• Essential to automatic test input generation
• Very large number of possible inputs (even for small programs)
• concentrate the test on interesting or critical parts of the program
• For some inputs the behavior is not specified
• select only the specified input space:
test precondition = specification precondition
• or exactly the opposite to check the robustness of the program

View Slide

Introduction State of the Art
Two ways to specify the input space
Declarative precondition vs. imperative construction
Declarative
t is a valid input iff:
• t is an array of integers
of size in 0..n
• for all i ∈ 0..n – 1,
t[i] ≤ t[i + 1]
• similar and in most
case identical to the
specification
precondition
Imperative
to construct a valid input t:
1 choose s in 0..n and
allocate t an array of size s
2 set m to MIN_INT
3 for i ∈ 0..n, choose t[i] in
m..MAX_INT and set m to t[i]
• explicitly bounds input space:
finitization (Korat[Milicevic et al.])
• still a lot of things are easier
done in a declarative manner
(Korat’s repOk)

View Slide

How to handle the precondition?
(1) Symbolically
• DSE tools compute path conditions as a conjunction of predicates
• simply add the precondition to the path condition
• very eﬃcient
• require to encode the precondition as a predicate
• not easy for software engineers
• not easy for solvers: may contain disjunctions, quantiﬁers, etc.

View Slide

Handling the precondition symbolically
(A1
) init., set precond., π := ε
(A2
) generate test t
(A5
) execute f on t
ﬁnish
ok
fail ok
fail
ok
no more paths in f

View Slide

How to handle the precondition?
(2) As a part of the program
if (precondition() == TRUE) {
function_under_test();
}
• simple enough
• DSE takes care of enforcing the precondition
• but the precondition code may contain multiple paths
• worst case: exponentially increase the number of paths to explore

View Slide

Outline
1 Introduction
2 Method
3 Experiments
4 Conclusion

View Slide

Method
Outline
1 Introduction
2 Method
3 Experiments
4 Conclusion

View Slide

Method
Late precondition method
• Code the precondition in the target language (here C)
• But explore the precondition by DSE after the program

View Slide

Method
(A
1
) init., π := ε
(A
2
) symb. exec. π in f (A
3
) generate test t
(A
5
) compute next π (A
4
) execute f on t
ﬁnish
fail
ok
no more paths in f
ok
ok
ok
fail

View Slide

Method
(A
1
) init., π := ε
(A
2
3
) generate test t
(A
5
4
) execute f on t
ﬁnish
fail
ok
no more paths in f
ok
ok
ok
fail
where f is the C function under test,
π the current partial path in f,
p is the C precondition,
and ρ the current partial path in p

View Slide

Method
(A
1
) init., π := ε
(A
2
) symb. exec. π in f
(A
5
4
) execute f on t
ﬁnish
fail
ok
no more paths in f

View Slide

Method
(A
1
) init., π := ε
(A
2
1
) ρ := ε
(A
2
) symb. exec. ρ in p (A
3
) generate test t
(A
5
) compute next ρ (A
4
) execute p on t
(A
5
4
) execute f on t
ﬁnish
fail
ok
no more paths in f
ok
ok
fail ok
fail
p is true on t
p is
false
on t
ok
no more paths in p

View Slide

Experiments
Outline
1 Introduction
2 Method
3 Experiments
4 Conclusion

View Slide

Experiments
Evaluation
Protocol
Measure the total test input generation time with:
• a single test input generation tool with some coverage criterium
• a number of programs with non-trivial preconditions
• diﬀerent kinds of precondition methods
Tool PathCrawler (k-path)
Prog. Merge, TriangMatrix, PermutOrder
Precond Late, Early, Native

View Slide

Experiments
Selected experiments
Program Merge TriangMatrix PermutOrder PermutOrder
Key part of
precond.
∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i

View Slide

Experiments
Key part of
precond.
Test generation time
Native 3m32s – 17.9s –
Early 117m43s 38.6s 23s 2m12s
Late 4m8s 27.5s 23.2s 25s

View Slide

Experiments
Key part of
precond.
Test generation time
Native 3m32s – 17.9s –
Early 117m43s 38.6s 23s 2m12s
Late 4m8s 27.5s 23.2s 25s
Number of paths
Native 8718 – 5153 –
Early 73644 4893 5179 14491
Late 8142 4093 6071 6027

View Slide

Experiments
Some more data on merge
Time (in s) for k-path test generation
k “Native” Early C Late C
precond precond precond
2 1.52 2.08 1.73
3 2.01 3.38 2.25
4 3.37 7.36 3.75
5 6.54 20.26 8.23
6 18.04 78.12 22.34
7 52.15 8m39 68.06
8 3m32 117m43 4m8
9 16m47 18m28

View Slide

Conclusion
Outline
1 Introduction
2 Method
3 Experiments
4 Conclusion

View Slide

Conclusion
Conclusion
• A method to handle a precondition in the target language
• Easy to write for developers
• main objective completed!
• Ensures that each path of the program is explored at most once
• leads to very satisfying performances (on merge and others)
• very important for thorough coverage criteria
• An implementation in PathCrawler for C
• Test it online! http://pathcrawler-online.com

View Slide

Conclusion
Perspectives
• Optimizations
• Memorize valid precondition paths instead of blindly going into the
precondition
• When possible (bounded loops) summarize the precondition to a
ﬁnite disjunction (SMART [Godefroid 2008])
• When the precondition fails ﬁnd and exploit the reason why (Korat)
• Explore the combination of multiple symbolic executions (algebra)

View Slide

That’s all folks!
Thanks!
Mickaël Delahaye
[email protected]
http://micdel.fr
Try PathCrawler online!
http://pathcrawler-online.com

View Slide

A Late Treatment of C Precondition in Dynamic Symbolic Execution Testing Tools (RV 2013)

A Late Treatment of C Precondition in Dynamic Symbolic Execution Testing Tools (RV 2013)

Mickaël Delahaye

Other Decks in Research

Featured

Transcript