Testing Tools Mickaël Delahaye1 Nikolai Kosmatov2 1 2 RV 2013 September 27, 2013 M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 1 / 20
is the way to go • Test input generation: no (smart) oracle • But how? • Black box vs. white box White box • Classic goal: code coverage • Path-oriented methods • Dynamic Symbolic Execution (DSE) • Combine dynamic and symbolic executions • Dynamic: on the ﬂy program exploration, use some concrete values (to speed things up or stay in supported theories) • Symbolic: guiding the exploration to new areas of the execution tree M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 2 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail ok M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ﬁnish ok fail ok fail ok no more paths in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
input space to test • Essential to automatic test input generation • Very large number of possible inputs (even for small programs) • concentrate the test on interesting or critical parts of the program • For some inputs the behavior is not speciﬁed • select only the speciﬁed input space: test precondition = speciﬁcation precondition • or exactly the opposite to check the robustness of the program M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 4 / 20
input space Declarative precondition vs. imperative construction Declarative t is a valid input iﬀ: • t is an array of integers of size in 0..n • for all i ∈ 0..n – 1, t[i] ≤ t[i + 1] • similar and in most case identical to the speciﬁcation precondition Imperative to construct a valid input t: 1 choose s in 0..n and allocate t an array of size s 2 set m to MIN_INT 3 for i ∈ 0..n, choose t[i] in m..MAX_INT and set m to t[i] • explicitly bounds input space: ﬁnitization (Korat[Milicevic et al.]) • still a lot of things are easier done in a declarative manner (Korat’s repOk) M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 5 / 20
(1) Symbolically • DSE tools compute path conditions as a conjunction of predicates • simply add the precondition to the path condition • very eﬃcient • require to encode the precondition as a predicate • not easy for software engineers • not easy for solvers: may contain disjunctions, quantiﬁers, etc. M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 6 / 20
is the C function under test and π the current partial path in f (A1 ) init., set precond., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ﬁnish ok fail ok fail ok no more paths in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 7 / 20
(2) As a part of the program if (precondition() == TRUE) { function_under_test(); } • simple enough • DSE takes care of enforcing the precondition • but the precondition code may contain multiple paths • worst case: exponentially increase the number of paths to explore M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 8 / 20
target language (here C) • But explore the precondition by DSE after the program M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 11 / 20
ε (A 2 ) symb. exec. π in f (A 3 ) generate test t (A 5 ) compute next π (A 4 ) execute f on t ﬁnish fail ok no more paths in f ok ok ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
ε (A 2 ) symb. exec. π in f (A 3 ) generate test t (A 5 ) compute next π (A 4 ) execute f on t ﬁnish fail ok no more paths in f ok ok ok fail where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
ε (A 2 ) symb. exec. π in f (A 5 ) compute next π (A 4 ) execute f on t ﬁnish fail ok no more paths in f where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
ε (A 2 ) symb. exec. π in f (A 1 ) ρ := ε (A 2 ) symb. exec. ρ in p (A 3 ) generate test t (A 5 ) compute next ρ (A 4 ) execute p on t (A 5 ) compute next π (A 4 ) execute f on t ﬁnish fail ok no more paths in f ok ok fail ok fail p is true on t p is false on t ok no more paths in p where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
ε (A 2 ) symb. exec. π in f (A 1 ) ρ := ε (A 2 ) symb. exec. ρ in p (A 3 ) generate test t (A 5 ) compute next ρ (A 4 ) execute p on t (A 5 ) compute next π (A 4 ) execute f on t ﬁnish fail ok no more paths in f ok ok fail ok fail p is true on t p is false on t ok no more paths in p where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
with: • a single test input generation tool with some coverage criterium • a number of programs with non-trivial preconditions • diﬀerent kinds of precondition methods Tool PathCrawler (k-path) Prog. Merge, TriangMatrix, PermutOrder Precond Late, Early, Native M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 14 / 20
of precond. ∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i Test generation time Native 3m32s – 17.9s – Early 117m43s 38.6s 23s 2m12s Late 4m8s 27.5s 23.2s 25s M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 15 / 20
of precond. ∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i Test generation time Native 3m32s – 17.9s – Early 117m43s 38.6s 23s 2m12s Late 4m8s 27.5s 23.2s 25s Number of paths Native 8718 – 5153 – Early 73644 4893 5179 14491 Late 8142 4093 6071 6027 M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 15 / 20
the target language • Easy to write for developers • main objective completed! • Ensures that each path of the program is explored at most once • leads to very satisfying performances (on merge and others) • very important for thorough coverage criteria • An implementation in PathCrawler for C • Test it online! http://pathcrawler-online.com M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 18 / 20
of blindly going into the precondition • When possible (bounded loops) summarize the precondition to a ﬁnite disjunction (SMART [Godefroid 2008]) • When the precondition fails ﬁnd and exploit the reason why (Korat) • Explore the combination of multiple symbolic executions (algebra) M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 19 / 20