A Late Treatment of C Precondition in Dynamic Symbolic Execution Testing Tools (RV 2013)

Transcript

1. A Late Treatment of C Precondition in Dynamic Symbolic Execution Testing Tools Mickaël Delahaye1 Nikolai Kosmatov2 1 2 RV 2013 September 27, 2013 M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 1 / 20
2. Introduction Context Automated test generation • Testing is hard: automation is the way to go • Test input generation: no (smart) oracle • But how? • Black box vs. white box White box • Classic goal: code coverage • Path-oriented methods • Dynamic Symbolic Execution (DSE) • Combine dynamic and symbolic executions • Dynamic: on the fly program exploration, use some concrete values (to speed things up or stay in supported theories) • Symbolic: guiding the exploration to new areas of the execution tree M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 2 / 20
3. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
4. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
5. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
6. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
7. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
8. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
9. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t ok fail ok fail ok M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
10. Introduction Context DSE process as implemented in PathCrawler f is the C function under test and π the current partial path in f (A1 ) init., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t finish ok fail ok fail ok no more paths in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 3 / 20
11. Introduction Context Test precondition • A means to define the input space to test • Essential to automatic test input generation • Very large number of possible inputs (even for small programs) • concentrate the test on interesting or critical parts of the program • For some inputs the behavior is not specified • select only the specified input space: test precondition = specification precondition • or exactly the opposite to check the robustness of the program M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 4 / 20
12. Introduction State of the Art Two ways to specify the input space Declarative precondition vs. imperative construction Declarative t is a valid input iff: • t is an array of integers of size in 0..n • for all i ∈ 0..n – 1, t[i] ≤ t[i + 1] • similar and in most case identical to the specification precondition Imperative to construct a valid input t: 1 choose s in 0..n and allocate t an array of size s 2 set m to MIN_INT 3 for i ∈ 0..n, choose t[i] in m..MAX_INT and set m to t[i] • explicitly bounds input space: finitization (Korat[Milicevic et al.]) • still a lot of things are easier done in a declarative manner (Korat’s repOk) M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 5 / 20
13. Introduction State of the Art How to handle the precondition? (1) Symbolically • DSE tools compute path conditions as a conjunction of predicates • simply add the precondition to the path condition • very efficient • require to encode the precondition as a predicate • not easy for software engineers • not easy for solvers: may contain disjunctions, quantifiers, etc. M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 6 / 20
14. Introduction State of the Art Handling the precondition symbolically f is the C function under test and π the current partial path in f (A1 ) init., set precond., π := ε (A2 ) symb. exec. π in f (A3 ) generate test t (A5 ) compute next π (A4 ) execute f on t finish ok fail ok fail ok no more paths in f M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 7 / 20
15. Introduction State of the Art How to handle the precondition? (2) As a part of the program if (precondition() == TRUE) { function_under_test(); } • simple enough • DSE takes care of enforcing the precondition • but the precondition code may contain multiple paths • worst case: exponentially increase the number of paths to explore M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 8 / 20
16. Introduction State of the Art Outline 1 Introduction 2 Method 3 Experiments 4 Conclusion M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 9 / 20
17. Method Outline 1 Introduction 2 Method 3 Experiments 4 Conclusion M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 10 / 20
18. Method Late precondition method • Code the precondition in the target language (here C) • But explore the precondition by DSE after the program M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 11 / 20
19. Method Late precondition method (A 1 ) init., π := ε (A 2 ) symb. exec. π in f (A 3 ) generate test t (A 5 ) compute next π (A 4 ) execute f on t finish fail ok no more paths in f ok ok ok fail M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
20. Method Late precondition method (A 1 ) init., π := ε (A 2 ) symb. exec. π in f (A 3 ) generate test t (A 5 ) compute next π (A 4 ) execute f on t finish fail ok no more paths in f ok ok ok fail where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
21. Method Late precondition method (A 1 ) init., π := ε (A 2 ) symb. exec. π in f (A 5 ) compute next π (A 4 ) execute f on t finish fail ok no more paths in f where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
22. Method Late precondition method (A 1 ) init., π := ε (A 2 ) symb. exec. π in f (A 1 ) ρ := ε (A 2 ) symb. exec. ρ in p (A 3 ) generate test t (A 5 ) compute next ρ (A 4 ) execute p on t (A 5 ) compute next π (A 4 ) execute f on t finish fail ok no more paths in f ok ok fail ok fail p is true on t p is false on t ok no more paths in p where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
23. Method Late precondition method (A 1 ) init., π := ε (A 2 ) symb. exec. π in f (A 1 ) ρ := ε (A 2 ) symb. exec. ρ in p (A 3 ) generate test t (A 5 ) compute next ρ (A 4 ) execute p on t (A 5 ) compute next π (A 4 ) execute f on t finish fail ok no more paths in f ok ok fail ok fail p is true on t p is false on t ok no more paths in p where f is the C function under test, π the current partial path in f, p is the C precondition, and ρ the current partial path in p M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 12 / 20
24. Experiments Outline 1 Introduction 2 Method 3 Experiments 4 Conclusion M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 13 / 20
25. Experiments Evaluation Protocol Measure the total test input generation time with: • a single test input generation tool with some coverage criterium • a number of programs with non-trivial preconditions • different kinds of precondition methods Tool PathCrawler (k-path) Prog. Merge, TriangMatrix, PermutOrder Precond Late, Early, Native M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 14 / 20
26. Experiments Selected experiments Program Merge TriangMatrix PermutOrder PermutOrder Key part of precond. ∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 15 / 20
27. Experiments Selected experiments Program Merge TriangMatrix PermutOrder PermutOrder Key part of precond. ∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i Test generation time Native 3m32s – 17.9s – Early 117m43s 38.6s 23s 2m12s Late 4m8s 27.5s 23.2s 25s M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 15 / 20
28. Experiments Selected experiments Program Merge TriangMatrix PermutOrder PermutOrder Key part of precond. ∀i, ti ≤ ti+1 ∀i ≥ j, Mij = 0 ∀i = j, pi = pj ∀i, ∃j, pj = i Test generation time Native 3m32s – 17.9s – Early 117m43s 38.6s 23s 2m12s Late 4m8s 27.5s 23.2s 25s Number of paths Native 8718 – 5153 – Early 73644 4893 5179 14491 Late 8142 4093 6071 6027 M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 15 / 20
29. Experiments Some more data on merge Time (in s) for k-path test generation k “Native” Early C Late C precond precond precond 2 1.52 2.08 1.73 3 2.01 3.38 2.25 4 3.37 7.36 3.75 5 6.54 20.26 8.23 6 18.04 78.12 22.34 7 52.15 8m39 68.06 8 3m32 117m43 4m8 9 16m47 18m28 M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 16 / 20
30. Conclusion Outline 1 Introduction 2 Method 3 Experiments 4 Conclusion M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 17 / 20
31. Conclusion Conclusion • A method to handle a precondition in the target language • Easy to write for developers • main objective completed! • Ensures that each path of the program is explored at most once • leads to very satisfying performances (on merge and others) • very important for thorough coverage criteria • An implementation in PathCrawler for C • Test it online! http://pathcrawler-online.com M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 18 / 20
32. Conclusion Perspectives • Optimizations • Memorize valid precondition paths instead of blindly going into the precondition • When possible (bounded loops) summarize the precondition to a finite disjunction (SMART [Godefroid 2008]) • When the precondition fails find and exploit the reason why (Korat) • Explore the combination of multiple symbolic executions (algebra) M. Delahaye, N. Kosmatov (LIG & CEA) Late Precondition in DSE RV 2013 19 / 20
33. That’s all folks! Thanks! Mickaël Delahaye mickael.delahaye@gmail.com http://micdel.fr Try PathCrawler online! http://pathcrawler-online.com