1 25-27 NOVEMBER SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS SPIDER: Specification-based Integration Defect Revealer Vladislav Feofilaktov, Vladimir Itsykson, SPbPU, JetBrains Research

2 Introduction Current practices of the software development: 1. Applications use external components and libraries 2. Mostly, external libraries are produced by third party developers Usually, these libraries are blackboxes that provide a public API

3 Problems with external libraries 1. Libraries are provided without docs or with badly docs 2. Descriptions of API calls and types aren’t enough 3. Examples are uninformative 4. Library component semantics are unclear Integration errors

4 // abstract file library FileReader file = new FileReader(path); // … file.close(); // … file.readLine(); // IOException: Stream closed Trivial example of integration error

5 Aim of the research 1. Develop the approach: a. Allows to find integration errors b. Must be statically 2. Develop the tool: a. Implement the approach b. Use java bytecode as a target c. Evaluate the tool on real projects

6 State of the art Existing approaches have some next restrictions: 1. Focuses only on the C language 2. Requires library source code to make analysis possible 3. Requires library code changes to write incode specification 4. Don’t allow to check a correctness of library usage

7 Our approach 1. Formalize the library semantics 2. Formal definition of integration errors 3. Apply static analysis methods for automatic detection of integration errors

8 Specification of external library LibSL (Library Specification Language) is a declarative language that allows writing components specification using extended finite state machines. LibSL allows to describe: 1. Structure of library 2. Public API functions 3. API functions behaviour 4. Library component’s behaviour Itsykson V. M. LibSL: Language for Specification of Software Libraries, Programmnaya Ingeneria, 2018, vol. 9, no. 5, pp. 209—220.

9 Simple example of LibSL specification libsl "1.0.0"; library File version "1.0.0"; types { File (org.jetbrains.research.File); String (string); } automaton org.jetbrains.research.File : File { initstate Init; state Open; finishstate Close; shift Init -> Open(open); shift Open -> Close(close); fun open(name: String) requires isNameEmpty: name != “”; fun close(); } Set LibSL version Specification for library ‘File’ version 1.0.0 Set semantic types and their java qualified names Description of an automaton with 1. initstate 2. simple state 3. finishstate 4. shift from state “Init” to state “Open” on calling function “open” and another one 5. function “open” with one String argument 6. precondition with name “isNameEmpty” 7. just a function

10 Definition of integration errors Types of integration errors: 1. API usage errors 2. Incorrect scenarios of library usage API usage errors: API function contracts violations Incorrect scenarios of library usage: 1. Incorrect order of API function calls 2. Incorrect finish state of automata

11 High-level scheme of our approach Static Analyzer Library Approximation Generator Error List Library Specification Library Main Application Library Approximation

12 Prototype implementation 1. Target platform JVM 2. LibSL parser 3. Static analyzer platform KEX We have developed prototype named SPIDER: Specification-based Integration Defect Revealer

13 Kex ● Java bytecode static analysis platform ● Supports few modes and allows writing new ones ● Uses special internal representation KFG ● Transforms source program to logical predicates ● Feed logical predicates to the SMT-solver A. Abdullin, M. Akhin and M. Belyaev, "Kex at the 2021 SBST Tool Competition," 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST), 2021, pp. 32-33, doi: 10.1109/SBST52555.2021.00014

14 The way to check correctness 1. Use LibSL specification as a library oracle 2. Modify KFG to informate KEX about correct library behaviour 3. KEX analyzes the modified KFG to find errors

15 Detailed scheme of SPIDER KEX Static Analyzer LibSL Parser KFG Builder Instrumentation Module Analysis Module Library Specification Library Compiled Binaries Main Application Compiled Binaries Error List ASG Library KFG Application KFG Instrumented Library KFG

16 Summary of SPIDER ● Implemented as KEX module ● Written in Kotlin ● Use LibSL parser* ● Use Kfg library to build KFG ● Z3 and boolector as SMT-solvers *https://github.com/vorpal-research/libsl

17 Evaluation ● Synthetic test projects ○ To verify the correctness of built model ○ To check KFG instrumentation ○ To verify detection of all types of errors ● Applicability approach to real-world library okHTTP* library with real curl implementation ○ To check the project without changes ○ To check the project with manually added errors *https://github.com/square/okhttp

18 Conclusion Results: ● An approach was developed to find integration errors ● The approach was implemented as the tool named SPIDER ● SPIDER was evaluated on synthetic and real projects Feature plans: ● Add rest integration error types to SPIDER ● Increase the performance ● Evaluate SPIDER on many open source projects

19 Thank You! Follow TMPA on Facebook TMPA-2021 Conference