Automated isolation for white-box test generation

D. Honfi, Z. Micskei. "Automated isolation for white-box test generation", Information and Software Technology 125 (2020), presented at ESEM 2021.

White-box test generation is a technique used for automatically selecting test inputs using only the code under test. A challenge when applying them is handling invocations to external modules or dependencies in the code under test. In this paper, we present an automated approach addressing the external dependency challenge for white-box test generation. We implemented the approach in a ready-to-use tool using Microsoft Pex as a test generator, and evaluated it on 10 open-source projects from GitHub having more than 38.000 lines of code in total.

Paper: https://doi.org/10.1016/j.infsof.2020.106319

Transcript

1. Automated isolation for white-box test generation Dávid Honfi ESEM’21 Journal-First Papers micskeiz mit.bme.hu/~micskeiz Information and Software Technology, 125, 2020. DOI: 10.1016/j.infsof.2020.106319 Zoltán Micskei davidhonfi
2. Automated isolation for white-box test generation 2 Generating tests/test inputs from the code • Numerous techniques and tools (→ see tool list) • Many challenges in real-world code • Handling dependencies and environment Motivation: white-box test generation bool TransferMoney(Token userToken, long amount, Account destination) { if (amount <= 0) throw new Exception("Invalid amount to transfer"); int balance = DB.RunQuery<int>("GetBalance", userToken); if (balance < amount) throw new Exception("Not enough balance"); TransferProcessor tp = new TransferProcessor(userToken); ProcessedTransfer pt = tp.Process(amount, destination); return pt.IsSuccess; } DB network
3. Automated isolation for white-box test generation 3 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper
4. Automated isolation for white-box test generation 4 Design of the approach & tool AutoIsolator tool
5. Automated isolation for white-box test generation 5 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper
6. Automated isolation for white-box test generation 6 • 10 open-source projects from GitHub – 7 187 methods – 2 596 handled by Pex • Measured – LoC (sum 38k, median ~11) – Cyclomatic complexity – External method invocations – External member access Overview of selected projects → Data in Brief paper Classes with more than 40 external method invocations! (median 4)
7. Automated isolation for white-box test generation 7 • Not supported: abstract/generic/nested class; not public method – 4591 out of the 7 187 method (!) • Mean statement coverage for remaining 2 596 methods: 52% Pex’s performance on projects (baseline) No tests for 1/3 of methods (object creation) 100% coverage for 1088 methods 2 685 failing tests out of 7 373 (unexpected exceptions) Classification of generated tests is not trivial! See SQJ 27:3
8. Automated isolation for white-box test generation 8 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper
9. Automated isolation for white-box test generation 9 RQ1: How does it improve statement and branch coverage? RQ2: How does it increase the time spent with test generation? Comparing AutoIsolator with Pex only Improving coverage in 195 cases with an average improvement of 52.95% 9.54 seconds on average (~ time for Pex’ generation)
10. Automated isolation for white-box test generation 10 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper Information and Software Technology, 125, 2020. DOI: 10.1016/j.infsof.2020.106319 Data in Brief, 31, 2020. DOI: 10.1016/j.dib.2020.105962 https://ftsrg.mit.bme.hu/autoisolator/