Slide 1

Slide 1 text

Automated isolation for white-box test generation Dávid Honfi ESEM’21 Journal-First Papers micskeiz mit.bme.hu/~micskeiz Information and Software Technology, 125, 2020. DOI: 10.1016/j.infsof.2020.106319 Zoltán Micskei davidhonfi

Slide 2

Slide 2 text

Automated isolation for white-box test generation 2 Generating tests/test inputs from the code • Numerous techniques and tools (→ see tool list) • Many challenges in real-world code • Handling dependencies and environment Motivation: white-box test generation bool TransferMoney(Token userToken, long amount, Account destination) { if (amount <= 0) throw new Exception("Invalid amount to transfer"); int balance = DB.RunQuery("GetBalance", userToken); if (balance < amount) throw new Exception("Not enough balance"); TransferProcessor tp = new TransferProcessor(userToken); ProcessedTransfer pt = tp.Process(amount, destination); return pt.IsSuccess; } DB network

Slide 3

Slide 3 text

Automated isolation for white-box test generation 3 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper

Slide 4

Slide 4 text

Automated isolation for white-box test generation 4 Design of the approach & tool AutoIsolator tool

Slide 5

Slide 5 text

Automated isolation for white-box test generation 5 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper

Slide 6

Slide 6 text

Automated isolation for white-box test generation 6 • 10 open-source projects from GitHub – 7 187 methods – 2 596 handled by Pex • Measured – LoC (sum 38k, median ~11) – Cyclomatic complexity – External method invocations – External member access Overview of selected projects → Data in Brief paper Classes with more than 40 external method invocations! (median 4)

Slide 7

Slide 7 text

Automated isolation for white-box test generation 7 • Not supported: abstract/generic/nested class; not public method – 4591 out of the 7 187 method (!) • Mean statement coverage for remaining 2 596 methods: 52% Pex’s performance on projects (baseline) No tests for 1/3 of methods (object creation) 100% coverage for 1088 methods 2 685 failing tests out of 7 373 (unexpected exceptions) Classification of generated tests is not trivial! See SQJ 27:3

Slide 8

Slide 8 text

Automated isolation for white-box test generation 8 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper

Slide 9

Slide 9 text

Automated isolation for white-box test generation 9 RQ1: How does it improve statement and branch coverage? RQ2: How does it increase the time spent with test generation? Comparing AutoIsolator with Pex only Improving coverage in 195 cases with an average improvement of 52.95% 9.54 seconds on average (~ time for Pex’ generation)

Slide 10

Slide 10 text

Automated isolation for white-box test generation 10 • Automatically isolating dependencies • Source code transformations • Implemented for IntelliTest / Pex Approach & AutoIsolator • Baseline with Pex only on 10 OS projects • Largest dataset about Pex’s performance • Showing current limitations Baseline measurements • Comparing AutoIsolator with baseline • Coverage improvement • Time taken Comparison & evaluation Main results of the paper Information and Software Technology, 125, 2020. DOI: 10.1016/j.infsof.2020.106319 Data in Brief, 31, 2020. DOI: 10.1016/j.dib.2020.105962 https://ftsrg.mit.bme.hu/autoisolator/