Testing in resource-constrained execution environments

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Kapfhammer2005/

Transcript

1. Testing in Resource Constrained Execution Environments Gregory M. Kapfhammer Department of Computer Science Allegheny College Mary Lou Soffa Department of Computer Science University of Virginia Daniel Mossé Department of Computer Science University of Pittsburgh Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 1/19
2. Contributions Use of native code unloading during test suite execution in a resource constrained environment Identification of the testing techniques that yield the greatest reduction in execution time and native code size Characterization of how software applications and test suites restrict and/or support resource constrained testing Cost-benefit analysis for the use of sample-based and exhaustive profiles of program testing behavior Executes test suites faster when memory resources are limited! Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 2/19
3. Test Suite Execution with a JVM Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap methodA testOne Input Output Byte Code T P During testing the JVM must manage limited resources Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 3/19
4. Resource Constrained Testing exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... Memory Resident Native Code Bodies All Tests size: 128 KB T 1 Tn ... Test Suite T P Program JIT compiler produces native code that exhausts limited memory resources Frequent invocation of GC increases testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 4/19
5. Test Suite Execution Strategies Omit tests? - Could reduce overall confidence in the correctness of P Use non-constrained environment? - Defects related to P’s interaction with environment might not be isolated Execute tests individually? - Might increase overall testing time and violate test order dependencies Unload with offline profile? - Not useful if P and T change frequently during regression testing Our Approach: Use online behavior profiles to guide the unloading of native code Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 5/19
6. Experiment Goals and Design Research Question: Can an adaptive code unloading JVM reduce time and space overheads associated with memory constrained testing? Experiment Metrics: percent reduction in time, T % R (P, T) and space, S% R (P, T) Jikes RVM 2.2.1, JUnit 3.8.1, GNU/Linux 2.4.18 Sample-based (S) and exhaustive (X) program profiles Timer (TM), garbage collection (GC), and code cache size (CS) triggers the unloading technique Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 6/19
7. Case Study Applications Name Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the MIN Jikes RVM heap size Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 7/19
8. Testing Time Overhead: Full RVM UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688 When memory is not constrained, testing time is acceptable Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 8/19
9. Testing Time Overhead: Min RVM UBS L SC S JD I Application, Min JVM 5 10 15 20 25 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376 Testing time increases significantly when memory is Min Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 9/19
10. Summary of Reductions for Library Name T % R (P, T) S% R (P, T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 10/19
11. Testing Time Overhead: Library S GC X GC S CS X CS S TM X TM Library 2 4 6 8 10 12 Execution Time sec S GC X GC S CS X CS S TM X TM 7.168 7.222 6.998 7.09 7.238 7.292 S vs. X: Similar reductions for all code unloading techniques Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 11/19
12. Testing Space Overhead: Library S GC X GC S CS X CS S TM X TM Library 20000 40000 60000 80000 Code Size KB S GC X GC S CS X CS S TM X TM 16052.7 26463.7 29212.1 30444.1 20612.2 28545.1 Code size reduction does not always yield best testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 12/19
13. Code Size Fluctuation: Library 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC S-GC causes code size fluctuation that increases testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 13/19
14. Summary of Reductions for ID Name T % R (P, T) S% R (P, T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 Unloading can decrease code size while still creating an overall increase in testing time Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 14/19
15. Number of Code Unloads 2.5 5 7.5 10 12.5 15 17.5 20 # Code Unloading Times ID JD S SC L UBS Application ID JD S SC L UBS 2.0 16.0 12.0 4.0 11.0 11.4 4.0 16.6 6.0 4.0 9.0 8.0 4.0 14.0 2.0 1.4 2.0 2.0 S GC S TM S CS All techniques cause ID to perform few unloading sessions Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 15/19
16. Number of Unloaded Code Bodies 500 1000 1500 2000 2500 # Unloaded Native Code Bodies ID JD S SC L UBS Application ID JD S SC L UBS 164.0 1970.2 589.8 163.0 542.8 469.4 635.8 2228.2 437.4 303.4 533.0 398.4 664.4 1953.8 326.6 194.0 272.4 238.6 S GC S TM S CS ID’s large working set forces unloading of many code bodies Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 16/19
17. Summary of Reductions Name T % R (P, T) S% R (P, T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 17/19
18. Conclusions and Future Work Dynamic compilation in JVMs can increase testing time if memory is constrained Adaptive unloading of native code often reduces memory overhead, avoids GC invocation, and reduces testing time Impact of unloading varies with respect to size of application’s working set and program testing behavior Regression test suite prioritization and reduction techniques that consider structural coverage and time and space overheads Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 18/19
19. Additional Resources Kapfhammer et al. Testing in Resource Constrained Execution Environments. In IEEE/ACM Automated Software Engineering. November 7 - 11, 2005. http : //cs.allegheny.edu/˜gkapfham/research/juggernaut/ Testing in Resource Constrained Execution Environments, IEEE/ACM ASE 2005, Nov. 7-11, 2005 – p. 19/19