Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution

Interested in learning more about this topic? Read the overview of my research to learn more: https://www.gregorykapfhammer.com/research/

Transcript

1. Empirical Evaluation of an Approach to Resource Constrained Test Suite Execution Gregory M. Kapfhammer Department of Computer Science Allegheny College (in conjunction with Mary Lou Soffa and Daniel Mossé) Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 1/30
2. Looking Ahead Use of native code unloading during test suite execution in a resource constrained environment Identification of the testing techniques that yield the greatest reduction in execution time and native code size Characterization of how software applications and test suites restrict and/or support resource constrained testing Cost-benefit analysis for the use of sample-based and exhaustive profiles of program testing behavior Executes test suites faster when memory resources are limited! Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 2/30
3. Traditional Testing Techniques Specification Testing Testing Random MTTF(P) = k Reliability Software Input Freq Testing Structural Input Space Different approaches to establishing a confidence in the correctness of and finding faults within software Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 3/30
4. Test Suite Execution setUp tearDown T1 m 1 setUp tearDown T m n n oracle oracle A test executes a method and uses an oracle to determine if the method’s output is correct Test suite execution frameworks exist for many different programming languages (e.g., JUnit for Java) Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 4/30
5. Test Suite Execution with a JVM Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap methodA testOne Input Output Byte Code T P During testing the JVM must manage limited resources Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 5/30
6. Resource Constrained Testing exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... Memory Resident Native Code Bodies All Tests size: 128 KB T 1 Tn ... Test Suite T P Program JIT compiler produces native code that exhausts limited memory resources Frequent invocation of GC increases testing time Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 6/30
7. Test Suite Execution Strategies Omit tests? - Could reduce overall confidence in the correctness of P Use non-constrained environment? - Defects related to P’s interaction with environment might not be isolated Execute tests individually? - Might increase overall testing time and violate test order dependencies Unload with offline profile? - Not useful if P and T change frequently during regression testing Our Approach: Use online behavior profiles to guide the unloading of native code Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 7/30
8. Testing with Native Code Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector Code Cache Size k Timer { When - three separate techniques and What - behavior profile stored in code body (Zhang and Krintz) Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 8/30
9. Unloading Configurations: GC and TM {S, X} − {GC, TM} = C, UC, U, H Parameter Meaning C init period (GC cycles, secs) UC init unload freq (GC cycles, secs) U non-init unload freq (GC cycles, secs) H heap residency threshold (%) Use GC execution or timer expiration to trigger code unloading Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 9/30
10. Unloading Configuration: CS {S, X} − CS = Zinit , Zincr , UCS Parameter Meaning Zinit init code cache size (bytes) Zincr code cache increment size (bytes) UCS unload session resize trigger (count) Unload when native code cache is full Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 10/30
11. Experiment Goals and Design Research Question: Can an adaptive code unloading JVM reduce time and space overheads associated with memory constrained testing? Experiment Metrics: percent reduction in time, T % R (P, T) and space, S% R (P, T) Jikes RVM 2.2.1, JUnit 3.8.1, GNU/Linux 2.4.18 Sample-based (S) and exhaustive (X) program profiles Timer (TM), garbage collection (GC), and code cache size (CS) triggers the unloading technique Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 11/30
12. Case Study Applications Name Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the MIN Jikes RVM heap size Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 12/30
13. Native Code Unloading Configurations Name GC CS TM UBS 4, 1, 1, 0.0 49370, 512, 5 3, .5, 1, 0.0 L 5, 1, 3, 0.0 49370, 512, 5 3, .5, 1, 0.0 SC 3, 1, 1, 0.0 49370, 512, 5 2, .5, 1, 0.0 S 4, 1, 1, 0.0 49370, 512, 5 3, .5, 1, 0.0 JD 8, 1, 4, 0.0 49370, 512, 5 3, .5, 1, 0.0 ID 1, 1, 3, 0.0 65536, 8192, 5 2, .5, 1, 0.0 Ten minutes (or less) of configuration time for each technique S and X use same configuration to a ensure fair comparison Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 13/30
14. Testing Time Overhead: Full RVM UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688 When memory is not constrained, testing time is acceptable Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 14/30
15. Testing Time Overhead: Min RVM UBS L SC S JD I Application, Min JVM 5 10 15 20 25 30 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376 Testing time increases significantly when memory is Min Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 15/30
16. Summary of Reductions for Library Name T % R (P, T) S% R (P, T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 16/30
17. Testing Time Overhead: Library S GC X GC S CS X CS S TM X TM Library 2 4 6 8 10 12 Execution Time sec S GC X GC S CS X CS S TM X TM 7.168 7.222 6.998 7.09 7.238 7.292 S vs. X: Similar reductions for all code unloading techniques Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 17/30
18. Testing Space Overhead: Library S GC X GC S CS X CS S TM X TM Library 20000 40000 60000 80000 Code Size KB S GC X GC S CS X CS S TM X TM 16052.7 26463.7 29212.1 30444.1 20612.2 28545.1 Code size reduction does not always yield best testing time Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 18/30
19. Code Size Fluctuation: Library 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC S-GC causes code size fluctuation that increases testing time Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 19/30
20. Summary of Reductions for Stack Name T % R (P, T) S% R (P, T) S-GC 24.3 79.0 X-GC 25.4 63.4 S-TM 25.0 64.9 X-TM 24.6 47.8 S-CS 24.7 61.6 X-CS 20.9 46.9 Across all applications: S-TM or S-CS normally produce the largest reduction in testing time overhead Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 20/30
21. Testing Time Overhead: Stack S GC X GC S CS X CS S TM X TM Stack 1 2 3 4 5 6 Execution Time sec S GC X GC S CS X CS S TM X TM 3.932 3.872 3.914 4.11 3.896 3.918 S vs. X: Similar time reductions for another application Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 21/30
22. Testing Space Overhead: Stack S GC X GC S CS X CS S TM X TM Stack 10000 20000 30000 40000 50000 60000 70000 Code Size KB S GC X GC S CS X CS S TM X TM 13646.5 23797.1 24999.4 34524.5 22794.2 33977.3 S-GC also produces best code size reduction without creating greatest reduction in testing time Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 22/30
23. Code Size Fluctuation: Stack 0 0.5 1 1.5 2 2.5 3 3.5 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC S-GC also causes code size fluctuations in other applications Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 23/30
24. Summary of Reductions for ID Name T % R (P, T) S% R (P, T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 Unloading can decrease native code size while still creating an overall increase in testing time Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 24/30
25. Number of Code Unloads 2.5 5 7.5 10 12.5 15 17.5 20 # Code Unloading Times ID JD S SC L UBS Application ID JD S SC L UBS 2.0 16.0 12.0 4.0 11.0 11.4 4.0 16.6 6.0 4.0 9.0 8.0 4.0 14.0 2.0 1.4 2.0 2.0 S GC S TM S CS All techniques cause ID to perform few unloading sessions Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 25/30
26. Number of Unloaded Code Bodies 500 1000 1500 2000 2500 # Unloaded Native Code Bodies ID JD S SC L UBS Application ID JD S SC L UBS 164.0 1970.2 589.8 163.0 542.8 469.4 635.8 2228.2 437.4 303.4 533.0 398.4 664.4 1953.8 326.6 194.0 272.4 238.6 S GC S TM S CS ID’s large working set forces unloading of many code bodies Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 26/30
27. Summary of Reductions Name T % R (P, T) S% R (P, T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 27/30
28. Conclusions Dynamic compilation in JVMs can increase testing time if memory is constrained Adaptive unloading of native code often reduces memory overhead, avoids GC invocation, and reduces testing time Impact of unloading varies with respect to size of application’s working set and program testing behavior Code unloading JVM can be rapidly configured to produce useful reductions in the time and space overheads of testing Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 28/30
29. Future Work Include new case study applications and test suites Experiments to measure the impact of garbage collection and heap compression algorithms (e.g., Jikes RVM MMTk) Regression test suite prioritization and reduction techniques that consider structural coverage and time or space overheads Real testing framework for emerging operating systems that support Java in ad hoc networks (e.g., MagnetOS) Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 29/30
30. Additional Resources Kapfhammer et al. Testing in Resource Constrained Execution Environments. In IEEE/ACM Automated Software Engineering. November 7 - 11, 2005. http : //cs.allegheny.edu/˜gkapfham/research/juggernaut/ Empirical Evaluation of an Approach to Resource Constrained ..., Univ. of Pittsburgh, Nov. 16, 2005 – p. 30/30