An experimental study of methods for executing test suites in memory constrained environments

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Bhadra2009/

Transcript

1. Introduction Automated Testing Empirical Evaluation Conclusion An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments Suvarshi Bhadra‡ and Alexander Conrad, Charles Hurkes, Brian Kirklin, Gregory M. Kapfhammer† ‡ Milcord LLC † Department of Computer Science Allegheny College 4th Workshop on the Automation of Software Test (AST 2009) May 18 - 19, 2009 1 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
2. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
3. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
4. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
5. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
6. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution setUp tearDown T1 m 1 oracle A test invokes one or more methods under test and uses an oracle to determine if the method’s output is correct 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
7. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution setUp tearDown T1 m 1 setUp tearDown T m n n oracle oracle A test invokes one or more methods under test and uses an oracle to determine if the method’s output is correct 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
8. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution setUp tearDown T1 m 1 setUp tearDown T m n n oracle oracle Test suite execution frameworks exist for many different programming languages (e.g., JUnit for Java) 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
9. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
10. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Heap methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
11. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
12. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints Tn m 3 m n T1 tearDown setUp m m 1 2 T2 oracle oracle oracle Startup: Store bytecodes and the initial objects 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
13. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Optimize: Create native code from bytecodes 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
14. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Threshold: Allocate too many additional objects 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
15. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Collection: Remove dead objects from the heap 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
16. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Problem: Collector does not remove native code! 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
17. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
18. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector k Timer { L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
19. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector Code Cache Size k Timer { L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
20. Introduction Automated Testing Empirical Evaluation Conclusion Case Study Applications Name Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the Min Jikes RVM heap size 7 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
21. Introduction Automated Testing Empirical Evaluation Conclusion Case Study Applications Name Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Future Work: Conduct experiments with larger applications 7 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
22. Introduction Automated Testing Empirical Evaluation Conclusion Testing Time Overhead: Full RVM UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688 When memory is not constrained, testing time is acceptable 8 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
23. Introduction Automated Testing Empirical Evaluation Conclusion Testing Time Overhead: Min RVM UBS L SC S JD I Application, Min JVM 5 10 15 20 25 30 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376 Testing time increases significantly when memory is Min 9 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
24. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for Library Technique T % R (P , T) S% R (P , T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing 10 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
25. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for Library Technique T % R (P , T) S% R (P , T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing 10 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
26. Introduction Automated Testing Empirical Evaluation Conclusion Code Size Fluctuations for Library 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC S-GC causes code size fluctuations that increase testing time 11 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
27. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for Identifier Technique T % R (P , T) S% R (P , T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 A decrease in native code size leads to an increase in test execution time! Why? Identifier has a large working set. 12 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
28. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for Identifier Technique T % R (P , T) S% R (P , T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 A decrease in native code size leads to an increase in test execution time! Why? Identifier has a large working set. 12 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
29. Introduction Automated Testing Empirical Evaluation Conclusion Improvements to Automated Testing Technique T % R (P , T) S% R (P , T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead 13 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
30. Introduction Automated Testing Empirical Evaluation Conclusion Improvements to Automated Testing Technique T % R (P , T) S% R (P , T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead 13 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
31. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
32. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
33. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
34. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
35. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that maximize an objective function like memory loads and unloads. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
36. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
37. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
38. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
39. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments