An experimental study of methods for executing test suites in memory constrained environments

An experimental study of methods for executing test suites in memory constrained environments

Interested in learning more about this topic? Visit this web site to read the paper: https://www.gregorykapfhammer.com/research/papers/Bhadra2009/

4ae30d49c8cc07e42d5a871efb9bcfba?s=128

Gregory Kapfhammer

May 18, 2009
Tweet

Transcript

  1. Introduction Automated Testing Empirical Evaluation Conclusion An Experimental Study of

    Methods for Executing Test Suites in Memory Constrained Environments Suvarshi Bhadra‡ and Alexander Conrad, Charles Hurkes, Brian Kirklin, Gregory M. Kapfhammer† ‡ Milcord LLC † Department of Computer Science Allegheny College 4th Workshop on the Automation of Software Test (AST 2009) May 18 - 19, 2009 1 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  2. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  3. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  4. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  5. Introduction Automated Testing Empirical Evaluation Conclusion Important Contributions Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implement and empirically evaluate the efficiency and effectiveness of techniques for automatically running test suites in memory constrained execution environments 2 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  6. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution

    setUp tearDown T1 m 1 oracle A test invokes one or more methods under test and uses an oracle to determine if the method’s output is correct 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  7. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution

    setUp tearDown T1 m 1 setUp tearDown T m n n oracle oracle A test invokes one or more methods under test and uses an oracle to determine if the method’s output is correct 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  8. Introduction Automated Testing Empirical Evaluation Conclusion Automated Test Suite Execution

    setUp tearDown T1 m 1 setUp tearDown T m n n oracle oracle Test suite execution frameworks exist for many different programming languages (e.g., JUnit for Java) 3 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  9. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges

    Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  10. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges

    Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Heap methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  11. Introduction Automated Testing Empirical Evaluation Conclusion Test Suite Execution Challenges

    Program Stack Fast? Interpreter? Machine Virtual JIT? Adaptive? Native Code Cache Heap methodA testOne Input Output Byte Code T P The virtual machine manages limited memory during testing 4 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  12. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints

    Tn m 3 m n T1 tearDown setUp m m 1 2 T2 oracle oracle oracle Startup: Store bytecodes and the initial objects 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  13. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints

    Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Optimize: Create native code from bytecodes 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  14. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints

    Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Threshold: Allocate too many additional objects 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  15. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints

    Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Collection: Remove dead objects from the heap 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  16. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Memory Constraints

    Tn m 3 m n m m 1 2 T2 T1 tearDown setUp oracle oracle oracle Problem: Collector does not remove native code! 5 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  17. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code

    Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  18. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code

    Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector k Timer { L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  19. Introduction Automated Testing Empirical Evaluation Conclusion Testing with Native Code

    Unloading exec time: 1% size: 100 KB inv ct: 8 inv ct: 1200 exec time: 15% size: 64 KB inv ct: 15 size: 50 KB exec time: 2% inv ct: 50 size: 75 KB exec time: 22% TE v inv ct: 1 exec time: 1% inv ct: 2 exec time: 2% ms m t ... Test Executor TE u ... All Tests size: 128 KB T 1 Tn ... Test Suite T P Program What to unload? When to unload? Garbage Collector Code Cache Size k Timer { L. Zhang and C. Krintz. ACM Trans. on Arch. and Code Optim. 6 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  20. Introduction Automated Testing Empirical Evaluation Conclusion Case Study Applications Name

    Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Empirically determined the Min Jikes RVM heap size 7 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  21. Introduction Automated Testing Empirical Evaluation Conclusion Case Study Applications Name

    Min Size (MB) # Tests NCSS UniqueBoundedStack (UBS) 8 24 362 Library (L) 8 53 551 ShoppingCart (SC) 8 20 229 Stack (S) 8 58 624 JDepend (JD) 10 53 2124 IDTable (ID) 11 24 315 Future Work: Conduct experiments with larger applications 7 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  22. Introduction Automated Testing Empirical Evaluation Conclusion Testing Time Overhead: Full

    RVM UBS L SC S JD I Application, Full RVM 1 2 3 4 5 Execution Time sec UBS L SC S JD I 0.47 0.52 0.468 0.48 3.67 0.688 When memory is not constrained, testing time is acceptable 8 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  23. Introduction Automated Testing Empirical Evaluation Conclusion Testing Time Overhead: Min

    RVM UBS L SC S JD I Application, Min JVM 5 10 15 20 25 30 Execution Time sec UBS L SC S JD I 4.276 10.644 3.686 5.196 21.108 3.376 Testing time increases significantly when memory is Min 9 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  24. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for

    Library Technique T % R (P , T) S% R (P , T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing 10 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  25. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for

    Library Technique T % R (P , T) S% R (P , T) S-GC 32.7 78.8 X-GC 32.1 65.0 S-TM 32.0 72.8 X-TM 31.5 62.3 S-CS 34.3 61.4 X-CS 33.4 59.8 Significant reductions in time and space required for testing 10 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  26. Introduction Automated Testing Empirical Evaluation Conclusion Code Size Fluctuations for

    Library 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC S-GC causes code size fluctuations that increase testing time 11 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  27. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for

    Identifier Technique T % R (P , T) S% R (P , T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 A decrease in native code size leads to an increase in test execution time! Why? Identifier has a large working set. 12 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  28. Introduction Automated Testing Empirical Evaluation Conclusion Summary of Reductions for

    Identifier Technique T % R (P , T) S% R (P , T) S-GC -1.1 42.5 X-GC -1.1 26.7 S-TM -1.2 44.5 X-TM -.29 28.8 S-CS -.77 51.4 X-CS -1.4 61.4 A decrease in native code size leads to an increase in test execution time! Why? Identifier has a large working set. 12 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  29. Introduction Automated Testing Empirical Evaluation Conclusion Improvements to Automated Testing

    Technique T % R (P , T) S% R (P , T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead 13 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  30. Introduction Automated Testing Empirical Evaluation Conclusion Improvements to Automated Testing

    Technique T % R (P , T) S% R (P , T) S-GC 16.1 68.4 X-GC 16.4 52.8 S-TM 17.1 62.6 X-TM 16.4 45.9 S-CS 17.6 58.8 X-CS 15.3 54.8 Across all applications, adaptive code unloading techniques reduce both testing time and space overhead 13 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  31. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and

    Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  32. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and

    Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  33. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and

    Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  34. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and

    Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Reduction discards some of the n tests in an attempt to decrease testing time while still preserving objectives like coverage or fault detection. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  35. Introduction Automated Testing Empirical Evaluation Conclusion Future Work: Reduction and

    Prioritization Before After Reduction Prunes the Test Suite Before After Prioritization Reorders the Tests It is expensive to run a test suite T = T1, . . . , Tn . Prioritization searches through the n! = n × n − 1 × . . . × 1 orderings for those that maximize an objective function like memory loads and unloads. 14 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  36. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  37. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  38. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments
  39. Introduction Automated Testing Empirical Evaluation Conclusion Concluding Remarks Program Test

    Executor Test Suite Test Results Coverage Report Automated Test Execution 0 1 2 3 4 5 6 7 Time sec 0 10000 20000 30000 40000 Code Size KB S CS S GC Detailed Empirical Results Implementation and empirical evaluation of methods for testing in memory constrained environments Aim to apply these methods to T-Mobile G1 with Google Android http://www.cs.allegheny.edu/~gkapfham/research/juggernaut/ 15 / 15 An Experimental Study of Methods for Executing Test Suites in Memory Constrained Environments