Splat

FYP research

Transcript

1. Splat: Simple Python Lazy Automated Tester Lee Wei Yeong lwy08@doc.ic.ac.uk tiny.cc/lwy08 1 Saturday, 3 November, 12
2. Introduction 2 2 Saturday, 3 November, 12
3. Motivation • writing unit tests ‣ crucial ‣ costly but invisible ‣ can be automated 3 3 Saturday, 3 November, 12
4. • dynamically typed • clean, concise & elegant syntax • builtin introspection & reflection facility 4 4 Saturday, 3 November, 12
5. Why bytecode? • “Growing numbers of... closed source applications... using...Python...” (Smith, 2010) • simple • fast 5 5 Saturday, 3 November, 12
6. Background 6 6 Saturday, 3 November, 12
7. Existing tools • Pythoscope • Pytestsgenerator • ...but neither works! 7 7 Saturday, 3 November, 12
8. Existing tools • Ruby Test case Generator (RuTeG) ‣ structural testing ‣ evolutionary search ‣ tests object-oriented programs 8 8 Saturday, 3 November, 12
9. Software testing 9 9 Saturday, 3 November, 12
10. Sample test target 10 10 Saturday, 3 November, 12
11. Sample unit test 11 11 Saturday, 3 November, 12
12. SPLAT 12 12 Saturday, 3 November, 12
13. Lazy instantiation • technique inspired by IRULAN • initially: None proxy metaclass wrapper • test data only generated for active variables 13 13 Saturday, 3 November, 12
14. Lazy instantiation 14 14 Saturday, 3 November, 12
15. Lazy instantiation 15 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE 15 Saturday, 3 November, 12
16. Lazy instantiation 16 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE function1 ( Param1(None), Param2(None), Param3(None) ) 16 Saturday, 3 November, 12
17. Lazy instantiation 17 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE function1 ( Param1(None), Param2(None), ‘default’ ) 17 Saturday, 3 November, 12
18. Lazy instantiation 18 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE function1 ( Param1(None), Param2(None), ‘default’ ) 18 Saturday, 3 November, 12
19. Lazy instantiation 19 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE TypeError! 19 Saturday, 3 November, 12
20. Lazy instantiation 20 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE function1 ( { ‘attr1’: Param1_attr1(None) }, Param2(None), ‘default’ ) 20 Saturday, 3 November, 12
21. Lazy instantiation 21 LOAD_GLOBAL (A) CALL_FUNCTION 0 STORE_FAST (arg1) LOAD_CONST (‘arg1’) LOAD_FAST (arg1) STORE_ATTR (attr1) LOAD_FAST (arg1) RETURN_VALUE function1 ( { ‘attr1’: Param1_attr1(None) }, None, ‘default’ ) 21 Saturday, 3 November, 12
22. Demonstration 22 22 Saturday, 3 November, 12
23. Architecture 23 23 Saturday, 3 November, 12
24. Challenges • Vague error messages • Lack of existing tool support • Test program contains imports 24 24 Saturday, 3 November, 12
25. Challenges • Relationship between input arguments • Various programming constructs • Range of test data values 25 25 Saturday, 3 November, 12
26. Limitations • Custom Exceptions, e.g. InvalidGraphType • Random functions • Generators 26 26 Saturday, 3 November, 12
27. Evaluation 27 27 Saturday, 3 November, 12
28. Criteria • Quality • Performance • Generality 28 28 Saturday, 3 November, 12
29. Results 29 Python package # 29 Saturday, 3 November, 12
30. Results 30 Mean number of iterations to achieve 100% coverage = 497 30 Saturday, 3 November, 12
31. Conclusion 31 31 Saturday, 3 November, 12
32. Future work • improve tool sophistication • optimise using PyPy • more comprehensive benchmarks 32 32 Saturday, 3 November, 12
33. evandrix.github.com/Splat 33 Saturday, 3 November, 12