Power Assert in Ruby

http://rubykaigi.org/2014/presentation/S-KazukiTsujimoto

Transcript

1. POWER ASSERT IN RUBY Kazuki Tsujimoto Nomura Research Institute, LTD.
2. SELF INTRODUCTION • A CRuby committer • Fix VM, GC bugs
3. AGENDA • Concept (13 slides) • Implementation (39 slides)
4. PROPOSAL NOTES • I'll give unduly deep explanation; not only of Ruby-level but VM/C-level
5. CONCEPT
6. TEST FRAMEWORKS unit spec test-unit ✔️ minitest ✔️ ✔️ RSpec ✔️
7. TEST FRAMEWORKS unit spec test-unit ✔️ minitest ✔️ ✔️ RSpec ✔️
8. TESTING WITH UNIT • Use assertions i = 0 assert_equal 0, i assert_include 3.times.to_a, 1
9. TESTING WITH UNIT • What is the difference? assert_include 3.times.to_a, 3 assert 3.times.to_a.include?(3)
10. TESTING WITH UNIT • Understandability of the report Failure: <[0, 1, 2]> expected to include <3>. Failure: <true> expected but was <false>
11. TESTING WITH UNIT • A number of assertions assert assert_alias_method assert_block assert_boolean assert_compare assert_const_defined assert_empty assert_equal a s s e r t _ f a i l _ a s s e r t i o n a s s e r t _ f a l s e a s s e r t _ i n _ d e l t a assert_in_epsilon assert_include assert_instance_of assert_kind_of assert_match assert_nil assert_no_match assert_not_const_defined assert_not_empty assert_not_equal assert_not_in_delta assert_not_in_epsilon assert_not_include assert_not_instance_of a s s e r t _ n o t _ k i n d _ o f a s s e r t _ n o t _ m a t c h a s s e r t _ n o t _ n i l assert_not_operator assert_not_predicate assert_not_respond_to assert_not_same assert_not_send assert_nothing_raised assert_nothing_thrown assert_operator assert_path_exist a s s e r t _ p a t h _ n o t _ e x i s t a s s e r t _ p r e d i c a t e a s s e r t _ r a i s e assert_raise_kind_of assert_raise_message assert_respond_to assert_same assert_send assert_throw assert_true
12. TESTING WITH POWER ASSERT • Use assert with a block • That's all assert { 3.times.to_a.include?(3) } Failure: assert { 3.times.to_a.include?(3) } | | | | | false | [0, 1, 2] #<Enumerator: 3:times>
13. POWER ASSERT IN GROOVY http://docs.codehaus.org/display/GROOVY/Groovy+1.7+release+notes
14. POWER_ASSERT FAMILY • power_assert • Requires CRuby 2.0.0 or later • test-unit-power_assert • test-unit 3(@ktou) • minitest-power_assert(@hsbt) • pry-power_assert(@spikeolaf)
15. DEBUGGING WITH POWER ASSERT • tap is frequently used for inspecting the value in method chains 3.times.to_a.tap {|i| p i }.include?(3) #=> [0, 1, 2]
16. DEBUGGING WITH POWER ASSERT • Extend p with power_assert to inspect all values p { 3.times.to_a.include?(3) } p { 3.times.to_a.include?(3) } | | | | | false | [0, 1, 2] #<Enumerator: 3:times>
17. DEBUGGING WITH POWER ASSERT module PowerP def p(*, &blk) if blk PowerAssert.start(blk, assertion_method: __callee__) do |pa| pa.yield puts pa.message_proc.call end else super end end end
18. IMPLEMENTATION
19. THE CHALLENGES OF IMPLEMENTATION • How to get values • How to get position information 3.to_s.length | | | 1 "3" to_s “3” length 1 Values to_s 2 length 7 Position info
20. HOW TO GET VALUES
21. TRACEPOINT
22. WHAT IS TRACEPOINT? • Tracing API which aims to replace set_trace_func • Introduced in Ruby 2.0
23. TRACEPOINT trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}
24. TRACEPOINT trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]} New event info • return_value • raised_exception ! Traditional event info • method_id • binding • etc.
25. OFF TOPIC: EXCEPTION#CAUSE • Exception#cause returns the previous exception at the time the exception was raised begin begin raise "A" rescue raise "B" end rescue => e p [e, e.cause] # => [#<RuntimeError: B>, #<RuntimeError: A>] end
26. OFF TOPIC: EXCEPTION#CAUSE • Implement Exception#cause using TracePoint class Exception attr_accessor :cause TracePoint.trace(:raise){|tp| e = tp.raised_exception e.cause = $! } end https://twitter.com/_ko1/statuses/409267459414179840
27. FILTERING RETURN VALUES • TracePoint captures return values of all method calls • We must filter them trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}
28. FILTERING RETURN VALUES • TracePoint captures return values of all method calls • We must filter them trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}
29. FILTERING RETURN VALUES • Use Binding#eval and Kernel#caller_locations • tp.binding.eval(‘caller_locations’).length returns the execution stack length of each method calls
30. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7} {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }
31. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7} {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }
32. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7} {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }
33. GETTING THE VALUE OF THE VARIABLE • Binding#eval is also used to get the value of the variable ary = 3.times.to_a assert { ary.include?(3) } Failure: assert { ary.include?(3) } | | | false [0, 1, 2]
34. TRAP OF POWER_ASSERT • The report does not contain the value of == require 'test-unit-power_assert' ... assert { [0, 1, 2].find {|i| i.odd? } == 3 } $ ruby ./test.rb Failure: assert { [0, 1, 2].find {|i| i.odd? } == 3 } | 1
35. WHAT SHOULD WE DO? • Require power_assert first $ ruby -rpower_assert ./test.rb Failure: assert { [0, 1, 2].find {|i| i.odd? } == 3 } | | | false 1
36. WHAT HAPPENED? • Minimum code for the problem trace = TracePoint.new(:return, :c_return) {|tp| p tp.return_value } trace.enable { 1 == 3 } (Expected) false (Actual)
37. INVESTIGATING BYTECODE • Use RubyVM::InstructionSequence.disasm puts RubyVM::InstructionSequence.disasm(Proc.new { [0, 1, 2].find {|i| i.odd? } == 3 }) 0000 trace 256 0002 trace 1 0004 duparray [0, 1, 2] 0006 send <callinfo!mid:find, argc:0, block:block> 0008 putobject 3 0010 opt_eq <callinfo!mid:==, argc:1, ARGS_SKIP> 0012 trace 512 0014 leave
38. INVESTIGATING BYTECODE • Use RubyVM::InstructionSequence.disasm puts RubyVM::InstructionSequence.disasm(Proc.new { [0, 1, 2].find {|i| i.odd? } == 3 }) 0000 trace 256 0002 trace 1 0004 duparray [0, 1, 2] 0006 send <callinfo!mid:find, argc:0, block:block> 0008 putobject 3 0010 opt_eq <callinfo!mid:==, argc:1, ARGS_SKIP> 0012 trace 512 0014 leave
39. OPT_EQ { val = opt_eq_func(recv, obj, ci); if (val == Qundef) { /* other */ PUSH(recv); PUSH(obj); CALL_SIMPLE_METHOD(recv); } }
40. OPT_EQ_FUNC VALUE opt_eq_func(VALUE recv, VALUE obj, CALL_INFO ci) { if (FIXNUM_2_P(recv, obj) && BASIC_OP_UNREDEFINED_P(BOP_EQ, FIXNUM_REDEFINED_OP_FLAG)) { return (recv == obj) ? Qtrue : Qfalse; } ...
41. SEND { ci->argc = ci->orig_argc; ci->blockptr = 0; vm_caller_setup_args(th, reg_cfp, ci); vm_search_method(ci, ci->recv = TOPN(ci->argc)); CALL_METHOD(ci); }
42. VM_CALL_CFUNC static VALUE vm_call_cfunc(rb_thread_t *th, rb_control_frame_t *reg_cfp, rb_call_info_t *ci) { ... val = vm_call_cfunc_latter(th, reg_cfp, ci); EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv, me->called_id, me->klass, val); ...
43. VM_CALL_CFUNC static VALUE vm_call_cfunc(rb_thread_t *th, rb_control_frame_t *reg_cfp, rb_call_info_t *ci) { ... val = vm_call_cfunc_latter(th, reg_cfp, ci); EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv, me->called_id, me->klass, val); ... TracePoint is invoked by EXEC_EVENT_HOOK
44. DISABLE VM OPTIMIZATION • Set optimization flags using RubyVM::InstructionSequence.compile_opti on= method • power_assert contains following code RubyVM::InstructionSequence.compile_option = { specialized_instruction: false }
45. DISABLE VM OPTIMIZATION • Only power_assert is compiled with optimization $ ruby -rpower_assert ./test.rb Compile option Test code power_assert Optimization: Enabled Compile Evaluate Disabled Compile Evaluate
46. DISABLE VM OPTIMIZATION • Both power_assert and test code are compiled with optimization $ ruby ./test.rb Compile option Test code power_assert Optimization: Enabled Compile Evaluate Compile Evaluate Evaluate Disabled
47. HOW TO GET POSITION INFORMATION
48. RIPPER
49. WHAT IS RIPPER? • A Ruby script parser • Ripper.sexp create S-exp tree including position information # assert { [0, 1, 2].find {|i| i.odd? } == 3 } [:program, [[:method_add_block, [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []], [:brace_block, nil, [[:binary, [:method_add_block, [:call, [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".", [:@ident, "find", [1, 19]]], [:brace_block, [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false], [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]], :==, [:@int, "3", [1, 41]]]]]]]]
50. WHAT IS RIPPER? • A Ruby script parser • Ripper.sexp create S-exp tree including position information # assert { [0, 1, 2].find {|i| i.odd? } == 3 } [:program, [[:method_add_block, [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []], [:brace_block, nil, [[:binary, [:method_add_block, [:call, [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".", [:@ident, "find", [1, 19]]], [:brace_block, [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false], [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]], :==, [:@int, "3", [1, 41]]]]]]]] "assert"
51. EXTRACT POSITION INFORMATION • Use multiple assignment tag, * = sexp case tag when :program _, ((tag0, (tag1, (tag2, (tag3, mname, _)), _), (tag4, _, ss))) = sexp if tag0 == :method_add_block and tag1 == :method_add_arg and tag2 == :fcall and tag3 == :@ident and mname == @assertion_method_name and (tag4 == :brace_block or tag4 == :do_block) ss.flat_map {|s| extract_idents(s) } else ...
52. EXTRACT POSITION INFORMATION • In early versions, use pattern matching instead match(sexp) do with(_[:program, _[_[:method_add_block, _[:method_add_arg, _[:fcall, _[:@ident, assertion_method.to_s, _]], _], _[Or(:brace_block, :do_block), _, ss]]]]) do ss.flat_map {|s| extract_idents(s) } end ...
53. OFF TOPIC: PATTERN MATCHING IN RUBY • Gems have been released one after another lately • k-tsj/pattern-match • egison/egison-ruby • todesking/patm • I hope Ruby supports pattern matching as a core feature • Welcome great ideas
54. CORNER CASES/LIMITATIONS # Case A assert { obj.foo } # Case B assert { obj.foo } # Case C assert { # comment obj.foo } # Case D assert { obj.foo obj.bar } # Case E foo { assert { obj.bar } } Which expressions will be power asserted?
55. CORNER CASES/LIMITATIONS # Case A assert { obj.foo } # Case B assert { obj.foo } # Case C assert { # comment obj.foo } # Case D assert { obj.foo obj.bar } # Case E foo { assert { obj.bar } } obj.foo obj.foo obj.foo obj.foo Which expressions will be power asserted?
56. CORNER CASES/LIMITATIONS • Reassignment of variables • LHS is 0 and RHS is 1, but eval('i') is 1 ! ! • Branch • TracePoint does not know which obj.foo is called assert { (i = 0) == (i = 1) } assert { cond ? obj.foo : obj.foo }
57. CONCLUSION
58. CONCLUSION • Enjoy programming with power_assert • Enjoy programming with TracePoint