Power Assert in Ruby

Transcript

1. POWER ASSERT IN RUBY Kazuki Tsujimoto Nomura Research Institute, LTD.

2. SELF INTRODUCTION • A CRuby committer • Fix VM, GC

   bugs

3. AGENDA • Concept (13 slides) • Implementation (39 slides)

4. PROPOSAL NOTES • I'll give unduly deep explanation; not only

   of Ruby-level but VM/C-level

5. CONCEPT

6. TEST FRAMEWORKS unit spec test-unit ✔️ minitest ✔️ ✔️ RSpec

   ✔️

7. TEST FRAMEWORKS unit spec test-unit ✔️ minitest ✔️ ✔️ RSpec

   ✔️

8. TESTING WITH UNIT • Use assertions i = 0 assert_equal

   0, i assert_include 3.times.to_a, 1

9. TESTING WITH UNIT • What is the difference? assert_include 3.times.to_a,

   3 assert 3.times.to_a.include?(3)

10. TESTING WITH UNIT • Understandability of the report Failure: <[0,

    1, 2]> expected to include <3>. Failure: <true> expected but was <false>

11. TESTING WITH UNIT • A number of assertions assert assert_alias_method

    assert_block assert_boolean assert_compare assert_const_defined assert_empty assert_equal a s s e r t _ f a i l _ a s s e r t i o n a s s e r t _ f a l s e a s s e r t _ i n _ d e l t a assert_in_epsilon assert_include assert_instance_of assert_kind_of assert_match assert_nil assert_no_match assert_not_const_defined assert_not_empty assert_not_equal assert_not_in_delta assert_not_in_epsilon assert_not_include assert_not_instance_of a s s e r t _ n o t _ k i n d _ o f a s s e r t _ n o t _ m a t c h a s s e r t _ n o t _ n i l assert_not_operator assert_not_predicate assert_not_respond_to assert_not_same assert_not_send assert_nothing_raised assert_nothing_thrown assert_operator assert_path_exist a s s e r t _ p a t h _ n o t _ e x i s t a s s e r t _ p r e d i c a t e a s s e r t _ r a i s e assert_raise_kind_of assert_raise_message assert_respond_to assert_same assert_send assert_throw assert_true

12. TESTING WITH POWER ASSERT • Use assert with a block

    • That's all assert { 3.times.to_a.include?(3) } Failure: assert { 3.times.to_a.include?(3) } | | | | | false | [0, 1, 2] #<Enumerator: 3:times>

13. POWER ASSERT IN GROOVY http://docs.codehaus.org/display/GROOVY/Groovy+1.7+release+notes

14. POWER_ASSERT FAMILY • power_assert • Requires CRuby 2.0.0 or later

    • test-unit-power_assert • test-unit 3(@ktou) • minitest-power_assert(@hsbt) • pry-power_assert(@spikeolaf)

15. DEBUGGING WITH POWER ASSERT • tap is frequently used for

    inspecting the value in method chains 3.times.to_a.tap {|i| p i }.include?(3) #=> [0, 1, 2]

16. DEBUGGING WITH POWER ASSERT • Extend p with power_assert to

    inspect all values p { 3.times.to_a.include?(3) } p { 3.times.to_a.include?(3) } | | | | | false | [0, 1, 2] #<Enumerator: 3:times>

17. DEBUGGING WITH POWER ASSERT module PowerP def p(*, &blk) if

    blk PowerAssert.start(blk, assertion_method: __callee__) do |pa| pa.yield puts pa.message_proc.call end else super end end end

18. IMPLEMENTATION

19. THE CHALLENGES OF IMPLEMENTATION • How to get values •

    How to get position information 3.to_s.length | | | 1 "3" to_s “3” length 1 Values to_s 2 length 7 Position info

20. HOW TO GET VALUES

21. TRACEPOINT

22. WHAT IS TRACEPOINT? • Tracing API which aims to replace

    set_trace_func • Introduced in Ruby 2.0

23. TRACEPOINT trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value:

    tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}

24. TRACEPOINT trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value:

    tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]} New event info • return_value • raised_exception ! Traditional event info • method_id • binding • etc.

25. OFF TOPIC: EXCEPTION#CAUSE • Exception#cause returns the previous exception at

    the time the exception was raised begin begin raise "A" rescue raise "B" end rescue => e p [e, e.cause] # => [#<RuntimeError: B>, #<RuntimeError: A>] end

26. OFF TOPIC: EXCEPTION#CAUSE • Implement Exception#cause using TracePoint class Exception

    attr_accessor :cause TracePoint.trace(:raise){|tp| e = tp.raised_exception e.cause = $! } end https://twitter.com/_ko1/statuses/409267459414179840

27. FILTERING RETURN VALUES • TracePoint captures return values of all

    method calls • We must filter them trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}

28. FILTERING RETURN VALUES • TracePoint captures return values of all

    method calls • We must filter them trace = TracePoint.new(:return, :c_return) {|tp| p id: tp.method_id, return_value: tp.return_value } trace.enable { 3.times.to_a } {:id=>:times, :return_value=>#<Enumerator: 3:times>} {:id=>:times, :return_value=>3} {:id=>:each, :return_value=>3} {:id=>:to_a, :return_value=>[0, 1, 2]}

29. FILTERING RETURN VALUES • Use Binding#eval and Kernel#caller_locations • tp.binding.eval(‘caller_locations’).length

    returns the execution stack length of each method calls

30. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7}

    {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }

31. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7}

    {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }

32. FILTERING RETURN VALUES {:id=>:times, :return_value=>#<Enumerator: 3:times>, :loc_len=>5} {:id=>:times, :return_value=>3, :loc_len=>7}

    {:id=>:each, :return_value=>3, :loc_len=>6} {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5} trace = TracePoint.new(:return, :c_return) {|tp| p tp.event, id: tp.method_id, return_value: tp.return_value, loc_len: tp.binding.eval('caller_locations').length } trace.enable { 3.times.to_a }

33. GETTING THE VALUE OF THE VARIABLE • Binding#eval is also

    used to get the value of the variable ary = 3.times.to_a assert { ary.include?(3) } Failure: assert { ary.include?(3) } | | | false [0, 1, 2]

34. TRAP OF POWER_ASSERT • The report does not contain the

    value of == require 'test-unit-power_assert' ... assert { [0, 1, 2].find {|i| i.odd? } == 3 } $ ruby ./test.rb Failure: assert { [0, 1, 2].find {|i| i.odd? } == 3 } | 1

35. WHAT SHOULD WE DO? • Require power_assert first $ ruby

    -rpower_assert ./test.rb Failure: assert { [0, 1, 2].find {|i| i.odd? } == 3 } | | | false 1

36. WHAT HAPPENED? • Minimum code for the problem trace =

    TracePoint.new(:return, :c_return) {|tp| p tp.return_value } trace.enable { 1 == 3 } (Expected) false (Actual)

37. INVESTIGATING BYTECODE • Use RubyVM::InstructionSequence.disasm puts RubyVM::InstructionSequence.disasm(Proc.new { [0, 1,

    2].find {|i| i.odd? } == 3 }) 0000 trace 256 0002 trace 1 0004 duparray [0, 1, 2] 0006 send <callinfo!mid:find, argc:0, block:block> 0008 putobject 3 0010 opt_eq <callinfo!mid:==, argc:1, ARGS_SKIP> 0012 trace 512 0014 leave

38. INVESTIGATING BYTECODE • Use RubyVM::InstructionSequence.disasm puts RubyVM::InstructionSequence.disasm(Proc.new { [0, 1,

    2].find {|i| i.odd? } == 3 }) 0000 trace 256 0002 trace 1 0004 duparray [0, 1, 2] 0006 send <callinfo!mid:find, argc:0, block:block> 0008 putobject 3 0010 opt_eq <callinfo!mid:==, argc:1, ARGS_SKIP> 0012 trace 512 0014 leave

39. OPT_EQ { val = opt_eq_func(recv, obj, ci); if (val ==

    Qundef) { /* other */ PUSH(recv); PUSH(obj); CALL_SIMPLE_METHOD(recv); } }

40. OPT_EQ_FUNC VALUE opt_eq_func(VALUE recv, VALUE obj, CALL_INFO ci) { if

    (FIXNUM_2_P(recv, obj) && BASIC_OP_UNREDEFINED_P(BOP_EQ, FIXNUM_REDEFINED_OP_FLAG)) { return (recv == obj) ? Qtrue : Qfalse; } ...

41. SEND { ci->argc = ci->orig_argc; ci->blockptr = 0; vm_caller_setup_args(th, reg_cfp,

    ci); vm_search_method(ci, ci->recv = TOPN(ci->argc)); CALL_METHOD(ci); }

42. VM_CALL_CFUNC static VALUE vm_call_cfunc(rb_thread_t *th, rb_control_frame_t *reg_cfp, rb_call_info_t *ci) {

    ... val = vm_call_cfunc_latter(th, reg_cfp, ci); EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv, me->called_id, me->klass, val); ...

43. VM_CALL_CFUNC static VALUE vm_call_cfunc(rb_thread_t *th, rb_control_frame_t *reg_cfp, rb_call_info_t *ci) {

    ... val = vm_call_cfunc_latter(th, reg_cfp, ci); EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv, me->called_id, me->klass, val); ... TracePoint is invoked by EXEC_EVENT_HOOK

44. DISABLE VM OPTIMIZATION • Set optimization flags using RubyVM::InstructionSequence.compile_opti on=

    method • power_assert contains following code RubyVM::InstructionSequence.compile_option = { specialized_instruction: false }

45. DISABLE VM OPTIMIZATION • Only power_assert is compiled with optimization

    $ ruby -rpower_assert ./test.rb Compile option Test code power_assert Optimization: Enabled Compile Evaluate Disabled Compile Evaluate

46. DISABLE VM OPTIMIZATION • Both power_assert and test code are

    compiled with optimization $ ruby ./test.rb Compile option Test code power_assert Optimization: Enabled Compile Evaluate Compile Evaluate Evaluate Disabled

47. HOW TO GET POSITION INFORMATION

48. RIPPER

49. WHAT IS RIPPER? • A Ruby script parser • Ripper.sexp

    create S-exp tree including position information # assert { [0, 1, 2].find {|i| i.odd? } == 3 } [:program, [[:method_add_block, [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []], [:brace_block, nil, [[:binary, [:method_add_block, [:call, [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".", [:@ident, "find", [1, 19]]], [:brace_block, [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false], [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]], :==, [:@int, "3", [1, 41]]]]]]]]

50. WHAT IS RIPPER? • A Ruby script parser • Ripper.sexp

    create S-exp tree including position information # assert { [0, 1, 2].find {|i| i.odd? } == 3 } [:program, [[:method_add_block, [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []], [:brace_block, nil, [[:binary, [:method_add_block, [:call, [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".", [:@ident, "find", [1, 19]]], [:brace_block, [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false], [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]], :==, [:@int, "3", [1, 41]]]]]]]] "assert"

51. EXTRACT POSITION INFORMATION • Use multiple assignment tag, * =

    sexp case tag when :program _, ((tag0, (tag1, (tag2, (tag3, mname, _)), _), (tag4, _, ss))) = sexp if tag0 == :method_add_block and tag1 == :method_add_arg and tag2 == :fcall and tag3 == :@ident and mname == @assertion_method_name and (tag4 == :brace_block or tag4 == :do_block) ss.flat_map {|s| extract_idents(s) } else ...

52. EXTRACT POSITION INFORMATION • In early versions, use pattern matching

    instead match(sexp) do with(_[:program, _[_[:method_add_block, _[:method_add_arg, _[:fcall, _[:@ident, assertion_method.to_s, _]], _], _[Or(:brace_block, :do_block), _, ss]]]]) do ss.flat_map {|s| extract_idents(s) } end ...

53. OFF TOPIC: PATTERN MATCHING IN RUBY • Gems have been

    released one after another lately • k-tsj/pattern-match • egison/egison-ruby • todesking/patm • I hope Ruby supports pattern matching as a core feature • Welcome great ideas

54. CORNER CASES/LIMITATIONS # Case A assert { obj.foo } #

    Case B assert { obj.foo } # Case C assert { # comment obj.foo } # Case D assert { obj.foo obj.bar } # Case E foo { assert { obj.bar } } Which expressions will be power asserted?

55. CORNER CASES/LIMITATIONS # Case A assert { obj.foo } #

    Case B assert { obj.foo } # Case C assert { # comment obj.foo } # Case D assert { obj.foo obj.bar } # Case E foo { assert { obj.bar } } obj.foo obj.foo obj.foo obj.foo Which expressions will be power asserted?

56. CORNER CASES/LIMITATIONS • Reassignment of variables • LHS is 0

    and RHS is 1, but eval('i') is 1 ! ! • Branch • TracePoint does not know which obj.foo is called assert { (i = 0) == (i = 1) } assert { cond ? obj.foo : obj.foo }

57. CONCLUSION

58. CONCLUSION • Enjoy programming with power_assert • Enjoy programming with

    TracePoint