Power Assert in Ruby

http://rubykaigi.org/2014/presentation/S-KazukiTsujimoto

Transcript

1. POWER ASSERT IN
   RUBY
   Kazuki Tsujimoto
   Nomura Research Institute, LTD.
   

   View Slide

2. SELF INTRODUCTION
   • A CRuby committer
   • Fix VM, GC bugs
   

   View Slide

3. AGENDA
   • Concept (13 slides)
   • Implementation (39 slides)
   

   View Slide

4. PROPOSAL NOTES
   • I'll give unduly deep explanation; not only of
   Ruby-level but VM/C-level
   

   View Slide

5. CONCEPT
   

   View Slide

6. TEST FRAMEWORKS
   unit spec
   test-unit ✔️
   minitest ✔️ ✔️
   RSpec ✔️
   

   View Slide

7. TEST FRAMEWORKS
   unit spec
   test-unit ✔️
   minitest ✔️ ✔️
   RSpec ✔️
   

   View Slide

8. TESTING WITH UNIT
   • Use assertions
   i = 0
   assert_equal 0, i
   assert_include 3.times.to_a, 1
   

   View Slide

9. TESTING WITH UNIT
   • What is the difference?
   assert_include 3.times.to_a, 3
   assert 3.times.to_a.include?(3)
   

   View Slide

10. TESTING WITH UNIT
    • Understandability of the report
    Failure:
    <[0, 1, 2]> expected to include
    <3>.
    Failure:
    expected but was
    
    

    View Slide

11. TESTING WITH UNIT
    • A number of assertions
    assert assert_alias_method assert_block assert_boolean
    assert_compare assert_const_defined assert_empty assert_equal
    a s s e r t _ f a i l _ a s s e r t i o n a s s e r t _ f a l s e a s s e r t _ i n _ d e l t a
    assert_in_epsilon assert_include assert_instance_of assert_kind_of
    assert_match assert_nil assert_no_match assert_not_const_defined
    assert_not_empty assert_not_equal assert_not_in_delta
    assert_not_in_epsilon assert_not_include assert_not_instance_of
    a s s e r t _ n o t _ k i n d _ o f a s s e r t _ n o t _ m a t c h a s s e r t _ n o t _ n i l
    assert_not_operator assert_not_predicate assert_not_respond_to
    assert_not_same assert_not_send assert_nothing_raised
    assert_nothing_thrown assert_operator assert_path_exist
    a s s e r t _ p a t h _ n o t _ e x i s t a s s e r t _ p r e d i c a t e a s s e r t _ r a i s e
    assert_raise_kind_of assert_raise_message assert_respond_to
    assert_same assert_send assert_throw assert_true
    

    View Slide

12. TESTING WITH
    POWER ASSERT
    • Use assert with a block
    • That's all
    assert { 3.times.to_a.include?(3) }
    Failure:
    assert { 3.times.to_a.include?(3) }
    | | |
    | | false
    | [0, 1, 2]
    #
    

    View Slide

13. POWER ASSERT IN GROOVY
    http://docs.codehaus.org/display/GROOVY/Groovy+1.7+release+notes
    

    View Slide

14. POWER_ASSERT FAMILY
    • power_assert
    • Requires CRuby 2.0.0 or later
    • test-unit-power_assert
    • test-unit 3(@ktou)
    • minitest-power_assert(@hsbt)
    • pry-power_assert(@spikeolaf)
    

    View Slide

15. DEBUGGING WITH
    POWER ASSERT
    • tap is frequently used for inspecting the
    value in method chains
    3.times.to_a.tap {|i| p i }.include?(3)
    #=> [0, 1, 2]
    

    View Slide

16. DEBUGGING WITH
    POWER ASSERT
    • Extend p with power_assert to inspect all
    values
    p { 3.times.to_a.include?(3) }
    p { 3.times.to_a.include?(3) }
    | | |
    | | false
    | [0, 1, 2]
    #
    

    View Slide

17. DEBUGGING WITH
    POWER ASSERT
    module PowerP
    def p(*, &blk)
    if blk
    PowerAssert.start(blk, assertion_method: __callee__) do |pa|
    pa.yield
    puts pa.message_proc.call
    end
    else
    super
    end
    end
    end
    

    View Slide

18. IMPLEMENTATION
    

    View Slide

19. THE CHALLENGES OF
    IMPLEMENTATION
    • How to get values
    • How to get position information
    3.to_s.length
    | |
    | 1
    "3"
    to_s
    “3”
    length
    1
    Values
    to_s
    2
    length
    7
    Position info
    

    View Slide

20. HOW TO GET
    VALUES
    

    View Slide

21. TRACEPOINT
    

    View Slide

22. WHAT IS TRACEPOINT?
    • Tracing API which aims to replace
    set_trace_func
    • Introduced in Ruby 2.0
    

    View Slide

23. TRACEPOINT
    trace = TracePoint.new(:return, :c_return) {|tp|
    p id: tp.method_id, return_value: tp.return_value
    }
    trace.enable {
    3.times.to_a
    }
    {:id=>:times, :return_value=>#}
    {:id=>:times, :return_value=>3}
    {:id=>:each, :return_value=>3}
    {:id=>:to_a, :return_value=>[0, 1, 2]}
    

    View Slide

24. TRACEPOINT
    trace = TracePoint.new(:return, :c_return) {|tp|
    p id: tp.method_id, return_value: tp.return_value
    }
    trace.enable {
    3.times.to_a
    }
    {:id=>:times, :return_value=>#}
    {:id=>:times, :return_value=>3}
    {:id=>:each, :return_value=>3}
    {:id=>:to_a, :return_value=>[0, 1, 2]}
    New event info
    • return_value
    • raised_exception
    !
    Traditional event info
    • method_id
    • binding
    • etc.
    

    View Slide

25. OFF TOPIC:
    EXCEPTION#CAUSE
    • Exception#cause returns the previous
    exception at the time the exception was
    raised
    begin
    begin
    raise "A"
    rescue
    raise "B"
    end
    rescue => e
    p [e, e.cause]
    # => [#, #]
    end
    

    View Slide

26. OFF TOPIC:
    EXCEPTION#CAUSE
    • Implement Exception#cause using TracePoint
    class Exception
    attr_accessor :cause
    TracePoint.trace(:raise){|tp|
    e = tp.raised_exception
    e.cause = $!
    }
    end
    https://twitter.com/_ko1/statuses/409267459414179840
    

    View Slide

27. FILTERING RETURN VALUES
    • TracePoint captures return values of all
    method calls
    • We must filter them
    trace = TracePoint.new(:return, :c_return) {|tp|
    p id: tp.method_id, return_value: tp.return_value
    }
    trace.enable {
    3.times.to_a
    }
    {:id=>:times, :return_value=>#}
    {:id=>:times, :return_value=>3}
    {:id=>:each, :return_value=>3}
    {:id=>:to_a, :return_value=>[0, 1, 2]}
    

    View Slide

28. FILTERING RETURN VALUES
    • TracePoint captures return values of all
    method calls
    • We must filter them
    trace = TracePoint.new(:return, :c_return) {|tp|
    p id: tp.method_id, return_value: tp.return_value
    }
    trace.enable {
    3.times.to_a
    }
    {:id=>:times, :return_value=>#}
    {:id=>:times, :return_value=>3}
    {:id=>:each, :return_value=>3}
    {:id=>:to_a, :return_value=>[0, 1, 2]}
    

    View Slide

29. FILTERING RETURN VALUES
    • Use Binding#eval and
    Kernel#caller_locations
    • tp.binding.eval(‘caller_locations’).length
    returns the execution stack length of each
    method calls
    

    View Slide

30. FILTERING RETURN VALUES
    {:id=>:times, :return_value=>#, :loc_len=>5}
    {:id=>:times, :return_value=>3, :loc_len=>7}
    {:id=>:each, :return_value=>3, :loc_len=>6}
    {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5}
    trace = TracePoint.new(:return, :c_return) {|tp|
    p tp.event, id: tp.method_id, return_value: tp.return_value,
    loc_len: tp.binding.eval('caller_locations').length
    }
    trace.enable {
    3.times.to_a
    }
    

    View Slide

31. FILTERING RETURN VALUES
    {:id=>:times, :return_value=>#, :loc_len=>5}
    {:id=>:times, :return_value=>3, :loc_len=>7}
    {:id=>:each, :return_value=>3, :loc_len=>6}
    {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5}
    trace = TracePoint.new(:return, :c_return) {|tp|
    p tp.event, id: tp.method_id, return_value: tp.return_value,
    loc_len: tp.binding.eval('caller_locations').length
    }
    trace.enable {
    3.times.to_a
    }
    

    View Slide

32. FILTERING RETURN VALUES
    {:id=>:times, :return_value=>#, :loc_len=>5}
    {:id=>:times, :return_value=>3, :loc_len=>7}
    {:id=>:each, :return_value=>3, :loc_len=>6}
    {:id=>:to_a, :return_value=>[0, 1, 2], :loc_len=>5}
    trace = TracePoint.new(:return, :c_return) {|tp|
    p tp.event, id: tp.method_id, return_value: tp.return_value,
    loc_len: tp.binding.eval('caller_locations').length
    }
    trace.enable {
    3.times.to_a
    }
    

    View Slide

33. GETTING THE VALUE OF THE
    VARIABLE
    • Binding#eval is also used to get the value of
    the variable
    ary = 3.times.to_a
    assert { ary.include?(3) }
    Failure:
    assert { ary.include?(3) }
    | |
    | false
    [0, 1, 2]
    

    View Slide

34. TRAP OF POWER_ASSERT
    • The report does not contain the value of ==
    require 'test-unit-power_assert'
    ...
    assert { [0, 1, 2].find {|i| i.odd? } == 3 }
    $ ruby ./test.rb
    Failure:
    assert { [0, 1, 2].find {|i| i.odd? } == 3 }
    |
    1
    

    View Slide

35. WHAT SHOULD WE DO?
    • Require power_assert first
    $ ruby -rpower_assert ./test.rb
    Failure:
    assert { [0, 1, 2].find {|i| i.odd? } == 3 }
    | |
    | false
    1
    

    View Slide

36. WHAT HAPPENED?
    • Minimum code for the problem
    trace = TracePoint.new(:return, :c_return) {|tp|
    p tp.return_value
    }
    trace.enable {
    1 == 3
    }
    (Expected)
    false
    (Actual)
    

    View Slide

37. INVESTIGATING BYTECODE
    • Use RubyVM::InstructionSequence.disasm
    puts RubyVM::InstructionSequence.disasm(Proc.new {
    [0, 1, 2].find {|i| i.odd? } == 3
    })
    0000 trace 256
    0002 trace 1
    0004 duparray [0, 1, 2]
    0006 send
    0008 putobject 3
    0010 opt_eq
    0012 trace 512
    0014 leave
    

    View Slide

38. INVESTIGATING BYTECODE
    • Use RubyVM::InstructionSequence.disasm
    puts RubyVM::InstructionSequence.disasm(Proc.new {
    [0, 1, 2].find {|i| i.odd? } == 3
    })
    0000 trace 256
    0002 trace 1
    0004 duparray [0, 1, 2]
    0006 send
    0008 putobject 3
    0010 opt_eq
    0012 trace 512
    0014 leave
    

    View Slide

39. OPT_EQ
    {
    val = opt_eq_func(recv, obj, ci);
    if (val == Qundef) {
    /* other */
    PUSH(recv);
    PUSH(obj);
    CALL_SIMPLE_METHOD(recv);
    }
    }
    

    View Slide

40. OPT_EQ_FUNC
    VALUE
    opt_eq_func(VALUE recv, VALUE obj, CALL_INFO ci)
    {
    if (FIXNUM_2_P(recv, obj) &&
    BASIC_OP_UNREDEFINED_P(BOP_EQ,
    FIXNUM_REDEFINED_OP_FLAG)) {
    return (recv == obj) ? Qtrue : Qfalse;
    }
    ...
    

    View Slide

41. SEND
    {
    ci->argc = ci->orig_argc;
    ci->blockptr = 0;
    vm_caller_setup_args(th, reg_cfp, ci);
    vm_search_method(ci, ci->recv = TOPN(ci->argc));
    CALL_METHOD(ci);
    }
    

    View Slide

42. VM_CALL_CFUNC
    static VALUE
    vm_call_cfunc(rb_thread_t *th, rb_control_frame_t
    *reg_cfp, rb_call_info_t *ci)
    {
    ...
    val = vm_call_cfunc_latter(th, reg_cfp, ci);
    EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv,
    me->called_id, me->klass, val);
    ...
    

    View Slide

43. VM_CALL_CFUNC
    static VALUE
    vm_call_cfunc(rb_thread_t *th, rb_control_frame_t
    *reg_cfp, rb_call_info_t *ci)
    {
    ...
    val = vm_call_cfunc_latter(th, reg_cfp, ci);
    EXEC_EVENT_HOOK(th, RUBY_EVENT_C_RETURN, recv,
    me->called_id, me->klass, val);
    ...
    TracePoint is invoked by
    EXEC_EVENT_HOOK
    

    View Slide

44. DISABLE VM OPTIMIZATION
    • Set optimization flags using
    RubyVM::InstructionSequence.compile_opti
    on= method
    • power_assert contains following code
    RubyVM::InstructionSequence.compile_option = {
    specialized_instruction: false
    }
    

    View Slide

45. DISABLE VM OPTIMIZATION
    • Only power_assert is compiled with
    optimization
    $ ruby -rpower_assert ./test.rb
    Compile option
    Test code
    power_assert
    Optimization: Enabled
    Compile Evaluate
    Disabled
    Compile Evaluate
    

    View Slide

46. DISABLE VM OPTIMIZATION
    • Both power_assert and test code are
    compiled with optimization
    $ ruby ./test.rb
    Compile option
    Test code
    power_assert
    Optimization: Enabled
    Compile Evaluate
    Compile Evaluate
    Evaluate
    Disabled
    

    View Slide

47. HOW TO GET POSITION
    INFORMATION
    

    View Slide

48. RIPPER
    

    View Slide

49. WHAT IS RIPPER?
    • A Ruby script parser
    • Ripper.sexp create S-exp tree including
    position information
    # assert { [0, 1, 2].find {|i| i.odd? } == 3 }
    [:program,
    [[:method_add_block,
    [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []],
    [:brace_block, nil,
    [[:binary,
    [:method_add_block,
    [:call,
    [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".",
    [:@ident, "find", [1, 19]]],
    [:brace_block,
    [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false],
    [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]],
    :==,
    [:@int, "3", [1, 41]]]]]]]]
    

    View Slide

50. WHAT IS RIPPER?
    • A Ruby script parser
    • Ripper.sexp create S-exp tree including
    position information
    # assert { [0, 1, 2].find {|i| i.odd? } == 3 }
    [:program,
    [[:method_add_block,
    [:method_add_arg, [:fcall, [:@ident, "assert", [1, 0]]], []],
    [:brace_block, nil,
    [[:binary,
    [:method_add_block,
    [:call,
    [:array,ɹ[[:@int, "0", [1, 10]], [:@int, "1", [1, 13]], [:@int, "2", [1, 16]]]], :".",
    [:@ident, "find", [1, 19]]],
    [:brace_block,
    [:block_var, [:params, [[:@ident, "i", [1, 26]]], nil, nil, nil, nil, nil, nil], false],
    [[:call, [:var_ref, [:@ident, "i", [1, 29]]], :”.", [:@ident, "odd?", [1, 31]]]]]],
    :==,
    [:@int, "3", [1, 41]]]]]]]]
    "assert"
    

    View Slide

51. EXTRACT
    POSITION INFORMATION
    • Use multiple assignment
    tag, * = sexp
    case tag
    when :program
    _, ((tag0, (tag1, (tag2, (tag3, mname, _)), _),
    (tag4, _, ss))) = sexp
    if tag0 == :method_add_block and tag1 == :method_add_arg and
    tag2 == :fcall and tag3 == :@ident and
    mname == @assertion_method_name and
    (tag4 == :brace_block or tag4 == :do_block)
    ss.flat_map {|s| extract_idents(s) }
    else
    ...
    

    View Slide

52. EXTRACT
    POSITION INFORMATION
    • In early versions, use pattern matching
    instead
    match(sexp) do
    with(_[:program,
    _[_[:method_add_block,
    _[:method_add_arg,
    _[:fcall, _[:@ident, assertion_method.to_s, _]], _],
    _[Or(:brace_block, :do_block), _, ss]]]]) do
    ss.flat_map {|s| extract_idents(s) }
    end
    ...
    

    View Slide

53. OFF TOPIC:
    PATTERN MATCHING IN RUBY
    • Gems have been released one after another
    lately
    • k-tsj/pattern-match
    • egison/egison-ruby
    • todesking/patm
    • I hope Ruby supports pattern matching as a
    core feature
    • Welcome great ideas
    

    View Slide

54. CORNER CASES/LIMITATIONS
    # Case A
    assert { obj.foo }
    # Case B
    assert {
    obj.foo
    }
    # Case C
    assert {
    # comment
    obj.foo
    }
    # Case D
    assert {
    obj.foo
    obj.bar
    }
    # Case E
    foo { assert { obj.bar } }
    Which expressions will be
    power asserted?
    

    View Slide

55. CORNER CASES/LIMITATIONS
    # Case A
    assert { obj.foo }
    # Case B
    assert {
    obj.foo
    }
    # Case C
    assert {
    # comment
    obj.foo
    }
    # Case D
    assert {
    obj.foo
    obj.bar
    }
    # Case E
    foo { assert { obj.bar } }
    obj.foo
    obj.foo
    obj.foo
    obj.foo
    Which expressions will be
    power asserted?
    

    View Slide

56. CORNER CASES/LIMITATIONS
    • Reassignment of variables
    • LHS is 0 and RHS is 1, but eval('i') is 1
    !
    !
    • Branch
    • TracePoint does not know which obj.foo is
    called
    assert { (i = 0) == (i = 1) }
    assert { cond ? obj.foo : obj.foo }
    

    View Slide

57. CONCLUSION
    

    View Slide

58. CONCLUSION
    • Enjoy programming with power_assert
    • Enjoy programming with TracePoint
    

    View Slide