Save 37% off PRO during our Black Friday Sale! »

Coverage.so, or RNode with column

B3ba3ccedfbf4d605f00bafd1a732529?s=47 yui-knk
November 29, 2017

Coverage.so, or RNode with column

B3ba3ccedfbf4d605f00bafd1a732529?s=128

yui-knk

November 29, 2017
Tweet

Transcript

  1.  ۚࢠ༤Ұ࿠ !ZVJLOL ϚωʔϑΥϫʔυ3VCZษڧձ ਐԽ͢Δ$PWFSBHFTPɺ ͋Δ͍͸/0%&ͱDPMVNOͷ͸ͳ͠

  2. raise MissingBeer unless 1

  3. ࣗݾ঺հ • Ҏલ͸ॏ޻ۀͳϝʔΧʔͰܦཧΛ͍ͯͨ͠ • ϚωʔϑΥʔϫʔυͰ͸ձܭνʔϜͰRailsΞϓϦΛॻ͍͍ͯΔ • Rubyίϛολʔ 2015/12~ • Asakusa.rb

    ॴଐ 2
  4. • Ruby 2.5ͷશମతͳ࿩͸͜ͷ͋ͱ͋Δ͸ͣ • ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱͷ࿩ 3

  5. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ $ git log --numstat --pretty="%H" --author="yui-knk" --since="2016-12-26" |\ awk

    'NF==3 {plus+=$1; minus+=$2; printf("%s %d %d\n", $3, $1, $2)}' |\ ruby -e 'ary=[]; while gets(); ary << $_.chomp.split(" ");end; \ ary.each_with_object({}) {|(n,p,m),h| h[n]||= [0,0];\ h[n] = [h[n][0] + p.to_i, h[n][1] + m.to_i]}.sort_by\ {|(k,(p,m))| -(p + m) }.each {|(k,(p,m))| puts "#{k} +#{p}, -#{m}"}' 4
  6. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ parse.y +2256, -1687 compile.c +148, -78 test/coverage/test_coverage.rb +100, -55

    node.h +34, -38 node.c +38, -18 ext/coverage/coverage.c +20, -12 thread.c +8, -6 ext/objspace/objspace.c +1, -5 test/ruby/test_iseq.rb +3, -1 test/ripper/test_lexer.rb +1, -1 test/ruby/test_string.rb +1, -1 insns.def +0, -2 ext/objspace/object_tracing.c +1, -1 doc/extension.ja.rdoc +1, -1 doc/extension.rdoc +1, -1 iseq.c +0, -1 5
  7. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ • (ओʹ)CoverageͷͨΊʹɺparserΛ͍͍ͬͯͨ͡ 6

  8. Contents • Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ

    Coverage.so • Branch Coverage ͷ࢓૊Έ • NODE ͱ column ͷ͸ͳ͠ • lexer ͱ parser • column ৘ใͷऔಘͱอ؅ • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ • ͓·͚ 7
  9. ஫ҙ఺ • ϋογϡλά => #mf_ruby (શ෦খจࣈ) • લ൒ => Ruby

    2.5Λ࢖͏͏͑Ͱ໾ʹཱͭ • ޙ൒ => Ruby 2.5Λ͍͡Δ͏͑Ͱ໾ʹཱͭ 8
  10. Coverage ͷ͸ͳ͠ 9

  11. Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ Coverage.so •

    Branch Coverage ͷ࢓૊Έ 10
  12. Coverage ͱ͸ • ςετίʔυΛධՁ͢Δͱ͖ʹࢀߟʹͳΔ৘ใ • ৄ͘͠͸ԕ౻(@mametter)͞ΜͷൃදΛ͝ཡ͍ͩ͘͞ • "An introduction and

    future of Ruby coverage library" (http:// rubykaigi.org/2017/presentations/mametter.html) 11
  13. Coverage ͷ࣮ྫ # Coverageऔಘର৅ͷίʔυ def target(bool) if bool 1 else

    0 end end target(true) # CoverageΛऔಘ͢ΔͨΊͷίʔυ require "coverage" Coverage.start require "target_file" # ର৅ͷϑΝΠϧΛload Coverage.result 12
  14. Coverage ͷ࣮ྫ # Coverage.result { "src/coverage_example.rb" => [1, 1, 1,

    nil, 0, nil, nil, nil, 1] } # ͦΕͬΆ͘mapping 1: def target(bool) 1: if bool 1: 1 nil: else 0: 0 nil: end nil: end nil: 1: target(true) 13
  15. Ruby 2.5ʹ͓͚Δ Coverage.so • Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ͞ΕΔ Ruby 2.4 Ruby

    2.5 Line Coverage o o Branch Coverage - o Method Coverage - o 14
  16. Method Coverageͱ͸ • ϝιου͕Կճݺͼग़͞Ε͔͕ͨΘ͔ΔΑ͏ʹͳΔ def target1 end def target2 end

    target2 0: def target1 : end : 1: def target2 : end : : target2 15
  17. Branch Coverageͱ͸ • Ͳͷ෼ذ͕Կճ࣮ߦ͞Ε͔ͨΘ͔ΔΑ͏ʹͳΔ a = 1 if a 1

    else 2 end : a = 1 : : if a 1: 1 : else 0: 2 : end 16
  18. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(1) • ޙஔif 1: a = true 1: b

    = false 1: 1 if a 1: 2 if b # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 17
  19. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(2) • ࡾ߲ԋࢉ 1: a = true 1: b

    = false 1: a ? 1 : 2 # a͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 1: b ? 1 : 2 # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 18
  20. Branch Coverageͷྫ(if/unless) • ޙஔif/unless΍ࡾ߲ԋࢉࢠʹ΋ରԠ : a = 1 : 1/0:

    10 if a == 1 0/1: 11 unless a == 1 0/1: (a == 2) ? :t : :f n/m -> n͕thenɺm͕elseͷճ਺ 19
  21. Branch Coverageͷྫ(while/until) : x = 3 : while x >

    0 3: x -= 1 : end : until x == 10 10: x += 1 : end : y = 3 3: y -= 1 while y > 0 10: y += 1 until y == 10 20
  22. Branch Coverageͷྫ(case) : x = 0 : : case x

    : when 0 1: 0 : when 1 0: 1 : end : : case : when x == 0 1: 0 : when x == 1 0: 1 : end 21
  23. Branch Coverageͷྫ(&.) : a = 10 : b = nil

    1/0: a&.abs 0/1: b&.hoo n/m -> n͕then(ϝιουΛcall)ɺm͕elseͷճ਺ 22
  24. Branch Coverage ͷ࢓૊Έ • ؅ཧςʔϒϧΛ༻ҙ • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ • ໋ྩ͕ɺ؅ཧςʔϒϧͷ֘౰ՕॴΛΠϯΫϦϝϯτ͢Δ

    a = 1 if a :A else :B end 23
  25. Branch Coverage ͷ࢓૊Έ (؅ཧςʔϒϧ) • ؅ཧςʔϒϧΛ༻ҙ [structure, counters] structure =

    [ [ :if , 2, 0, 6, 3, # type, first_lineno, first_column, last_lineno, last_column :then, 3, 2, 3, 4, 0, # type, first_lineno, first_column, last_lineno, last_column, counter_index :else, 5, 2, 5, 4, 1 # type, first_lineno, first_column, last_lineno, last_column, counter_index ], ... ] counters = [0, 0] # [:then_counter, :else_counter] 24
  26. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ src = <<~SRC

    a = 1 if a :A else :B end SRC # Coverage OFF puts RubyVM::InstructionSequence.compile(src).disasm require "coverage" ENV["COVERAGE_EXPERIMENTAL_MODE"] = "true" Coverage.start(branches: true) # Coverage ON puts RubyVM::InstructionSequence.compile(src).disasm 25
  27. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕OFFͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local

    table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 10 0007 putobject :A ( 3)[Li] 0009 leave ( 5) 0010 putobject :B[Li] 0012 leave 26
  28. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local

    table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 27
  29. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local

    table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 28
  30. Branch Coverage ͷ࢓૊Έ (count up) trace2 131072, 1 # =>

    counter_idx = 0 trace2 131072, 17 # => counter_idx = 1 • ୈ2Φϖϥϯυ(1΍17)͕counter_idxΛද͢ • counter_idx * 16 + COVERAGE_INDEX_BRANCHES • COVERAGE_INDEX_BRANCHES͸1 29
  31. Branch Coverage ͷ࢓૊Έ (count up) == disasm: #<ISeq:<compiled>@<compiled>>================================ local table

    (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • a͕false΋͘͠͸nilͳΒ0013ʹͱͿ • ͦΕҎ֎ͷͱ͖͸Կ΋ͤͣɺ࣍ͷ໋ྩ΁͍͘ 30
  32. Branch Coverage ͷ࢓૊Έ (count up) == disasm: #<ISeq:<compiled>@<compiled>>================================ local table

    (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • ࠓճͷέʔεͰ͸a = 1ͳͷͰɺtrace2 131072, 1͕࣮ߦ͞Ε Δ -> :then_counter͕૿Ճ͢Δ 31
  33. Branch Coverage ͷ࢓૊Έ (݁Ռ) [structure, counters] structure = [[:if, 2,

    0, 6, 3, :then, 3, 2, 3, 4, 0, :else, 5, 2, 5, 4, 1], ...] counters = [1, 0] # [:then_counter, :else_counter] => # ͦΕͬΆ͘mapping : a = 1 : if a 1: :A : else 0: :B : end 32
  34. column৘ใ͕΄͍͠έʔε • औಘͨ͠Coverage݁ՌΛՄࢹԽ͢Δͱ͖ • :Aͱ͔:BͷҐஔ͕Θ͔Δͱศར a, b = 1, 1

    (a == b) ? :A : :B 33
  35. column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1,

    3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2] => { # 2ߦ໨ [:then, 1, 2] => 0, # 2ߦ໨ [:else, 2, 2] => 1 # 2ߦ໨ }, [:if, 3, 2] => { # 2ߦ໨ [:then, 4, 2] => 1, # 2ߦ໨ [:else, 5, 2] => 0 # 2ߦ໨ } } 34
  36. column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1,

    3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2, 12, 2, 30] => { # [2.12-2.30] [:then, 1, 2, 23, 2, 25] => 0, # [2.23-2.25] [:else, 2, 2, 28, 2, 30] => 1 # [2.28-2.30] }, [:if, 3, 2, 0, 2, 36] => { # [2.0 -2.36] [:then, 4, 2, 12, 2, 30] => 1, # [2.12-2.30] [:else, 5, 2, 34, 2, 36] => 0 # [2.34-2.36] } } 35
  37. NODE ͱ column ͷ͸ͳ͠ 36

  38. NODE ͱ column ͷ͸ͳ͠ • column ͱ͸ͳʹ͔ • column ৘ใͷऔಘͱอ؅

    • lexer ͱ parser • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ 37
  39. column ͱ͸ͳʹ͔ • ࣜ΍จͷҐஔ৘ใ͸ྫ͑͹ҎԼͷ2ͭͰදݱͰ͖Δ • ߦ൪߸ (lineno) • ߦ಄͔Βͷoffset (column)

    a = 1 if a # [3.0-7.3] 1 # [4.2-4.3] else 10 # [6.2-6.4] end 38
  40. column ͱ͸ͳʹ͔ • ߦͷઌ಄͔Β / ϑΝΠϧͷઌ಄͔Β • ݱঢ়͸ߦͷઌ಄͔Β • 0-based

    / 1-based • ݴޠ΍ΤσΟλʹΑͬͯҟͳΔ • ݱঢ়͸ 0-based • byte਺ / จࣈ਺ • ݱঢ়͸byte਺ • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby-lang.org/projects/ruby-trunk/wiki/Node- position-memo 39
  41. column ৘ใͷऔಘͱอ؅ • Ruby 2.4·Ͱ͸columnͷ৘ใΛอ؅͍ͯ͠ͳ͔ͬͨ(linenoͷΈ Λอ͍࣋ͯ͠Δ) • Ruby 2.5ͰରԠͨ͜͠ͱ •

    columnΛه࿥͢ΔྖҬΛ(ԕ౻͞Μ͕)֬อ(লུ) • ద੾ͳcolumn৘ใΛॻ͖ࠐΉ 40
  42. column ৘ใͷऔಘͱอ؅ (Ruby 2.4) $ ruby --dump=p -e '1 +

    2' # @ NODE_SCOPE (line: 1) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_CALL (line: 1) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 41
  43. column ৘ใͷऔಘͱอ؅ (Ruby 2.5) $ ruby --dump=p -e '1 +

    2' # @ NODE_SCOPE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_OPCALL (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 4, last_lineno: 1, last_column: 5) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 42
  44. lexer ͱ parser • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) • lexer (parse.y yylex) •

    ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) • parser (parse.c yyparse) • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ 43
  45. lexer ͱ parser # e.g. 1 + 2\n 44

  46. lexer ͱ parser (ࣈ۟ղੳ) • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) $ ruby --dump=y -e

    '1 + 2' tINTEGER : 1 '+' : '+' tINTEGER : 2 '\n' : '\n' 45
  47. lexer ͱ parser (ߏจղੳ) • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) $ ruby --dump=y -e

    '1 + 2' 46
  48. lexer ͱ parser (ߏจղੳ) # Shifting token tINTEGER (1) tINTEGER

    simple_numeric numeric literal primary arg 47
  49. lexer ͱ parser (ߏจղੳ) # Shifting token '+' arg '+'

    # Shifting token tINTEGER (2) arg '+' tINTEGER arg '+' simple_numeric arg '+' numeric arg '+' literal arg '+' primary arg '+' arg arg expr stmt top_stmt top_stmts 48
  50. lexer ͱ parser (ߏจղੳ) # Shifting token '\n' top_stmts '\n'

    top_stmts term top_stmts terms top_stmts opt_terms top_compstmt program # ׬ྃ 49
  51. lexer ͱ parser • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ $ ruby --dump=p -e '1

    + 2' NODE_SCOPE NODE_PRELUDE NODE_OPCALL (:+) NODE_LIT (1) NODE_ARRAY NODE_LIT (2) 50
  52. lexer ͱ parser • ৄ͘͠͸ parse.y ΛಡΜͰ͍ͩ͘͞ • https://github.com/ruby/ruby/blob/trunk/parse.y 51

  53. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • ͋ΔtokenΛೝࣝͨ͠ͱ͖ͷɺlexerͷoffset৘ใ • tokp - lex_pbeg͓Αͼlex_p

    - lex_pbeg • ࠓͷtokenΛ੾ΓऔΓऴΘͬͯɺ࣍ͷtokenΛ੾ΓऔΔલʹͲ͜ ͔ʹୀආ͢Δඞཁ͕͋Δ 52
  54. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) /* parse.y */ /* Structure of

    Lexer Buffer: lex_pbeg tokp lex_p lex_pend | | | | |-----------+--------------+------------| |<------------>| token */ 53
  55. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • 1 + 2Ͱ+ͷtokenΛೝࣝͨ͠ॠؒ 1 +

    2 ^ ^^ ^ | || +- lex_pend | |+---- lex_p | +----- tokp +-------- lex_pbeg 54
  56. columnͷऔಘํ๏ • lexer͔Βparserʹlocation৘ใΛ౉͢ • parser͔ΒNODEʹlocation৘ใΛ౉͢ 55

  57. columnͷऔಘํ๏ • Ґஔ৘ใ͸YYLTYPEͱ͍͏ߏ଄ମͰ؅ཧ͍ͯ͠Δ • ։࢝(first)ͱऴྃ(last)ͷ৘ใΛอ༗͍ͯ͠Δ #define YYLTYPE rb_code_range_t typedef struct

    rb_code_location_struct { int lineno; int column; } rb_code_location_t; typedef struct rb_code_range_struct { rb_code_location_t first_loc; rb_code_location_t last_loc; } rb_code_range_t; 56
  58. columnͷऔಘํ๏ (lexer -> parser) • yylexͷͳ͔ͰҐஔ৘ใΛ౎౓ୀආ͢Δ( YYLTYPE *yylloc ) •

    Ҿ਺ʹYYLTYPE *yylloc͕௥Ճ͞Εͨ • RUBY_SET_YYLLOCΛݺͼग़ͯ͠yyllocΛઃఆ͢Δ 57
  59. columnͷऔಘํ๏ (lexer -> parser) -static int -yylex(YYSTYPE *lval, struct parser_params

    *parser) +static enum yytokentype +yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *parser) { - int t; + enum yytokentype t; parser->lval = lval; lval->val = Qundef; t = parser_yylex(parser); if (has_delayed_token()) dispatch_delayed_token(t); else if (t != 0) dispatch_scan_event(t); + RUBY_SET_YYLLOC(*yylloc); + return t; } parser_yylex͕࣮ࡍʹtokenΛೝࣝ͢ΔͨΊͷؔ਺ 58
  60. columnͷऔಘํ๏ (lexer -> parser) #define RUBY_SET_YYLLOC(Current) \ rb_parser_set_location(parser, &(Current)) void

    rb_parser_set_location(struct parser_params *parser, YYLTYPE *yylloc) { yylloc->first_loc.lineno = ruby_sourceline; yylloc->first_loc.column = (int)(parser->tokp - lex_pbeg); yylloc->last_loc.lineno = ruby_sourceline; yylloc->last_loc.column = (int)(lex_p - lex_pbeg); } 59
  61. columnͷऔಘํ๏ (parser -> NODE) • ֤ΞΫγϣϯͰ͸Bisonͷ@nΛར༻ͯ͠tokenͷҐஔ৘ใΛऔಘ ͢Δ (YYLTYPE *yylsp) •

    ओʹ࢖͏ͷ͸@$ (YYLTYPE yyloc) 60
  62. columnͷऔಘํ๏ (parser -> NODE) |---| @$ (1.0-1.5) 1 + 2

    ^ ^ ^ | | +- @3 (1.4-1.5) | +--- @2 (1.2-1.3) +----- @1 (1.0-1.1) | arg '+' arg { - $$ = call_bin_op($1, '+', $3); + $$ = call_bin_op($1, '+', $3, &@$); } 61
  63. columnͷऔಘํ๏ (parser -> NODE) static NODE * call_bin_op_gen(struct parser_params *parser,

    NODE *recv, ID id, NODE *arg1, const YYLTYPE *location) { NODE *expr; value_expr(recv); value_expr(arg1); expr = NEW_OPCALL(recv, id, new_list(arg1, location)); /* NODEͷੜ੒ */ fixpos(expr, recv); expr->nd_loc = *location; /* locationΛηοτ */ return expr; } 62
  64. ޻෉͕͍Βͳ͍ NODE (NODE_OPCALL) • NODE_OPCALLΛ࡞Δͱ͖ʹඞཁͳ৘ใ͕શ෦ͦΖ͍ͬͯΔ • NODE_LIT (1) • mid

    (:+) • NODE_ARRAY (2) 1 + 2 ----------- arg '+' arg 63
  65. ޻෉͕ඞཁͳ NODE (NODE_*ASGNܥ) • NODE_LASGN (ϩʔΧϧม਺୅ೖ)΍NODE_IASGN (Πϯελϯεม਺ ୅ೖ) @a =

    1 64
  66. • ୅ೖͷӈล͕ܾ·Δ·͑ʹNODE_*ASGNΛੜ੒͢ΔͷͰɺӈล͕ ܾ·Δͱ͖ʹߋ৽͕ඞཁ NODE_IASGN @a ---------- ------------- lhs : user_variable

    { $$ = assignable(var_field($1), 0, &@$); } => NODE_IASGN NODE_IASGN = 1 ---------- ------------------------ arg : lhs '=' arg_rhs { $$ = node_assign($1, $3, &@$); } 65
  67. static NODE * node_assign_gen(struct parser_params *parser, NODE *lhs, NODE *rhs,

    const YYLTYPE *location) { if (!lhs) return 0; switch (nd_type(lhs)) { case NODE_GASGN: case NODE_IASGN: ... lhs->nd_value = rhs; lhs->nd_loc = *location; break; ... } return lhs; } 66
  68. ޻෉͕ඞཁͳ NODE (NODE_ITER) • NODE_ITER (ϒϩοΫ෇͖ϝιουݺͼग़͠) 3.times { foo }

    NODE_ITER NODE_CALL (:times) NODE_LIT (3) NODE_SCOPE NODE_VCALL (:foo) 67
  69. ޻෉͕ඞཁͳ NODE (NODE_ITER) • ϒϩοΫͷ෦෼ͰNODE_ITERΛੜ੒͢ΔͷͰɺ͋ͱ͔ΒNODEͷ ։࢝ҐஔΛमਖ਼͢Δඞཁ͕͋Δ 68

  70. 3.times { foo } NODE_CALL 3 . times ----------- --------------------------------------------------

    method_call | primary_value call_op operation2 {} opt_paren_args { $$ = new_qcall($2, $1, $3, $5, &@$); nd_set_line($$, $<num>4); } NODE_ITER { NODE_ITER } ----------- --------------------- brace_block : '{' {} brace_body '}' { $$ = $3; nd_set_line($$, $<num>2); } NODE_ITER NODE_CALL NODE_ITER --------- ----------------------- primary | method_call brace_block { block_dup_check($1->nd_args, $2); $2->nd_iter = $1; $2->nd_loc = @$; $$ = $2; } 69
  71. test ʹ͍ͭͯ • 2017/11/27ݱࡏɺtrunkʹtest͸commit͞Ε͍ͯͳ͍ 70

  72. test ʹ͍ͭͯ • test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ • ະॳظԽͷNODEΛݕग़͢Δ • NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ)

    71
  73. test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ node = AST.parse("1 + 2") # => #<AST(NODE_SCOPE(0) 1:0,

    1:5): > node.children[1].children[0] # => #<AST(NODE_OPCALL(35) 1:0, 1:5): > node.children[1].children[0].children # => [#<AST(NODE_LIT(58) 1:0, 1:1): >, #<AST(NODE_ARRAY(41) 1:0, 1:5): >] 72
  74. ະॳظԽͷNODEΛݕग़͢Δ • linenoΛ0, columnΛ-1ͰॳظԽͯ͋͠Δ NODE_IF (line: 1, first_lineno: 0, first_column:

    -1, last_lineno: 0, last_column: -1) 73
  75. ະॳظԽͷNODEΛݕग़͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $

    ruby ast_validation.rb "Total 0 count errors." 74
  76. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ 3.times { foo } NODE_ITER [1.0-1.15] -> [1.0-1.13]ΛؚΉ NODE_CALL

    (:times) [1.0-1.9] -> [1.0-1.1]ΛؚΉ NODE_LIT (3) [1.0-1.1] NODE_SCOPE [1.8-1.15] -> [1.10-1.13]ΛؚΉ NODE_VCALL (:foo) [1.10-1.13] 75
  77. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $

    ruby ast_validation.rb "test/ruby/beginmainend.rb has 1 errors" [["NODE_ARRAY", 1]] "test/ruby/sentence.rb has 1 errors" [["NODE_ARGS", 1]] ... "test/ruby/test_yield.rb has 1 errors" [["NODE_ARGS", 1]] "Total 552 count errors." 76
  78. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • "NODE_ARRAY"΍"NODE_DSTR"ͳͲ͕໰୊ • "NODE_ARRAY" • ྫ͑͹ [1,2,3] 77

  79. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ຤ඌ௥هͷͨͼʹશ෦ͷNODEͷlast locationΛߋ৽͢Δʁ 78

  80. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| |__________| ----> |___________| 79

  81. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________|

    | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| |__________| ----> |___________| 80
  82. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________|

    | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(3)| |__________| ----> |___________| 81
  83. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ݱঢ়͸node.cಉ༷ʹ్தͷNODE_ARRAYΛඈ͹͍ͯ͠Δ node = AST.parse("[1, 2, 3]") # =>

    #<AST(NODE_SCOPE(0) 1:0, 1:9): > node.children[1].children[0] # => #<AST(NODE_ARRAY(41) 1:1, 1:8): > node.children[1].children[0].children # => [ # #<AST(NODE_LIT(58) 1:1, 1:2): >, # #<AST(NODE_LIT(58) 1:4, 1:5): >, # #<AST(NODE_LIT(58) 1:7, 1:8): >, nil # ] 82
  84. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) 1 + 2 NODE_OPCALL (:+) [1.0-1.5] -> (1)ͱ(2)͕ඃ͍ͬͯͳ͍ NODE_LIT

    (1) [1.0-1.1] -> (1) NODE_ARRAY [1.4-1.5] -> (2) NODE_LIT (2) [1.4-1.5] 83
  85. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) • "NODE_OP_ASGN*"ܥ͕໰୊ • ྫ͑͹ @a ||= 1 ("NODE_OP_ASGN_OR") 84

  86. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) NODE_OP_ASGN_OR [1.0-1.8] -> (1)ͱ(2)͕ඃ͍ͬͯΔ NODE_IVAR (@a) [1.0-1.2] -> (1)

    NODE_IASGN (@a) [1.0-1.8] -> (2) NODE_LIT (1) [1.7-1.8] @a ||= 1 -- <-- NODE_IVAR -------- <-- NODE_IASGN 85
  87. ·ͱΊ 86

  88. ·ͱΊ • Ruby 2.5Ͱ͸Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ ͞ΕΔ • Ruby 2.5Ͱ͸NODE͕։࢝ɾऴྃҐஔΛ࣋ͭΑ͏ʹͳΔ

    • Ruby 2.5ͷϦϦʔε·Ͱʹ͍Ζ͍Ζमਖ਼Λ͍ΕΔ 87
  89. ँࣙ • @mametter • @nobu • @ko1 • @shyouhei •

    @takeshinoda • @hkdnet • @HaiTo • @littlestarling 88
  90. Ruby 2.5ϦϦʔε·ͰͷಓͷΓ • testΛcommit͢Δ • test͕௨ΔΑ͏ʹ͢Δ • Method CoverageͰ΋column͕ͱΕΔΑ͏ʹ͢Δ 89

  91. ࠓޙͷ֦ு(ͷՄೳੑ) • &&΍||ͷΧόϨοδ΋ͱΕΔΑ͏ʹ͢Δ • NoMethodErrorͷ৘ใΛ૿΍͢ • power_assertͷ৘ใΛ૿΍͢ 90

  92. ࢀߟ৘ใ • "Add branch coverage" https://bugs.ruby-lang.org/issues/13901 • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby- lang.org/projects/ruby-trunk/wiki/Node-position-memo

    • https://www.gnu.org/software/bison/manual/html_node/Token- Locations.html#Token-Locations • http://i.loveruby.net/ja/rhg/book/ ͷ"ୈ 2 ෦ʮߏจղੳʯ" 91
  93. ͓ΘΓ 92

  94. ͓·͚ 93

  95. heredocʹ͍ͭͯ 94

  96. heredocʹ͍ͭͯ • heredocศརͰ͢ΑͶ def upcase(s) s.upcase end upcase(<<STR1) abcde STR1

    # => "ABCDE\n" 95
  97. heredocʹ͍ͭͯ • heredocศར... def concat(s, t) s + t end

    concat(<<STR1, <<STR2) abcde STR1 12345 STR2 # => "abcde\n12345\n" 96
  98. heredocʹ͍ͭͯ • Ͳ͜·Ͱ͕heredocͷൣғʁ def concat(s, t) s + t end

    concat(<<STR1, <<STR2) abcde # ͔͜͜Β fghij # ͜͜·Ͱͱ STR1 12345 # ͔͜͜Β 67890 # ͜͜·Ͱʁ STR2 97
  99. ఏҊ • heredocͷNODEͷൣғΛ <<STR1 ʹ͠·ͤΜ͔ʁ • test͕༰қʹͳΔ • ࣮ࡍͷจࣈྻͷൣғ͸୅දҐஔ(ະ࣮૷)ͳͲͰද͢ concat(<<STR1,

    <<STR2) # [1.7-1.13] [1.15-1.21] abcde fghij STR1 12345 67890 STR2 98
  100. heredocͷ࢓૊Έ • ·ͣී௨ʹconcat(·ͰಡΉ concat( 99

  101. heredocͷ࢓૊Έ • <<STR1ΛಡΉͱɺheredocΛಡΈऔΔঢ়ଶʹͳΔ concat(<<STR1 100

  102. heredocͷ࢓૊Έ • STR1·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<<STR1 abcde fghij STR1 101

  103. heredocͷ࢓૊Έ • ,ΛಡΉ concat(<<STR1, abcde fghij STR1 102

  104. heredocͷ࢓૊Έ • <<STR2ΛಡΉͱɺheredocΛಡΈऔΔঢ়ଶʹͳΔ concat(<<STR1, <<STR2 abcde fghij STR1 103

  105. heredocͷ࢓૊Έ • STR2·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<<STR1, <<STR2 abcde fghij STR1 12345 67890

    STR2 104
  106. heredocͷ࢓૊Έ • lex_strterm ͷঢ়ଶʹΑͬͯɺݱࡏheredocΛղੳ͍ͯ͠Δ͔൑ அ͍ͯ͠Δ static enum yytokentype parser_yylex(struct parser_params

    *parser) { ... if (lex_strterm) { if (lex_strterm->flags & STRTERM_HEREDOC) { return here_document(&lex_strterm->u.heredoc); } else { token_flush(parser); return parse_string(&lex_strterm->u.literal); } } ... switch (c = nextc()) { case '\0': /* NUL */ case '\004': /* ^D */ ... 105
  107. heredocͷ࢓૊Έ • lex_strterm Λઃఆ͍ͯ͠Δίʔυ (parser_heredoc_identifier) • lex_goto_eol Ͱߦ຤·Ͱjump͍ͯ͠Δ • ͜ΕʹΑͬͯ࣍ͷtokenऔಘ࣌ʹ࣍ͷߦ͔Β͸͡ΊΔ͜ͱ͕Ͱ

    ͖Δ • lex_strterm ʹheredocΛൈ͚ͨ࣌ʹඞཁͳ৘ใΛୀආ͢Δ 106
  108. heredocͷ࢓૊Έ static enum yytokentype parser_heredoc_identifier(struct parser_params *parser) { ... lex_goto_eol(parser);

    /* ߦ຤·Ͱjump͍ͯ͠Δ */ lex_strterm = (rb_strterm_t*)rb_imemo_new(imemo_parser_strterm, STR_NEW(tok(), toklen()), /* term */ lex_lastline, /* lastline */ len, /* lastidx */ ruby_sourceline); /* ෮ؼޙʹඞཁͳ৘ใΛୀආ */ lex_strterm->flags |= STRTERM_HEREDOC; token_flush(parser); heredoc_indent = indent; heredoc_line_indent = 0; return token; /* ͨͱ͑͹ɺtSTRING_BEG */ } 107
  109. ఏҊ (࣮૷) • lex_strterm ʹୀආͯ͋͠Δ৘ใΛ࢖͑͹ɺ <<STR1 ͷҐஔ͕Θ ͔Δ͸ͣ 108

  110. #define YYLLOC_DEFAULT(Current, Rhs, N) \ do \ if (N) \

    - { \ - (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ - (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ - } \ + if (lex_strterm && (lex_strterm->flags & STRTERM_HEREDOC)) \ + { \ + rb_strterm_heredoc_t here = lex_strterm->u.heredoc; \ + (Current).first_loc.lineno = (int)here.sourceline; \ + (Current).first_loc.column = (int)(here.u3.lastidx - RSTRING_LEN(here.term));\ + (Current).last_loc.lineno = (int)here.sourceline; \ + (Current).last_loc.column = (int)(here.u3.lastidx);\ + } \ + else \ + { \ + (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ + (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ + } \ else \ RUBY_SET_YYLLOC(Current); \ while (0) 109
  111. ͩΊͦ͏ͳέʔε (1) "#{<<~"A"}" #{:b if :a} f = 11 A

    110
  112. ͩΊͦ͏ͳέʔε (2) "#{<<~"A"}" #{<<~B} 12 13 B f = 11

    A 111
  113. parse.y͸ѱຐ৓ͳͷ͔ʁ • ҎԼͷ఺Λআ͚͹ɺͦ͜·Ͱѱຐ৓Ͱ͸ͳ͍ͷͰ͸? • Ұ෦ripperͱϑΝΠϧΛڞ༗͍ͯ͠Δ • ߦ਺͕ଟ͍ • parser_paramsʹ͍Ζ͍Ζͳϑϥά͕͋Δ •

    lex_stateʹ͍Ζ͍Ζͳϑϥά͕͋Δ • Rubyʹ͸heredoc͕͋Δ 112