Coverage.so, or RNode with column

Transcript

1. 
   ۚࢠ༤Ұ࿠
   !ZVJLOL
   
   ϚωʔϑΥϫʔυ3VCZษڧձ ਐԽ͢Δ
   $PWFSBHFTPɺ
   ͋Δ͍͸
   /0%&
   ͱ
   DPMVNO
   ͷ͸ͳ͠
2. raise MissingBeer unless 1
3. ࣗݾ঺հ • Ҏલ͸ॏ޻ۀͳϝʔΧʔͰܦཧΛ͍ͯͨ͠ • ϚωʔϑΥʔϫʔυͰ͸ձܭνʔϜͰRailsΞϓϦΛॻ͍͍ͯΔ • Rubyίϛολʔ 2015/12~ • Asakusa.rb ॴଐ 2
4. • Ruby 2.5ͷશମతͳ࿩͸͜ͷ͋ͱ͋Δ͸ͣ • ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱͷ࿩ 3
5. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ $ git log --numstat --pretty="%H" --author="yui-knk" --since="2016-12-26" |\ awk 'NF==3 {plus+=$1; minus+=$2; printf("%s %d %d\n", $3, $1, $2)}' |\ ruby -e 'ary=[]; while gets(); ary << $_.chomp.split(" ");end; \ ary.each_with_object({}) {|(n,p,m),h| h[n]||= [0,0];\ h[n] = [h[n][0] + p.to_i, h[n][1] + m.to_i]}.sort_by\ {|(k,(p,m))| -(p + m) }.each {|(k,(p,m))| puts "#{k} +#{p}, -#{m}"}' 4
6. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ parse.y +2256, -1687 compile.c +148, -78 test/coverage/test_coverage.rb +100, -55 node.h +34, -38 node.c +38, -18 ext/coverage/coverage.c +20, -12 thread.c +8, -6 ext/objspace/objspace.c +1, -5 test/ruby/test_iseq.rb +3, -1 test/ripper/test_lexer.rb +1, -1 test/ruby/test_string.rb +1, -1 insns.def +0, -2 ext/objspace/object_tracing.c +1, -1 doc/extension.ja.rdoc +1, -1 doc/extension.rdoc +1, -1 iseq.c +0, -1 5
7. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ • (ओʹ)CoverageͷͨΊʹɺparserΛ͍͍ͬͯͨ͡ 6
8. Contents • Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ Coverage.so • Branch Coverage ͷ࢓૊Έ • NODE ͱ column ͷ͸ͳ͠ • lexer ͱ parser • column ৘ใͷऔಘͱอ؅ • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ • ͓·͚ 7
9. ஫ҙ఺ • ϋογϡλά => #mf_ruby (શ෦খจࣈ) • લ൒ => Ruby 2.5Λ࢖͏͏͑Ͱ໾ʹཱͭ • ޙ൒ => Ruby 2.5Λ͍͡Δ͏͑Ͱ໾ʹཱͭ 8
10. Coverage ͷ͸ͳ͠ 9
11. Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ Coverage.so • Branch Coverage ͷ࢓૊Έ 10
12. Coverage ͱ͸ • ςετίʔυΛධՁ͢Δͱ͖ʹࢀߟʹͳΔ৘ใ • ৄ͘͠͸ԕ౻(@mametter)͞ΜͷൃදΛ͝ཡ͍ͩ͘͞ • "An introduction and future of Ruby coverage library" (http:// rubykaigi.org/2017/presentations/mametter.html) 11
13. Coverage ͷ࣮ྫ # Coverageऔಘର৅ͷίʔυ def target(bool) if bool 1 else 0 end end target(true) # CoverageΛऔಘ͢ΔͨΊͷίʔυ require "coverage" Coverage.start require "target_file" # ର৅ͷϑΝΠϧΛload Coverage.result 12
14. Coverage ͷ࣮ྫ # Coverage.result { "src/coverage_example.rb" => [1, 1, 1, nil, 0, nil, nil, nil, 1] } # ͦΕͬΆ͘mapping 1: def target(bool) 1: if bool 1: 1 nil: else 0: 0 nil: end nil: end nil: 1: target(true) 13
15. Ruby 2.5ʹ͓͚Δ Coverage.so • Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ͞ΕΔ Ruby 2.4 Ruby 2.5 Line Coverage o o Branch Coverage - o Method Coverage - o 14
16. Method Coverageͱ͸ • ϝιου͕Կճݺͼग़͞Ε͔͕ͨΘ͔ΔΑ͏ʹͳΔ def target1 end def target2 end target2 0: def target1 : end : 1: def target2 : end : : target2 15
17. Branch Coverageͱ͸ • Ͳͷ෼ذ͕Կճ࣮ߦ͞Ε͔ͨΘ͔ΔΑ͏ʹͳΔ a = 1 if a 1 else 2 end : a = 1 : : if a 1: 1 : else 0: 2 : end 16
18. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(1) • ޙஔif 1: a = true 1: b = false 1: 1 if a 1: 2 if b # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 17
19. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(2) • ࡾ߲ԋࢉ 1: a = true 1: b = false 1: a ? 1 : 2 # a͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 1: b ? 1 : 2 # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 18
20. Branch Coverageͷྫ(if/unless) • ޙஔif/unless΍ࡾ߲ԋࢉࢠʹ΋ରԠ : a = 1 : 1/0: 10 if a == 1 0/1: 11 unless a == 1 0/1: (a == 2) ? :t : :f n/m -> n͕thenɺm͕elseͷճ਺ 19
21. Branch Coverageͷྫ(while/until) : x = 3 : while x > 0 3: x -= 1 : end : until x == 10 10: x += 1 : end : y = 3 3: y -= 1 while y > 0 10: y += 1 until y == 10 20
22. Branch Coverageͷྫ(case) : x = 0 : : case x : when 0 1: 0 : when 1 0: 1 : end : : case : when x == 0 1: 0 : when x == 1 0: 1 : end 21
23. Branch Coverageͷྫ(&.) : a = 10 : b = nil 1/0: a&.abs 0/1: b&.hoo n/m -> n͕then(ϝιουΛcall)ɺm͕elseͷճ਺ 22
24. Branch Coverage ͷ࢓૊Έ • ؅ཧςʔϒϧΛ༻ҙ • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ • ໋ྩ͕ɺ؅ཧςʔϒϧͷ֘౰ՕॴΛΠϯΫϦϝϯτ͢Δ a = 1 if a :A else :B end 23
25. Branch Coverage ͷ࢓૊Έ (؅ཧςʔϒϧ) • ؅ཧςʔϒϧΛ༻ҙ [structure, counters] structure = [ [ :if , 2, 0, 6, 3, # type, first_lineno, first_column, last_lineno, last_column :then, 3, 2, 3, 4, 0, # type, first_lineno, first_column, last_lineno, last_column, counter_index :else, 5, 2, 5, 4, 1 # type, first_lineno, first_column, last_lineno, last_column, counter_index ], ... ] counters = [0, 0] # [:then_counter, :else_counter] 24
26. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ src = <<~SRC a = 1 if a :A else :B end SRC # Coverage OFF puts RubyVM::InstructionSequence.compile(src).disasm require "coverage" ENV["COVERAGE_EXPERIMENTAL_MODE"] = "true" Coverage.start(branches: true) # Coverage ON puts RubyVM::InstructionSequence.compile(src).disasm 25
27. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕OFFͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 10 0007 putobject :A ( 3)[Li] 0009 leave ( 5) 0010 putobject :B[Li] 0012 leave 26
28. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 27
29. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #<ISeq:<compiled>@<compiled>>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 28
30. Branch Coverage ͷ࢓૊Έ (count up) trace2 131072, 1 # => counter_idx = 0 trace2 131072, 17 # => counter_idx = 1 • ୈ2Φϖϥϯυ(1΍17)͕counter_idxΛද͢ • counter_idx * 16 + COVERAGE_INDEX_BRANCHES • COVERAGE_INDEX_BRANCHES͸1 29
31. Branch Coverage ͷ࢓૊Έ (count up) == disasm: #<ISeq:<compiled>@<compiled>>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • a͕false΋͘͠͸nilͳΒ0013ʹͱͿ • ͦΕҎ֎ͷͱ͖͸Կ΋ͤͣɺ࣍ͷ໋ྩ΁͍͘ 30
32. Branch Coverage ͷ࢓૊Έ (count up) == disasm: #<ISeq:<compiled>@<compiled>>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • ࠓճͷέʔεͰ͸a = 1ͳͷͰɺtrace2 131072, 1͕࣮ߦ͞Ε Δ -> :then_counter͕૿Ճ͢Δ 31
33. Branch Coverage ͷ࢓૊Έ (݁Ռ) [structure, counters] structure = [[:if, 2, 0, 6, 3, :then, 3, 2, 3, 4, 0, :else, 5, 2, 5, 4, 1], ...] counters = [1, 0] # [:then_counter, :else_counter] => # ͦΕͬΆ͘mapping : a = 1 : if a 1: :A : else 0: :B : end 32
34. column৘ใ͕΄͍͠έʔε • औಘͨ͠Coverage݁ՌΛՄࢹԽ͢Δͱ͖ • :Aͱ͔:BͷҐஔ͕Θ͔Δͱศར a, b = 1, 1 (a == b) ? :A : :B 33
35. column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1, 3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2] => { # 2ߦ໨ [:then, 1, 2] => 0, # 2ߦ໨ [:else, 2, 2] => 1 # 2ߦ໨ }, [:if, 3, 2] => { # 2ߦ໨ [:then, 4, 2] => 1, # 2ߦ໨ [:else, 5, 2] => 0 # 2ߦ໨ } } 34
36. column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1, 3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2, 12, 2, 30] => { # [2.12-2.30] [:then, 1, 2, 23, 2, 25] => 0, # [2.23-2.25] [:else, 2, 2, 28, 2, 30] => 1 # [2.28-2.30] }, [:if, 3, 2, 0, 2, 36] => { # [2.0 -2.36] [:then, 4, 2, 12, 2, 30] => 1, # [2.12-2.30] [:else, 5, 2, 34, 2, 36] => 0 # [2.34-2.36] } } 35
37. NODE ͱ column ͷ͸ͳ͠ 36
38. NODE ͱ column ͷ͸ͳ͠ • column ͱ͸ͳʹ͔ • column ৘ใͷऔಘͱอ؅ • lexer ͱ parser • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ 37
39. column ͱ͸ͳʹ͔ • ࣜ΍จͷҐஔ৘ใ͸ྫ͑͹ҎԼͷ2ͭͰදݱͰ͖Δ • ߦ൪߸ (lineno) • ߦ಄͔Βͷoffset (column) a = 1 if a # [3.0-7.3] 1 # [4.2-4.3] else 10 # [6.2-6.4] end 38
40. column ͱ͸ͳʹ͔ • ߦͷઌ಄͔Β / ϑΝΠϧͷઌ಄͔Β • ݱঢ়͸ߦͷઌ಄͔Β • 0-based / 1-based • ݴޠ΍ΤσΟλʹΑͬͯҟͳΔ • ݱঢ়͸ 0-based • byte਺ / จࣈ਺ • ݱঢ়͸byte਺ • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby-lang.org/projects/ruby-trunk/wiki/Node- position-memo 39
41. column ৘ใͷऔಘͱอ؅ • Ruby 2.4·Ͱ͸columnͷ৘ใΛอ؅͍ͯ͠ͳ͔ͬͨ(linenoͷΈ Λอ͍࣋ͯ͠Δ) • Ruby 2.5ͰରԠͨ͜͠ͱ • columnΛه࿥͢ΔྖҬΛ(ԕ౻͞Μ͕)֬อ(লུ) • ద੾ͳcolumn৘ใΛॻ͖ࠐΉ 40
42. column ৘ใͷऔಘͱอ؅ (Ruby 2.4) $ ruby --dump=p -e '1 + 2' # @ NODE_SCOPE (line: 1) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_CALL (line: 1) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 41
43. column ৘ใͷऔಘͱอ؅ (Ruby 2.5) $ ruby --dump=p -e '1 + 2' # @ NODE_SCOPE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_OPCALL (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 4, last_lineno: 1, last_column: 5) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 42
44. lexer ͱ parser • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) • lexer (parse.y yylex) • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) • parser (parse.c yyparse) • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ 43
45. lexer ͱ parser # e.g. 1 + 2\n 44
46. lexer ͱ parser (ࣈ۟ղੳ) • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) $ ruby --dump=y -e '1 + 2' tINTEGER : 1 '+' : '+' tINTEGER : 2 '\n' : '\n' 45
47. lexer ͱ parser (ߏจղੳ) • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) $ ruby --dump=y -e '1 + 2' 46
48. lexer ͱ parser (ߏจղੳ) # Shifting token tINTEGER (1) tINTEGER simple_numeric numeric literal primary arg 47
49. lexer ͱ parser (ߏจղੳ) # Shifting token '+' arg '+' # Shifting token tINTEGER (2) arg '+' tINTEGER arg '+' simple_numeric arg '+' numeric arg '+' literal arg '+' primary arg '+' arg arg expr stmt top_stmt top_stmts 48
50. lexer ͱ parser (ߏจղੳ) # Shifting token '\n' top_stmts '\n' top_stmts term top_stmts terms top_stmts opt_terms top_compstmt program # ׬ྃ 49
51. lexer ͱ parser • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ $ ruby --dump=p -e '1 + 2' NODE_SCOPE NODE_PRELUDE NODE_OPCALL (:+) NODE_LIT (1) NODE_ARRAY NODE_LIT (2) 50
52. lexer ͱ parser • ৄ͘͠͸ parse.y ΛಡΜͰ͍ͩ͘͞ • https://github.com/ruby/ruby/blob/trunk/parse.y 51
53. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • ͋ΔtokenΛೝࣝͨ͠ͱ͖ͷɺlexerͷoffset৘ใ • tokp - lex_pbeg͓Αͼlex_p - lex_pbeg • ࠓͷtokenΛ੾ΓऔΓऴΘͬͯɺ࣍ͷtokenΛ੾ΓऔΔલʹͲ͜ ͔ʹୀආ͢Δඞཁ͕͋Δ 52
54. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) /* parse.y */ /* Structure of Lexer Buffer: lex_pbeg tokp lex_p lex_pend | | | | |-----------+--------------+------------| |<------------>| token */ 53
55. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • 1 + 2Ͱ+ͷtokenΛೝࣝͨ͠ॠؒ 1 + 2 ^ ^^ ^ | || +- lex_pend | |+---- lex_p | +----- tokp +-------- lex_pbeg 54
56. columnͷऔಘํ๏ • lexer͔Βparserʹlocation৘ใΛ౉͢ • parser͔ΒNODEʹlocation৘ใΛ౉͢ 55
57. columnͷऔಘํ๏ • Ґஔ৘ใ͸YYLTYPEͱ͍͏ߏ଄ମͰ؅ཧ͍ͯ͠Δ • ։࢝(first)ͱऴྃ(last)ͷ৘ใΛอ༗͍ͯ͠Δ #define YYLTYPE rb_code_range_t typedef struct rb_code_location_struct { int lineno; int column; } rb_code_location_t; typedef struct rb_code_range_struct { rb_code_location_t first_loc; rb_code_location_t last_loc; } rb_code_range_t; 56
58. columnͷऔಘํ๏ (lexer -> parser) • yylexͷͳ͔ͰҐஔ৘ใΛ౎౓ୀආ͢Δ( YYLTYPE *yylloc ) • Ҿ਺ʹYYLTYPE *yylloc͕௥Ճ͞Εͨ • RUBY_SET_YYLLOCΛݺͼग़ͯ͠yyllocΛઃఆ͢Δ 57
59. columnͷऔಘํ๏ (lexer -> parser) -static int -yylex(YYSTYPE *lval, struct parser_params *parser) +static enum yytokentype +yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *parser) { - int t; + enum yytokentype t; parser->lval = lval; lval->val = Qundef; t = parser_yylex(parser); if (has_delayed_token()) dispatch_delayed_token(t); else if (t != 0) dispatch_scan_event(t); + RUBY_SET_YYLLOC(*yylloc); + return t; } parser_yylex͕࣮ࡍʹtokenΛೝࣝ͢ΔͨΊͷؔ਺ 58
60. columnͷऔಘํ๏ (lexer -> parser) #define RUBY_SET_YYLLOC(Current) \ rb_parser_set_location(parser, &(Current)) void rb_parser_set_location(struct parser_params *parser, YYLTYPE *yylloc) { yylloc->first_loc.lineno = ruby_sourceline; yylloc->first_loc.column = (int)(parser->tokp - lex_pbeg); yylloc->last_loc.lineno = ruby_sourceline; yylloc->last_loc.column = (int)(lex_p - lex_pbeg); } 59
61. columnͷऔಘํ๏ (parser -> NODE) • ֤ΞΫγϣϯͰ͸Bisonͷ@nΛར༻ͯ͠tokenͷҐஔ৘ใΛऔಘ ͢Δ (YYLTYPE *yylsp) • ओʹ࢖͏ͷ͸@$ (YYLTYPE yyloc) 60
62. columnͷऔಘํ๏ (parser -> NODE) |---| @$ (1.0-1.5) 1 + 2 ^ ^ ^ | | +- @3 (1.4-1.5) | +--- @2 (1.2-1.3) +----- @1 (1.0-1.1) | arg '+' arg { - $$ = call_bin_op($1, '+', $3); + $$ = call_bin_op($1, '+', $3, &@$); } 61
63. columnͷऔಘํ๏ (parser -> NODE) static NODE * call_bin_op_gen(struct parser_params *parser, NODE *recv, ID id, NODE *arg1, const YYLTYPE *location) { NODE *expr; value_expr(recv); value_expr(arg1); expr = NEW_OPCALL(recv, id, new_list(arg1, location)); /* NODEͷੜ੒ */ fixpos(expr, recv); expr->nd_loc = *location; /* locationΛηοτ */ return expr; } 62
64. ޻෉͕͍Βͳ͍ NODE (NODE_OPCALL) • NODE_OPCALLΛ࡞Δͱ͖ʹඞཁͳ৘ใ͕શ෦ͦΖ͍ͬͯΔ • NODE_LIT (1) • mid (:+) • NODE_ARRAY (2) 1 + 2 ----------- arg '+' arg 63
65. ޻෉͕ඞཁͳ NODE (NODE_*ASGNܥ) • NODE_LASGN (ϩʔΧϧม਺୅ೖ)΍NODE_IASGN (Πϯελϯεม਺ ୅ೖ) @a = 1 64
66. • ୅ೖͷӈล͕ܾ·Δ·͑ʹNODE_*ASGNΛੜ੒͢ΔͷͰɺӈล͕ ܾ·Δͱ͖ʹߋ৽͕ඞཁ NODE_IASGN @a ---------- ------------- lhs : user_variable { $$ = assignable(var_field($1), 0, &@$); } => NODE_IASGN NODE_IASGN = 1 ---------- ------------------------ arg : lhs '=' arg_rhs { $$ = node_assign($1, $3, &@$); } 65
67. static NODE * node_assign_gen(struct parser_params *parser, NODE *lhs, NODE *rhs, const YYLTYPE *location) { if (!lhs) return 0; switch (nd_type(lhs)) { case NODE_GASGN: case NODE_IASGN: ... lhs->nd_value = rhs; lhs->nd_loc = *location; break; ... } return lhs; } 66
68. ޻෉͕ඞཁͳ NODE (NODE_ITER) • NODE_ITER (ϒϩοΫ෇͖ϝιουݺͼग़͠) 3.times { foo } NODE_ITER NODE_CALL (:times) NODE_LIT (3) NODE_SCOPE NODE_VCALL (:foo) 67
69. ޻෉͕ඞཁͳ NODE (NODE_ITER) • ϒϩοΫͷ෦෼ͰNODE_ITERΛੜ੒͢ΔͷͰɺ͋ͱ͔ΒNODEͷ ։࢝ҐஔΛमਖ਼͢Δඞཁ͕͋Δ 68
70. 3.times { foo } NODE_CALL 3 . times ----------- -------------------------------------------------- method_call | primary_value call_op operation2 {} opt_paren_args { $$ = new_qcall($2, $1, $3, $5, &@$); nd_set_line($$, $<num>4); } NODE_ITER { NODE_ITER } ----------- --------------------- brace_block : '{' {} brace_body '}' { $$ = $3; nd_set_line($$, $<num>2); } NODE_ITER NODE_CALL NODE_ITER --------- ----------------------- primary | method_call brace_block { block_dup_check($1->nd_args, $2); $2->nd_iter = $1; $2->nd_loc = @$; $$ = $2; } 69
71. test ʹ͍ͭͯ • 2017/11/27ݱࡏɺtrunkʹtest͸commit͞Ε͍ͯͳ͍ 70
72. test ʹ͍ͭͯ • test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ • ະॳظԽͷNODEΛݕग़͢Δ • NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) 71
73. test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ node = AST.parse("1 + 2") # => #<AST(NODE_SCOPE(0) 1:0, 1:5): > node.children[1].children[0] # => #<AST(NODE_OPCALL(35) 1:0, 1:5): > node.children[1].children[0].children # => [#<AST(NODE_LIT(58) 1:0, 1:1): >, #<AST(NODE_ARRAY(41) 1:0, 1:5): >] 72
74. ະॳظԽͷNODEΛݕग़͢Δ • linenoΛ0, columnΛ-1ͰॳظԽͯ͋͠Δ NODE_IF (line: 1, first_lineno: 0, first_column: -1, last_lineno: 0, last_column: -1) 73
75. ະॳظԽͷNODEΛݕग़͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $ ruby ast_validation.rb "Total 0 count errors." 74
76. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ 3.times { foo } NODE_ITER [1.0-1.15] -> [1.0-1.13]ΛؚΉ NODE_CALL (:times) [1.0-1.9] -> [1.0-1.1]ΛؚΉ NODE_LIT (3) [1.0-1.1] NODE_SCOPE [1.8-1.15] -> [1.10-1.13]ΛؚΉ NODE_VCALL (:foo) [1.10-1.13] 75
77. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $ ruby ast_validation.rb "test/ruby/beginmainend.rb has 1 errors" [["NODE_ARRAY", 1]] "test/ruby/sentence.rb has 1 errors" [["NODE_ARGS", 1]] ... "test/ruby/test_yield.rb has 1 errors" [["NODE_ARGS", 1]] "Total 552 count errors." 76
78. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • "NODE_ARRAY"΍"NODE_DSTR"ͳͲ͕໰୊ • "NODE_ARRAY" • ྫ͑͹ [1,2,3] 77
79. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ຤ඌ௥هͷͨͼʹશ෦ͷNODEͷlast locationΛߋ৽͢Δʁ 78
80. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| |__________| ----> |___________| 79
81. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| |__________| ----> |___________| 80
82. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(3)| |__________| ----> |___________| 81
83. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ݱঢ়͸node.cಉ༷ʹ్தͷNODE_ARRAYΛඈ͹͍ͯ͠Δ node = AST.parse("[1, 2, 3]") # => #<AST(NODE_SCOPE(0) 1:0, 1:9): > node.children[1].children[0] # => #<AST(NODE_ARRAY(41) 1:1, 1:8): > node.children[1].children[0].children # => [ # #<AST(NODE_LIT(58) 1:1, 1:2): >, # #<AST(NODE_LIT(58) 1:4, 1:5): >, # #<AST(NODE_LIT(58) 1:7, 1:8): >, nil # ] 82
84. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) 1 + 2 NODE_OPCALL (:+) [1.0-1.5] -> (1)ͱ(2)͕ඃ͍ͬͯͳ͍ NODE_LIT (1) [1.0-1.1] -> (1) NODE_ARRAY [1.4-1.5] -> (2) NODE_LIT (2) [1.4-1.5] 83
85. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) • "NODE_OP_ASGN*"ܥ͕໰୊ • ྫ͑͹ @a ||= 1 ("NODE_OP_ASGN_OR") 84
86. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) NODE_OP_ASGN_OR [1.0-1.8] -> (1)ͱ(2)͕ඃ͍ͬͯΔ NODE_IVAR (@a) [1.0-1.2] -> (1) NODE_IASGN (@a) [1.0-1.8] -> (2) NODE_LIT (1) [1.7-1.8] @a ||= 1 -- <-- NODE_IVAR -------- <-- NODE_IASGN 85
87. ·ͱΊ 86
88. ·ͱΊ • Ruby 2.5Ͱ͸Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ ͞ΕΔ • Ruby 2.5Ͱ͸NODE͕։࢝ɾऴྃҐஔΛ࣋ͭΑ͏ʹͳΔ • Ruby 2.5ͷϦϦʔε·Ͱʹ͍Ζ͍Ζमਖ਼Λ͍ΕΔ 87
89. ँࣙ • @mametter • @nobu • @ko1 • @shyouhei • @takeshinoda • @hkdnet • @HaiTo • @littlestarling 88
90. Ruby 2.5ϦϦʔε·ͰͷಓͷΓ • testΛcommit͢Δ • test͕௨ΔΑ͏ʹ͢Δ • Method CoverageͰ΋column͕ͱΕΔΑ͏ʹ͢Δ 89
91. ࠓޙͷ֦ு(ͷՄೳੑ) • &&΍||ͷΧόϨοδ΋ͱΕΔΑ͏ʹ͢Δ • NoMethodErrorͷ৘ใΛ૿΍͢ • power_assertͷ৘ใΛ૿΍͢ 90
92. ࢀߟ৘ใ • "Add branch coverage" https://bugs.ruby-lang.org/issues/13901 • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby- lang.org/projects/ruby-trunk/wiki/Node-position-memo • https://www.gnu.org/software/bison/manual/html_node/Token- Locations.html#Token-Locations • http://i.loveruby.net/ja/rhg/book/ ͷ"ୈ 2 ෦ʮߏจղੳʯ" 91
93. ͓ΘΓ 92
94. ͓·͚ 93
95. heredocʹ͍ͭͯ 94
96. heredocʹ͍ͭͯ • heredocศརͰ͢ΑͶ def upcase(s) s.upcase end upcase(<<STR1) abcde STR1 # => "ABCDE\n" 95
97. heredocʹ͍ͭͯ • heredocศར... def concat(s, t) s + t end concat(<<STR1, <<STR2) abcde STR1 12345 STR2 # => "abcde\n12345\n" 96
98. heredocʹ͍ͭͯ • Ͳ͜·Ͱ͕heredocͷൣғʁ def concat(s, t) s + t end concat(<<STR1, <<STR2) abcde # ͔͜͜Β fghij # ͜͜·Ͱͱ STR1 12345 # ͔͜͜Β 67890 # ͜͜·Ͱʁ STR2 97
99. ఏҊ • heredocͷNODEͷൣғΛ <<STR1 ʹ͠·ͤΜ͔ʁ • test͕༰қʹͳΔ • ࣮ࡍͷจࣈྻͷൣғ͸୅දҐஔ(ະ࣮૷)ͳͲͰද͢ concat(<<STR1, <<STR2) # [1.7-1.13] [1.15-1.21] abcde fghij STR1 12345 67890 STR2 98
100. heredocͷ࢓૊Έ • ·ͣී௨ʹconcat(·ͰಡΉ concat( 99
101. heredocͷ࢓૊Έ • <<STR1ΛಡΉͱɺheredocΛಡΈऔΔঢ়ଶʹͳΔ concat(<<STR1 100
102. heredocͷ࢓૊Έ • STR1·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<<STR1 abcde fghij STR1 101
103. heredocͷ࢓૊Έ • ,ΛಡΉ concat(<<STR1, abcde fghij STR1 102
104. heredocͷ࢓૊Έ • <<STR2ΛಡΉͱɺheredocΛಡΈऔΔঢ়ଶʹͳΔ concat(<<STR1, <<STR2 abcde fghij STR1 103
105. heredocͷ࢓૊Έ • STR2·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<<STR1, <<STR2 abcde fghij STR1 12345 67890 STR2 104
106. heredocͷ࢓૊Έ • lex_strterm ͷঢ়ଶʹΑͬͯɺݱࡏheredocΛղੳ͍ͯ͠Δ͔൑ அ͍ͯ͠Δ static enum yytokentype parser_yylex(struct parser_params *parser) { ... if (lex_strterm) { if (lex_strterm->flags & STRTERM_HEREDOC) { return here_document(&lex_strterm->u.heredoc); } else { token_flush(parser); return parse_string(&lex_strterm->u.literal); } } ... switch (c = nextc()) { case '\0': /* NUL */ case '\004': /* ^D */ ... 105
107. heredocͷ࢓૊Έ • lex_strterm Λઃఆ͍ͯ͠Δίʔυ (parser_heredoc_identifier) • lex_goto_eol Ͱߦ຤·Ͱjump͍ͯ͠Δ • ͜ΕʹΑͬͯ࣍ͷtokenऔಘ࣌ʹ࣍ͷߦ͔Β͸͡ΊΔ͜ͱ͕Ͱ ͖Δ • lex_strterm ʹheredocΛൈ͚ͨ࣌ʹඞཁͳ৘ใΛୀආ͢Δ 106
108. heredocͷ࢓૊Έ static enum yytokentype parser_heredoc_identifier(struct parser_params *parser) { ... lex_goto_eol(parser); /* ߦ຤·Ͱjump͍ͯ͠Δ */ lex_strterm = (rb_strterm_t*)rb_imemo_new(imemo_parser_strterm, STR_NEW(tok(), toklen()), /* term */ lex_lastline, /* lastline */ len, /* lastidx */ ruby_sourceline); /* ෮ؼޙʹඞཁͳ৘ใΛୀආ */ lex_strterm->flags |= STRTERM_HEREDOC; token_flush(parser); heredoc_indent = indent; heredoc_line_indent = 0; return token; /* ͨͱ͑͹ɺtSTRING_BEG */ } 107
109. ఏҊ (࣮૷) • lex_strterm ʹୀආͯ͋͠Δ৘ใΛ࢖͑͹ɺ <<STR1 ͷҐஔ͕Θ ͔Δ͸ͣ 108
110. #define YYLLOC_DEFAULT(Current, Rhs, N) \ do \ if (N) \ - { \ - (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ - (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ - } \ + if (lex_strterm && (lex_strterm->flags & STRTERM_HEREDOC)) \ + { \ + rb_strterm_heredoc_t here = lex_strterm->u.heredoc; \ + (Current).first_loc.lineno = (int)here.sourceline; \ + (Current).first_loc.column = (int)(here.u3.lastidx - RSTRING_LEN(here.term));\ + (Current).last_loc.lineno = (int)here.sourceline; \ + (Current).last_loc.column = (int)(here.u3.lastidx);\ + } \ + else \ + { \ + (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ + (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ + } \ else \ RUBY_SET_YYLLOC(Current); \ while (0) 109
111. ͩΊͦ͏ͳέʔε (1) "#{<<~"A"}" #{:b if :a} f = 11 A 110
112. ͩΊͦ͏ͳέʔε (2) "#{<<~"A"}" #{<<~B} 12 13 B f = 11 A 111
113. parse.y͸ѱຐ৓ͳͷ͔ʁ • ҎԼͷ఺Λআ͚͹ɺͦ͜·Ͱѱຐ৓Ͱ͸ͳ͍ͷͰ͸? • Ұ෦ripperͱϑΝΠϧΛڞ༗͍ͯ͠Δ • ߦ਺͕ଟ͍ • parser_paramsʹ͍Ζ͍Ζͳϑϥά͕͋Δ • lex_stateʹ͍Ζ͍Ζͳϑϥά͕͋Δ • Rubyʹ͸heredoc͕͋Δ 112