Tweet
Tweet

Slide 1

Slide 1 text

ۚࢠ༤Ұ࿠ !ZVJLOL ϚωʔϑΥϫʔυ3VCZษڧձ ਐԽ͢Δ$PWFSBHFTPɺ ͋Δ͍͸/0%&ͱDPMVNOͷ͸ͳ͠

Slide 2

Slide 2 text

raise MissingBeer unless 1

Slide 3

Slide 3 text

ࣗݾ঺հ • Ҏલ͸ॏ޻ۀͳϝʔΧʔͰܦཧΛ͍ͯͨ͠ • ϚωʔϑΥʔϫʔυͰ͸ձܭνʔϜͰRailsΞϓϦΛॻ͍͍ͯΔ • Rubyίϛολʔ 2015/12~ • Asakusa.rb ॴଐ 2

Slide 4

Slide 4 text

• Ruby 2.5ͷશମతͳ࿩͸͜ͷ͋ͱ͋Δ͸ͣ • ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱͷ࿩ 3

Slide 5

Slide 5 text

ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ $ git log --numstat --pretty="%H" --author="yui-knk" --since="2016-12-26" |\ awk 'NF==3 {plus+=$1; minus+=$2; printf("%s %d %d\n", $3, $1, $2)}' |\ ruby -e 'ary=[]; while gets(); ary << $_.chomp.split(" ");end; \ ary.each_with_object({}) {|(n,p,m),h| h[n]||= [0,0];\ h[n] = [h[n][0] + p.to_i, h[n][1] + m.to_i]}.sort_by\ {|(k,(p,m))| -(p + m) }.each {|(k,(p,m))| puts "#{k} +#{p}, -#{m}"}' 4

Slide 6

Slide 6 text

ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ parse.y +2256, -1687 compile.c +148, -78 test/coverage/test_coverage.rb +100, -55 node.h +34, -38 node.c +38, -18 ext/coverage/coverage.c +20, -12 thread.c +8, -6 ext/objspace/objspace.c +1, -5 test/ruby/test_iseq.rb +3, -1 test/ripper/test_lexer.rb +1, -1 test/ruby/test_string.rb +1, -1 insns.def +0, -2 ext/objspace/object_tracing.c +1, -1 doc/extension.ja.rdoc +1, -1 doc/extension.rdoc +1, -1 iseq.c +0, -1 5

Slide 7

Slide 7 text

ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ • (ओʹ)CoverageͷͨΊʹɺparserΛ͍͍ͬͯͨ͡ 6

Slide 8

Slide 8 text

Contents • Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ Coverage.so • Branch Coverage ͷ࢓૊Έ • NODE ͱ column ͷ͸ͳ͠ • lexer ͱ parser • column ৘ใͷऔಘͱอ؅ • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ • ͓·͚ 7

Slide 9

Slide 9 text

஫ҙ఺ • ϋογϡλά => #mf_ruby (શ෦খจࣈ) • લ൒ => Ruby 2.5Λ࢖͏͏͑Ͱ໾ʹཱͭ • ޙ൒ => Ruby 2.5Λ͍͡Δ͏͑Ͱ໾ʹཱͭ 8

Slide 10

Slide 10 text

Coverage ͷ͸ͳ͠ 9

Slide 11

Slide 11 text

Coverage ͷ͸ͳ͠ • Coverage ͱ͸ • Ruby 2.5ʹ͓͚Δ Coverage.so • Branch Coverage ͷ࢓૊Έ 10

Slide 12

Slide 12 text

Coverage ͱ͸ • ςετίʔυΛධՁ͢Δͱ͖ʹࢀߟʹͳΔ৘ใ • ৄ͘͠͸ԕ౻(@mametter)͞ΜͷൃදΛ͝ཡ͍ͩ͘͞ • "An introduction and future of Ruby coverage library" (http:// rubykaigi.org/2017/presentations/mametter.html) 11

Slide 13

Slide 13 text

Coverage ͷ࣮ྫ # Coverageऔಘର৅ͷίʔυ def target(bool) if bool 1 else 0 end end target(true) # CoverageΛऔಘ͢ΔͨΊͷίʔυ require "coverage" Coverage.start require "target_file" # ର৅ͷϑΝΠϧΛload Coverage.result 12

Slide 14

Slide 14 text

Coverage ͷ࣮ྫ # Coverage.result { "src/coverage_example.rb" => [1, 1, 1, nil, 0, nil, nil, nil, 1] } # ͦΕͬΆ͘mapping 1: def target(bool) 1: if bool 1: 1 nil: else 0: 0 nil: end nil: end nil: 1: target(true) 13

Slide 15

Slide 15 text

Ruby 2.5ʹ͓͚Δ Coverage.so • Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ͞ΕΔ Ruby 2.4 Ruby 2.5 Line Coverage o o Branch Coverage - o Method Coverage - o 14

Slide 16

Slide 16 text

Method Coverageͱ͸ • ϝιου͕Կճݺͼग़͞Ε͔͕ͨΘ͔ΔΑ͏ʹͳΔ def target1 end def target2 end target2 0: def target1 : end : 1: def target2 : end : : target2 15

Slide 17

Slide 17 text

Branch Coverageͱ͸ • Ͳͷ෼ذ͕Կճ࣮ߦ͞Ε͔ͨΘ͔ΔΑ͏ʹͳΔ a = 1 if a 1 else 2 end : a = 1 : : if a 1: 1 : else 0: 2 : end 16

Slide 18

Slide 18 text

Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(1) • ޙஔif 1: a = true 1: b = false 1: 1 if a 1: 2 if b # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 17

Slide 19

Slide 19 text

Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(2) • ࡾ߲ԋࢉ 1: a = true 1: b = false 1: a ? 1 : 2 # a͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 1: b ? 1 : 2 # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ 18

Slide 20

Slide 20 text

Branch Coverageͷྫ(if/unless) • ޙஔif/unless΍ࡾ߲ԋࢉࢠʹ΋ରԠ : a = 1 : 1/0: 10 if a == 1 0/1: 11 unless a == 1 0/1: (a == 2) ? :t : :f n/m -> n͕thenɺm͕elseͷճ਺ 19

Slide 21

Slide 21 text

Branch Coverageͷྫ(while/until) : x = 3 : while x > 0 3: x -= 1 : end : until x == 10 10: x += 1 : end : y = 3 3: y -= 1 while y > 0 10: y += 1 until y == 10 20

Slide 22

Slide 22 text

Branch Coverageͷྫ(case) : x = 0 : : case x : when 0 1: 0 : when 1 0: 1 : end : : case : when x == 0 1: 0 : when x == 1 0: 1 : end 21

Slide 23

Slide 23 text

Branch Coverageͷྫ(&.) : a = 10 : b = nil 1/0: a&.abs 0/1: b&.hoo n/m -> n͕then(ϝιουΛcall)ɺm͕elseͷճ਺ 22

Slide 24

Slide 24 text

Branch Coverage ͷ࢓૊Έ • ؅ཧςʔϒϧΛ༻ҙ • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ • ໋ྩ͕ɺ؅ཧςʔϒϧͷ֘౰ՕॴΛΠϯΫϦϝϯτ͢Δ a = 1 if a :A else :B end 23

Slide 25

Slide 25 text

Branch Coverage ͷ࢓૊Έ (؅ཧςʔϒϧ) • ؅ཧςʔϒϧΛ༻ҙ [structure, counters] structure = [ [ :if , 2, 0, 6, 3, # type, first_lineno, first_column, last_lineno, last_column :then, 3, 2, 3, 4, 0, # type, first_lineno, first_column, last_lineno, last_column, counter_index :else, 5, 2, 5, 4, 1 # type, first_lineno, first_column, last_lineno, last_column, counter_index ], ... ] counters = [0, 0] # [:then_counter, :else_counter] 24

Slide 26

Slide 26 text

Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ src = <<~SRC a = 1 if a :A else :B end SRC # Coverage OFF puts RubyVM::InstructionSequence.compile(src).disasm require "coverage" ENV["COVERAGE_EXPERIMENTAL_MODE"] = "true" Coverage.start(branches: true) # Coverage ON puts RubyVM::InstructionSequence.compile(src).disasm 25

Slide 27

Slide 27 text

Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕OFFͷͱ͖ == disasm: #@>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 10 0007 putobject :A ( 3)[Li] 0009 leave ( 5) 0010 putobject :B[Li] 0012 leave 26

Slide 28

Slide 28 text

Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #@>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 27

Slide 29

Slide 29 text

Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ) • Coverage͕ONͷͱ͖ == disasm: #@>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave 28

Slide 30

Slide 30 text

Branch Coverage ͷ࢓૊Έ (count up) trace2 131072, 1 # => counter_idx = 0 trace2 131072, 17 # => counter_idx = 1 • ୈ2Φϖϥϯυ(1΍17)͕counter_idxΛද͢ • counter_idx * 16 + COVERAGE_INDEX_BRANCHES • COVERAGE_INDEX_BRANCHES͸1 29

Slide 31

Slide 31 text

Branch Coverage ͷ࢓૊Έ (count up) == disasm: #@>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • a͕false΋͘͠͸nilͳΒ0013ʹͱͿ • ͦΕҎ֎ͷͱ͖͸Կ΋ͤͣɺ࣍ͷ໋ྩ΁͍͘ 30

Slide 32

Slide 32 text

Branch Coverage ͷ࢓૊Έ (count up) == disasm: #@>================================ local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1]) [ 1] a 0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li] 0001 setlocal_OP__WC__0 a 0003 getlocal_OP__WC__0 a ( 2)[Li] 0005 branchunless 13 0007 trace2 131072, 1 ( 3) 0010 putobject :A[Li] 0012 leave ( 5) 0013 trace2 131072, 17 0016 putobject :B[Li] 0018 leave • ࠓճͷέʔεͰ͸a = 1ͳͷͰɺtrace2 131072, 1͕࣮ߦ͞Ε Δ -> :then_counter͕૿Ճ͢Δ 31

Slide 33

Slide 33 text

Branch Coverage ͷ࢓૊Έ (݁Ռ) [structure, counters] structure = [[:if, 2, 0, 6, 3, :then, 3, 2, 3, 4, 0, :else, 5, 2, 5, 4, 1], ...] counters = [1, 0] # [:then_counter, :else_counter] => # ͦΕͬΆ͘mapping : a = 1 : if a 1: :A : else 0: :B : end 32

Slide 34

Slide 34 text

column৘ใ͕΄͍͠έʔε • औಘͨ͠Coverage݁ՌΛՄࢹԽ͢Δͱ͖ • :Aͱ͔:BͷҐஔ͕Θ͔Δͱศར a, b = 1, 1 (a == b) ? :A : :B 33

Slide 35

Slide 35 text

column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1, 3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2] => { # 2ߦ໨ [:then, 1, 2] => 0, # 2ߦ໨ [:else, 2, 2] => 1 # 2ߦ໨ }, [:if, 3, 2] => { # 2ߦ໨ [:then, 4, 2] => 1, # 2ߦ໨ [:else, 5, 2] => 0 # 2ߦ໨ } } 34

Slide 36

Slide 36 text

column৘ใ͕΄͍͠έʔε • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖ a, b, c, d = 1, 1, 3, 4 (a == b) ? ((c == d) ? :A : :B) : :C { [:if, 0, 2, 12, 2, 30] => { # [2.12-2.30] [:then, 1, 2, 23, 2, 25] => 0, # [2.23-2.25] [:else, 2, 2, 28, 2, 30] => 1 # [2.28-2.30] }, [:if, 3, 2, 0, 2, 36] => { # [2.0 -2.36] [:then, 4, 2, 12, 2, 30] => 1, # [2.12-2.30] [:else, 5, 2, 34, 2, 36] => 0 # [2.34-2.36] } } 35

Slide 37

Slide 37 text

NODE ͱ column ͷ͸ͳ͠ 36

Slide 38

Slide 38 text

NODE ͱ column ͷ͸ͳ͠ • column ͱ͸ͳʹ͔ • column ৘ใͷऔಘͱอ؅ • lexer ͱ parser • ޻෉͕ඞཁͳ NODE • test ʹ͍ͭͯ 37

Slide 39

Slide 39 text

column ͱ͸ͳʹ͔ • ࣜ΍จͷҐஔ৘ใ͸ྫ͑͹ҎԼͷ2ͭͰදݱͰ͖Δ • ߦ൪߸ (lineno) • ߦ಄͔Βͷoffset (column) a = 1 if a # [3.0-7.3] 1 # [4.2-4.3] else 10 # [6.2-6.4] end 38

Slide 40

Slide 40 text

column ͱ͸ͳʹ͔ • ߦͷઌ಄͔Β / ϑΝΠϧͷઌ಄͔Β • ݱঢ়͸ߦͷઌ಄͔Β • 0-based / 1-based • ݴޠ΍ΤσΟλʹΑͬͯҟͳΔ • ݱঢ়͸ 0-based • byte਺ / จࣈ਺ • ݱঢ়͸byte਺ • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby-lang.org/projects/ruby-trunk/wiki/Node- position-memo 39

Slide 41

Slide 41 text

column ৘ใͷऔಘͱอ؅ • Ruby 2.4·Ͱ͸columnͷ৘ใΛอ؅͍ͯ͠ͳ͔ͬͨ(linenoͷΈ Λอ͍࣋ͯ͠Δ) • Ruby 2.5ͰରԠͨ͜͠ͱ • columnΛه࿥͢ΔྖҬΛ(ԕ౻͞Μ͕)֬อ(লུ) • ద੾ͳcolumn৘ใΛॻ͖ࠐΉ 40

Slide 42

Slide 42 text

column ৘ใͷऔಘͱอ؅ (Ruby 2.4) $ ruby --dump=p -e '1 + 2' # @ NODE_SCOPE (line: 1) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_CALL (line: 1) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 41

Slide 43

Slide 43 text

column ৘ใͷऔಘͱอ؅ (Ruby 2.5) $ ruby --dump=p -e '1 + 2' # @ NODE_SCOPE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_PRELUDE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # +- nd_head: # | (null node) # +- nd_body: # | @ NODE_OPCALL (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_mid: :+ # | +- nd_recv: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 1) # | | +- nd_lit: 1 # | +- nd_args: # | @ NODE_ARRAY (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5) # | +- nd_alen: 1 # | +- nd_head: # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 4, last_lineno: 1, last_column: 5) # | | +- nd_lit: 2 # | +- nd_next: # | (null node) # +- nd_compile_option: # +- coverage_enabled: false 42

Slide 44

Slide 44 text

lexer ͱ parser • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) • lexer (parse.y yylex) • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) • parser (parse.c yyparse) • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ 43

Slide 45

Slide 45 text

lexer ͱ parser # e.g. 1 + 2\n 44

Slide 46

Slide 46 text

lexer ͱ parser (ࣈ۟ղੳ) • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ) $ ruby --dump=y -e '1 + 2' tINTEGER : 1 '+' : '+' tINTEGER : 2 '\n' : '\n' 45

Slide 47

Slide 47 text

lexer ͱ parser (ߏจղੳ) • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ) $ ruby --dump=y -e '1 + 2' 46

Slide 48

Slide 48 text

lexer ͱ parser (ߏจղੳ) # Shifting token tINTEGER (1) tINTEGER simple_numeric numeric literal primary arg 47

Slide 49

Slide 49 text

lexer ͱ parser (ߏจղੳ) # Shifting token '+' arg '+' # Shifting token tINTEGER (2) arg '+' tINTEGER arg '+' simple_numeric arg '+' numeric arg '+' literal arg '+' primary arg '+' arg arg expr stmt top_stmt top_stmts 48

Slide 50

Slide 50 text

lexer ͱ parser (ߏจղੳ) # Shifting token '\n' top_stmts '\n' top_stmts term top_stmts terms top_stmts opt_terms top_compstmt program # ׬ྃ 49

Slide 51

Slide 51 text

lexer ͱ parser • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ $ ruby --dump=p -e '1 + 2' NODE_SCOPE NODE_PRELUDE NODE_OPCALL (:+) NODE_LIT (1) NODE_ARRAY NODE_LIT (2) 50

Slide 52

Slide 52 text

lexer ͱ parser • ৄ͘͠͸ parse.y ΛಡΜͰ͍ͩ͘͞ • https://github.com/ruby/ruby/blob/trunk/parse.y 51

Slide 53

Slide 53 text

column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • ͋ΔtokenΛೝࣝͨ͠ͱ͖ͷɺlexerͷoffset৘ใ • tokp - lex_pbeg͓Αͼlex_p - lex_pbeg • ࠓͷtokenΛ੾ΓऔΓऴΘͬͯɺ࣍ͷtokenΛ੾ΓऔΔલʹͲ͜ ͔ʹୀආ͢Δඞཁ͕͋Δ 52

Slide 54

Slide 54 text

column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) /* parse.y */ /* Structure of Lexer Buffer: lex_pbeg tokp lex_p lex_pend | | | | |-----------+--------------+------------| |<------------>| token */ 53

Slide 55

Slide 55 text

column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ ͨͱ͖) • 1 + 2Ͱ+ͷtokenΛೝࣝͨ͠ॠؒ 1 + 2 ^ ^^ ^ | || +- lex_pend | |+---- lex_p | +----- tokp +-------- lex_pbeg 54

Slide 56

Slide 56 text

columnͷऔಘํ๏ • lexer͔Βparserʹlocation৘ใΛ౉͢ • parser͔ΒNODEʹlocation৘ใΛ౉͢ 55

Slide 57

Slide 57 text

columnͷऔಘํ๏ • Ґஔ৘ใ͸YYLTYPEͱ͍͏ߏ଄ମͰ؅ཧ͍ͯ͠Δ • ։࢝(first)ͱऴྃ(last)ͷ৘ใΛอ༗͍ͯ͠Δ #define YYLTYPE rb_code_range_t typedef struct rb_code_location_struct { int lineno; int column; } rb_code_location_t; typedef struct rb_code_range_struct { rb_code_location_t first_loc; rb_code_location_t last_loc; } rb_code_range_t; 56

Slide 58

Slide 58 text

columnͷऔಘํ๏ (lexer -> parser) • yylexͷͳ͔ͰҐஔ৘ใΛ౎౓ୀආ͢Δ( YYLTYPE *yylloc ) • Ҿ਺ʹYYLTYPE *yylloc͕௥Ճ͞Εͨ • RUBY_SET_YYLLOCΛݺͼग़ͯ͠yyllocΛઃఆ͢Δ 57

Slide 59

Slide 59 text

columnͷऔಘํ๏ (lexer -> parser) -static int -yylex(YYSTYPE *lval, struct parser_params *parser) +static enum yytokentype +yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *parser) { - int t; + enum yytokentype t; parser->lval = lval; lval->val = Qundef; t = parser_yylex(parser); if (has_delayed_token()) dispatch_delayed_token(t); else if (t != 0) dispatch_scan_event(t); + RUBY_SET_YYLLOC(*yylloc); + return t; } parser_yylex͕࣮ࡍʹtokenΛೝࣝ͢ΔͨΊͷؔ਺ 58

Slide 60

Slide 60 text

columnͷऔಘํ๏ (lexer -> parser) #define RUBY_SET_YYLLOC(Current) \ rb_parser_set_location(parser, &(Current)) void rb_parser_set_location(struct parser_params *parser, YYLTYPE *yylloc) { yylloc->first_loc.lineno = ruby_sourceline; yylloc->first_loc.column = (int)(parser->tokp - lex_pbeg); yylloc->last_loc.lineno = ruby_sourceline; yylloc->last_loc.column = (int)(lex_p - lex_pbeg); } 59

Slide 61

Slide 61 text

columnͷऔಘํ๏ (parser -> NODE) • ֤ΞΫγϣϯͰ͸Bisonͷ@nΛར༻ͯ͠tokenͷҐஔ৘ใΛऔಘ ͢Δ (YYLTYPE *yylsp) • ओʹ࢖͏ͷ͸@$ (YYLTYPE yyloc) 60

Slide 62

Slide 62 text

columnͷऔಘํ๏ (parser -> NODE) |---| @$ (1.0-1.5) 1 + 2 ^ ^ ^ | | +- @3 (1.4-1.5) | +--- @2 (1.2-1.3) +----- @1 (1.0-1.1) | arg '+' arg { - $$ = call_bin_op($1, '+', $3); + $$ = call_bin_op($1, '+', $3, &@$); } 61

Slide 63

Slide 63 text

columnͷऔಘํ๏ (parser -> NODE) static NODE * call_bin_op_gen(struct parser_params *parser, NODE *recv, ID id, NODE *arg1, const YYLTYPE *location) { NODE *expr; value_expr(recv); value_expr(arg1); expr = NEW_OPCALL(recv, id, new_list(arg1, location)); /* NODEͷੜ੒ */ fixpos(expr, recv); expr->nd_loc = *location; /* locationΛηοτ */ return expr; } 62

Slide 64

Slide 64 text

޻෉͕͍Βͳ͍ NODE (NODE_OPCALL) • NODE_OPCALLΛ࡞Δͱ͖ʹඞཁͳ৘ใ͕શ෦ͦΖ͍ͬͯΔ • NODE_LIT (1) • mid (:+) • NODE_ARRAY (2) 1 + 2 ----------- arg '+' arg 63

Slide 65

Slide 65 text

޻෉͕ඞཁͳ NODE (NODE_*ASGNܥ) • NODE_LASGN (ϩʔΧϧม਺୅ೖ)΍NODE_IASGN (Πϯελϯεม਺ ୅ೖ) @a = 1 64

Slide 66

Slide 66 text

• ୅ೖͷӈล͕ܾ·Δ·͑ʹNODE_*ASGNΛੜ੒͢ΔͷͰɺӈล͕ ܾ·Δͱ͖ʹߋ৽͕ඞཁ NODE_IASGN @a ---------- ------------- lhs : user_variable { $$ = assignable(var_field($1), 0, &@$); } => NODE_IASGN NODE_IASGN = 1 ---------- ------------------------ arg : lhs '=' arg_rhs { $$ = node_assign($1, $3, &@$); } 65

Slide 67

Slide 67 text

static NODE * node_assign_gen(struct parser_params *parser, NODE *lhs, NODE *rhs, const YYLTYPE *location) { if (!lhs) return 0; switch (nd_type(lhs)) { case NODE_GASGN: case NODE_IASGN: ... lhs->nd_value = rhs; lhs->nd_loc = *location; break; ... } return lhs; } 66

Slide 68

Slide 68 text

޻෉͕ඞཁͳ NODE (NODE_ITER) • NODE_ITER (ϒϩοΫ෇͖ϝιουݺͼग़͠) 3.times { foo } NODE_ITER NODE_CALL (:times) NODE_LIT (3) NODE_SCOPE NODE_VCALL (:foo) 67

Slide 69

Slide 69 text

޻෉͕ඞཁͳ NODE (NODE_ITER) • ϒϩοΫͷ෦෼ͰNODE_ITERΛੜ੒͢ΔͷͰɺ͋ͱ͔ΒNODEͷ ։࢝ҐஔΛमਖ਼͢Δඞཁ͕͋Δ 68

Slide 70

Slide 70 text

3.times { foo } NODE_CALL 3 . times ----------- -------------------------------------------------- method_call | primary_value call_op operation2 {} opt_paren_args { $$ = new_qcall($2, $1, $3, $5, &@$); nd_set_line($$, $4); } NODE_ITER { NODE_ITER } ----------- --------------------- brace_block : '{' {} brace_body '}' { $$ = $3; nd_set_line($$, $2); } NODE_ITER NODE_CALL NODE_ITER --------- ----------------------- primary | method_call brace_block { block_dup_check($1->nd_args, $2); $2->nd_iter = $1; $2->nd_loc = @$; $$ = $2; } 69

Slide 71

Slide 71 text

test ʹ͍ͭͯ • 2017/11/27ݱࡏɺtrunkʹtest͸commit͞Ε͍ͯͳ͍ 70

Slide 72

Slide 72 text

test ʹ͍ͭͯ • test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ • ະॳظԽͷNODEΛݕग़͢Δ • NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) 71

Slide 73

Slide 73 text

test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘ node = AST.parse("1 + 2") # => # node.children[1].children[0] # => # node.children[1].children[0].children # => [#, #] 72

Slide 74

Slide 74 text

ະॳظԽͷNODEΛݕग़͢Δ • linenoΛ0, columnΛ-1ͰॳظԽͯ͋͠Δ NODE_IF (line: 1, first_lineno: 0, first_column: -1, last_lineno: 0, last_column: -1) 73

Slide 75

Slide 75 text

ະॳظԽͷNODEΛݕग़͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $ ruby ast_validation.rb "Total 0 count errors." 74

Slide 76

Slide 76 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ 3.times { foo } NODE_ITER [1.0-1.15] -> [1.0-1.13]ΛؚΉ NODE_CALL (:times) [1.0-1.9] -> [1.0-1.1]ΛؚΉ NODE_LIT (3) [1.0-1.1] NODE_SCOPE [1.8-1.15] -> [1.10-1.13]ΛؚΉ NODE_VCALL (:foo) [1.10-1.13] 75

Slide 77

Slide 77 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ (ݱঢ়) • r60918 (2017/11/27) • "test/ruby/*.rb"Λର৅ -> ! $ ruby ast_validation.rb "test/ruby/beginmainend.rb has 1 errors" [["NODE_ARRAY", 1]] "test/ruby/sentence.rb has 1 errors" [["NODE_ARGS", 1]] ... "test/ruby/test_yield.rb has 1 errors" [["NODE_ARGS", 1]] "Total 552 count errors." 76

Slide 78

Slide 78 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • "NODE_ARRAY"΍"NODE_DSTR"ͳͲ͕໰୊ • "NODE_ARRAY" • ྫ͑͹ [1,2,3] 77

Slide 79

Slide 79 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ຤ඌ௥هͷͨͼʹશ෦ͷNODEͷlast locationΛߋ৽͢Δʁ 78

Slide 80

Slide 80 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| |__________| ----> |___________| 79

Slide 81

Slide 81 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| |__________| ----> |___________| 80

Slide 82

Slide 82 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ __________ ___________ |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(2)| # Ґஔ৘ใͷߋ৽͕ඞཁ |__________| ----> |___________| | _____v____ ___________ |NODE_ARRAY| |NODE_LIT(3)| |__________| ----> |___________| 81

Slide 83

Slide 83 text

NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ • ݱঢ়͸node.cಉ༷ʹ్தͷNODE_ARRAYΛඈ͹͍ͯ͠Δ node = AST.parse("[1, 2, 3]") # => # node.children[1].children[0] # => # node.children[1].children[0].children # => [ # #, # #, # #, nil # ] 82

Slide 84

Slide 84 text

NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) 1 + 2 NODE_OPCALL (:+) [1.0-1.5] -> (1)ͱ(2)͕ඃ͍ͬͯͳ͍ NODE_LIT (1) [1.0-1.1] -> (1) NODE_ARRAY [1.4-1.5] -> (2) NODE_LIT (2) [1.4-1.5] 83

Slide 85

Slide 85 text

NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) • "NODE_OP_ASGN*"ܥ͕໰୊ • ྫ͑͹ @a ||= 1 ("NODE_OP_ASGN_OR") 84

Slide 86

Slide 86 text

NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ) NODE_OP_ASGN_OR [1.0-1.8] -> (1)ͱ(2)͕ඃ͍ͬͯΔ NODE_IVAR (@a) [1.0-1.2] -> (1) NODE_IASGN (@a) [1.0-1.8] -> (2) NODE_LIT (1) [1.7-1.8] @a ||= 1 -- <-- NODE_IVAR -------- <-- NODE_IASGN 85

Slide 87

Slide 87 text

·ͱΊ 86

Slide 88

Slide 88 text

·ͱΊ • Ruby 2.5Ͱ͸Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ ͞ΕΔ • Ruby 2.5Ͱ͸NODE͕։࢝ɾऴྃҐஔΛ࣋ͭΑ͏ʹͳΔ • Ruby 2.5ͷϦϦʔε·Ͱʹ͍Ζ͍Ζमਖ਼Λ͍ΕΔ 87

Slide 89

Slide 89 text

ँࣙ • @mametter • @nobu • @ko1 • @shyouhei • @takeshinoda • @hkdnet • @HaiTo • @littlestarling 88

Slide 90

Slide 90 text

Ruby 2.5ϦϦʔε·ͰͷಓͷΓ • testΛcommit͢Δ • test͕௨ΔΑ͏ʹ͢Δ • Method CoverageͰ΋column͕ͱΕΔΑ͏ʹ͢Δ 89

Slide 91

Slide 91 text

ࠓޙͷ֦ு(ͷՄೳੑ) • &&΍||ͷΧόϨοδ΋ͱΕΔΑ͏ʹ͢Δ • NoMethodErrorͷ৘ใΛ૿΍͢ • power_assertͷ৘ใΛ૿΍͢ 90

Slide 92

Slide 92 text

ࢀߟ৘ใ • "Add branch coverage" https://bugs.ruby-lang.org/issues/13901 • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby- lang.org/projects/ruby-trunk/wiki/Node-position-memo • https://www.gnu.org/software/bison/manual/html_node/Token- Locations.html#Token-Locations • http://i.loveruby.net/ja/rhg/book/ ͷ"ୈ 2 ෦ʮߏจղੳʯ" 91

Slide 93

Slide 93 text

͓ΘΓ 92

Slide 94

Slide 94 text

͓·͚ 93

Slide 95

Slide 95 text

heredocʹ͍ͭͯ 94

Slide 96

Slide 96 text

heredocʹ͍ͭͯ • heredocศརͰ͢ΑͶ def upcase(s) s.upcase end upcase(< "ABCDE\n" 95

Slide 97

Slide 97 text

heredocʹ͍ͭͯ • heredocศར... def concat(s, t) s + t end concat(< "abcde\n12345\n" 96

Slide 98

Slide 98 text

heredocʹ͍ͭͯ • Ͳ͜·Ͱ͕heredocͷൣғʁ def concat(s, t) s + t end concat(<

Slide 99

Slide 99 text

ఏҊ • heredocͷNODEͷൣғΛ <

Slide 100

Slide 100 text

heredocͷ࢓૊Έ • ·ͣී௨ʹconcat(·ͰಡΉ concat( 99

Slide 101

Slide 101 text

heredocͷ࢓૊Έ • <

Slide 102

Slide 102 text

heredocͷ࢓૊Έ • STR1·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<

Slide 103

Slide 103 text

heredocͷ࢓૊Έ • ,ΛಡΉ concat(<

Slide 104

Slide 104 text

heredocͷ࢓૊Έ • <

Slide 105

Slide 105 text

heredocͷ࢓૊Έ • STR2·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ concat(<

Slide 106

Slide 106 text

heredocͷ࢓૊Έ • lex_strterm ͷঢ়ଶʹΑͬͯɺݱࡏheredocΛղੳ͍ͯ͠Δ͔൑ அ͍ͯ͠Δ static enum yytokentype parser_yylex(struct parser_params *parser) { ... if (lex_strterm) { if (lex_strterm->flags & STRTERM_HEREDOC) { return here_document(&lex_strterm->u.heredoc); } else { token_flush(parser); return parse_string(&lex_strterm->u.literal); } } ... switch (c = nextc()) { case '\0': /* NUL */ case '\004': /* ^D */ ... 105

Slide 107

Slide 107 text

heredocͷ࢓૊Έ • lex_strterm Λઃఆ͍ͯ͠Δίʔυ (parser_heredoc_identifier) • lex_goto_eol Ͱߦ຤·Ͱjump͍ͯ͠Δ • ͜ΕʹΑͬͯ࣍ͷtokenऔಘ࣌ʹ࣍ͷߦ͔Β͸͡ΊΔ͜ͱ͕Ͱ ͖Δ • lex_strterm ʹheredocΛൈ͚ͨ࣌ʹඞཁͳ৘ใΛୀආ͢Δ 106

Slide 108

Slide 108 text

heredocͷ࢓૊Έ static enum yytokentype parser_heredoc_identifier(struct parser_params *parser) { ... lex_goto_eol(parser); /* ߦ຤·Ͱjump͍ͯ͠Δ */ lex_strterm = (rb_strterm_t*)rb_imemo_new(imemo_parser_strterm, STR_NEW(tok(), toklen()), /* term */ lex_lastline, /* lastline */ len, /* lastidx */ ruby_sourceline); /* ෮ؼޙʹඞཁͳ৘ใΛୀආ */ lex_strterm->flags |= STRTERM_HEREDOC; token_flush(parser); heredoc_indent = indent; heredoc_line_indent = 0; return token; /* ͨͱ͑͹ɺtSTRING_BEG */ } 107

Slide 109

Slide 109 text

ఏҊ (࣮૷) • lex_strterm ʹୀආͯ͋͠Δ৘ใΛ࢖͑͹ɺ <

Slide 110

Slide 110 text

#define YYLLOC_DEFAULT(Current, Rhs, N) \ do \ if (N) \ - { \ - (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ - (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ - } \ + if (lex_strterm && (lex_strterm->flags & STRTERM_HEREDOC)) \ + { \ + rb_strterm_heredoc_t here = lex_strterm->u.heredoc; \ + (Current).first_loc.lineno = (int)here.sourceline; \ + (Current).first_loc.column = (int)(here.u3.lastidx - RSTRING_LEN(here.term));\ + (Current).last_loc.lineno = (int)here.sourceline; \ + (Current).last_loc.column = (int)(here.u3.lastidx);\ + } \ + else \ + { \ + (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \ + (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \ + } \ else \ RUBY_SET_YYLLOC(Current); \ while (0) 109

Slide 111

Slide 111 text

ͩΊͦ͏ͳέʔε (1) "#{<<~"A"}" #{:b if :a} f = 11 A 110

Slide 112

Slide 112 text

ͩΊͦ͏ͳέʔε (2) "#{<<~"A"}" #{<<~B} 12 13 B f = 11 A 111

Slide 113

Slide 113 text

parse.y͸ѱຐ৓ͳͷ͔ʁ • ҎԼͷ఺Λআ͚͹ɺͦ͜·Ͱѱຐ৓Ͱ͸ͳ͍ͷͰ͸? • Ұ෦ripperͱϑΝΠϧΛڞ༗͍ͯ͠Δ • ߦ਺͕ଟ͍ • parser_paramsʹ͍Ζ͍Ζͳϑϥά͕͋Δ • lex_stateʹ͍Ζ͍Ζͳϑϥά͕͋Δ • Rubyʹ͸heredoc͕͋Δ 112