Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Coverage.so, or RNode with column

yui-knk
November 29, 2017

Coverage.so, or RNode with column

yui-knk

November 29, 2017
Tweet

More Decks by yui-knk

Other Decks in Programming

Transcript


  1. ۚࢠ༤Ұ࿠ !ZVJLOL
    ϚωʔϑΥϫʔυ3VCZษڧձ
    ਐԽ͢Δ$PWFSBHFTPɺ
    ͋Δ͍͸/0%&ͱDPMVNOͷ͸ͳ͠

    View Slide

  2. raise MissingBeer unless
    1

    View Slide

  3. ࣗݾ঺հ
    • Ҏલ͸ॏ޻ۀͳϝʔΧʔͰܦཧΛ͍ͯͨ͠
    • ϚωʔϑΥʔϫʔυͰ͸ձܭνʔϜͰRailsΞϓϦΛॻ͍͍ͯΔ
    • Rubyίϛολʔ 2015/12~
    • Asakusa.rb ॴଐ
    2

    View Slide

  4. • Ruby 2.5ͷશମతͳ࿩͸͜ͷ͋ͱ͋Δ͸ͣ
    • ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱͷ࿩
    3

    View Slide

  5. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ
    $ git log --numstat --pretty="%H" --author="yui-knk" --since="2016-12-26" |\
    awk 'NF==3 {plus+=$1; minus+=$2; printf("%s %d %d\n", $3, $1, $2)}' |\
    ruby -e 'ary=[]; while gets(); ary << $_.chomp.split(" ");end; \
    ary.each_with_object({}) {|(n,p,m),h| h[n]||= [0,0];\
    h[n] = [h[n][0] + p.to_i, h[n][1] + m.to_i]}.sort_by\
    {|(k,(p,m))| -(p + m) }.each {|(k,(p,m))| puts "#{k} +#{p}, -#{m}"}'
    4

    View Slide

  6. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ
    parse.y +2256, -1687
    compile.c +148, -78
    test/coverage/test_coverage.rb +100, -55
    node.h +34, -38
    node.c +38, -18
    ext/coverage/coverage.c +20, -12
    thread.c +8, -6
    ext/objspace/objspace.c +1, -5
    test/ruby/test_iseq.rb +3, -1
    test/ripper/test_lexer.rb +1, -1
    test/ruby/test_string.rb +1, -1
    insns.def +0, -2
    ext/objspace/object_tracing.c +1, -1
    doc/extension.ja.rdoc +1, -1
    doc/extension.rdoc +1, -1
    iseq.c +0, -1
    5

    View Slide

  7. ࣗ෼͕ࠓ೥ͱΓ͘Μͩ͜ͱ
    • (ओʹ)CoverageͷͨΊʹɺparserΛ͍͍ͬͯͨ͡
    6

    View Slide

  8. Contents
    • Coverage ͷ͸ͳ͠
    • Coverage ͱ͸
    • Ruby 2.5ʹ͓͚Δ Coverage.so
    • Branch Coverage ͷ࢓૊Έ
    • NODE ͱ column ͷ͸ͳ͠
    • lexer ͱ parser
    • column ৘ใͷऔಘͱอ؅
    • ޻෉͕ඞཁͳ NODE
    • test ʹ͍ͭͯ
    • ͓·͚
    7

    View Slide

  9. ஫ҙ఺
    • ϋογϡλά => #mf_ruby (શ෦খจࣈ)
    • લ൒ => Ruby 2.5Λ࢖͏͏͑Ͱ໾ʹཱͭ
    • ޙ൒ => Ruby 2.5Λ͍͡Δ͏͑Ͱ໾ʹཱͭ
    8

    View Slide

  10. Coverage ͷ͸ͳ͠
    9

    View Slide

  11. Coverage ͷ͸ͳ͠
    • Coverage ͱ͸
    • Ruby 2.5ʹ͓͚Δ Coverage.so
    • Branch Coverage ͷ࢓૊Έ
    10

    View Slide

  12. Coverage ͱ͸
    • ςετίʔυΛධՁ͢Δͱ͖ʹࢀߟʹͳΔ৘ใ
    • ৄ͘͠͸ԕ౻(@mametter)͞ΜͷൃදΛ͝ཡ͍ͩ͘͞
    • "An introduction and future of Ruby coverage library" (http://
    rubykaigi.org/2017/presentations/mametter.html)
    11

    View Slide

  13. Coverage ͷ࣮ྫ
    # Coverageऔಘର৅ͷίʔυ
    def target(bool)
    if bool
    1
    else
    0
    end
    end
    target(true)
    # CoverageΛऔಘ͢ΔͨΊͷίʔυ
    require "coverage"
    Coverage.start
    require "target_file" # ର৅ͷϑΝΠϧΛload
    Coverage.result
    12

    View Slide

  14. Coverage ͷ࣮ྫ
    # Coverage.result
    {
    "src/coverage_example.rb" => [1, 1, 1, nil, 0, nil, nil, nil, 1]
    }
    # ͦΕͬΆ͘mapping
    1: def target(bool)
    1: if bool
    1: 1
    nil: else
    0: 0
    nil: end
    nil: end
    nil:
    1: target(true)
    13

    View Slide

  15. Ruby 2.5ʹ͓͚Δ Coverage.so
    • Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ͞ΕΔ
    Ruby 2.4 Ruby 2.5
    Line Coverage o o
    Branch Coverage - o
    Method Coverage - o
    14

    View Slide

  16. Method Coverageͱ͸
    • ϝιου͕Կճݺͼग़͞Ε͔͕ͨΘ͔ΔΑ͏ʹͳΔ
    def target1
    end
    def target2
    end
    target2
    0: def target1
    : end
    :
    1: def target2
    : end
    :
    : target2
    15

    View Slide

  17. Branch Coverageͱ͸
    • Ͳͷ෼ذ͕Կճ࣮ߦ͞Ε͔ͨΘ͔ΔΑ͏ʹͳΔ
    a = 1
    if a
    1
    else
    2
    end
    : a = 1
    :
    : if a
    1: 1
    : else
    0: 2
    : end
    16

    View Slide

  18. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(1)
    • ޙஔif
    1: a = true
    1: b = false
    1: 1 if a
    1: 2 if b # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ
    17

    View Slide

  19. Line CoverageͩͱऔΕͳ͔ͬͨ৘ใ(2)
    • ࡾ߲ԋࢉ
    1: a = true
    1: b = false
    1: a ? 1 : 2 # a͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ
    1: b ? 1 : 2 # b͕ධՁ͞ΕΔ͔Βpassͨ͜͠ͱʹͳΔ
    18

    View Slide

  20. Branch Coverageͷྫ(if/unless)
    • ޙஔif/unless΍ࡾ߲ԋࢉࢠʹ΋ରԠ
    : a = 1
    :
    1/0: 10 if a == 1
    0/1: 11 unless a == 1
    0/1: (a == 2) ? :t : :f
    n/m -> n͕thenɺm͕elseͷճ਺
    19

    View Slide

  21. Branch Coverageͷྫ(while/until)
    : x = 3
    : while x > 0
    3: x -= 1
    : end
    : until x == 10
    10: x += 1
    : end
    : y = 3
    3: y -= 1 while y > 0
    10: y += 1 until y == 10
    20

    View Slide

  22. Branch Coverageͷྫ(case)
    : x = 0
    :
    : case x
    : when 0
    1: 0
    : when 1
    0: 1
    : end
    :
    : case
    : when x == 0
    1: 0
    : when x == 1
    0: 1
    : end
    21

    View Slide

  23. Branch Coverageͷྫ(&.)
    : a = 10
    : b = nil
    1/0: a&.abs
    0/1: b&.hoo
    n/m -> n͕then(ϝιουΛcall)ɺm͕elseͷճ਺
    22

    View Slide

  24. Branch Coverage ͷ࢓૊Έ
    • ؅ཧςʔϒϧΛ༻ҙ
    • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ
    • ໋ྩ͕ɺ؅ཧςʔϒϧͷ֘౰ՕॴΛΠϯΫϦϝϯτ͢Δ
    a = 1
    if a
    :A
    else
    :B
    end
    23

    View Slide

  25. Branch Coverage ͷ࢓૊Έ (؅ཧςʔϒϧ)
    • ؅ཧςʔϒϧΛ༻ҙ
    [structure, counters]
    structure = [
    [
    :if , 2, 0, 6, 3, # type, first_lineno, first_column, last_lineno, last_column
    :then, 3, 2, 3, 4, 0, # type, first_lineno, first_column, last_lineno, last_column, counter_index
    :else, 5, 2, 5, 4, 1 # type, first_lineno, first_column, last_lineno, last_column, counter_index
    ], ...
    ]
    counters = [0, 0] # [:then_counter, :else_counter]
    24

    View Slide

  26. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ)
    • byte codeʹಛघͳ໋ྩΛຒΊࠐΉ
    src = <<~SRC
    a = 1
    if a
    :A
    else
    :B
    end
    SRC
    # Coverage OFF
    puts RubyVM::InstructionSequence.compile(src).disasm
    require "coverage"
    ENV["COVERAGE_EXPERIMENTAL_MODE"] = "true"
    Coverage.start(branches: true)
    # Coverage ON
    puts RubyVM::InstructionSequence.compile(src).disasm
    25

    View Slide

  27. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ)
    • Coverage͕OFFͷͱ͖
    == disasm: #@>================================
    local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
    [ 1] a
    0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li]
    0001 setlocal_OP__WC__0 a
    0003 getlocal_OP__WC__0 a ( 2)[Li]
    0005 branchunless 10
    0007 putobject :A ( 3)[Li]
    0009 leave ( 5)
    0010 putobject :B[Li]
    0012 leave
    26

    View Slide

  28. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ)
    • Coverage͕ONͷͱ͖
    == disasm: #@>================================
    local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
    [ 1] a
    0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li]
    0001 setlocal_OP__WC__0 a
    0003 getlocal_OP__WC__0 a ( 2)[Li]
    0005 branchunless 13
    0007 trace2 131072, 1 ( 3)
    0010 putobject :A[Li]
    0012 leave ( 5)
    0013 trace2 131072, 17
    0016 putobject :B[Li]
    0018 leave
    27

    View Slide

  29. Branch Coverage ͷ࢓૊Έ (໋ྩຒΊࠐΈ)
    • Coverage͕ONͷͱ͖
    == disasm: #@>================================
    local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
    [ 1] a
    0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li]
    0001 setlocal_OP__WC__0 a
    0003 getlocal_OP__WC__0 a ( 2)[Li]
    0005 branchunless 13
    0007 trace2 131072, 1 ( 3)
    0010 putobject :A[Li]
    0012 leave ( 5)
    0013 trace2 131072, 17
    0016 putobject :B[Li]
    0018 leave
    28

    View Slide

  30. Branch Coverage ͷ࢓૊Έ (count up)
    trace2 131072, 1 # => counter_idx = 0
    trace2 131072, 17 # => counter_idx = 1
    • ୈ2Φϖϥϯυ(1΍17)͕counter_idxΛද͢
    • counter_idx * 16 + COVERAGE_INDEX_BRANCHES
    • COVERAGE_INDEX_BRANCHES͸1
    29

    View Slide

  31. Branch Coverage ͷ࢓૊Έ (count up)
    == disasm: #@>================================
    local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
    [ 1] a
    0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li]
    0001 setlocal_OP__WC__0 a
    0003 getlocal_OP__WC__0 a ( 2)[Li]
    0005 branchunless 13
    0007 trace2 131072, 1 ( 3)
    0010 putobject :A[Li]
    0012 leave ( 5)
    0013 trace2 131072, 17
    0016 putobject :B[Li]
    0018 leave
    • a͕false΋͘͠͸nilͳΒ0013ʹͱͿ
    • ͦΕҎ֎ͷͱ͖͸Կ΋ͤͣɺ࣍ͷ໋ྩ΁͍͘
    30

    View Slide

  32. Branch Coverage ͷ࢓૊Έ (count up)
    == disasm: #@>================================
    local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
    [ 1] a
    0000 putobject_OP_INT2FIX_O_1_C_ ( 1)[Li]
    0001 setlocal_OP__WC__0 a
    0003 getlocal_OP__WC__0 a ( 2)[Li]
    0005 branchunless 13
    0007 trace2 131072, 1 ( 3)
    0010 putobject :A[Li]
    0012 leave ( 5)
    0013 trace2 131072, 17
    0016 putobject :B[Li]
    0018 leave
    • ࠓճͷέʔεͰ͸a = 1ͳͷͰɺtrace2 131072, 1͕࣮ߦ͞Ε
    Δ -> :then_counter͕૿Ճ͢Δ
    31

    View Slide

  33. Branch Coverage ͷ࢓૊Έ (݁Ռ)
    [structure, counters]
    structure = [[:if, 2, 0, 6, 3, :then, 3, 2, 3, 4, 0, :else, 5, 2, 5, 4, 1], ...]
    counters = [1, 0] # [:then_counter, :else_counter]
    =>
    # ͦΕͬΆ͘mapping
    : a = 1
    : if a
    1: :A
    : else
    0: :B
    : end
    32

    View Slide

  34. column৘ใ͕΄͍͠έʔε
    • औಘͨ͠Coverage݁ՌΛՄࢹԽ͢Δͱ͖
    • :Aͱ͔:BͷҐஔ͕Θ͔Δͱศར
    a, b = 1, 1
    (a == b) ? :A : :B
    33

    View Slide

  35. column৘ใ͕΄͍͠έʔε
    • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖
    a, b, c, d = 1, 1, 3, 4
    (a == b) ? ((c == d) ? :A : :B) : :C
    {
    [:if, 0, 2] => { # 2ߦ໨
    [:then, 1, 2] => 0, # 2ߦ໨
    [:else, 2, 2] => 1 # 2ߦ໨
    },
    [:if, 3, 2] => { # 2ߦ໨
    [:then, 4, 2] => 1, # 2ߦ໨
    [:else, 5, 2] => 0 # 2ߦ໨
    }
    }
    34

    View Slide

  36. column৘ใ͕΄͍͠έʔε
    • 1ߦʹෳ਺ͷ෼ذ͕ଘࡏ͢Δͱ͖
    a, b, c, d = 1, 1, 3, 4
    (a == b) ? ((c == d) ? :A : :B) : :C
    {
    [:if, 0, 2, 12, 2, 30] => { # [2.12-2.30]
    [:then, 1, 2, 23, 2, 25] => 0, # [2.23-2.25]
    [:else, 2, 2, 28, 2, 30] => 1 # [2.28-2.30]
    },
    [:if, 3, 2, 0, 2, 36] => { # [2.0 -2.36]
    [:then, 4, 2, 12, 2, 30] => 1, # [2.12-2.30]
    [:else, 5, 2, 34, 2, 36] => 0 # [2.34-2.36]
    }
    }
    35

    View Slide

  37. NODE ͱ column ͷ͸ͳ͠
    36

    View Slide

  38. NODE ͱ column ͷ͸ͳ͠
    • column ͱ͸ͳʹ͔
    • column ৘ใͷऔಘͱอ؅
    • lexer ͱ parser
    • ޻෉͕ඞཁͳ NODE
    • test ʹ͍ͭͯ
    37

    View Slide

  39. column ͱ͸ͳʹ͔
    • ࣜ΍จͷҐஔ৘ใ͸ྫ͑͹ҎԼͷ2ͭͰදݱͰ͖Δ
    • ߦ൪߸ (lineno)
    • ߦ಄͔Βͷoffset (column)
    a = 1
    if a # [3.0-7.3]
    1 # [4.2-4.3]
    else
    10 # [6.2-6.4]
    end
    38

    View Slide

  40. column ͱ͸ͳʹ͔
    • ߦͷઌ಄͔Β / ϑΝΠϧͷઌ಄͔Β
    • ݱঢ়͸ߦͷઌ಄͔Β
    • 0-based / 1-based
    • ݴޠ΍ΤσΟλʹΑͬͯҟͳΔ
    • ݱঢ়͸ 0-based
    • byte਺ / จࣈ਺
    • ݱঢ়͸byte਺
    • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby-lang.org/projects/ruby-trunk/wiki/Node-
    position-memo
    39

    View Slide

  41. column ৘ใͷऔಘͱอ؅
    • Ruby 2.4·Ͱ͸columnͷ৘ใΛอ؅͍ͯ͠ͳ͔ͬͨ(linenoͷΈ
    Λอ͍࣋ͯ͠Δ)
    • Ruby 2.5ͰରԠͨ͜͠ͱ
    • columnΛه࿥͢ΔྖҬΛ(ԕ౻͞Μ͕)֬อ(লུ)
    • ద੾ͳcolumn৘ใΛॻ͖ࠐΉ
    40

    View Slide

  42. column ৘ใͷऔಘͱอ؅ (Ruby 2.4)
    $ ruby --dump=p -e '1 + 2'
    # @ NODE_SCOPE (line: 1)
    # +- nd_tbl: (empty)
    # +- nd_args:
    # | (null node)
    # +- nd_body:
    # @ NODE_PRELUDE (line: 1)
    # +- nd_head:
    # | (null node)
    # +- nd_body:
    # | @ NODE_CALL (line: 1)
    # | +- nd_mid: :+
    # | +- nd_recv:
    # | | @ NODE_LIT (line: 1)
    # | | +- nd_lit: 1
    # | +- nd_args:
    # | @ NODE_ARRAY (line: 1)
    # | +- nd_alen: 1
    # | +- nd_head:
    # | | @ NODE_LIT (line: 1)
    # | | +- nd_lit: 2
    # | +- nd_next:
    # | (null node)
    # +- nd_compile_option:
    # +- coverage_enabled: false
    41

    View Slide

  43. column ৘ใͷऔಘͱอ؅ (Ruby 2.5)
    $ ruby --dump=p -e '1 + 2'
    # @ NODE_SCOPE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5)
    # +- nd_tbl: (empty)
    # +- nd_args:
    # | (null node)
    # +- nd_body:
    # @ NODE_PRELUDE (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5)
    # +- nd_head:
    # | (null node)
    # +- nd_body:
    # | @ NODE_OPCALL (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5)
    # | +- nd_mid: :+
    # | +- nd_recv:
    # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 1)
    # | | +- nd_lit: 1
    # | +- nd_args:
    # | @ NODE_ARRAY (line: 1, first_lineno: 1, first_column: 0, last_lineno: 1, last_column: 5)
    # | +- nd_alen: 1
    # | +- nd_head:
    # | | @ NODE_LIT (line: 1, first_lineno: 1, first_column: 4, last_lineno: 1, last_column: 5)
    # | | +- nd_lit: 2
    # | +- nd_next:
    # | (null node)
    # +- nd_compile_option:
    # +- coverage_enabled: false
    42

    View Slide

  44. lexer ͱ parser
    • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ)
    • lexer (parse.y yylex)
    • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ)
    • parser (parse.c yyparse)
    • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ
    43

    View Slide

  45. lexer ͱ parser
    # e.g.
    1 + 2\n
    44

    View Slide

  46. lexer ͱ parser (ࣈ۟ղੳ)
    • ೖྗ͞ΕͨจࣈྻΛద੾ʹτʔΫϯʹ෼ׂ͢Δ(ࣈ۟ղੳ)
    $ ruby --dump=y -e '1 + 2'
    tINTEGER : 1
    '+' : '+'
    tINTEGER : 2
    '\n' : '\n'
    45

    View Slide

  47. lexer ͱ parser (ߏจղੳ)
    • ෼ׂ͞ΕͨτʔΫϯ͔ΒߏจΛ૊Έ্͛Δ(ߏจղੳ)
    $ ruby --dump=y -e '1 + 2'
    46

    View Slide

  48. lexer ͱ parser (ߏจղੳ)
    # Shifting token tINTEGER (1)
    tINTEGER
    simple_numeric
    numeric
    literal
    primary
    arg
    47

    View Slide

  49. lexer ͱ parser (ߏจղੳ)
    # Shifting token '+'
    arg '+'
    # Shifting token tINTEGER (2)
    arg '+' tINTEGER
    arg '+' simple_numeric
    arg '+' numeric
    arg '+' literal
    arg '+' primary
    arg '+' arg
    arg
    expr
    stmt
    top_stmt
    top_stmts
    48

    View Slide

  50. lexer ͱ parser (ߏจղੳ)
    # Shifting token '\n'
    top_stmts '\n'
    top_stmts term
    top_stmts terms
    top_stmts opt_terms
    top_compstmt
    program # ׬ྃ
    49

    View Slide

  51. lexer ͱ parser
    • ૊Έ্͛Δͱ͖ʹɺAST(ந৅ߏจ໦)Λͭ͘Δ
    $ ruby --dump=p -e '1 + 2'
    NODE_SCOPE
    NODE_PRELUDE
    NODE_OPCALL (:+)
    NODE_LIT (1)
    NODE_ARRAY
    NODE_LIT (2)
    50

    View Slide

  52. lexer ͱ parser
    • ৄ͘͠͸ parse.y ΛಡΜͰ͍ͩ͘͞
    • https://github.com/ruby/ruby/blob/trunk/parse.y
    51

    View Slide

  53. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ
    ͨͱ͖)
    • ͋ΔtokenΛೝࣝͨ͠ͱ͖ͷɺlexerͷoffset৘ใ
    • tokp - lex_pbeg͓Αͼlex_p - lex_pbeg
    • ࠓͷtokenΛ੾ΓऔΓऴΘͬͯɺ࣍ͷtokenΛ੾ΓऔΔલʹͲ͜
    ͔ʹୀආ͢Δඞཁ͕͋Δ
    52

    View Slide

  54. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ
    ͨͱ͖)
    /* parse.y */
    /*
    Structure of Lexer Buffer:
    lex_pbeg tokp lex_p lex_pend
    | | | |
    |-----------+--------------+------------|
    |<------------>|
    token
    */
    53

    View Slide

  55. column ͱ͸ͳʹ͔ (ϓϩάϥϜͷ಺෦͔ΒΈ
    ͨͱ͖)
    • 1 + 2Ͱ+ͷtokenΛೝࣝͨ͠ॠؒ
    1 + 2
    ^ ^^ ^
    | || +- lex_pend
    | |+---- lex_p
    | +----- tokp
    +-------- lex_pbeg
    54

    View Slide

  56. columnͷऔಘํ๏
    • lexer͔Βparserʹlocation৘ใΛ౉͢
    • parser͔ΒNODEʹlocation৘ใΛ౉͢
    55

    View Slide

  57. columnͷऔಘํ๏
    • Ґஔ৘ใ͸YYLTYPEͱ͍͏ߏ଄ମͰ؅ཧ͍ͯ͠Δ
    • ։࢝(first)ͱऴྃ(last)ͷ৘ใΛอ༗͍ͯ͠Δ
    #define YYLTYPE rb_code_range_t
    typedef struct rb_code_location_struct {
    int lineno;
    int column;
    } rb_code_location_t;
    typedef struct rb_code_range_struct {
    rb_code_location_t first_loc;
    rb_code_location_t last_loc;
    } rb_code_range_t;
    56

    View Slide

  58. columnͷऔಘํ๏ (lexer -> parser)

    yylexͷͳ͔ͰҐஔ৘ใΛ౎౓ୀආ͢Δ(
    YYLTYPE *yylloc
    )
    • Ҿ਺ʹYYLTYPE *yylloc͕௥Ճ͞Εͨ

    RUBY_SET_YYLLOCΛݺͼग़ͯ͠yyllocΛઃఆ͢Δ
    57

    View Slide

  59. columnͷऔಘํ๏ (lexer -> parser)
    -static int
    -yylex(YYSTYPE *lval, struct parser_params *parser)
    +static enum yytokentype
    +yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *parser)
    {
    - int t;
    + enum yytokentype t;
    parser->lval = lval;
    lval->val = Qundef;
    t = parser_yylex(parser);
    if (has_delayed_token())
    dispatch_delayed_token(t);
    else if (t != 0)
    dispatch_scan_event(t);
    + RUBY_SET_YYLLOC(*yylloc);
    +
    return t;
    }
    parser_yylex͕࣮ࡍʹtokenΛೝࣝ͢ΔͨΊͷؔ਺
    58

    View Slide

  60. columnͷऔಘํ๏ (lexer -> parser)
    #define RUBY_SET_YYLLOC(Current) \
    rb_parser_set_location(parser, &(Current))
    void
    rb_parser_set_location(struct parser_params *parser, YYLTYPE *yylloc)
    {
    yylloc->first_loc.lineno = ruby_sourceline;
    yylloc->first_loc.column = (int)(parser->tokp - lex_pbeg);
    yylloc->last_loc.lineno = ruby_sourceline;
    yylloc->last_loc.column = (int)(lex_p - lex_pbeg);
    }
    59

    View Slide

  61. columnͷऔಘํ๏ (parser -> NODE)
    • ֤ΞΫγϣϯͰ͸Bisonͷ@nΛར༻ͯ͠tokenͷҐஔ৘ใΛऔಘ
    ͢Δ (YYLTYPE *yylsp)
    • ओʹ࢖͏ͷ͸@$ (YYLTYPE yyloc)
    60

    View Slide

  62. columnͷऔಘํ๏ (parser -> NODE)
    |---| @$ (1.0-1.5)
    1 + 2
    ^ ^ ^
    | | +- @3 (1.4-1.5)
    | +--- @2 (1.2-1.3)
    +----- @1 (1.0-1.1)
    | arg '+' arg
    {
    - $$ = call_bin_op($1, '+', $3);
    + $$ = call_bin_op($1, '+', $3, &@$);
    }
    61

    View Slide

  63. columnͷऔಘํ๏ (parser -> NODE)
    static NODE *
    call_bin_op_gen(struct parser_params *parser, NODE *recv, ID id, NODE *arg1, const YYLTYPE *location)
    {
    NODE *expr;
    value_expr(recv);
    value_expr(arg1);
    expr = NEW_OPCALL(recv, id, new_list(arg1, location)); /* NODEͷੜ੒ */
    fixpos(expr, recv);
    expr->nd_loc = *location; /* locationΛηοτ */
    return expr;
    }
    62

    View Slide

  64. ޻෉͕͍Βͳ͍ NODE (NODE_OPCALL)
    • NODE_OPCALLΛ࡞Δͱ͖ʹඞཁͳ৘ใ͕શ෦ͦΖ͍ͬͯΔ
    • NODE_LIT (1)
    • mid (:+)
    • NODE_ARRAY (2)
    1 + 2
    -----------
    arg '+' arg
    63

    View Slide

  65. ޻෉͕ඞཁͳ NODE (NODE_*ASGNܥ)

    NODE_LASGN
    (ϩʔΧϧม਺୅ೖ)΍NODE_IASGN
    (Πϯελϯεม਺
    ୅ೖ)
    @a = 1
    64

    View Slide

  66. • ୅ೖͷӈล͕ܾ·Δ·͑ʹNODE_*ASGNΛੜ੒͢ΔͷͰɺӈล͕
    ܾ·Δͱ͖ʹߋ৽͕ඞཁ
    NODE_IASGN @a
    ---------- -------------
    lhs : user_variable
    {
    $$ = assignable(var_field($1), 0, &@$);
    }
    =>
    NODE_IASGN NODE_IASGN = 1
    ---------- ------------------------
    arg : lhs '=' arg_rhs
    {
    $$ = node_assign($1, $3, &@$);
    }
    65

    View Slide

  67. static NODE *
    node_assign_gen(struct parser_params *parser, NODE *lhs, NODE *rhs, const YYLTYPE *location)
    {
    if (!lhs) return 0;
    switch (nd_type(lhs)) {
    case NODE_GASGN:
    case NODE_IASGN:
    ...
    lhs->nd_value = rhs;
    lhs->nd_loc = *location;
    break;
    ...
    }
    return lhs;
    }
    66

    View Slide

  68. ޻෉͕ඞཁͳ NODE (NODE_ITER)

    NODE_ITER
    (ϒϩοΫ෇͖ϝιουݺͼग़͠)
    3.times { foo }
    NODE_ITER
    NODE_CALL (:times)
    NODE_LIT (3)
    NODE_SCOPE
    NODE_VCALL (:foo)
    67

    View Slide

  69. ޻෉͕ඞཁͳ NODE (NODE_ITER)
    • ϒϩοΫͷ෦෼ͰNODE_ITERΛੜ੒͢ΔͷͰɺ͋ͱ͔ΒNODEͷ
    ։࢝ҐஔΛमਖ਼͢Δඞཁ͕͋Δ
    68

    View Slide

  70. 3.times { foo }
    NODE_CALL 3 . times
    ----------- --------------------------------------------------
    method_call | primary_value call_op operation2 {} opt_paren_args
    {
    $$ = new_qcall($2, $1, $3, $5, &@$);
    nd_set_line($$, $4);
    }
    NODE_ITER { NODE_ITER }
    ----------- ---------------------
    brace_block : '{' {} brace_body '}'
    {
    $$ = $3;
    nd_set_line($$, $2);
    }
    NODE_ITER NODE_CALL NODE_ITER
    --------- -----------------------
    primary | method_call brace_block
    {
    block_dup_check($1->nd_args, $2);
    $2->nd_iter = $1;
    $2->nd_loc = @$;
    $$ = $2;
    }
    69

    View Slide

  71. test ʹ͍ͭͯ
    • 2017/11/27ݱࡏɺtrunkʹtest͸commit͞Ε͍ͯͳ͍
    70

    View Slide

  72. test ʹ͍ͭͯ
    • test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘
    • ະॳظԽͷNODEΛݕग़͢Δ
    • NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    • NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ)
    71

    View Slide

  73. test͚࣌ͩ༗ޮͳ֦ுϥΠϒϥϦΛॻ͘
    node = AST.parse("1 + 2")
    # => #
    node.children[1].children[0]
    # => #
    node.children[1].children[0].children
    # => [#, #]
    72

    View Slide

  74. ະॳظԽͷNODEΛݕग़͢Δ
    • linenoΛ0, columnΛ-1ͰॳظԽͯ͋͠Δ
    NODE_IF (line: 1, first_lineno: 0, first_column: -1, last_lineno: 0, last_column: -1)
    73

    View Slide

  75. ະॳظԽͷNODEΛݕग़͢Δ (ݱঢ়)
    • r60918 (2017/11/27)
    • "test/ruby/*.rb"Λର৅ ->
    !
    $ ruby ast_validation.rb
    "Total 0 count errors."
    74

    View Slide

  76. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    3.times { foo }
    NODE_ITER [1.0-1.15] -> [1.0-1.13]ΛؚΉ
    NODE_CALL (:times) [1.0-1.9] -> [1.0-1.1]ΛؚΉ
    NODE_LIT (3) [1.0-1.1]
    NODE_SCOPE [1.8-1.15] -> [1.10-1.13]ΛؚΉ
    NODE_VCALL (:foo) [1.10-1.13]
    75

    View Slide

  77. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ (ݱঢ়)
    • r60918 (2017/11/27)
    • "test/ruby/*.rb"Λର৅ ->
    !
    $ ruby ast_validation.rb
    "test/ruby/beginmainend.rb has 1 errors"
    [["NODE_ARRAY", 1]]
    "test/ruby/sentence.rb has 1 errors"
    [["NODE_ARGS", 1]]
    ...
    "test/ruby/test_yield.rb has 1 errors"
    [["NODE_ARGS", 1]]
    "Total 552 count errors."
    76

    View Slide

  78. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    • "NODE_ARRAY"΍"NODE_DSTR"ͳͲ͕໰୊
    • "NODE_ARRAY"
    • ྫ͑͹ [1,2,3]
    77

    View Slide

  79. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    • ຤ඌ௥هͷͨͼʹશ෦ͷNODEͷlast locationΛߋ৽͢Δʁ
    78

    View Slide

  80. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    __________ ___________
    |NODE_ARRAY| |NODE_LIT(1)|
    |__________| ----> |___________|
    79

    View Slide

  81. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    __________ ___________
    |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ
    |__________| ----> |___________|
    |
    _____v____ ___________
    |NODE_ARRAY| |NODE_LIT(2)|
    |__________| ----> |___________|
    80

    View Slide

  82. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    __________ ___________
    |NODE_ARRAY| |NODE_LIT(1)| # Ґஔ৘ใͷߋ৽͕ඞཁ
    |__________| ----> |___________|
    |
    _____v____ ___________
    |NODE_ARRAY| |NODE_LIT(2)| # Ґஔ৘ใͷߋ৽͕ඞཁ
    |__________| ----> |___________|
    |
    _____v____ ___________
    |NODE_ARRAY| |NODE_LIT(3)|
    |__________| ----> |___________|
    81

    View Slide

  83. NODEͷ਌ࢠؔ܎ΛνΣοΫ͢Δ
    • ݱঢ়͸node.cಉ༷ʹ్தͷNODE_ARRAYΛඈ͹͍ͯ͠Δ
    node = AST.parse("[1, 2, 3]")
    # => #
    node.children[1].children[0]
    # => #
    node.children[1].children[0].children
    # => [
    # #,
    # #,
    # #, nil
    # ]
    82

    View Slide

  84. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ)
    1 + 2
    NODE_OPCALL (:+) [1.0-1.5] -> (1)ͱ(2)͕ඃ͍ͬͯͳ͍
    NODE_LIT (1) [1.0-1.1] -> (1)
    NODE_ARRAY [1.4-1.5] -> (2)
    NODE_LIT (2) [1.4-1.5]
    83

    View Slide

  85. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ)
    • "NODE_OP_ASGN*"ܥ͕໰୊
    • ྫ͑͹ @a ||= 1 ("NODE_OP_ASGN_OR")
    84

    View Slide

  86. NODEͷܑఋؔ܎ΛνΣοΫ͢Δ(༧ఆ)
    NODE_OP_ASGN_OR [1.0-1.8] -> (1)ͱ(2)͕ඃ͍ͬͯΔ
    NODE_IVAR (@a) [1.0-1.2] -> (1)
    NODE_IASGN (@a) [1.0-1.8] -> (2)
    NODE_LIT (1) [1.7-1.8]
    @a ||= 1
    -- <-- NODE_IVAR
    -------- <-- NODE_IASGN
    85

    View Slide

  87. ·ͱΊ
    86

    View Slide

  88. ·ͱΊ
    • Ruby 2.5Ͱ͸Branch CoverageͱMethod Coverage͕৽ͨʹಋೖ
    ͞ΕΔ
    • Ruby 2.5Ͱ͸NODE͕։࢝ɾऴྃҐஔΛ࣋ͭΑ͏ʹͳΔ
    • Ruby 2.5ͷϦϦʔε·Ͱʹ͍Ζ͍Ζमਖ਼Λ͍ΕΔ
    87

    View Slide

  89. ँࣙ
    • @mametter
    • @nobu
    • @ko1
    • @shyouhei
    • @takeshinoda
    • @hkdnet
    • @HaiTo
    • @littlestarling
    88

    View Slide

  90. Ruby 2.5ϦϦʔε·ͰͷಓͷΓ
    • testΛcommit͢Δ
    • test͕௨ΔΑ͏ʹ͢Δ
    • Method CoverageͰ΋column͕ͱΕΔΑ͏ʹ͢Δ
    89

    View Slide

  91. ࠓޙͷ֦ு(ͷՄೳੑ)
    • &&΍||ͷΧόϨοδ΋ͱΕΔΑ͏ʹ͢Δ
    • NoMethodErrorͷ৘ใΛ૿΍͢
    • power_assertͷ৘ใΛ૿΍͢
    90

    View Slide

  92. ࢀߟ৘ใ
    • "Add branch coverage" https://bugs.ruby-lang.org/issues/13901
    • "ߏจ໦ʹৄࡉͳҐஔ৘ใΛ΋ͨͤΔܭը" https://bugs.ruby-
    lang.org/projects/ruby-trunk/wiki/Node-position-memo
    • https://www.gnu.org/software/bison/manual/html_node/Token-
    Locations.html#Token-Locations
    • http://i.loveruby.net/ja/rhg/book/ ͷ"ୈ 2 ෦ʮߏจղੳʯ"
    91

    View Slide

  93. ͓ΘΓ
    92

    View Slide

  94. ͓·͚
    93

    View Slide

  95. heredocʹ͍ͭͯ
    94

    View Slide

  96. heredocʹ͍ͭͯ
    • heredocศརͰ͢ΑͶ
    def upcase(s)
    s.upcase
    end
    upcase(<abcde
    STR1
    # => "ABCDE\n"
    95

    View Slide

  97. heredocʹ͍ͭͯ
    • heredocศར...
    def concat(s, t)
    s + t
    end
    concat(<abcde
    STR1
    12345
    STR2
    # => "abcde\n12345\n"
    96

    View Slide

  98. heredocʹ͍ͭͯ
    • Ͳ͜·Ͱ͕heredocͷൣғʁ
    def concat(s, t)
    s + t
    end
    concat(<abcde # ͔͜͜Β
    fghij # ͜͜·Ͱͱ
    STR1
    12345 # ͔͜͜Β
    67890 # ͜͜·Ͱʁ
    STR2
    97

    View Slide

  99. ఏҊ
    • heredocͷNODEͷൣғΛ <• test͕༰қʹͳΔ
    • ࣮ࡍͷจࣈྻͷൣғ͸୅දҐஔ(ະ࣮૷)ͳͲͰද͢
    concat(<abcde
    fghij
    STR1
    12345
    67890
    STR2
    98

    View Slide

  100. heredocͷ࢓૊Έ
    • ·ͣී௨ʹconcat(·ͰಡΉ
    concat(
    99

    View Slide

  101. heredocͷ࢓૊Έ
    • <concat(<100

    View Slide

  102. heredocͷ࢓૊Έ
    • STR1·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ
    concat(<abcde
    fghij
    STR1
    101

    View Slide

  103. heredocͷ࢓૊Έ
    • ,ΛಡΉ
    concat(<abcde
    fghij
    STR1
    102

    View Slide

  104. heredocͷ࢓૊Έ
    • <concat(<abcde
    fghij
    STR1
    103

    View Slide

  105. heredocͷ࢓૊Έ
    • STR2·ͰಡΈࠐΜͰɺheredocϞʔυΛൈ͚Δ
    concat(<abcde
    fghij
    STR1
    12345
    67890
    STR2
    104

    View Slide

  106. heredocͷ࢓૊Έ
    • lex_strterm ͷঢ়ଶʹΑͬͯɺݱࡏheredocΛղੳ͍ͯ͠Δ͔൑
    அ͍ͯ͠Δ
    static enum yytokentype
    parser_yylex(struct parser_params *parser)
    {
    ...
    if (lex_strterm) {
    if (lex_strterm->flags & STRTERM_HEREDOC) {
    return here_document(&lex_strterm->u.heredoc);
    }
    else {
    token_flush(parser);
    return parse_string(&lex_strterm->u.literal);
    }
    }
    ...
    switch (c = nextc()) {
    case '\0': /* NUL */
    case '\004': /* ^D */
    ...
    105

    View Slide

  107. heredocͷ࢓૊Έ
    • lex_strterm Λઃఆ͍ͯ͠Δίʔυ
    (parser_heredoc_identifier)
    • lex_goto_eol Ͱߦ຤·Ͱjump͍ͯ͠Δ
    • ͜ΕʹΑͬͯ࣍ͷtokenऔಘ࣌ʹ࣍ͷߦ͔Β͸͡ΊΔ͜ͱ͕Ͱ
    ͖Δ
    • lex_strterm ʹheredocΛൈ͚ͨ࣌ʹඞཁͳ৘ใΛୀආ͢Δ
    106

    View Slide

  108. heredocͷ࢓૊Έ
    static enum yytokentype
    parser_heredoc_identifier(struct parser_params *parser)
    {
    ...
    lex_goto_eol(parser); /* ߦ຤·Ͱjump͍ͯ͠Δ */
    lex_strterm = (rb_strterm_t*)rb_imemo_new(imemo_parser_strterm,
    STR_NEW(tok(), toklen()), /* term */
    lex_lastline, /* lastline */
    len, /* lastidx */
    ruby_sourceline); /* ෮ؼޙʹඞཁͳ৘ใΛୀආ */
    lex_strterm->flags |= STRTERM_HEREDOC;
    token_flush(parser);
    heredoc_indent = indent;
    heredoc_line_indent = 0;
    return token; /* ͨͱ͑͹ɺtSTRING_BEG */
    }
    107

    View Slide

  109. ఏҊ (࣮૷)
    • lex_strterm ʹୀආͯ͋͠Δ৘ใΛ࢖͑͹ɺ <͔Δ͸ͣ
    108

    View Slide

  110. #define YYLLOC_DEFAULT(Current, Rhs, N) \
    do \
    if (N) \
    - { \
    - (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \
    - (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \
    - } \
    + if (lex_strterm && (lex_strterm->flags & STRTERM_HEREDOC)) \
    + { \
    + rb_strterm_heredoc_t here = lex_strterm->u.heredoc; \
    + (Current).first_loc.lineno = (int)here.sourceline; \
    + (Current).first_loc.column = (int)(here.u3.lastidx - RSTRING_LEN(here.term));\
    + (Current).last_loc.lineno = (int)here.sourceline; \
    + (Current).last_loc.column = (int)(here.u3.lastidx);\
    + } \
    + else \
    + { \
    + (Current).first_loc = YYRHSLOC(Rhs, 1).first_loc; \
    + (Current).last_loc = YYRHSLOC(Rhs, N).last_loc; \
    + } \
    else \
    RUBY_SET_YYLLOC(Current); \
    while (0)
    109

    View Slide

  111. ͩΊͦ͏ͳέʔε (1)
    "#{<<~"A"}"
    #{:b if :a}
    f = 11
    A
    110

    View Slide

  112. ͩΊͦ͏ͳέʔε (2)
    "#{<<~"A"}"
    #{<<~B}
    12
    13
    B
    f = 11
    A
    111

    View Slide

  113. parse.y͸ѱຐ৓ͳͷ͔ʁ
    • ҎԼͷ఺Λআ͚͹ɺͦ͜·Ͱѱຐ৓Ͱ͸ͳ͍ͷͰ͸?
    • Ұ෦ripperͱϑΝΠϧΛڞ༗͍ͯ͠Δ
    • ߦ਺͕ଟ͍
    • parser_paramsʹ͍Ζ͍Ζͳϑϥά͕͋Δ
    • lex_stateʹ͍Ζ͍Ζͳϑϥά͕͋Δ
    • Rubyʹ͸heredoc͕͋Δ
    112

    View Slide