What is expected?

B3ba3ccedfbf4d605f00bafd1a732529?s=47 yui-knk
December 14, 2019

What is expected?

B3ba3ccedfbf4d605f00bafd1a732529?s=128

yui-knk

December 14, 2019
Tweet

Transcript

  1. What is expected? ۚࢠ༤Ұ࿠(@yui-knk) 2019/12/14 ฏ੒Rubyձٞ01 Keynote

  2. ࣗݾ঺հ • ۚࢠ ༤Ұ࿠ • Arm Treasure Data ॴଐ •

    Audience νʔϜ (RailsΞϓϦΛॻ͍͍ͯ·͢) • CRuby Committer 2015/12~ • GitHub (yui-knk)
  3. 8FBSFIJSJOH https://www.treasuredata.com/company/careers/jobs/?team=Engineering

  4. 8FBSFIJSJOH https://www.treasuredata.com/company/careers/jobs/?team=Engineering

  5. What is expected?

  6. What is expected? • ^^^ͷ෦෼ʹ͸ͲͷΑ͏ͳ” จࣈྻ”͕͸͍Δ͔ʁ def xxx ^^^

  7. What is expected? • “m1”ͱ͍͏໊લͷϝιου Λఆٛ͢Δ def m1 end

  8. What is expected? • “def”͕͘Δ͜ͱ΋͋Δ def def

  9. What is expected? • “def”͕͘Δ͜ͱ΋͋Δ def def def def a;

    end
  10. What is expected? $ ruby -wc -e 'def def a;

    end' Syntax OK
  11. What is expected? • ໊લ͕”def"Ͱ1Ҿ਺"a"ΛͱΔϝιουΛఆ͍ٛͯ͠Δ • 2ͭ໨ͷ`def`͸`reswords`Ͱɺ͜Ε͸`fname`Ͱ`k_def fname ...` def

    def a ^^^ ^ | |-- args +------ method name end
  12. What is expected? • ͋Δ࣌఺Ͱ࣍ʹདྷΔ͜ͱ͕Ͱ͖Δ”จࣈྻ” (expected tokens)ΛͲ͏΍ͬͯ஌Δ͜ͱ͕Ͱ͖Δͷ͔ʁ

  13. How Ruby script is processed

  14. How Ruby script is processed 4UFQ *OQVU 0VUQVU %FCVH 4PVSDF

    5PLFOJ[BUJPO 3VCZTDSJQU 5PLFOT EVNQZ QBSTFZ 1BSTJOH 5PLFOT "45 EVNQQ QBSTFZ $PNQJMF "45 #ZUFDPEF EVNQJ DPNQJMFD Parsing ___ \ Ruby script -> Tokens -> AST -> Byte code (insns / ISeq) __/ __/ Tokenization Compile
  15. Ruby script 1 + 2

  16. Tokenization Parsing ___ \ Ruby script -> Tokens -> AST

    -> Byte code (insns / ISeq) __/ __/ Tokenization Compile
  17. Tokenization • Token͸ҎԼͷ2ͭͷ৘ใΛ΋ͭ • a token type (tINTEGER) • a

    semantic value (1) 1 + 2 ^ ^ ^^ | | |+--- '\n' / "end-of-input" | | +---- tINTEGER (2) | +------ '+' +-------- tINTEGER (1)
  18. Tokenization $ ruby --dump=y -e '1 + 2' | grep

    Shifting Shifting token "integer literal" (1.0-1.1: 1) Shifting token '+' (1.2-1.3: ) Shifting token "integer literal" (1.4-1.5: 2) Shifting token '\n' (1.5-1.5: ) Shifting token "end-of-input" (1.5-1.5: ) On Ruby 2.7.0preview3
  19. Parsing Parsing ___ \ Ruby script -> Tokens -> AST

    -> Byte code (insns / ISeq) __/ __/ Tokenization Compile
  20. Parsing • Tokenྻ͕Rubyͷߏจʹ߹க͢Δ͔ΛνΣοΫ͢Δ • AST (Abstract Syntax Tree)Λߏங͢Δ

  21. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y
  22. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y W W W Rules
  23. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y
  24. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y 1 2.1 3r 4i
  25. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y 1
  26. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y 1
  27. Parsing numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ;

    simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; %% program : { } top_compstmt parse.y Goal
  28. Parsing $ ruby --dump=y -e '1 + 2' # Shifting

    token “integer literal” (1) “integer literal” simple_numeric numeric literal primary arg
  29. Parsing # Shifting token '+' arg '+' # Shifting token

    “integer literal” (2) arg '+' “integer literal” arg '+' simple_numeric arg '+' numeric arg '+' literal arg '+' primary arg '+' arg arg expr stmt top_stmt top_stmts
  30. Parsing # Shifting token '\n' top_stmts '\n' top_stmts term top_stmts

    terms top_stmts opt_terms top_compstmt program # Completed # Shifting token "end-of-input"
  31. Build AST $ ruby --dump=p -e '1 + 2' NODE_SCOPE

    NODE_OPCALL (:+) NODE_LIT (1) NODE_LIST NODE_LIT (2) NODE_SCOPE NODE_OPCALL (:+) NODE_LIT (1) NODE_LIST NODE_LIT (2)
  32. Compile Parsing ___ \ Ruby script -> Tokens -> AST

    -> Byte code (insns / ISeq) __/ __/ Tokenization Compile
  33. Compile • ίϯύΠϧΛ͢Δ • See “compile.c” $ ruby --dump=i -e

    '1 + 2' == disasm: #<ISeq:<main>@-e:1 (1,0)-(1,5)> (catch: FALSE) 0000 putobject_INT2FIX_1_ ( 1)[Li] 0001 putobject 2 0003 opt_plus <calldata!mid:+, argc:1, ARGS_SIMPLE> 0005 leave
  34. 1BSTFSJTUPEBZ`TUPQJD • expected tokens͸ߏจղੳ্ͷ໰୊ Parsing ___ \ Ruby script ->

    Tokens -> AST -> Byte code (insns / ISeq) __/ __/ Tokenization Compile
  35. What is parser?

  36. • ೖྗจࣈྻ͕จ๏͔ΒͲͷΑ͏ʹੜ੒Ͱ͖Δ͔Λܭࢉ͢ Δ͜ͱ • RubyͰ͸GNU BisonΛ͔ͭͬͯLALR(1)ύʔβΛੜ੒͢Δ • GNU Bison͸parse.yΛೖྗͱͯ͠ύʔβͷCίʔυΛ ग़ྗ͢Δ

    • ߏจղੳදΛͭͬͯ͘ϓογϡμ΢ϯΦʔτϚτϯΛར༻ ͯ͠ߏจΛղੳ͢Δ What is parser?
  37. • ੜ੒نଇ (Production Rule): • ྫ) simple_numeric: tINTEGER • ͖͞΄ͲͷྫͰ͸programΛى఺ʹੜ੒نଇΛద༻ͯ͠

    ͍ͬͯ࡞Δ͜ͱͷͰ͖Δ΋ͷ͕ɺͦͷจ๏ʹΑͬͯఆ ٛ͞ΕΔݴޠͱͳΔ simple_numeric : tINTEGER | tFLOAT | tRATIONAL | tIMAGINARY ; 8IBUJTHSBNNBS
  38. • ඇऴ୺ه߸ (Nonterminal): RuleͷࠨลʹදΕΔه߸ • ྫ) program, numeric, simple_numeric •

    ӈลʹ΋දΕΔ͜ͱ͕͋Δ • ऴ୺ه߸ (Terminal): Ruleͷӈลʹ͔͠දΕͳ͍ه߸ • ྫ) tINTEGER, tUMINUS_NUM, tLOWEST numeric : simple_numeric | tUMINUS_NUM simple_numeric %prec tLOWEST ; 8IBUJTHSBNNBS ऴ୺ه߸ ඇऴ୺ه߸
  39. How parser works? L : L ';' E /* Rule

    1 */ | E /* Rule 2 */ ; E : E ',' P /* Rule 3 */ | P /* Rule 4 */ ; P : 'a' /* Rule 5 */ | '(' M ')' /* Rule 6 */ ; M : /* nothing */ /* Rule 7 */ | L /* Rule 8 */ ; a;()
  40. • Shift: ࣍ͷtokenΛstackʹϓογϡ͢Δ • Reduce: ͋ΔϧʔϧΛ࢖ͬͯӈ͔ΒnݸͷtokenΛஔ͖׵͑Δ How parser works? "a"

    ";" "(" ")" $end # Shift "a" ";" "(" ")" $end # Reduce by rule 5 P ";" "(" ")" $end # Reduce by rule 4 E ";" "(" ")" $end # Reduce by rule 2 L ";" "(" ")" $end # Shift L ";" "(" ")" $end # Shift L ";" "(" ")" $end # Reduce by rule 7 L ";" "(" M ")" $end # Shift L ";" "(" M ")" $end # Reduce by rule 6 L ";" P $end # Reduce by rule 4 L ";" E $end # Reduce by rule 1 L $end # Shift L $end # accept Rule 6. P: (M) ʹΑΔreduce stack ೖྗτʔΫϯ
  41. How parser works? • Shift/ReduceΛͦΕͧΕ͍ͭߦ͏΂͖͔ʁ • ߏจղੳදʹج͍ͮͯ൑அ͢Δ

  42. -"-3  UBCMF • sN: Shiftͯ͠NΛstackʹpush • rN: ϧʔϧNͰReduce͢Δ •

    acc: डཧ • ۭന: ߏจΤϥʔ • GOTO: ඇऴ୺ه߸༻ɻNΛstack ʹpush https://www.cs.uic.edu/~spopuri/cparser.html#dragonbook-tables
  43. )PXUPDSFBUFUIFUBCMF • Ruleͷӈลʹ ‘.’ ΛՃ͑ͨ΋ͷΛߟ͑Δ 1~4 • ‘.’ ͸ruleͷͲ͜·ͰಡΜ͔ͩΛ͋ΒΘ͢ •

    2Ͱ͋Ε͹࣍ʹ ‘;’ ͕͘Δ͜ͱΛظ଴͍ͯ͠ΔͷͰɺ ‘;’ ͳ Βshift͢Δ • 4ͳΒreduceͯ͠Lʹ͢Δ • ͜ΕΒΛLR(0)߲ͱ͍ͬͨΓ͢Δ 0 L: L ';' E 1 L: . L ';' E 2 L: L . ';' E 3 L: L ';' . E 4 L: L ';' E .
  44. )PXUPDSFBUFUIFUBCMF • LALR(1)ͷ৔߹LR(1)߲Λ࢖͏ • LR(0)߲ʹ͞Βʹtoken 1ͭΛઌಡΈͨ͠΋ͷ L: E . /

    [$end, ';', ')']
  45. )PXUPDSFBUFUIFUBCMF • 1ͷ৔߹ɺ࣍ʹظ଴͞ΕΔͷ͸E • E͸ผͷruleͰreduce͢Δ͜ͱʹΑͬͯൃੜ͢Δ͔΋͠Ε ͳ͍(3, 4) • P͸ผͷruleͰ… (5,

    6) • ‘a’ ΍ ‘(’ ͸ऴ୺ه߸ͳͷͰผͷrule͔Β͸ൃੜ͠ͳ͍ • ߲͸͍͔ͭ͘ͷάϧʔϓʹ෼͚ΒΕΔ 1 L: L ';' . E 3 E: . E ',' P 4 | . P 5 P: . 'a' 6 | . '(' M ')'
  46. )PXUPDSFBUFUIFUBCMF

  47. )PXUPDSFBUFUIFUBCMF ॳظঢ়ଶ डཧ

  48. )PXUPDSFBUFUIFUBCMF PΛshift

  49. )PXUPDSFBUFUIFUBCMF reduce

  50. )PXUPDSFBUFUIFUBCMF shift΋͘͠͸reduce

  51. )PXUPHFUFYQFDUFE UPLFOT • state 3ʹ͍ΔͳΒɺ‘;’ ‘,’ ‘)’ ‘$’͕expected tokensʹͳΔ https://www.cs.uic.edu/~spopuri/cparser.html#dragonbook-tables

  52. Bison’s compressed table

  53. 5BCMFJTTQBSTF • actionςʔϒϧ͸ 37/78 = 47% ͔͠ຒ·ͬͯͳ͍ • GOTOςʔϒϧ͸ 10/52

    = 19% ͔͠ຒ·ͬͯͳ͍ • Ruby 2.7.3 pre3Ͱ͸ 1234 state, 411 symbols https://www.cs.uic.edu/~spopuri/cparser.html#dragonbook-tables
  54. Compress table (1) • Default Reductions, Default GOTOsʹΑΔѹॖ • actionςʔϒϧ͸ԣํ޲ɺ

    GOTOςʔϒϧ͸ॎํ޲ʹѹ ॖ • state 5 -> r3 • PͷGOTO -> 9 https://www.cs.uic.edu/~spopuri/cparser.html#dragonbook-tables
  55. Compress table (2) • Default ReductionsΛಋೖͯ͠΋ɺ·ͩ sparse • double displacementʹΑΔѹॖ

    https://www.cs.uic.edu/~spopuri/cparser.html#table-compression ΛҰ෦मਖ਼
  56. Compress table (2) • ࣮ࡍͷ஋Λ΋ͭyytable, guard tableͰ͋Δyycheck, offset Λ؅ཧ͢Δyypactͷ3ͭͰදݱ͢Δ 0:

    [ , , , , , 1, 2, ] 2: [ , , , , , 1, 2, ] 3: [ 8, , , 9, , , , ] 4: [ , , , , 10, , , ] 6: [ , , , 9, , , , ] 7: [ , , , , , , , 11] 9: [ , , , , , 1, 2, ] 10: [ , , , , , 1, 2, ] 12: [ , , , , 10, , , ] yycheck [0, 5, 6, 3, 7, 4, 3, 2, 9, -1, -1, -1, 10] yytable [8, 1, 2, 9, 11, 10, 9, 6, 12, 0, 0, 0, 13] yypact [-4, -5, -4, 0, 1, -5, 3, -3, -5, -4, -4, -5, 1, -5]
  57. • state 0ͷέʔεΛߟ͑ͯΈΔ Compress table (2) 0: [ , ,

    , , , 1, 2, ] 2: [ , , , , , 1, 2, ] 3: [ 8, , , 9, , , , ] 4: [ , , , , 10, , , ] 6: [ , , , 9, , , , ] 7: [ , , , , , , , 11] 9: [ , , , , , 1, 2, ] 10: [ , , , , , 1, 2, ] 12: [ , , , , 10, , , ] yycheck [0, 5, 6, 3, 7, 4, 3, 2, 9, -1, -1, -1, 10] yytable [8, 1, 2, 9, 11, 10, 9, 6, 12, 0, 0, 0, 13] yypact [-4, -5, -4, 0, 1, -5, 3, -3, -5, -4, -4, -5, 1, -5] yypact[0] = -4
  58. Compress table (2) index = 5 Ͱ͸yycheckͷ஋ͱҰக͢Δ 0: [ ,

    , , , , 1, 2, ] 2: [ , , , , , 1, 2, ] 3: [ 8, , , 9, , , , ] 4: [ , , , , 10, , , ] 6: [ , , , 9, , , , ] 7: [ , , , , , , , 11] 9: [ , , , , , 1, 2, ] 10: [ , , , , , 1, 2, ] 12: [ , , , , 10, , , ] yycheck [0, 5, 6, 3, 7, 4, 3, 2, 9, -1, -1, -1, 10] yytable [8, 1, 2, 9, 11, 10, 9, 6, 12, 0, 0, 0, 13] yypact [-4, -5, -4, 0, 1, -5, 3, -3, -5, -4, -4, -5, 1, -5]
  59. Compress table (2) index = 5 ͷ஋͸ 1 (= yytable[1])

    0: [ , , , , , 1, 2, ] 2: [ , , , , , 1, 2, ] 3: [ 8, , , 9, , , , ] 4: [ , , , , 10, , , ] 6: [ , , , 9, , , , ] 7: [ , , , , , , , 11] 9: [ , , , , , 1, 2, ] 10: [ , , , , , 1, 2, ] 12: [ , , , , 10, , , ] yycheck [0, 5, 6, 3, 7, 4, 3, 2, 9, -1, -1, -1, 10] yytable [8, 1, 2, 9, 11, 10, 9, 6, 12, 0, 0, 0, 13] yypact [-4, -5, -4, 0, 1, -5, 3, -3, -5, -4, -4, -5, 1, -5]
  60. DPNQSFTTFEUBCMFͷಛ௃ • double displacement • ෮ݩՄೳͳͷͰ໰୊ͳ͍ • default reductions •

    errorൃੜ͕஗Ԇͯ͠expected tokens͕มΘͬͯ͠·͏ • expected tokensͷܭࢉ͕ͦͷ࣌఺ͷstate stackʹґଘ ͢Δ
  61. &SSPSൃੜͷ஗Ԇ • ҎԼͷྫ͸ຊ౰ʹ”when”͚ͩͳͷ͔ʁ $ ruby -e 'case a;' -e:1: syntax

    error, unexpected end-of-input, expecting `when'
  62. &SSPSൃੜͷ஗Ԇ • “in”΋ॻ͘͜ͱ͕Ͱ͖Δ $ ruby -wce 'case a; in b;

    end' -e:1: warning: Pattern matching is experimental, and the behavior may change in future versions of Ruby! Syntax OK
  63. &SSPSൃੜͷ஗Ԇ case a; ^ State 375 737 opt_terms: terms .

    ["`when'", "`in'"] ... $default reduce using rule 737 (opt_terms) State 587 ... $default reduce using rule 330 (@18) State 717 331 primary: k_case expr_value opt_terms @18 . case_body k_end 367 k_when: . "`when'" 464 case_body: . k_when case_args then compstmt cases "`when'" shift, and go to state 719 k_when go to state 720 case_body go to state 841 default reduceʹΑͬͯstate 717·ͰҠಈ͢Δ
  64. 4UBDL΁ͷґଘ

  65. 4UBDL΁ͷґଘ • ‘;’ ‘,’ ‘$end’͸ຊདྷͳΒerror • ຊདྷerror͔Ͳ͏͔Λ֬ೝ͢Δ ͨΊʹ࣮ࡍʹreduce͢Δඞཁ ͕͋Δ •

    ͜ͷܭࢉͷ݁Ռ͸ͦͷ࣌఺ͷ stackʹґଘ͢Δ • ςετͷෳࡶ͕͞૿͢ https://www.cs.uic.edu/~spopuri/cparser.html#modified-tables ΛҰ෦मਖ਼
  66. 8PSLBSPVOE • lr.default-reduction Λ accepting ʹ͢Δ͜ͱͰdefault reduce͞ΕΔ෦෼ΛݮΒ͢͜ͱ͕Ͱ͖Δ • https://www.gnu.org/software/bison/manual/html_node/ Default-Reductions.html

  67. 1JUGBMMPGUIFXPSLBSPVOE • ruby͕buildͰ͖ͳ͘ͳΔ • ຊདྷ͸tLABELͱ͍͏τʔΫϯʹͳΔ $ ./miniruby -v <internal:gc>:33: syntax

    error, unexpected ':' def self.start full_mark: true, immediate_mark: true, ...
  68. • tLABELΛు͖ग़͢ʹ͸EXPR_LABEL|EXPR_ENDFN͕ඞ ཁ 1JUGBMMPGUIFXPSLBSPVOE #define IS_LABEL_POSSIBLE() (\ (IS_lex_state(EXPR_LABEL|EXPR_ENDFN) && !cmd_state)

    || \ IS_ARG()) static enum yytokentype parse_ident(struct parser_params *p, int c, int cmd_state) { ... if (IS_LABEL_POSSIBLE()) { if (IS_LABEL_SUFFIX(0)) { SET_LEX_STATE(EXPR_ARG|EXPR_LABELED); nextc(p); set_yylval_name(TOK_INTERN()); return tLABEL; } }
  69. • fnameͷ࣍ͷΞΫγϣϯͰlex_stateΛηοτ͍ͯ͠Δ 1JUGBMMPGUIFXPSLBSPVOE | k_def singleton dot_or_colon {SET_LEX_STATE(EXPR_FNAME);} fname {

    $<num>4 = p->in_def; p->in_def = 1; SET_LEX_STATE(EXPR_ENDFN|EXPR_LABEL); /* force for args */ local_push(p, 0); $<id>$ = p->cur_arg; p->cur_arg = 0; } f_arglist
  70. #FGPSF • ͱΓ͋͑ͣΞΫγϣϯ(@26)Λ࣮ߦ͢Δ State 858 346 @26: . %empty 347

    primary: k_def singleton dot_or_colon @25 fname . @26 f_arglist bodystmt k_end $default reduce using rule 346 (@26) @26 go to state 961
  71. "GUFS • ΞΫγϣϯΛ࣮ߦ͢ΔલʹtLABELΛཁٻ͢ΔΑ͏ʹͳͬ ͨ State 858 346 @26: . %empty

    ["local variable or method", "global variable", "instance variable", "constant", "class variable", tLABEL, "**", "(", "*", "**arg", "&", '&', '*', '(', ';', '\n'] 347 primary: k_def singleton dot_or_colon @25 fname . @26 f_arglist bodystmt k_end "local variable or method" reduce using rule 346 (@26) ... tLABEL reduce using rule 346 (@26) ... @26 go to state 961
  72. Current implementation

  73. static enum yytokentype -yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *p)

    +yylex(YYSTYPE *lval, YYLTYPE *yylloc, struct parser_params *p, int yystate, short *yyss, short *yyssp) { enum yytokentype t; + VALUE yysstack; p->lval = lval; lval->val = Qundef; + + yysstack = yysstack_new(yyss, yyssp); + + if (p->debug) { + VALUE tokens = expected_tokens(yystate, yysstack); + rb_parser_printf(p, "\nexpected_tokens (state = %d): %"PRIsVALUE"\n", yystate, tokens); + } + stackΛίϐʔ͢Δ stateͱstackΛҾ਺ʹ௥Ճ https://github.com/ruby/ruby/compare/master...yui-knk:feature/ expected_tokens_v2_7_0_preview3_heisei_01?expand=1
  74. /* See also: yysyntax_error and yybackup */ static VALUE expected_tokens(const

    int yystate, VALUE yysstack) { VALUE ary = rb_ary_new(); for (int yytoken = 0; yytoken < YYNTOKENS; ++yytoken) { push_expected_token(ary, yystate, yytoken, rb_ary_dup(yysstack)); } return ary; } શtokenʹରͯ͠push_expected_tokenΛݺͼग़͢
  75. static void push_expected_token(VALUE ary, const int yystate, const int yytoken,

    VALUE yysstack) { int yyn = yypact[yystate]; /* See: yydefault label */ if (yypact_value_is_default(yyn)) { int new_state; if ((new_state = default_reduce(yystate, yytoken, yysstack)) >= 0) { /* yysstack is changed */ push_expected_token(ary, new_state, yytoken, yysstack); } return; } default reduction ͷͱ͖ ࣮ࡍʹreduceͯࣗ͠਎Λ࠶ؼݺͼग़͠
  76. yyn += yytoken; if (yyn < 0 || YYLAST <

    yyn || yycheck[yyn] != yytoken) { int new_state; if ((new_state = default_reduce(yystate, yytoken, yysstack)) >= 0) { /* yysstack is changed */ push_expected_token(ary, new_state, yytoken, yysstack); } return; } default reduction ͷͱ͖ ࣮ࡍʹreduceͯࣗ͠਎Λ࠶ؼݺͼग़͠
  77. yyn = yytable[yyn]; if (yyn <= 0) { if (!yytable_value_is_error(yyn))

    { rb_ary_push(ary, rb_str_new2(yytname[yytoken])); return; } } else { rb_ary_push(ary, rb_str_new2(yytname[yytoken])); return; } } reduction ͷͱ͖ shift ͷͱ͖ expected tokensʹ௥Ճ͢Δ
  78. $ ./miniruby --dump=y -e 'case a;' ... Entering state 375

    Reading a token: expected_tokens (state = 375): ["\"`when'\"", "\"`in'\"", "';'"] ... -e:1: syntax error, unexpected end-of-input, expecting `when' ...
  79. • yypact΍yytableͳͲBisonͷ࣮૷ʹڧ͘ґଘ͢Δ࣮૷ʹ ͳ͍ͬͯΔ yyn += yytoken; if (yyn < 0

    || YYLAST < yyn || yycheck[yyn] != yytoken) { int new_state; if ((new_state = default_reduce(yystate, yytoken, yysstack)) >= 0) { /* yysstack is changed */ push_expected_token(ary, new_state, yytoken, yysstack); } return; }
  80. • ੈͷதʹ͸ tool/ytab.sed ͷΑ͏ͳίʔυ΋͋Δ #!/bin/sed -f # This file is

    used when generating code for the Ruby parser. ... s/^yysyntax_error (/&struct parser_params *p, / s/ yysyntax_error (/&p, / s/\( YYFPRINTF *(\)yyoutput,/\1p,/ s/\( YYFPRINTF *(\)yyo,/\1p,/ s/\( YYFPRINTF *(\)stderr,/\1p,/ s/\( YYDPRINTF *((\)stderr,/\1p,/ s/^\([ ]*\)\(yyerror[ ]*([ ]*parser,\)/\1parser_\2/ s!^ *extern char \*getenv();!/* & */! s/^\(#.*\)".*\.tab\.c"/\1"parse.c"/ /^\(#.*\)".*\.y"/s:\\\\:/:g
  81. 'VUVSFXPSL • ςετํ๏Λߟ͑Δඞཁ͕͋Δ • yyss, yysspʹґଘͨ͠ίʔυ͸SEGV͢ΔՄೳੑ͕͋Δͷ Ͱผͷํ๏Λߟ͑Δ

  82. ·ͱΊ • expected tokens͸parserͷ৘ใ͔Βܾఆ͢Δ͜ͱ͕Ͱ͖ Δ • Default ReductionsΛ͏·͘ѻ͏ͷ͸೉͍͠ • ҰํͰRuby͸Default

    Reductionsͷڍಈʹґଘ͍ͯ͠Δͷ Ͱ͏·͘෇͖߹͍ͬͯ͘ඞཁ͕͋Δ
  83. "DLOPXMFEHFNFOUT • @takeshinoda and @hkdnet • Reviewing the slide

  84. ࢀߟจݙ • Rubyॲཧܥશൠ • http://i.loveruby.net/ja/rhg/book/ • "Rubyͷ͘͠ΈɹRuby Under a Microscope”

    • Parser • http://i.loveruby.net/ja/rhg/book/ • "ίϯύΠϥ―ݪཧɾٕ๏ɾπʔϧ (Information & Computing)” • Bison • https://www.cs.uic.edu/~spopuri/cparser.html
  85. Thank you !!