Declarative parse.y

Fukuoka.rb #333 Ninja Talk 大会
https://fukuokarb.connpass.com/event/302318/

ruby/lrama
https://github.com/ruby/lrama

Transcript

1. Declarative parse.y November 29, 2023 Fukuoka.rb #333 @yui-knk Yuichiro Kaneko
2. The Bison Slayer https://twitter.com/kakutani/status/1657762294431105025
3. About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf (Twitter) • The author of ruby/lrama LALR parser generator • CRuby committer, mainly develop parser related features • Code positions to RNode (2018, Ruby 2.6) • Coverage • Error reporting • RubyVM::AbstractSyntaxTree (2018, Ruby 2.6) • keep_tokens & error_tolerant option (2022, Ruby 3.2)
4. Fukuoka.rb #333 ͓ΊͰͱ͏͍͟͝·͢🎉
5. https://docs.google.com/presentation/d/1E4v9WPHBLjtvkN7QqulHPGJzKkwIweVfcaMsIQ984_Q/ edit#slide=id.g29612ae1b1e_0_0
6. Declarative • Declarative = “એݴత” • ྫ • SQL • ར఺ • ਺ֶత/࿦ཧֶͳߏ଄Λഎܠʹ࣋ͭ͜ͱ͕ଟ͍ • SQLͷ৔߹͸ؔ܎୅਺ • ڊਓͷݞʹ৐Δ͜ͱ͕Ͱ͖Δ • ղ͘΂͖໰୊ʹϑΥʔΧεͰ͖Δ
7. Declarativeͳจ๏ͱparser • BNF (Backus-Naur form) • ڞ௨ݴޠ(Ͱ͢ΑͶ) • ΦʔτϚτϯ • ܭࢉػͷ਺ֶతͳϞσϧ • ༗ݶΦʔτϚτϯ/ϓογϡμ΢ϯɾΦʔτϚτϯ/ઢܗ߆ଋΦʔτϚτϯ/ νϡʔϦϯάϚγϯ … • ܗࣜݴޠཧ࿦ • (ϓϩάϥϛϯά)ݴޠΛϞσϧԽͨ͠΋ͷ • ਖ਼نݴޠ/จ຺ࣗ༝ݴޠ/จ຺ґଘݴޠ/ؼೲతՄࢉݴޠ …
8. Declarativeʁ • arg_value, *arg_value, * ͷ͍ͣΕ͔͕ ‘,’ ۠੾ΓͰฒΜͰ͍Δͱ͍͏ϧʔ ϧ • declarative͚ͩͲprimitiveͳͷͰݟ௨͕͠ѱ͍ • ͜ͷޙͷ ydah͞Μͷൃදʹظ଴
9. parse.y͸declarative͔ʁ • ΋͜͠Μͳײ͡ͷϧʔϧͩͬͨΒɺͦΕ͸ͱͯ΋declarative
10. ݱ࣮͸Ͳ͏͔
11. ͳΜ͔ࢥ͍ͬͯͨͷͱҧ͏
12. ϧʔϧͱΞΫγϣϯ • ϧʔϧ: BNF • ΞΫγϣϯ: Cݴޠ ϧʔϧ ΞΫγϣϯ
13. ASTΛੜ੒͢ΔϩδοΫ • Node΍location৘ ใͳͲΛ͍͍ͬͯ͡ Δ
14. ίϝϯτʹજΉRipperͷӨ • ໨ΛڽΒ͢ͱίϝϯ τʹࠞͬͯ͟ripper ͷίʔυ͕͋Δ!!
15. ม਺ςʔϒϧΛ؅ཧ͢ΔϩδοΫ • localม਺ͷςʔϒ ϧΛpush, popͯ͠ ͍Δ
16. ঢ়ଶΛ؅ཧ͢ΔϩδοΫ • in_def΍in_classͱ ͍͏ม਺Λૢ࡞ͯ͠ ͍Δ • in_def΍in_class͸ def಺෦Ͱͷclassఆ ٛΛνΣοΫ͢Δͷ ʹར༻ͨ͠Γ͢Δ
17. parse.y for Under graduate • ΞΫγϣϯ(Cݴޠ)ͱ͍͏ͷ͸׬શʹCRubyݻ༗ͷϩδοΫ • parse.yͷ೉͠͞͸ΞΫγϣϯ(Cݴޠ)ʹ͋ΔͷͰ͸ͳ͍͔ • ΑΓdeclarativeʹ͍ͯ͘͜͠ͱͰɺڭՊॻͷ஌ࣝͱparse.yͷ࣮૷ͷဃ཭ Λͳ͍ͯ͘͘͠
18. ΞΫγϣϯͷத਎ • ASTΛߏங͢Δ • Ripper༻ͷϩδοΫ • ҙຯղੳ • def಺෦ͰͷclassఆٛͳͲͳͲ • https://i.loveruby.net/ja/rhg/book/syntree.html “ҙຯղੳ” ΋ࢀর • ͜ΕΛdeclarativeʹͰ͖ͳ͍͔?
19. ม਺ҰͭͰstackΛ؅ཧ͢Δ • k_classΛड͚ࡼʹ͠ ͯड͚౉͞ΕΔctxt (context) • ͜Ε͔Β্ॻ͖͞Ε Δ͜ͱʹඋ͑ͯݱ࣌ ఺ͷctxtΛk_classͷ semantic valueʹୀ ආ͢Δ 1. ctxtΛୀආ
20. ม਺ҰͭͰstackΛ؅ཧ͢Δ • classఆٛͷύʔεॲ ཧͷதͰctxtΛ্ॻ ͖͢Δ 1. ctxtΛୀආ 2. ctxtΛ্ॻ͖
21. ม਺ҰͭͰstackΛ؅ཧ͢Δ • classఆٛͷύʔεॲ ཧ͕ऴΘͬͨͷͰ k_classʹୀආ͓ͯ͠ ͍ͨctxtΛ෮چ͢Δ 1. ctxtΛୀආ 3. ctxtΛϦετΞ 2. ctxtΛ্ॻ͖
22. ௨ৗͷstackͷ࢖͍ํ A A B A Push Pop
23. ม਺ҰͭͰstackΛ؅ཧ͢Δ A k_class A ctxt B k_class A k_class A A ctxt ctxt ctxt ctxt A A ୀආ ্ॻ͖ ϦετΞ ୀආઌ͕ফ͑Δ
24. ݱ࣮͸ݫ͍͠
25. ϧʔϧͱΞΫγϣϯ • ϧʔϧ: • BNF • Lrama͕ίϯτϩʔϧͰ͖Δ • ܗࣜݴޠཧ࿦΍ΦʔτϚτϯͷཧ࿦ʹै͏ඞཁ͕͋Δ • ΞΫγϣϯ • Cݴޠ • Lrama͕ίϯτϩʔϧͰ͖ͳ͍ • ࣮࣭ԿͰ΋Ͱ͖Δ
26. LR parserΛ৴͡Α
27. LR parser͸stackߏ଄ class A body end def m1 body end class B body end
28. StackΛ௥Ճͯ͠ΈΔ class A body end def m1 body end class B body end in_class in_def Ture Ture Ture False True False
29. PDAͷࣗવͳ֦ு • Pushdown automaton͸༗ݶΦʔτϚτϯʹελοΫΛ͚ͭͨ΋ͷ • LR parser͸Pushdown automatonͷߏ଄Λ͍ͯ͠Δ • ΦʔτϚτϯͱΦʔτϚτϯΛ૊Έ߹Θͤͨ΋ͷ͸ΦʔτϚτϯ
30. ΦʔτϚτϯͷ߹੒ x0 x1 x2 x3 a b c a d y0 y1 y2 y3 a a b a e ฏੴ๜඙ (2019) ܗࣜతϞσϧԽ ཭ࢄࣄ৅ʗ࣮࣌ؒʗϋΠϒϦουγεςϜͷϞσϧԽͱղੳ. ৿๺ग़൛. P.25 ਤ2.7Λࢀߟʹ࡞੒
31. ΦʔτϚτϯͷ߹੒ (x0, y0) (x0, y1) (x0, y3) (x0, y2) (x1, y0) (x1, y1) (x1, y3) (x1, y2) (x2, y0) (x2, y1) (x2, y3) (x2, y2) (x3, y0) (x3, y1) (x3, y3) (x3, y2) a c d c d c d e e e ฏੴ๜඙ (2019) ܗࣜతϞσϧԽ ཭ࢄࣄ৅ʗ࣮࣌ؒʗϋΠϒϦουγεςϜͷϞσϧԽͱղੳ. ৿๺ग़൛. P.25 ਤ2.9Λࢀߟʹ࡞੒
32. ruby/lrama #231 • Parser stateΛruleͷ Ұ෦ͱͯ͠هड़Ͱ͖Δ Α͏ʹ͢Δػೳ • ೥಺ʹೖΕΒΕΔ͔!? https://github.com/ruby/lrama/pull/231
33. Parser States (lex context) • ͜ͷล͸enum΍bool஋ͳͷͰදݱͰ͖ͦ͏
34. shareable_constant_value • Magic comment͸lexer͔Βparserʹ౉Βͳ͍ͷͰɺlexerଆ͔Βݱࡏͷ stack topͷ஋Λߋ৽Ͱ͖Δඞཁ͕͋Δ
35. in_rescue • retryΛॻ͍͍͍ͯͷ͸rescueͱ else (΋͘͠͸ ensure)ͷ͚ؒͩ • ͦͷͨΊࠓͷঢ়ଶΛ؅ཧ͢Δ
36. Nested in_rescue • ͜ΕΒ͸શͯ֎ଆͷrescueͷதʹ͍Δ ͷͰvalidʹͳΔ https://github.com/ruby/ruby/blob/v3_3_0_preview3/test/ruby/test_ast.rb#L287-L306
37. ಛผͳঢ়ଶΛ௥Ճ͢Δ • ಁաతʹͦͷલͷঢ়ଶΛࢀর͢Δঢ়ଶ(ε)Λ༻ҙ͢Δ • ௚લ͕after_resuceͳΒЏΛੵΉ • ݱࡏͷঢ়ଶΛධՁ͢Δͱ͖ʹ͸ЏΛແࢹͯ͠࠷ॳʹग़ݱ͢Δঢ়ଶΛΈΔΑ ͏ʹ͢Δ before_rescue after_rescue before_rescue after_rescue ε
38. next/break/redo • next/break/redoΛॻ͚Δ৔ॴʹ੍ݶ͕͋Δ
39. retryͱҟͳΔ࣮૷… • next/break/redoͷ؅ཧ͸in_def΍in_class ͱ͸ҧ͏࣮૷ʹͳ͍ͬͯΔ • ͜͜Ͱ͍͏node͸listͩͱࢥ͍ͬͯͩ͘͞
40. retryͷ৔߹ • retryͷ৔߹͸”retry”͕ग़͖ͯͨͱ͖ʹ in_rescue ΛνΣοΫͯ͠ɺ rescueͷ௚ޙग़ͳ͍৔߹͸Τϥʔʹ͍ͯ͠Δ
41. next/break/redoͷ৔߹ • stmt͕֬ఆͨ͠ஈ֊ͰνΣοΫ͍ͯ͠Δ • parse.yͷ࣮૷ʹແବ͕ͳ͍ͱԾఆͨ͠ͱ ͖ɺͳͥ͜͏ͳ͍ͬͯΔ͔෼͔Γ·͢ ͔ʁ
42. ޙஔͷwhile • ޙஔͷwhile/untilͷ৔߹ɺͦͷ௚લͷstmtͰ͸next/break/redoΛॻ͘ ͜ͱ͕Ͱ͖Δ
43. modi fi er_whileͷ࣮૷ • stmtͷ࣌఺Ͱ͸Τϥʔʹ͍͍͔ͯ͠ΛܾఆͰ͖ͳ͍ • ૺ۰ͨ͠next/break/redoΛͲ͔͜ʹ͓࣋ͬͯ͘ඞཁ͕͋Δ
44. node͸list
45. node͸list struct parser_params { rb_node_exits_t *exits; } struct RNode_EXITS { NODE *nd_chain; } struct RNode_BREAK { NODE *nd_chain; } struct RNode_BREAK { NODE *nd_chain; }
46. https://twitter.com/spikeolaf/status/1707031682648555756
47. ελοΫʹϦετΛࡌͤΔ • ཧղͯ͠͠·͑͹ϦετͱελοΫͷ૊Έ߹ΘͤͰ͔͠ͳ͍ • Lramaʹ૊ΈࠐΈͰϦετ(or ഑ྻ)Λ࣮૷ͯ͠ελοΫͰ؅ཧ͢Ε͹Α͍ k_whileʹ͸֎ଆͷ exits͕อଘ͞Ε͍ͯΔ whileશମ͕͓Θͬͨ Βrestore
48. ΑΓdeclarativeͳparse.y • ৽͍͠ঢ়ଶΛpush͢ΔՕॴ(allow_exits)ͱpop͢ΔՕॴ (restore_block_exit)͕1ͭͷϧʔϧ಺ʹด͍ͯ͡ΔͷͰɺ͜ͷϧʔϧ͚ͩ ΛݟΕ͹Α͍
49. ΑΓdeclarativeͳparse.y • push͢ΔՕॴ(allow_exits)ͱpop͢ΔՕॴ(restore_block_exit)͕2ͭͷ ϧʔϧʹ·͕͍ͨͬͯΔͷͰಡΈʹ͍͘ • ࣮ࡍ͸300ߦ͘Β͍཭Ε͍ͯΔ
50. ΑΓdeclarativeͳparse.y • ελοΫʹ৽͍͠ঢ়ଶΛpushͯ͠ɺ͜ͷϧʔϧͷύʔε͕͓ΘͬͨΒࣗಈ Ͱpopͯ͠΄͍͠ • Πϝʔδͱͯ͠͸around_actionΈ͍ͨͳ΍ͭ • ͍͍ײ͡ͷsyntaxΛ୳͠த • ΋ͱͷΦʔτϚτϯΛͲ͏มߋ͢Δͷ͔ਫ਼៛Խ͢Δඞཁ͕͋Γͦ͏
51. Parser States (local_vars) • parser stateքͷϘε • ԾҾ਺ɺlocalม਺ɺม਺͕ ࢖༻͞Ε͍ͯΔ͔൱͔ɺ numbered parameterʹؔ ͢Δ͋Ε͜ΕΛ1ͭʹ·ͱΊ ͨເͷΑ͏ͳߏ଄ମ • ελοΫͱू߹(Set)ͷ૊Έ ߹Θͤ
52. • Declarative • ਺ֶత/࿦ཧֶͳߏ଄Λഎܠʹ࣋ͭ͜ͱ͕ଟ͍ • ΦʔτϚτϯ΍ܗࣜݴޠཧ࿦ͷݞʹ৐Δ • ϧʔϧ(BNF)ͱΞΫγϣϯ(Cݴޠ) • ҙຯղੳʹ஫໨ ·ͱΊ
53. • ҙຯղੳ • ૉ௚ʹparserͱ౷߹Ͱ͖Δ (in_def΍in_class) • ͪΐͬͱ޻෉͕ඞཁ (shareable_constant_value΍in_rescue) • Ϧονͳσʔλߏ଄͕ඞཁ • Ϧετ (next/break/redo) • Set (local_vars) ·ͱΊ
54. Thank you!!