LALR parser generatorの作り方

BuriKaigi 2024
https://burikaigi.dev/

Transcript

1. LALR parser generatorͷ࡞Γํ January 20, 2024 BuriKaigi 2024 @yui-knk Yuichiro Kaneko
2. About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf (Twitter) • Treasure Data • Engineering Manager of Applications Backend • The author of ruby/lrama LALR parser generator • CRuby committer, mainly develop parser related features
3. PR: We are hiring!! • https://www.treasuredata.com/company/jobs/
4. Parserͱ͸ ͍ͧ͘๺཮ɺ͍ͧ͘෋ࢁ
5. Parserͷ໾ׂ • ೖྗ͞Εͨจࣈྻʹߏ଄Λ༩͑Δ Class Method Method Assignment @name Call Name capitalize
6. Lexerͷ໾ׂ • ॲཧܥ(ruby)͔ΒΈΔͱͨͩͷόΠτྻ • ·ͣจࣈྻΛద੾ͳ୯ҐͰ۠੾Δඞཁ͕͋Δ • Ruby͸͞·͟·ͳEncodingΛαϙʔτ͍ͯ͠Δ 636c61737320477265657465720a2020646566 20696e697469616c697a65286e616d65290a20 202020406e616d65203d206e616d652e636170 6974616c697a650a2020656e640a0a2020646 5662073616c7574650a2020202070757473202 248656c6c6f20237b406e616d657d21220a202 0656e640a656e640a class Greeter def
7. ParserͱLexer • Lexer͕tokenΛ੾Γग़͠ɺParser͕ߏ଄Խ͢Δ Class Method Method Assignm @name Call Name capitaliz class Greeter def Lexer Parser
8. ͍·RubyͰParser͕೤͍ https://twitter.com/OssVision/status/1735433960191299602
9. ͍·RubyͰParser͕೤͍ • Bison͕ਏ͍ • Bisonͷversion͸؀ڥʹΑͬͯҟͳΔ • ෳ਺ͷBisonͷversionΛαϙʔτ͢ΔͨΊʹɺ৽͍͠ػೳ͕࢖͑ͳ͍ • Rubyͷparser͸ෳࡶͰɺϝϯςφϯεੑ͕௕Β͘໰୊ࢹ͞Ε͖ͯͨ • Language Server Protocolͷ୆಄ • RubyͰ͸RBS(ܕ)΍RubyCop(੩తղੳث)ͱ͍ͬͨπʔϧ͕׆ൃʹ։ൃ ͞Ε͍ͯΔ • ͦͷ݁ՌɺϢʔβʔͷೖྗதͷϓϩάϥϜͱ͍͏ෆ׬શͳೖྗΛύʔε͢ Δඞཁ͕Ͱ͖ͯͨ (ΤϥʔτϨϥϯτͳύʔαʔ) • ͜ΕΒΛղܾ͢ΔͨΊʹParserʹେن໛ͳվળΛ͍Ε͍ͯΔ
10. Parserͷ࡞Γํ • खॻ͖parser • Parser GeneratorΛར༻ͯ͠ੜ੒͢Δ • Ruby Ͱ͸ͪ͜Βͷํ๏Λ࠾༻͍ͯ͠Δ • Yacc, Bison, ANTLR ͳͲ • Lrama΋parser generator
11. Parser Generator • ઃఆϑΝΠϧΛ΋ͱʹparserΛੜ੒͢Δπʔϧ • RubyͰ͸GNU BisonΛ͍··Ͱ࢖͖ͬͯͨ • Ruby 3.3Ͱ͸BisonΛLramaʹஔ͖׵͑ͨ • https://github.com/ruby/lrama ઃఆϑΝΠϧ parse.c Bison Lrama
12. ઃఆϑΝΠϧͷྫ • BNFͰจ๏Λهड़͢Δ • ͱͯ΋Θ͔Γ΍͍͢
13. Parser Generatorͷར఺ • จ๏͕ཧղ͠΍͍͢ • จ๏ఆٛͱύʔαʔͷ࣮૷ʹဃ཭͕ͳ͍ • จ๏ͷมߋʹରͯ͠ϑΟʔυόοΫΛಘΔ͜ͱ͕Ͱ͖Δ • ίϯϐϡʔλαΠΤϯεͷཧ࿦ʹج͍͍ͮͯΔ • ΤϥʔτϨϥϯτͳύʔαʔΛจ๏ఆ͔ٛΒࣗಈੜ੒Ͱ͖Δ
14. LR parser͸͍͍ͧ • Bisonʹ͍Ζ͍Ζͱػೳ͕ෆ଍͍ͯ͠ΔͷͰ • RubyͰLrama LR parser generatorΛ࣮૷ͯ͠ • RubyͰLramaΛ࢖͏Α͏ʹ͢Δ͘Β͍ʹਪ͍ͯ͠Δ
15. Parser Generatorͷ ࡞Γํ ෋ࢁʹண͍ͨʂ
16. ΞʔΩςΫνϟ • Frontend, Backend, Code Generator͔ΒͳΔ ઃఆϑΝΠϧ Parser Frontend Backend Code 
 Generator Parser Generator
17. Frontend • LexerͱParserΛ༻͍ͯઃఆϑΝΠϧΛ಺෦తͳσʔλߏ଄ʹม׵͢Δ • σʔλߏ଄ͷओ໾͸Rule ઃఆϑΝΠϧ ಺෦දݱ
18. Action • Actionͷ෦෼΋ղੳ͢Δඞཁ͕͋Δ • $$ͳͲͷಛघͳม਺Ͱtokenͷ஋΍Ґஔ৘ใʹΞΫηεͰ͖Δ • จ๏ϑΝΠϧͱ͸ผͷLexerΛ༻ҙ͢Δͷ͕Α͍ • ม਺͸Codeੜ੒࣌ʹparser internalͳม਺ʹஔ͖׵͑Δ
19. Backend • Rule͔ΒState MachineΛੜ੒͢Δ • ߏจղੳදͱ͍͏ͷ͸ཁ͢ΔʹΦʔτϚτϯ ߏจղੳද
20. • ֤RuleΛΦʔτϚτϯʹม׵͢Δ • શͯͷΦʔτϚτϯΛ߹੒ͨ͠΋ͷ͕ߏจղੳද LR parser͸stackΛ΋ͬͨDAF class A body end def m1 body end class B body end
21. LALRҎ֎ͷΞϧΰϦζϜ • ߏจղੳͷͨΊͷΦʔτϚτϯͷ࡞Γํ͸͍Ζ͍Ζ͋Δ • LR(0), SLR(1), LALR(1), LR(1), IELR(1) ͳͲͷΞϧΰϦζϜ͕͋Δ • ղੳՄೳͳݴޠ΍ඞཁͳϝϞϦ͕ͦΕͧΕҟͳΔ • Rule͔ΒΦʔτϚτϯΛ࡞ΔͷͰɺΞϧΰϦζϜͷબ୒͕จ๏ϑΝΠϧͷ γϯλοΫεͱಠཱ͍ͯ͠Δ
22. ߴ଎ͳLook-Aheadू߹ͷܭࢉ • LALR(1)Λ࣮૷͢Δͱ͖ʹ໰୊ʹͳΔͷ͕ɺޮ཰తʹLook-Aheadू߹Λ ܭࢉ͢Δ͜ͱ • “Ef fi cient Computation of LALR(1) Look-Ahead Sets” ͱ͍͏࿦จͷΞ ϧΰϦζϜΛ࢖༻͢ΔͱΑ͍ • https://dl.acm.org/doi/pdf/10.1145/69622.357187
23. Code Generator • State MachineΛλʔήοτͷݴޠʹ߹Θͤ ࣮ͯ૷͢Δ • tableΛࠓͷstateͱtokenͰݕࡧͯ࣍͠ʹ΍Δ ΂͖͜ͱΛܾΊΔ • shift, reduce, accept, error
24. Templateʹ஋ΛຒΊࠐΉ • ࣮ࡍʹ΍Δ͜ͱ͸ඞཁͳม਺Λ੔ཧͯ͠templateʹຒΊࠐΉ࡞ۀ • LramaͩͱERB, Bisonͩͱm4 • ERB͸Ғେ
25. εύʔε(ૄ)ͳߏจղੳද • ॎ͕ঢ়ଶ਺ɺԣ͕τʔΫϯͷछྨͱ͍͏େ͖ͳςʔϒϧ • ઌ΄Ͳͷྫͩͱ70/238Ϛε͔͠࢖͍ͬͯͳ͍ (29%͘Β͍)
26. εύʔε(ૄ)ͳߏจղੳද • ޓ͍ҧ͍ʹͯ͠1ͭͷ഑ྻʹ·ͱΊΔ • ίϯύΫτσʔλߏ଄ͰΑ͘ͳΒͳ͍ͩΖ͏͔?
27. ΞʔΩςΫνϟʔ • RuleͱState Machine͕ͦΕͧΕͷίϯϙʔωϯτؒͷΠϯλʔϑΣΠε ઃఆϑΝΠϧ ύʔαʔ Frontend Backend Code 
 Generator Rule State Machine
28. Parser GeneratorΛ ֦ு͢Δ ͔ʹඒຯͦ͠͏…
29. Named References • Tokenͷ஋ͳͲʹΞΫηε͢Δͱ͖ʹ$1, $2Ͱ͸ͳ͘ɺ$cpath, $bodyͱ ໊લͰΞΫηεͰ͖Δ • Lexerͷ࣮૷͚ͩͰ࣮ݱͰ͖ΔͷͰFrontend͚ͩͷมߋͰ࣮ݱͰ͖ͨ
30. Parameterizing Rules • ෳ਺ճͷ܁Γฦ͠ͱ͍͏ͷ͸จ๏ఆ্ٛΑ͘ग़ ͯ͘Δ • ॻ͖ํͷύλʔϯ͕ܾ·͍ͬͯΔͷͰ͋Ε͹ɺͦ ΕΛந৅Խͯ͠ॻ͖͍ͨ • LramaͰ͸࣮૷ͣΈ
31. Parameterizing Rules • ઃఆϑΝΠϧ͔ΒRuleͷσʔλߏ଄Λͭ͘Δͱ͖ʹల։͢Δ͚ͩͳͷͰɺ Frontend͚ͩͷมߋͰ࣮ݱͰ͖Δ
32. %after-shift • RipperͷΑ͏ͳ໘ന͍ػೳΛ࣮૷͠Α͏ͱ͢ΔͱShift͢Δॠؒ΍ Reduce͢ΔॠؒʹcallbackΛ͜͞͠Έͨ͘ͳΔ • Frontendͷparser/lexerͱCode GeneratorͷtemplateΛ͍͡Ε͹Ͱ͖ Δ
33. ·ͱΊ ͦΖͦΖᲳ͕৯΂͍ͨ…
34. ·ͱΊ • LR parser͸͍͍ͧ • LR parser generator͸3ͭͷίϯϙʔωϯτ͔ΒͳΓɺݴޠॲཧܥʹߏ଄͕ࣅ ͍ͯΔ • ෼ׂ͞Ε͍ͯΔͨΊػೳ௥Ճͷࡍʹඞཁͳίϯϙʔωϯτ͚ͩΛมߋ͢Ε͹͍͍ • Lrama parser generator͸ΨϯΨϯ։ൃத • ͔Ͷ͜ʹ͖ͬͰRubyͷparserͷ։ൃঢ়گΛ·ͱΊ͍ͯΔ • https://yui-knk.hatenablog.com/ • ruby-jpͱ͍͏slackͷ #lr-parser νϟωϧʹීஈ͍Δ
35. RubyKaigi 2024
36. Thank you!!