Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
LALR parser generatorの作り方
Search
yui-knk
January 20, 2024
Programming
6
6.2k
LALR parser generatorの作り方
BuriKaigi 2024
https://burikaigi.dev/
yui-knk
January 20, 2024
Tweet
Share
More Decks by yui-knk
See All by yui-knk
Ruby's Line Breaks
yui_knk
4
4.3k
What is Parser
yui_knk
11
5.2k
Ruby Parser progress report 2024
yui_knk
2
410
最高の構文木の設計 2024年版
yui_knk
9
5.8k
Converting AST
yui_knk
4
350
My favorite script, "dsl.rb"
yui_knk
2
1.2k
Rearchitect Ripper
yui_knk
2
1.2k
The grand strategy of Ruby Parser
yui_knk
7
10k
Declarative parse.y
yui_knk
2
5.6k
Other Decks in Programming
See All in Programming
スタートアップの急成長を支えるプラットフォームエンジニアリングと組織戦略
sutochin26
0
170
アンドパッドの Go 勉強会「 gopher 会」とその内容の紹介
andpad
0
280
“いい感じ“な定量評価を求めて - Four Keysとアウトカムの間の探求 -
nealle
0
360
童醫院敏捷轉型的實踐經驗
cclai999
0
210
AWS CDKの推しポイント 〜CloudFormationと比較してみた〜
akihisaikeda
3
320
Team operations that are not burdened by SRE
kazatohiei
1
290
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
250
Kotlin エンジニアへ送る:Swift 案件に参加させられる日に備えて~似てるけど色々違う Swift の仕様 / from Kotlin to Swift
lovee
1
260
git worktree × Claude Code × MCP ~生成AI時代の並列開発フロー~
hisuzuya
1
520
PipeCDのプラグイン化で目指すところ
warashi
1
230
Hypervel - A Coroutine Framework for Laravel Artisans
albertcht
1
110
Result型で“失敗”を型にするPHPコードの書き方
kajitack
4
550
Featured
See All Featured
Become a Pro
speakerdeck
PRO
28
5.4k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Building an army of robots
kneath
306
45k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Music & Morning Musume
bryan
46
6.6k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.8k
Code Reviewing Like a Champion
maltzj
524
40k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Transcript
LALR parser generatorͷ࡞Γํ January 20, 2024 BuriKaigi 2024 @yui-knk Yuichiro
Kaneko
About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf
(Twitter) • Treasure Data • Engineering Manager of Applications Backend • The author of ruby/lrama LALR parser generator • CRuby committer, mainly develop parser related features
PR: We are hiring!! • https://www.treasuredata.com/company/jobs/
Parserͱ ͍ͧ͘ɺ͍ͧ͘ࢁ
Parserͷׂ • ೖྗ͞ΕͨจࣈྻʹߏΛ༩͑Δ Class Method Method Assignment @name Call Name
capitalize
Lexerͷׂ • ॲཧܥ(ruby)͔ΒΈΔͱͨͩͷόΠτྻ • ·ͣจࣈྻΛదͳ୯ҐͰ۠Δඞཁ͕͋Δ • Ruby͞·͟·ͳEncodingΛαϙʔτ͍ͯ͠Δ 636c61737320477265657465720a2020646566 20696e697469616c697a65286e616d65290a20 202020406e616d65203d206e616d652e636170
6974616c697a650a2020656e640a0a2020646 5662073616c7574650a2020202070757473202 248656c6c6f20237b406e616d657d21220a202 0656e640a656e640a class Greeter def
ParserͱLexer • Lexer͕tokenΛΓग़͠ɺParser͕ߏԽ͢Δ Class Method Method Assignm @name Call Name
capitaliz class Greeter def Lexer Parser
͍·RubyͰParser͕͍ https://twitter.com/OssVision/status/1735433960191299602
͍·RubyͰParser͕͍ • Bison͕ਏ͍ • BisonͷversionڥʹΑͬͯҟͳΔ • ෳͷBisonͷversionΛαϙʔτ͢ΔͨΊʹɺ৽͍͠ػೳ͕͑ͳ͍ • RubyͷparserෳࡶͰɺϝϯςφϯεੑ͕Β͘ࢹ͞Ε͖ͯͨ •
Language Server Protocolͷ಄ • RubyͰRBS(ܕ)RubyCop(੩తղੳث)ͱ͍ͬͨπʔϧ͕׆ൃʹ։ൃ ͞Ε͍ͯΔ • ͦͷ݁ՌɺϢʔβʔͷೖྗதͷϓϩάϥϜͱ͍͏ෆશͳೖྗΛύʔε͢ Δඞཁ͕Ͱ͖ͯͨ (ΤϥʔτϨϥϯτͳύʔαʔ) • ͜ΕΒΛղܾ͢ΔͨΊʹParserʹେنͳվળΛ͍Ε͍ͯΔ
Parserͷ࡞Γํ • खॻ͖parser • Parser GeneratorΛར༻ͯ͠ੜ͢Δ • Ruby Ͱͪ͜Βͷํ๏Λ࠾༻͍ͯ͠Δ •
Yacc, Bison, ANTLR ͳͲ • Lramaparser generator
Parser Generator • ઃఆϑΝΠϧΛͱʹparserΛੜ͢Δπʔϧ • RubyͰGNU BisonΛ͍··Ͱ͖ͬͯͨ • Ruby 3.3ͰBisonΛLramaʹஔ͖͑ͨ
• https://github.com/ruby/lrama ઃఆϑΝΠϧ parse.c Bison Lrama
ઃఆϑΝΠϧͷྫ • BNFͰจ๏Λهड़͢Δ • ͱͯΘ͔Γ͍͢
Parser Generatorͷར • จ๏͕ཧղ͍͢͠ • จ๏ఆٛͱύʔαʔͷ࣮ʹဃ͕ͳ͍ • จ๏ͷมߋʹରͯ͠ϑΟʔυόοΫΛಘΔ͜ͱ͕Ͱ͖Δ • ίϯϐϡʔλαΠΤϯεͷཧʹج͍͍ͮͯΔ
• ΤϥʔτϨϥϯτͳύʔαʔΛจ๏ఆ͔ٛΒࣗಈੜͰ͖Δ
LR parser͍͍ͧ • Bisonʹ͍Ζ͍Ζͱػೳ͕ෆ͍ͯ͠ΔͷͰ • RubyͰLrama LR parser generatorΛ࣮ͯ͠ •
RubyͰLramaΛ͏Α͏ʹ͢Δ͘Β͍ʹਪ͍ͯ͠Δ
Parser Generatorͷ ࡞Γํ ࢁʹண͍ͨʂ
ΞʔΩςΫνϟ • Frontend, Backend, Code Generator͔ΒͳΔ ઃఆϑΝΠϧ Parser Frontend Backend
Code Generator Parser Generator
Frontend • LexerͱParserΛ༻͍ͯઃఆϑΝΠϧΛ෦తͳσʔλߏʹม͢Δ • σʔλߏͷओRule ઃఆϑΝΠϧ ෦දݱ
Action • Actionͷ෦ղੳ͢Δඞཁ͕͋Δ • $$ͳͲͷಛघͳมͰtokenͷҐஔใʹΞΫηεͰ͖Δ • จ๏ϑΝΠϧͱผͷLexerΛ༻ҙ͢Δͷ͕Α͍ • มCodeੜ࣌ʹparser internalͳมʹஔ͖͑Δ
Backend • Rule͔ΒState MachineΛੜ͢Δ • ߏจղੳදͱ͍͏ͷཁ͢ΔʹΦʔτϚτϯ ߏจղੳද
• ֤RuleΛΦʔτϚτϯʹม͢Δ • શͯͷΦʔτϚτϯΛ߹ͨ͠ͷ͕ߏจղੳද LR parserstackΛͬͨDAF class A body end
def m1 body end class B body end
LALRҎ֎ͷΞϧΰϦζϜ • ߏจղੳͷͨΊͷΦʔτϚτϯͷ࡞Γํ͍Ζ͍Ζ͋Δ • LR(0), SLR(1), LALR(1), LR(1), IELR(1) ͳͲͷΞϧΰϦζϜ͕͋Δ
• ղੳՄೳͳݴޠඞཁͳϝϞϦ͕ͦΕͧΕҟͳΔ • Rule͔ΒΦʔτϚτϯΛ࡞ΔͷͰɺΞϧΰϦζϜͷબ͕จ๏ϑΝΠϧͷ γϯλοΫεͱಠཱ͍ͯ͠Δ
ߴͳLook-Aheadू߹ͷܭࢉ • LALR(1)Λ࣮͢Δͱ͖ʹʹͳΔͷ͕ɺޮతʹLook-Aheadू߹Λ ܭࢉ͢Δ͜ͱ • “Ef fi cient Computation of
LALR(1) Look-Ahead Sets” ͱ͍͏จͷΞ ϧΰϦζϜΛ༻͢ΔͱΑ͍ • https://dl.acm.org/doi/pdf/10.1145/69622.357187
Code Generator • State MachineΛλʔήοτͷݴޠʹ߹Θͤ ࣮ͯ͢Δ • tableΛࠓͷstateͱtokenͰݕࡧͯ࣍͠ʹΔ ͖͜ͱΛܾΊΔ •
shift, reduce, accept, error
TemplateʹΛຒΊࠐΉ • ࣮ࡍʹΔ͜ͱඞཁͳมΛཧͯ͠templateʹຒΊࠐΉ࡞ۀ • LramaͩͱERB, Bisonͩͱm4 • ERBҒେ
εύʔε(ૄ)ͳߏจղੳද • ॎ͕ঢ়ଶɺԣ͕τʔΫϯͷछྨͱ͍͏େ͖ͳςʔϒϧ • ઌ΄Ͳͷྫͩͱ70/238Ϛε͔͍ͬͯ͠ͳ͍ (29%͘Β͍)
εύʔε(ૄ)ͳߏจղੳද • ޓ͍ҧ͍ʹͯ͠1ͭͷྻʹ·ͱΊΔ • ίϯύΫτσʔλߏͰΑ͘ͳΒͳ͍ͩΖ͏͔?
ΞʔΩςΫνϟʔ • RuleͱState Machine͕ͦΕͧΕͷίϯϙʔωϯτؒͷΠϯλʔϑΣΠε ઃఆϑΝΠϧ ύʔαʔ Frontend Backend Code
Generator Rule State Machine
Parser GeneratorΛ ֦ு͢Δ ͔ʹඒຯͦ͠͏…
Named References • TokenͷͳͲʹΞΫηε͢Δͱ͖ʹ$1, $2Ͱͳ͘ɺ$cpath, $bodyͱ ໊લͰΞΫηεͰ͖Δ • Lexerͷ࣮͚ͩͰ࣮ݱͰ͖ΔͷͰFrontend͚ͩͷมߋͰ࣮ݱͰ͖ͨ
Parameterizing Rules • ෳճͷ܁Γฦ͠ͱ͍͏ͷจ๏ఆ্ٛΑ͘ग़ ͯ͘Δ • ॻ͖ํͷύλʔϯ͕ܾ·͍ͬͯΔͷͰ͋Εɺͦ ΕΛநԽͯ͠ॻ͖͍ͨ • LramaͰ࣮ͣΈ
Parameterizing Rules • ઃఆϑΝΠϧ͔ΒRuleͷσʔλߏΛͭ͘Δͱ͖ʹల։͢Δ͚ͩͳͷͰɺ Frontend͚ͩͷมߋͰ࣮ݱͰ͖Δ
%after-shift • RipperͷΑ͏ͳ໘ന͍ػೳΛ࣮͠Α͏ͱ͢ΔͱShift͢Δॠؒ Reduce͢ΔॠؒʹcallbackΛ͜͞͠Έͨ͘ͳΔ • Frontendͷparser/lexerͱCode GeneratorͷtemplateΛ͍͡ΕͰ͖ Δ
·ͱΊ ͦΖͦΖᲳ͕৯͍ͨ…
·ͱΊ • LR parser͍͍ͧ • LR parser generator3ͭͷίϯϙʔωϯτ͔ΒͳΓɺݴޠॲཧܥʹߏ͕ࣅ ͍ͯΔ •
ׂ͞Ε͍ͯΔͨΊػೳՃͷࡍʹඞཁͳίϯϙʔωϯτ͚ͩΛมߋ͢Ε͍͍ • Lrama parser generatorΨϯΨϯ։ൃத • ͔Ͷ͜ʹ͖ͬͰRubyͷparserͷ։ൃঢ়گΛ·ͱΊ͍ͯΔ • https://yui-knk.hatenablog.com/ • ruby-jpͱ͍͏slackͷ #lr-parser νϟωϧʹීஈ͍Δ
RubyKaigi 2024
Thank you!!