Ruby Parser progress report 2025

RubyKaigi 2025 follow up
https://rhc.connpass.com/event/356128/

リンク:
Understanding Ruby Grammar Through Conflicts in RubyConf Taiwan 2025
https://speakerdeck.com/yui_knk/understanding-ruby-grammar-through-conflicts

Transcript

1. Ruby Parser progress report 2025 April 30, 2025 in RubyKaigi 2025 follow up @yui-knk Yuichiro Kaneko
2. ✦ Yuichiro Kaneko ✦ yui-knk (GitHub) / spikeolaf (Twitter) ✦ Treasure Data ✦ Engineering Manager of Applications Backend ✦ CRuby committer, mainly develop parser generator and parser ✦ Lrama LALR (1) parser generator (2023, Ruby 3.3) ✦ Love LR parser About me
3. ✦ ٕज़ॻయ18Ͱ৽ץΛ൦෍ ✦ ෺ཧຊ ¥1,000 ✦ ॏ൛ग़དྷʂʂ ✦ ୈ3࡮ ✦ ిࢠຊ ¥500 ✦ ٕज़ॻయͷαΠτ͔ΒߪೖՄೳ ਐḿ 1.ʮLR parserೖ໳ʯ
4. ✦ ୈ1ষ LR parserͷ֓࿦ ✦ ୈ2ষ ޮ཰తͳLookaheadͷܭࢉ ʮLR parserೖ໳ʯ
5. ಡऀ͔Βͷײ૝ https://x.com/junk0612/status/1929128786701832610 https://x.com/junk0612/status/1929158546244956317
6. ಡऀ͔Βͷײ૝ https://ydah.hatenablog.com/entry/2025/07/24/215617
7. ✦ େ޷ධʂʂ ✦ ຊ೔΋ઈࢍൢചதʂʂ ݁࿦
8. ✦ C107ʹਃ͠ࠐΈΛͨ͠ ✦ ৽ץ(!?)΋༧ఆத ✦ IELR? ✦ ଐੑจ๏? C107
9. ਐḿ 2. RubyConf Taiwan 2025
10. “Understanding Ruby Grammar Through Con fl icts” https://speakerdeck.com/yui_knk/understanding-ruby-grammar-through-conflicts
11. ✦ ΈΜͳ͕ࢥ͍ͭ͘Ͱ͋Ζ͏จ๏ͷมߋΛ͍ΕͯΈΔ ✦ ίϯϑϦΫτ͕͓͖ΔͷͰղੳ͢Δ ✦ Rubyͷจ๏ͷσβΠϯ΍੍໿Λߟ͑Δ ✦ Arguments and parameters ✦ Command call ✦ Endless method de fi nitions ✦ Pattern matching “Understanding Ruby Grammar Through Con fl icts”
12. • [Bug #17925] Pattern matching syntax using semicolon one-line • https://bugs.ruby-lang.org/issues/17925 • [Bug #18080] Syntax error on one-line pattern matching • https://bugs.ruby-lang.org/issues/18080 • [Bug #21378] variable pinning does not look for method arguments • https://bugs.ruby-lang.org/issues/21378 • [Bug #21097] `x = a rescue b in c` and `def f = a rescue b in c` parsed differently between parse.y and prism • https://bugs.ruby-lang.org/issues/21097 Pattern matchingʹؔ͢Δνέοτ
13. ✦ endless method de fi nitionͱpattern matchingΛ૊Έ߹ΘͤΔͱ… ✦ େ͖͘“=”Ͱ෼ׂͯ͠΄͍͕͠ɺ࣮ࡍ͸“in”Ͱ෼ׂ͞Εͯ͠·͏ [Bug #21378]
14. ✦ ·ͣݕ౼͢Δͷ͸exprʹͳ͍ͬͯΔpattern matchingΛargʹͯ͠ΈΔ͜ͱ ✦ ཁ͢ΔʹҾ਺ʹॻ͚ΔΑ͏ʹมߋ͢Δ ✦ ͱͯ΋Α͘ίϯϑϦΫτ͢Δ ✦ “=>” ✦ “,” ✦ “|” ✦ “^” ✦ (“..” and “…”) ✦ (“**”) Ξϓϩʔν
15. • pattern matchingͷύλʔϯͰ͸ “[]” ΍ “{}” ͕লུՄೳ • ͱ͍͏͜ͱ͸Ͳ͜ͰҾ਺͕੾ΕΔ͔෼͔Βͳ͘ͳΔ Con fl icts on “,”
16. ✦ pattern matching: “|” (pipe)͸ෳ਺ͷύλʔϯΛ۠੾Δͷʹ࢖ΘΕ Δ ✦ binary operator: “|” ͸ೋ߲ԋࢉࢠͰ΋͋Δ Con fl icts on “|” Pattern 1 Pattern 2 Pattern matching arg
17. ✦ “^” (caret)ͰίϯϑϦΫτ͕͓͖Δͷ͸ͪΐͬͱෆࢥٞ ✦ pattern matching: “^” ͸variable pinningʹ࢖ΘΕΔ͕ɺ͜Ε͸୯ ߲ԋࢉࢠ ✦ binary operator: “^” ͸ೋ߲ԋࢉࢠͰ΋͋Δ Con fl icts on “^”
18. ✦ pattern matchingͰ͸trailing commaΛॻ͘͜ͱ͕Ͱ͖Δ ✦ trailing commaͷ༗ແͰڍಈ͕มΘΔ trailing comma in pattern matching
19. ✦ ୯ʹVariable pinningΛ͍ͯ͠Δͷ͔ɺͦΕͱ΋trailing commaͱ ೋ߲ԋࢉࢠͷ૊Έ߹Θͤͳͷ͔͕۠ผͰ͖ͳ͘ͳΔ trailing comma in pattern matching array pattern w/ variable pinning pattern matching w/ trailing comma arg Binary Operator
20. ✦ ᐆດੑͱ͍͏ͷ͸parser΍parser generatorͷ໰୊Ͱ͸ͳ͍ ✦ จ๏ͷ΋ͭੑ࣭ ✦ ᐆດͳέʔεʹ͍ͭͯͲͷΑ͏ʹղऍ͞ΕΔ͔ܾΊΔ͜ͱ͸Մೳ จ๏ͷᐆດੑ
21. ✦ ํ๏1: “,” ͕ଓ͘ݶΓpattern matchingͰ͋ Δͱղऍ͢Δ ✦ ෳ਺ͷҾ਺͕͋Γɺͦ͜ʹpattern matchingؚ͕·ΕΔέʔεΛͲ͏͢Δ͔ ✦ 1. pattern matching͸࠷ޙͷҾ਺ʹ͚ͩॻ ͘͜ͱ͕Ͱ͖Δ ✦ 2. pattern matchingͷͱ͖͸Ҿ਺͸1ͭͩ ͚ॻ͚Δ Con fl ictʹରԠ͢Δ
22. ✦ ํ๏2: Ҿ਺ͱͯ͠ॻ͘͜ͱ͕Ͱ͖Δpattern matchingͷจ๏ʹ੍ ݶΛ͍ΕΔ ✦ ύλʔϯʹ͓͚ΔΧοίͷলུΛېࢭ͢Δ ✦ “,”ʹ͍ͭͯ͸ղܾ͢Δ ✦ “|”͸? Con fl ictʹରԠ͢Δ
23. ✦ ํ๏3: ͦͷΑ͏ͳॻ͖ํ͸ېࢭ͢Δ ✦ ํ๏1: ʮ͑ɺͦ͏͍͏;͏ʹղऍ͞ΕΔͱࢥ͍ͬͯͳ͔ͬͨʯ ✦ ํ๏2: ʮ͑ɺͲ͏ͯ͜͠ͷέʔεͰ͸Χοί͕লུͰ͖ͳ͍ͷʯ Con fl ictʹରԠ͢Δ
24. • “command_asgn” Λ֦ுͯ͠one-line pattern matchingΛؚΉendless method de fi nitionΛڐՄ͢Δ • ͋Θͤͯ༏ઌ౓ͷఆٛ΋ௐ੔͢Δ ผͷΞϓϩʔν
25. ✦ Ҿ਺ͷཁૉ(arg)ͱ͍͏ࠞࡶͨ͠ͱ͜Ζʹpattern matchingͷΑ͏ ͳϦονͳจ๏ΛೖΕΑ͏ͱ͢Δͷʹແཧ͕͋Δ ✦ τοϓϨϕϧ(stmt)͚ͩʹݶఆ͢Δ͜ͱͰᐆດੑΛ͚͞Δͷ͸Ͳ ͏ͩΖ͏͔ʁ ✦ ํ๏2ʹ͍ۙ ཁ͢Δʹ?
26. ✦ Ұ෦ͷendless method de fi nition͸ϝιουͷҾ਺ʹͰ͖ͳ͍ ✦ ͭ·ΓprivateϝιουͷҾ਺ʹͰ͖ͳ͍ [Feature #17398]
27. ✦ ݸʑͷҾ਺Ͱ͸ͳ͘ɺҾ਺શମΛද͢ϧʔϧ ✦ endless method de fi nition͕1ͭͷͱ͖͸Ҿ਺ʹͰ͖Δɻͱ͍͏ ϧʔϧΛ௥Ճ͢Δ “call_args”ͷ֦ு
28. ✦ endless method de fi nitionͷbodyʹෳ਺ͷҾ਺͕ฒͿͱ͖͸body ʹଐ͢ΔΑ͏ʹղऍ͢Δ ✦ ͜Ε͸ίϚϯυ(Χοίͳ͠ϝιουݺͼग़͠)͕1ͭͷͱ͖͚ͩ Ҿ਺ʹͰ͖Δͱ͍͏طଘͷڍಈʹ͋Θͤͨ(ͭ΋Γ) “call_args”
29. ✦ 615 commits since the last release ✦ ίϛοτͷ30%͘Β͍͕0.7.1޲͚ ਐḿ 3. Lrama 0.7.1
30. ✦ IELRͷߴ଎Խ (by junk0612) ✦ ࣮ઓ౤ೖͰ͖Δ଎౓ʹͳͬͨͷͰɺLrama΋LR(1) parser parser generatorΛ ໊৐Δͱ͖͕͖ͨͷͰ͸? ✦ Support parameterized rule's arguments include inline (by ydah) ✦ Grammar fi leͰڐՄ͢Δsyntaxͷ֦ு (by ydah) ✦ Output fi leͷ֦ॆ ✦ Con fl ictʹؔ͢Δ৘ใ ✦ ༏ઌ౓ʹؔ͢Δ৘ใ Lrama 0.7.1
31. ✦ Bugsʹ͕͋Δจ๏ؔ܎ͷνέοτ ✦ ৽͍͠จ๏ͷఏҊ ✦ طଘͷจ๏ͷมߋ ✦ ͜͏͍͏;͏ʹॻ͚ͳ͍ͷ͔ʁ ✦ ༏ઌ౓͕ࢥ͍ͬͯͨ΋ͷͱҧ͏ ✦ จ๏Λมߋ͠ͳ͕ΒίϯϑϦΫτΛ֬ೝ͢Δ࡞ۀ Con fl ictʹؔ͢Δ৘ใ
32. ✦ ঢ়ଶ͝ͱʹͲͷτʔΫϯͰίϯϑϦΫτ͍ͯ͠Δ͔දࣔ͢ΔΑ͏ ʹͳͬͨ Con fl ictʹؔ͢Δ৘ใ(1)
33. ✦ ࣮ࡍ͸͜ͷ͘Β͍ ৭ʑͳભҠ͕͋Δͷ Ͱɺ໨ࢹͰͷ֬ೝ͸ ແཧ͕͋Δ Con fl ictʹؔ͢Δ৘ใ(1)
34. ✦ ίϯϑϦΫτ͍ͯ͠ΔτʔΫϯ͕Ͳͷঢ়ଶ͔Β͖͍ͯΔ͔දࣔ͢ ΔΑ͏ʹͳͬͨ Con fl ictʹؔ͢Δ৘ใ(2)
35. ✦ ঢ়ଶ6Ͱexprʹreduce͢Δͱɺstate 0΍state 5ͰexprʹΑΔભҠ ͕͓͖Δ ✦ ͦͷ࣍ͷτʔΫϯ͕ ‘+’ ʹͳ͍ͬͯͯɺ͜Ε͕ঢ়ଶ6ͷlookahead tokenͰ͋ΔͱΘ͔Δ Con fl ictʹؔ͢Δ৘ใ(2)
36. ✦ ࣮૷: lookahead tokenͷܭࢉͷ࠷ॳʹDirect Readू߹ΛٻΊΔ ͕ɺͦ͜Ͱ୅ΘΓʹͦͷGotoࣗ਎͔ΒͳΔू߹Λ࢖͏Α͏ʹ͢ Δ Con fl ictʹؔ͢Δ৘ใ(2)
37. ✦ ಺༰ ✦ ୈ1ষ LR parserͷ֓࿦ ✦ ୈ2ষ ޮ཰తͳLookaheadͷܭࢉ ✦ ෺ཧຊ ¥1,000 ✦ େ޷ධʂʂ ✦ ຊ೔΋ઈࢍൢചதʂ ʮޮ཰తͳLookaheadͷܭࢉʯͱ͍͑͹
38. ✦ ༏ઌ౓ఆ͕ٛͲͷঢ়ଶͰ࢖ΘΕ͍ͯΔ͔දࣔ͢ΔΑ͏ʹͳͬͨ ༏ઌ౓ʹؔ͢Δ৘ใ
39. ✦ ྫ͑͹ tASSOC (“=>”) ʹؔ͢Δ༏ઌ౓Λ௥Ճͨ͠ͱ͖ʹɺ૝ఆ ֎ͷ৔ॴͰ࢖ΘΕͯͳ͍͔ʁΛ஌Γ͍ͨ ✦ tASSOC͸Ҿ਺ͷߏจͰ΋࢖ΘΕ͍ͯΔ ༏ઌ౓ʹؔ͢Δ৘ใ
40. [Bug #21097]
41. ✦ ‘=’ ͱ “rescue” ͱ ‘in’ (ͱ “=>” ?)ͷ༏ઌ౓ͷ໰୊ ✦ Rubyʹ͓͍ͯଟ͘ͷ୅ೖ͸Ҿ਺ͷཁૉ(arg)Ͱ͋Δ ✦ pattern matchingΛargʹ͠Α͏ͱ͢Δͱແཧ͕͋Δ ✦ ͜Ε΋·ͨstmtϨϕϧͰͷΈڐՄ͢Δํ޲͕Α͍ͷͰ͸ [Bug #21097]
42. ✦ "'rescue' modi fi er”͕ࠨ݁߹ͳͷͰendless commandͱendless argΛލ͍Ͱ݁߹ଇ͕͸ͨΒͨ͘Ί ✦ ͳΔ΄Ͳʁ ✦ ΋͏গ͠ࢼߦࡨޡ͕ඞཁͦ͏ ࣮૷ͯ͠ΈΔͱ
43. ✦ RubyKaigi 2025Ͱ͸վߦΛ୊ࡐʹɺҰ෦ͷlex stateʹ͍ͭͯ৮Ε ͨ RubyKaigi 2025
44. ✦ akr͞Μ͕ઌߦݚڀʹؔͯ͠ϒϩάΛॻ͍͍ͯΔ ✦ 2023೥ͷ࿦จ͸“ઌߦ”ݚڀͳͷ͔…? lex stateͷਐḿ͸? http://www.a-k-r.org/d/2025-08.html
45. ✦ དྷ೥ʹ͸Կ͔ܗʹ͍ͨ͠… lex stateͷਐḿ͸? https://x.com/spikeolaf/status/1960354386045211099
46. ✦ parser͔Βlexerʹvalidͳlookahead tokenͷू߹Λ౉͢Ξϓϩʔν ✦ LALR͸ෳ਺ͷঢ়ଶΛmerge͢ΔͷͰຊདྷ(Canonical LRతʹ ͸)invalidͳtokenͰ΋reduceΛਐΊΔ͜ͱ͕͋Δ ✦ lexerͷ࢖༻͢Δvalid lookahead tokenʹج͍ͮͯಛఆͷঢ়ଶΛ merge͠ͳ͍Α͏ʹ͢ΔͱΑ͍ͷͰ͸? ✦ IELRͷܭࢉ଎౓͕े෼ʹૣ͘ͳͬͨΒݕূ͍ͨ͠ ✦ IELRͷܭࢉ଎౓͕े෼ʹૣ͘ͳͬͨ…!!! LALRͷ໰୊఺
47. ✦ ਐḿ ✦ ʮLR parserೖ໳ʯΛॻ͍ͨ ✦ RubyConf Taiwan 2025Ͱొஃͨ͠ ✦ pattern matching΍endless method de fi nitionΛΊ͙Δจ๏มߋ ʹ͍ͭͯߟ࡯ͨ͠ ✦ Lrama 0.7.1 ✦ Output fi leͷ֦ॆΛͨ͠ ·ͱΊ
48. ✦ ͜Ε͔Β ✦ Lrama 0.7.1ΛϦϦʔε͢Δ ✦ Lrama 0.7.2Ͱ͸Grammar fi leͷߏจղੳΛASTʹ͍ͨ͠ ✦ “จ๏Λ͍ͬͯ͡Con fl ictͱٔΕΔপ”ʹ΋ͬͱਓΛམͱͨ͢Ίͷ͍Ζ͍Ζ ✦ IELRΛ༻͍ͯ੔ཧͰ͖ͦ͏ͳ෦෼Λ੔ཧ͢Δ ✦ [Bug #21097]ͷ੔ཧ ✦ lexer stateͷ੔ཧ & lexer stateΛফ͢ ·ͱΊ