Ruby Parser progress report 2024

RubyKaigi 2024 follow up
https://rhc.connpass.com/event/320709/

リンク:
Ruby Parser開発日誌 (19) - 最高の構文木の設計 2024年版
https://yui-knk.hatenablog.com/entry/2024/08/23/113543

Transcript

1. Ruby Parser progress report 2024 August 31, 2024 in RubyKaigi 2024 follow up @yui-knk Yuichiro Kaneko
2. About me Yuichiro Kaneko yui-knk (GitHub) / spikeolaf (Twitter) Treasure Data Engineering Manager of Applications Backend CRuby committer, mainly develop parser generator and parser Lrama LALR (1) parser generator (2023, Ruby 3.3) Love LR parser
3. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan The Bison Slayer The parser monster Parser界の黎明卿 The world is now in the great age of parsers. People are setting sail into the vast sea of parsers. - RubyKaigi 2023 LT- Yuichiro Kaneko https://twitter.com/kakutani/status/1657762294431105025/ NEW !!! NEW !!!
4. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan
5. The grand strategy of Ruby Parser The grand strategy of Ruby Parser Long term goals Provide platform for LSP and other tools Provide Universal parser Keep both Ruby grammar and parser to be maintainable Solution LR parser and parser generator are the best approach for Ruby
6. Universal Parser Decouple AST from imemo Remove Object from Node Refactoring Ripper LSP Optimize Node memory management Delete parser level optimization Union to Struct (Node) User friendly node structure parse.y for Under graduate More declarative parser Ef fi cient data structure (Cactuses) Delete operation support Integration to parse.y More accurate recovery Parameterizing rules Replace hand written parser with Racc User de fi ned stack Scanner state update syntax Scannerless parser IELR RBS Error tolerance Parser Generator (Lrama) Parser ✅ ✅ ✅ ✅ ✅ ✅ ✅ 💪 💪 💪 💪 💪 💪 💪
7. Universal Parser Decouple AST from imemo Remove Object from Node Refactoring Ripper LSP Optimize Node memory management Delete parser level optimization Union to Struct (Node) User friendly node structure parse.y for Under graduate More declarative parser Ef fi cient data structure (Cactuses) Delete operation support Integration to parse.y More accurate recovery Parameterizing rules Replace hand written parser with Racc User de fi ned stack Scanner state update syntax Scannerless parser IELR RBS Error tolerance Parser Generator (Lrama) Parser ✅ ✅ ✅ ✅ ✅ ✅ ✅ 💪 💪 💪 💪 💪 💪 💪
8. Ruby 3.4 Universal Parser Fine-grained Error Tolerance parser Compatibility layer for Prism interfaces
9. yui-knk/ast_to_prism https://github.com/yui-knk/ast_to_prism
10. 構文木とは !? طଘͷߏจ໦ͱͲͷล͕ҧ͏ͷͩΖ͏ ? parser gem͸Ͳ͏͍͚ͯͨͬ͠ ?? ߏจ໦ʹਅཧͷΑ͏ͳ΋ͷ͸ͳ͍ͷ͔ ???
11. 構文木の真理を求めて ϢʔεέʔεΛूΊΔ ઃܭΛߟ͑Δ ઃܭ͕ϢʔεέʔεΛͲͷ͘Β͍ΧόʔͰ͖͍ͯΔ͔ධՁ͢Δ
12. https://yui-knk.hatenablog.com/entry/2024/08/23/113543 3万字越え。本にできるのでは? https://x.com/inao/status/1827130119665938491
13. https://yui-knk.hatenablog.com/entry/2024/08/23/113543
14. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan 10分でわかる 構文木のユースケースと設計
15. 構文木のユースケース ίʔυΛ࣮ߦ͍ͨ͠ compile.c ܕγεςϜ ίʔυΛղੳ͍ͨ͠ LSP (ruby-lsp) Linter & Code Formatter (RuboCop)
16. コードを実行したい ಉ͡ҙຯͷNode͸ͻͱͭʹ·ͱ·͍ͬͯͯ΄͍͠ if, ޙஔif, ࡾ߲ԋࢉࢠͷif => ಉ͡Node ߏจ໦͸্͔ΒԼ΁ḷΓ͍ͨ ίʔυͷҙຯʹؔ܎ͷͳ͍ཁૉ͸ෆཁ ۭനͱ͔վߦͱ͔Χοί͸ߏจ໦ʹ͸͍Βͳ͍ ͱݴ͍ͭͭඞཁʹԠ֦ͯ͡ு͖ͯͨ͠ UNLESS_NODEͷ௥Ճ (Ruby 2.5͔Β) ৄࡉͳҐஔ৘ใͷ௥Ճ (Ruby 2.5͔Β)
17. コードを解析したい τʔΫϯͷ৘ใ͕΄͍͠ (Syntax Highlight) ίϝϯτΛղੳ͍ͨ͠ (LSP DocumentLink) ߏจ໦Ͱࢠ͔Β਌ΛḷΓ͍ͨ (LSP SelectionRange) ίʔυΛॻ͖׵͍͑ͨ (LSP & Code Formatter)
18. 子から親を辿りたい LSP SelectionRange ίʔυͷબ୒ൣғΛม͑Δػೳ
19. 子から親を辿りたい NODE_CLASS NODE_DEF obj.m1
20. コードを書き換えたい Style::IfInsideElse ͱ͍͏Cop͕͋Δ ωετͨ͠ifͷ಺ଆʹ͋ΔifΛ֎ଆͷelsifʹ͢Δ https://github.com/rubocop/rubocop/blob/v1.65.1/lib/rubocop/cop/style/if_inside_else.rb#L10-L29
21. RuboCop の書き換え if condition_a action_a else if condition_b action_b else action_c end end if condition_a action_a elsif condition_b if condition_b action_b else action_c end end if condition_a action_a elsif condition_b action_b action_b else action_c end end if condition_a action_a elsif condition_b action_b action_b else action_c end if condition_a action_a elsif condition_b action_b else action_c end 1. elseΛelsif΁ 2. if condition_b Λ࡟আ 3. ༨෼ͳendΛ ࡟আ 4.ॏෳ͍ͯ͠Δaction_b Λ࡟আ
22. TreeRewriterの問題点 #1 ࣮૷͕ෳࡶ TreeRewriter͕௚઀จࣈྻΛॻ͖׵͑ΔΘ͚Ͱ͸ͳ͍ TreeRewriter::ActionͷΠϯελϯεΛͭͬͯ͘ɺ࠷ޙʹҰؾʹมߋΛՃ͑ Δ Action. :replace (2, 0)-(2, 4) “elsif condition_b” Action. :replace (3, 2)-(3, 16) “action_b” Action. :replace (7, 0)-(7, 6) “” Action. :replace (4, 0)-(4, 13) “”
23. Actionを用いる理由 #1 จࣈྻΛ౎౓ॻ͖׵͑Δͱίετ͕ߴ͍͔Β ͲͪΒͷέʔε΋elseҎ߱ͷจࣈྻΛҠಈ(ίϐʔ)͠ͳ͍ͱ͍ ͚ͳ͍ if condition_a action_a else action_b end if condition_a action_a action_b end elseΛ࡟আ if condition_a action_a else action_b end if condition_a action_a elsif action_b end elsifʹஔ׵
24. Actionを用いる理由 #2 จࣈྻΛ௚઀ॻ͖׵͑ΔͱଞͷϊʔυʹӨڹ͢Δ͔Β if condition_a action_a else action_b end Parser::Source::Bu ff er if condition_a action_a action_b end Parser::Source::Bu ff er NODE_VCALL action_b Range (3, 2)-(3, 10) elseΛ࡟আ
25. ॻ͖׵͑࣌ͷૢ࡞͕൥ࡶ ֤ૢ࡞ͰࠓͲͷΑ͏ͳঢ়ଶ͔ཧղ͠ͳ͍ͱ͍͚ͳ͍ TreeRewriterの問題点 #2 if condition_a action_a else if condition_b action_b else action_c end end if condition_a action_a elsif condition_b if condition_b action_b else action_c end end if condition_a action_a elsif condition_b action_b action_b else action_c end end if condition_a action_a elsif condition_b action_b action_b else action_c end if condition_a action_a elsif condition_b action_b else action_c end 1. elseΛelsif΁ 2. if condition_b Λ࡟আ 3. ༨෼ͳendΛ ࡟আ 4.ॏෳ͍ͯ͠Δaction_b Λ࡟আ
26. 構文木を書き換えよう! ߏจ໦ͷߏ଄Λੜ͔͍ͨ͠ NODE_IFΛNODE_ELSIFʹͯ͠ɺNODE_ELSEΛ͚͢ NODE_IF condition_a action_a NODE_ELSE NODE_IF condition_b action_b NODE_ELSE action_c NODE_IF condition_a action_a NODE_ELSIF condition_b action_b action_c
27. 構文木からソースコードへ ߏจ໦Λॻ͖׵͑ͨΒ͋ͱ͸ιʔείʔυΛੜ੒͢Δ͚ͩ ۭന΍վߦͳͲASTͰ͸͚͍ܽͯΔ৘ใ͕͋Δ NODE_IF condition_a action_a NODE_ELSIF condition_b action_b action_c
28. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan 簡単だとおもった？
29. 問 1. 位置情報を与えよ NodeͷΠϯελϯε࡞੒࣌ʹRangeΛ౉͢ඞཁ͕͋Δ փ৭ͷ৽͍͠ίʔυͷҐஔ৘ใΛܭࢉͤΑ if condition_a action_a else if condition_b action_b else action_c end end if condition_a action_a elsif condition_b action_b else action_c end
30. 問 1. 位置情報を与えよ ੜ੒͞ΕΔίʔυΛΠϝʔδ͠ͳ͕ΒҐஔ৘ใΛߟ͑ͳ͍ͱ͍ ͚ͳ͍ͷͰ൥ࡶ ͔ͤͬ͘ߏจ໦ͷॻ͖׵͑ʹͨ͠ͷʹʂ if condition_a action_a else if condition_b action_b else action_c end end if condition_a action_a elsif condition_b action_b else action_c end ։࢝Ґஔ͸ else ͷҐஔ ऴྃҐஔ(ߦ)͸ “action_cͷߦ - 1” ऴྃҐஔ(ΧϥϜ)͸ “action_cͷ຤ඌ - 2”
31. 問 2. 他のNodeの位置を修正せよ มߋͨ͠ߏจ໦Ҏ߱ͷશͯͷϊʔυͷҐஔ৘ใ͕มԽ͢Δ ؔ৺ͷ͋ΔIF NODEͷܑఋཁૉʹ΋Өڹ͢Δ NODE_CLASS condition_a action_a NODE_ELSIF NODE_DEF NODE_IF NODE_DEF expr1 expr2 expr2 expr2 condition_b action_b action_c ߋ৽!!
32. Migration Tool ίʔυͷॻ͖׵͑͸LSP΍Code Formatter͚ͩͷ࢓ࣄͰ͸ͳ͍ TranspecͳͲͷmigration toolͰ΋ίʔυͷॻ͖׵͑Λߦ͏ඞཁ͕ ͋Δ http://yujinakayama.me/transpec/
33. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan 構文木の設計と実装
34. 具象構文木: コードを復元する ۩৅ߏจ໦ (Concrete Syntax Tree: CST) ׅހͳͲASTͰ͸ࣦΘΕͯ͠·͏৘ใ΋࢒ͨ͠ߏจ໦ AST͕ҙຯ(Semantics)ʹ஫໨͍ͯ͠Δͷʹରͯ͠ɺCST͸ߏจ(Syntax)ʹ஫ ໨͍ͯ͠Δ ಛ௃ τʔΫϯΛද͢σʔλߏ଄Λಋೖ͢Δ lexerͰམͱͯ͠͠·͏৘ใΛτʔΫϯʹඥ͚ͮΔ ϊʔυ͕τʔΫϯΛ࣋ͭΑ͏ʹ͢Δ ߏจ໦͔Β΋ͱͷίʔυΛ׬શʹ෮ݩͰ͖ΔΑ͏ʹ͢Δ
35. 具象構文木: コードを復元する ۩৅ߏจ໦ (Concrete Syntax Tree: CST) ׅހͳͲASTͰ͸ࣦΘΕͯ͠·͏৘ใ΋࢒ͨ͠ߏจ໦ AST͕ҙຯ(Semantics)ʹ஫໨͍ͯ͠Δͷʹରͯ͠ɺCST͸ߏจ(Syntax)ʹ஫ ໨͍ͯ͠Δ ಛ௃ τʔΫϯΛද͢σʔλߏ଄Λಋೖ͢Δ lexerͰམͱͯ͠͠·͏৘ใΛτʔΫϯʹඥ͚ͮΔ ϊʔυ͕τʔΫϯΛ࣋ͭΑ͏ʹ͢Δ ߏจ໦͔Β΋ͱͷίʔυΛ׬શʹ෮ݩͰ͖ΔΑ͏ʹ͢Δ ιʔείʔυͷ׬શͳ৘ใΛ࣋ͬͨߏจ໦
36. Trivia LexerͰམͱͯ͠͠·͏৘ใΛTriviaͱΑͿ ۭന, վߦ, ίϝϯτͳͲ͕Trivia Trivia (comment) Trivia (spaces) Trivia (new line)
37. Node, Token and Trivia τʔΫϯ͸લޙʹTriviaΛ΋ͭ ϊʔυ͸ϊʔυ / τʔΫϯΛ΋ͭ NODE_IF IF cond action_a END Token NODE ຌྫ space (1) NL (1) + space (2) NL (1) Trivia
38. 構文木からコードへ ਂ͞༏ઌ୳ࡧͭͭ͠จࣈྻΛdump͢Δͱ΋ͱͷίʔυʹͳΔ Token NODE ຌྫ NODE_IF IF cond action_a END space (1) NL (1) + space (2) NL (1) Trivia
39. Red Green Tree: 編集容易な木 C# (Roslyn)ͷൃ໌ Swift (SwiftSyntax)΍ rust-analyzer (LSP)Ͱ΋࢖ΘΕ͍ͯΔ ߏจ໦ΛRed NodeͱGreen Nodeͱ͍͏2ͭͷσʔλߏ଄Ͱද ݱ͢Δ swift-syntaxΛಡ΋͏ʂ https://github.com/swiftlang/swift-syntax
40. Red Green Tree Green Node ࢠ΁ͷࢀরΛ΋ͭ ࣗ਎ͷ෯(width)Λ΋ͭ Red Node ਌΁ͷࢀরΛ΋ͭ Ґஔ৘ใ(offset)Λ΋ͭ Token Green NODE ຌྫ Red NODE NODE_IF width: 90 IF width: 3 NODE_IF width: 56 condition_a width: 11 action_a width: 11 NODE_ELSE width: 61 END width: 4 ELSE width: 5 NODE_IF o ff set: 0 NODE_ELSE o ff set: 25 NODE_IF o ff set: 30
41. Offsetを持っている場合 มԽͷ͋ͬͨϊʔυ/τʔΫϯͷޙଓͷ͢΂ͯͷཁૉʹӨڹ͢Δ ӨڹΛड͚Δཁૉ NODE_IF o ff set: 0 IF o ff set: 0 NODE_IF o ff set: 30 condition_a o ff set: 3 action_a o ff set: 14 NODE_ELSE o ff set: 25 END o ff set: 86 ELSE o ff set: 25 IF o ff set: 30 condition_b o ff set: 35 action_b o ff set: 46 -> 47 END o ff set: 79 NODE_ELSE o ff set: 59 action_c o ff set: 66 ELSE o ff set: 59 ߋ৽!!
42. 幅を持っている場合 มԽͷ͋ͬͨϊʔυ/τʔΫϯͷ਌ཁૉʹӨڹ͕ݶΒΕΔ ࢠཁૉ͔ΒܭࢉՄೳ ӨڹΛड͚Δཁૉ NODE_IF width: 90 -> 91 IF width: 3 NODE_IF width: 56 -> 57 condition_a width: 11 action_a width: 11 NODE_ELSE width: 61 -> 62 END width: 4 ELSE width: 5 IF width: 5 condition_b width: 11 action_b width: 13 -> 14 END width: 7 NODE_ELSE width: 20 action_c width: 13 ELSE width: 7 ߋ৽!!
43. まとめ ίʔυΛ࣮ߦ͍ͨ͠: compile.c ίʔυΛղੳ͍ͨ͠: LSP, Linter & Code Formatter ςΩετϕʔεͰͷίʔυͷॻ͖׵͕͑೉͍͠ ߏจ໦ϕʔεͷίʔυॻ͖׵͑΁ ߏจ໦͔ΒίʔυΛ෮ݩ͍ͨ͠ ۩৅ߏจ໦ !! ߏจ໦ॻ͖׵͑ͷӨڹൣғΛڱ͍ͨ͘͠ Red Green Tree !!
44. May 15th - 17th, 2024 NAHA CULTURAL ARTS THEATER NAHArt, Okinawa, Japan 進捗が出た結果、 やることが増えた!!
45. 話したいことはまだまだある Red Green Tree͸immutableͰpersistentͳσʔλߏ଄ มߋલޙͷߏจ໦ΛͦΕͧΕอ࣋͢Δ͜ͱͰࠩ෼ΛදࣔͰ͖Δ ΑΓޮ཰తʹߏจ໦Λॻ͖׵͑ΔͨΊʹ൚༻ͷRewriterΛఏڙ͢Δͷ͕Α͍ Open questions ηϛίϩϯɺվߦͳͲ͸trivia͔? ώΞυΩϡϝϯτΛ͏·͘ѻ͑Δ͔? LexerͱparserͷΠϯλʔϑΣΠε͸ࠓޙ΋෼཭͞Ε͍ͯΔͷ͕Α͍ͷ͔? ॻ͖׵͑ޙͷߏจ໦͕valid͔νΣοΫ͢Δ͜ͱ͸Ͱ͖Δ͔?