Ruby Parser progress report

RubyKaigi 2023 follow up
https://rhc.connpass.com/event/288535/

ruby/lrama
https://github.com/ruby/lrama

RubyKaigi 2023
https://rubykaigi.org/2023/presentations/spikeolaf.html

Transcript

1. Ruby Parser progress report August 19, 2023 @yui-knk Yuichiro Kaneko
2. The Bison Slayer https://twitter.com/kakutani/status/1657762294431105025
3. About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf (Twitter) • The author of ruby/lrama LALR parser generator • CRuby committer, mainly develop parser related features • Code positions to RNode (2018, Ruby 2.6) • Coverage • Error reporting • RubyVM::AbstractSyntaxTree (2018, Ruby 2.6) • keep_tokens & error_tolerant option (2022, Ruby 3.2)
4. Ruby Method Karuta https://twitter.com/spikeolaf/status/1658104271580045312
5. Parser generator • Lrama creates “parse.c” from “parse.y” • Parser generator creates parser from DSL fi le parse.y parse.c Lrama
6. Lrama v0.5.4 released 🎉
7. Brief history of release • v0.5.4. Counterexamples @yui-knk • v0.5.3. Merge error_recovery branch by @alitaso345 • v0.5.2. Named references by @junk0612 • v0.5.1. Add RBS and Steep check by @Little-Rubyist • v0.5.0. Add stdin mode by @nobu • v0.4.0. First ruby/ruby migrated version (<- RubyKaigi 2023)
8. 15 Contributors !!! • Many of them are not CRuby committers • For some of them, Lrama is a fi rst OSS project they send PR :)
9. What’s the problem? • Error-tolerant parser • For LSP • Universal Parser • It’s tough task to write your own parser for Ruby • Maintainability • “parse.y” seems dif fi cult
10. Error-tolerant parser
11. Progress • error_recovery branch has been merged https://github.com/ruby/lrama/pull/44
12. Interface is implemented • Runtime con fi gures • YYERROR_RECOVERY_ENABLED: Enable/Disable the feature • YYMAXREPAIR: How deep search candidates for recovery https://github.com/ruby/lrama/pull/74
13. What’s next? • Integrate it with Ruby • I’m now fi ghting with memory related bugs…
14. Universal Parser
15. Universal Parser mode • Development has been started https://github.com/ruby/ruby/pull/7927
16. Progress • char(s) functions are reimplemented https://github.com/ruby/ruby/pull/8044
17. What’s next? • Replacing Ruby objects in AST node with Strings
18. Maintainability
19. Why LR parser is the best • BNF is good interface • BNF is common knowledge https://dev.mysql.com/doc/refman/8.0/en/select.html https://datatracker.ietf.org/doc/html/rfc3986
20. Why LR parser is the best • BNF is good interface • BNF is common knowledge • Bene fi ts derived from LR parser • Auto detection of grammar rule con fl icts • Error-tolerant parser without detailed grammar knowledge
21. Why “parse.y” is dif fi cult? 1. “parse.y” is large (about 15,000 lines) 2. LALR is dif fi cult, e.g. S/R con fl ict, R/R con fl ict 3. Bison doesn’t provide syntax sugar like option, list 4. It’s a mixture of parser and ripper 5. Parser and Lexer are tightly-coupled
22. S/R con fl ict, R/R con fl ict • Counterexamples is implemented from v0.5.4
23. How to resolve con fl ict • Add %prec to resolve con fl ict
24. Not enough • We can show how to resolve it (sometimes).
25. Still not enough • There might be a better syntax to direct how to solve the con fl ict
26. How to support new syntax • kaneko: How do you come up with new syntax? • nobu: I remember BNF of Ruby then it’s easy to fi nd new syntax idea. Once come up with an idea, I check it by changing “parse.y” • kaneko: ?? • nobu: ??
27. More easy approach • (New) syntax for reducing indent of nested module • Finding syntax which is not valid now • “module X in Consts”
28. module’s case • There is no “`in`” after M
29. class’s case • There is “`in`” after “class C < D”
30. IDE • Report fi le is primitive and static • More interactive tool is better, e.g. irb commands, LSP
31. Maintainability -> Language Designer’s Developer Experience
32. Only primitive syntax • https://github.com/ruby/racc/pull/222 • Menhir provides “Parameterizing rules”
33. mixture of parser and ripper • One of complexities of Ripper is that we use only one semantic value stack to mange both (1) semantic value and (2) returned value of callback methods
34. mixture of parser and ripper • Implement User de fi ned stack for Lrama • Can manage VALUE, the result of callback method call, on another stack
35. Hand written parser -> Racc https://github.com/ruby/lrama/pull/62
36. Parser and Lexer are tightly-coupled • Today I focus on “enum lex_state_e state”. • “1 || 2”. “||” is tOROP • “a do || end”. “||” is ‘|’ and ‘|’ • Lexer checks EXPR_BEG to decide which token generated • I have been wondering why such communication is needed, because parser knows current situation. Parser knows it never accept “||” after “a do”.
37. Pseudo-Scannerless Minimal LR(1) • Joel Denny “PSLR(1): Pseudo-Scannerless Minimal LR(1) for the Deterministic Parsing of Composite Languages”, 2010 • “vector<list<string>> v;”. “>>” is an example of this paper • > Nevertheless, traditional scanner and parser generators attempt to generate loosely coupled scanners and parsers, so the user must maintain these tightly coupled scanner and parser speci fi cations separately but consistently. • > Scanner and parser speci fi cations would be signi fi cantly more maintainable if all sub-language transitions were instead computed from a grammar by a parser generator and recognized automatically by the scanner using the parser’s stack.
38. Lex state • State of the lexer by which lexer determines which token type is generated • Parser updates lex state • Lexer updates and uses lex state • Example: keyword_if & modi fi er_if
39. Move lexer logic to parser • Only parser updates lex state • Lexer has very few logic
40. Automaton & Automaton • LR parser manages automaton • Lex state is also automaton • automaton + automaton = automaton
41. In the future • Lexer (C code) is a source of Ruby speci fi c knowledge. Then developers need to learn the knowledge • Scannerless parser can remove such Ruby speci fi c knowledge. • Once scannerless parser is covered by textbooks of computer science, developers can manage “parse.y” by just textbooks knowledge
42. Why “parse.y” is dif fi cult? 1. “parse.y” is large (about 15,000 lines) 2. LALR is dif fi cult, e.g. S/R con fl ict, R/R con fl ict • Counterexamples • More hints for Con fl ict Resolution & new syntax discussion 3. Bison doesn’t provide syntax sugar like option, list • option, list and so on 4. It’s a mixture of parser and ripper • User de fi ned stack 5. Parser and Lexer are tightly-coupled • Moving lexer logic into parser • Scannerless LR • IELR(1) • Lex state management in parser
43. Summary • Error-tolerant parser • Next action: Integration with Ruby • Universal Parser • Next action: Remove Ruby object from AST node
44. • Maintainability • Ruby speci fi c knowledge from parse.y • Scannerless LR • Lex state management by parser • Improve Language Designer’s Developer Experience • Fine grained con fl ict resolution • Guide for how to resolve con fl ict Summary
45. Thank you!!
46. References • Yuichiro Kaneko “The future vision of Ruby Parser”, 2023 https://rubykaigi.org/ 2023/presentations/spikeolaf.html • Llama LALR parser generator https://github.com/ruby/lrama • Chinawat Isradisaikul and Andrew C. Myers. “Finding Counterexamples from Parsing Con fl icts”, 2015 https://www.cs.cornell.edu/andru/papers/cupex/ cupex.pdf • Menhir Reference Manual (version 20230608) “5.2 Parameterizing rules” http:// gallium.inria.fr/~fpottier/menhir/manual.html#sec32 • Joel Denny “PSLR(1): Pseudo-Scannerless Minimal LR(1) for the Deterministic Parsing of Composite Languages”, 2010 https://tigerprints.clemson.edu/cgi/ viewcontent.cgi?article=1519&context=all_dissertations
47. References • Joel E. Denny “The IELR(1) algorithm for generating minimal LR(1) parser tables for non-LR(1) grammars with con fl ict resolution”, 2010 https://core.ac.uk/ download/pdf/82047055.pdf