Prism

This deck presented the prism parser as Ruby World 2023.

Transcript

1. Kevin Newton Prism Ruby World 2023
2. What is prism?
3. Overview
4. Overview Prism is a new Ruby parser built from scratch with the following goals:
5. Overview Prism is a new Ruby parser built from scratch with the following goals: • Compatibility - must match CRuby semantics exactly in order to merge
6. Overview Prism is a new Ruby parser built from scratch with the following goals: • Compatibility - must match CRuby semantics exactly in order to merge • Maintainability - must be as maintainable as possible
7. Overview Prism is a new Ruby parser built from scratch with the following goals: • Compatibility - must match CRuby semantics exactly in order to merge • Maintainability - must be as maintainable as possible • Error tolerance - must be able to recover from errors and continue parsing
8. Overview Prism is a new Ruby parser built from scratch with the following goals: • Compatibility - must match CRuby semantics exactly in order to merge • Maintainability - must be as maintainable as possible • Error tolerance - must be able to recover from errors and continue parsing • Portability - must be able to be run from any ruby platform
9. Overview Prism is a new Ruby parser built from scratch with the following goals: • Compatibility - must match CRuby semantics exactly in order to merge • Maintainability - must be as maintainable as possible • Error tolerance - must be able to recover from errors and continue parsing • Portability - must be able to be run from any ruby platform • Performance - must be as fast and memory e ffi cient as possible
10. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01
11. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit
12. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team
13. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit
14. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit 👶
15. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit 👶
16. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit 👶
17. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit 👶
18. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit 👶
19. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit 👶
20. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit Rename to prism 👶
21. How was it built?
22. Design
23. Design Approach
24. • Comparative analysis with existing trees Approach Design
25. • Comparative analysis with existing trees • "Semantic" versus "syntactic" trees Approach Design
26. • Comparative analysis with existing trees • "Semantic" versus "syntactic" trees • Provide for all consumers Approach Design
27. Implementations Design
28. • Make compilation as simple as possible Implementations Design
29. • Make compilation as simple as possible • Never have to check child to compile current node Implementations Design
30. • Make compilation as simple as possible • Never have to check child to compile current node • Cover as much common functionality as possible Implementations Design
31. Tooling Design
32. • Named fi elds and known types Tooling Design
33. • Named fi elds and known types • Nodes with enough information to round-trip Tooling Design
34. • Named fi elds and known types • Nodes with enough information to round-trip • Location information for as much as possible Tooling Design
35. Maintenance Design
36. • Template as much as possible Maintenance Design
37. • Template as much as possible • Provide as much documentation as possible Maintenance Design
38. • Template as much as possible • Provide as much documentation as possible • Build up as big of a test suite as possible Maintenance Design
39. Challenges
40. Challenges • Escaping - required to properly fi nd string-like terminators
41. Challenges • Escaping - required to properly fi nd string-like terminators • Regular expressions - required because of named capture groups
42. Challenges • Escaping - required to properly fi nd string-like terminators • Regular expressions - required because of named capture groups • Encoding support - required to properly fi nd string-like terminators
43. Challenges • Escaping - required to properly fi nd string-like terminators • Regular expressions - required because of named capture groups • Encoding support - required to properly fi nd string-like terminators • Heredocs - very complicated semantics, can be almost anywhere in source
44. Testing
45. Testing • Lexer testing - testing against ripper lex
46. Testing • Lexer testing - testing against ripper lex • Snapshot testing - protect from regressions, check parser structure
47. Testing • Lexer testing - testing against ripper lex • Snapshot testing - protect from regressions, check parser structure • Brute-force testing - all escape sequences in all contexts
48. Testing • Lexer testing - testing against ripper lex • Snapshot testing - protect from regressions, check parser structure • Brute-force testing - all escape sequences in all contexts • Fuzzing - random input, check for crashes
49. Testing • Lexer testing - testing against ripper lex • Snapshot testing - protect from regressions, check parser structure • Brute-force testing - all escape sequences in all contexts • Fuzzing - random input, check for crashes • Grammar-based fuzzing - random input, check for crashes and parse errors
50. Integrations
51. Integrations • C - CRuby, Sorbet, Artichoke, Natalie, C/C++/Rust/Zig tooling
52. Integrations • C - CRuby, Sorbet, Artichoke, Natalie, C/C++/Rust/Zig tooling • Ruby - Natalie, tooling
53. Integrations • C - CRuby, Sorbet, Artichoke, Natalie, C/C++/Rust/Zig tooling • Ruby - Natalie, tooling • Serialization - JRuby, Tru ffl eRuby, JavaScript tooling
54. Why build this?
55. Fractured ecosystem Ruby parsers
56. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y
57. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y
58. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser
59. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby
60. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet
61. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser
62. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser
63. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby
64. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby • oracle/tru ff l eruby
65. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby
66. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby
67. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby • natalie-lang/natalie
68. Fractured ecosystem Ruby parsers • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
69. Fractured ecosystem Ruby syntax trees
70. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree
71. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP}
72. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree
73. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser
74. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby
75. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet
76. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast
77. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor
78. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast
79. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e
80. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter
81. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby
82. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie
84. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie
85. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
86. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
87. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
88. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
89. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
90. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
91. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
92. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
93. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
94. Fractured ecosystem Ruby syntax trees • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie • ruby/prism
95. Fractured ecosystem Ruby syntax trees • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
96. Fractured ecosystem Ruby syntax trees • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
97. Fractured ecosystem Ruby syntax trees • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
98. ecosystem Ruby syntax trees • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
99. Separate understandings
100. Separate understandings • Every tool represents Ruby di ff erently
101. Separate understandings • Every tool represents Ruby di ff erently • Every tool requires contributors to relearn
102. Separate understandings • Every tool represents Ruby di ff erently • Every tool requires contributors to relearn • None of this knowledge translates to contributing to CRuby
103. Maintenance burden
104. Maintenance burden • Every parser has to update every time new syntax is introduced
105. Maintenance burden • Every parser has to update every time new syntax is introduced • Very minor changes have very major consequences
106. Maintenance burden • Every parser has to update every time new syntax is introduced • Very minor changes have very major consequences • Helping Ruby can inadvertently hurt Ruby tooling
107. What is the future of prism?
108. The future of prism
109. The future of prism • Improved error tolerance and messaging
110. The future of prism • Improved error tolerance and messaging • Performance enhancements (SIMD instructions, arena allocation, etc.)
111. The future of prism • Improved error tolerance and messaging • Performance enhancements (SIMD instructions, arena allocation, etc.) • Multi-version support
112. The future of prism • Improved error tolerance and messaging • Performance enhancements (SIMD instructions, arena allocation, etc.) • Multi-version support • Further integrations into the ecosystem
113. The future of prism • Improved error tolerance and messaging • Performance enhancements (SIMD instructions, arena allocation, etc.) • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools
114. The future of prism • Improved error tolerance and messaging • Performance enhancements (SIMD instructions, arena allocation, etc.) • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools • Contributor community around a single tool
115. Thank you!