The Future of Understanding Ruby

For decades, the Ruby community has been fractured in the way that it parses and understands Ruby code. After countless tools have been developed and person-hours have been invested, we still don't have a common language for understanding Ruby code. No longer! Starting in Ruby 3.3, we will have a single API for parsing and understanding Ruby code. This talk will cover the history of how we got here, what is getting built today, and what you can expect from this exciting future.

Transcript

3. /1+2/
4. match /1+2/
5. match /1+2 /
6. match /1+2 #/
7. /\d+/ =~ "3" match /1+2 #/
8. /(?<match>\d+)/ =~ "3" match /1+2 #/
9. * = <<~EOF, /(?<match>\d+)/ =~ "3" matchy match EOF match /1+2 #/
10. * = <<~EOF, /(?<m\ matchy match EOF atch>\d+)/ =~ "3" match /1+2 #/
11. What does this code do?
12. What does this code do?
13. What does this code do? How would you fi nd out?
14. Dependencies
16. Applications
17. External libraries Applications
18. Standard libraries External libraries Applications
19. Byte code/IR Standard libraries External libraries Applications
20. Compiler Byte code/IR Standard libraries External libraries Applications
21. Parser Compiler Byte code/IR Standard libraries External libraries Applications
22. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
23. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
24. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
25. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications def ❨╯°□°❩╯︵┻━┻ puts "Calm down, yo." end
26. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications def ❨╯°□°❩╯︵┻━┻ puts "Calm down, yo." end
27. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications [[KEYWORD_DEF, "def"], [IDENTIFIER, "❨╯°□°❩╯︵┻━┻"], [NEWLINE, "\n"], [IDENTIFIER, "puts"], [STRING_BEGIN, "\""], [STRING_CONTENT, "Calm down, yo."], [STRING_END, "\""], [NEWLINE, "\n"], [KEYWORD_END, "end"], [NEWLINE, "\n"], [EOF, ""]]
28. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
29. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
30. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
31. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself
32. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself
33. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.”
34. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.”
35. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts
36. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts
37. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts leave
38. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications putself putstring "Calm down, yo.” send :puts leave
39. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications putself putstring "Calm down, yo.” send :puts leave
40. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
41. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
42. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
43. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
44. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
45. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
46. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
47. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
48. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications Sidekiq.❨╯°□°❩╯︵┻━┻
49. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
50. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications
51. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y
52. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c
53. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV
54. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib
55. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org
56. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org
57. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org
58. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y
59. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java
60. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR
61. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib
62. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib rubygems.org*
63. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib rubygems.org* Applications*
64. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org
65. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y
66. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java
67. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e
68. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib
69. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib rubygems.org*
70. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib rubygems.org* Applications*
71. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org
72. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser
73. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb
74. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb
75. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib
76. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib rubygems.org*
77. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib rubygems.org* Applications*
78. Ruby source
79. Ruby source parse.y compile.c YARV
80. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR
81. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e
82. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
83. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
84. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
85. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
86. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
87. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
88. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
89. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
90. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
91. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
92. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
93. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
94. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
95. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
96. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
97. Ruby source
99. Ruby source
100. Ruby source Parser
101. Ruby source Parser Compiler
102. Ruby source Parser Compiler Byte code/IR
103. Ruby source Parser Compiler Byte code/IR
104. Ruby source Parser Compiler Byte code/IR parse.y
105. Ruby source Parser Compiler Byte code/IR parse.y compile.c
106. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
107. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
108. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV parser gem
109. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV parser gem rubocop
110. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
111. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV ruby_parser gem
112. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV ruby_parser gem fasterer
113. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
114. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV tree-sitter-ruby
115. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV tree-sitter-ruby vscode-ruby
116. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
117. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV ripper
118. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV ripper syntax_tree
119. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
120. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV
121. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV RubyVM::ISeq
122. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV RubyVM::ISeq typeprof
123. Ruby source Parser Compiler Byte code/IR parse.y compile.c YARV RubyVM::ISeq tenderjit
124. Ruby source
125. Ruby source parser gem
126. Ruby source parser gem ruby_parser gem
127. Ruby source parser gem ruby_parser gem tree-sitter-ruby
128. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper
129. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop
130. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph
131. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser
132. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next
133. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard
134. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fasterer
135. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fasterer
136. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay fasterer
137. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer
138. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby
139. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree
140. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt
141. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb
142. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo
143. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo prettier
144. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo prettier
145. Ruby source
147. Di ff erent representations lead to di ff erent assumptions.
148. Different representations
149. Different representations • Every tool represents Ruby di ff erently
150. Different representations • Every tool represents Ruby di ff erently • Every tool requires contributors to relearn
151. Different representations • Every tool represents Ruby di ff erently • Every tool requires contributors to relearn • None of this knowledge translates to contributing to CRuby
152. Maintaining a Ruby parser is a massive maintenance burden.
153. Maintenance burden
154. Maintenance burden • Every parser has to update every time new syntax is introduced
155. Maintenance burden • Every parser has to update every time new syntax is introduced • Very minor changes have very major consequences
156. Maintenance burden • Every parser has to update every time new syntax is introduced • Very minor changes have very major consequences • Helping Ruby can inadvertently hurt the Ruby ecosystem
157. Fractured ecosystem
158. Fractured ecosystem
159. Fractured ecosystem Ruby p a rsers
160. Fractured ecosystem • ruby/ruby/parse.y Ruby p a rsers
161. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y Ruby p a rsers
162. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser Ruby p a rsers
163. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby Ruby p a rsers
164. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet Ruby p a rsers
165. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser Ruby p a rsers
166. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers
167. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby
168. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby
169. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby
170. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby
171. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby • natalie-lang/natalie_parser
172. Fractured ecosystem Synt a x trees
173. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree Synt a x trees
174. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} Synt a x trees
175. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree Synt a x trees
176. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser Synt a x trees
177. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby Synt a x trees
178. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet Synt a x trees
179. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees
180. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor
181. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast
182. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e
183. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter
184. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby
185. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser
187. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser
188. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
189. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
190. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
191. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
192. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
193. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
194. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
195. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
196. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
197. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree • lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
198. Fractured ecosystem • lib-ruby-parser/lib-ruby-parser • mruby/mruby Synt a x trees • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
199. Fractured ecosystem • ruby/prism • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby Synt a x trees
200. Fractured ecosystem • ruby/prism • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby Synt a x trees
201. ecosystem • ruby/prism • lib-ruby-parser/lib-ruby-parser • mruby/mruby • tree-sitter/tree-sitter • sisshiki1969/ruruby Synt a x trees
202. Prism
203. Prism
204. Prism • Compatibility
205. Prism • Compatibility • Maintainability
206. Prism • Compatibility • Maintainability • Error tolerance
207. Prism • Compatibility • Maintainability • Error tolerance • Portability
208. Prism • Compatibility • Maintainability • Error tolerance • Portability • Performance
209. Timeline
210. Timeline
211. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01
212. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit
213. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team
214. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit
215. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit 👶
216. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit 👶
217. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit 👶
218. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit 👶
219. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit 👶
220. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit 👶
221. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit Rename to prism 👶
222. How was it built?
223. Design
224. Design Appro a ch
225. Design • Comparative analysis with existing trees Appro a ch
226. Design • Comparative analysis with existing trees • "Semantic" versus "syntactic" trees Appro a ch
227. Design • Comparative analysis with existing trees • "Semantic" versus "syntactic" trees • Provide enough value for all consumers Appro a ch
228. Design Implement a tions
229. Design Implement a tions • Make compilation as simple as possible
230. Design Implement a tions • Make compilation as simple as possible • Never have to check child to compile current node
231. Design Implement a tions • Make compilation as simple as possible • Never have to check child to compile current node • Cover as much common functionality as possible
232. Design Tooling
233. Design Tooling • Named fi elds and known types
234. Design Tooling • Named fi elds and known types • Nodes with enough information to round-trip
235. Design Tooling • Named fi elds and known types • Nodes with enough information to round-trip • Location information for as much as possible
236. Design M a inten a nce
237. Design M a inten a nce • Template as much as possible
238. Design M a inten a nce • Template as much as possible • Provide as much documentation as possible
239. Design M a inten a nce • Template as much as possible • Provide as much documentation as possible • Build up as big of a test suite as possible
240. Challenges
241. Challenges
242. Challenges Encoding
243. Challenges Encoding • # -*- encoding: Shift_JIS -*-
244. Challenges Encoding • # -*- encoding: Shift_JIS -*- • 90 possible encodings for source fi les
245. Challenges Encoding • # -*- encoding: Shift_JIS -*- • 90 possible encodings for source fi les • 154 aliases for those possible encodings
246. Challenges Encoding • # -*- encoding: Shift_JIS -*- • 90 possible encodings for source fi les • 154 aliases for those possible encodings • => we must support them all
247. Challenges Esc a ping
248. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))
249. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal)
250. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding)
251. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding)
252. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding) • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x
253. Challenges Esc a ping • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding) • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x • => brute force testing
254. Challenges Regul a r Expressions
255. Challenges Regul a r Expressions • You must know local variables to parse correctly
256. Challenges Regul a r Expressions • You must know local variables to parse correctly • /(?<foo>\d+)/ can make local variables
257. Challenges Regul a r Expressions • You must know local variables to parse correctly • /(?<foo>\d+)/ can make local variables • => we need a regex parser
258. Challenges Heredocs
259. Challenges Heredocs • Declarations function as any other expression
260. Challenges Heredocs • Declarations function as any other expression • => every newline, you must check for heredoc contents
261. What’s the current status?
262. Current status
263. Current status • C API • CRuby, Natalie, Rust tooling • Sorbet, Artichoke, C/C++/Rust/Zig tools
264. Current status void Init_ISeq(void) { rb_define_singleton_method( rb_cISeq, "compile_prism", iseqw_s_compile_prism, -1 ); } CRuby
265. Current status class Natalie::Parser def ast Prism .parse(@code_str, filepath: @path) .value .accept(PathVisitor.new(@path)) end end N a t a lie
266. Current status • C API • CRuby, Natalie, Rust tooling • Sorbet, Artichoke, C/C++/Rust/Zig tools
267. Current status • C API • CRuby, Natalie, Rust tooling • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools
268. Current status def parse(source_buffer) @source_buffer = source_buffer build_ast( ::Prism.parse( source_buffer.source, source_buffer.name ).value ) end p a rser-prism
269. Current status • C API • CRuby, Natalie, Rust tooling • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools
270. Current status • C API • CRuby, Natalie, Rust tooling • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools • Serialization API • JRuby, Tru ffl eRuby, JavaScript
271. public interface ParserBindingPrism { void pm_serialize_parse( Buffer buffer, @In byte[] source, int size, @In byte[] metadata ); } Current status JRuby
272. JNIEXPORT jbyteArray JNICALL Java_org_prism_Parser_parseAndSerialize(...) { ... pm_parse_serialize((uint8_t*) bytes, size, &buffer, NULL); ... } Current status Truf f leRuby
273. Current status J a v a Script export async function loadPrism() { const fileURL = new URL("prism.wasm", import.meta.url); const wasm = await WebAssembly.compile( await readFile(fileURLToPath(fileURL)) ); const wasi = new WASI({ version: "preview1" }); const imports = wasi.getImportObject(); const instance = await WebAssembly.instantiate(wasm, imports); wasi.initialize(instance); return function parse(source) { return parsePrism(instance.exports, source); } }
274. What’s next?
275. What’s next?
276. What’s next? • Improved error tolerance and messaging
277. What’s next? • Improved error tolerance and messaging • Performance enhancements
278. What’s next? • Improved error tolerance and messaging • Performance enhancements • Multi-version support
279. What’s next? • Improved error tolerance and messaging • Performance enhancements • Multi-version support • Further integrations into the ecosystem
280. What’s next? • Improved error tolerance and messaging • Performance enhancements • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools
281. What’s next? • Improved error tolerance and messaging • Performance enhancements • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools • Contributor community around a single tool
282. I have worked on parsing Ruby for 6 years.
283. Parsing Ruby is not fun.
284. Building Ruby tools is very fun.
288. We can have all of these tools, and more.
289. * = <<~EOF, /(?<m\ matchy match EOF atch>\d+)/ =~ "3" match /1+2 #/
290. The Future of Understanding Ruby
291. • An ecosystem of runtimes, that get to work on what makes each one unique. The Future of Understanding Ruby
292. • An ecosystem of runtimes, that get to work on what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. The Future of Understanding Ruby
293. • An ecosystem of runtimes, that get to work on what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. • Lower barriers to entry for contributions around the ecosystem. The Future of Understanding Ruby
294. • An ecosystem of runtimes, that get to work on what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. • Lower barriers to entry for contributions around the ecosystem. • A community that works together to empower every tool and everyone. The Future of Understanding Ruby
295. Let’s build it together.