Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

The Future of Understanding Ruby

Kevin Newton
November 14, 2023

The Future of Understanding Ruby

For decades, the Ruby community has been fractured in the way that it parses and understands Ruby code. After countless tools have been developed and person-hours have been invested, we still don't have a common language for understanding Ruby code. No longer! Starting in Ruby 3.3, we will have a single API for parsing and understanding Ruby code. This talk will cover the history of how we got here, what is getting built today, and what you can expect from this exciting future.

Kevin Newton

November 14, 2023
Tweet

More Decks by Kevin Newton

Other Decks in Technology

Transcript

  1. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications def ❨╯°□°❩╯︵┻━┻ puts "Calm down, yo." end
  2. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications def ❨╯°□°❩╯︵┻━┻ puts "Calm down, yo." end
  3. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications [[KEYWORD_DEF, "def"], [IDENTIFIER, "❨╯°□°❩╯︵┻━┻"], [NEWLINE, "\n"], [IDENTIFIER, "puts"], [STRING_BEGIN, "\""], [STRING_CONTENT, "Calm down, yo."], [STRING_END, "\""], [NEWLINE, "\n"], [KEYWORD_END, "end"], [NEWLINE, "\n"], [EOF, ""]]
  4. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
  5. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
  6. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts
  7. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself
  8. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself
  9. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.”
  10. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.”
  11. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts
  12. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts
  13. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications @ DefNode (location: (1,0)-(3,3)) ├── name: :❨╯°□°❩╯︵┻━┻ └── body: @ StatementsNode (location: (2,2)-(2,23)) └── body: (length: 1) └── @ CallNode (location: (2,2)-(2,23)) ├── arguments: │ @ ArgumentsNode (location: (2,7)-(2,23)) │ └── arguments: (length: 1) │ └── @ StringNode (location: (2,7)-(2,23)) │ └── unescaped: "Calm down, yo." └── name: :puts putself putstring "Calm down, yo.” send :puts leave
  14. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications putself putstring "Calm down, yo.” send :puts leave
  15. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications putself putstring "Calm down, yo.” send :puts leave
  16. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  17. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  18. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  19. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  20. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  21. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications == disasm: #<ISeq:❨╯°□°❩╯︵┻━┻> 0000 putself 0001 putstring "Calm down, yo." 0003 send <cdata!mid:puts, argc:1> 0005 leave
  22. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications Sidekiq.❨╯°□°❩╯︵┻━┻
  23. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org
  24. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org
  25. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org
  26. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y
  27. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java
  28. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR
  29. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib
  30. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib rubygems.org*
  31. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y IRBuilder.java JRuby IR JRuby stdlib rubygems.org* Applications*
  32. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org
  33. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y
  34. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java
  35. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e
  36. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib
  37. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib rubygems.org*
  38. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org RubyParser.y Translator.java Tru ff l e TRuby stdlib rubygems.org* Applications*
  39. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org
  40. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser
  41. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb
  42. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb
  43. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib
  44. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib rubygems.org*
  45. Ruby source Parser Compiler Byte code/IR Standard libraries External libraries

    Applications parse.y compile.c YARV CRuby stdlib rubygems.org natalie_parser pass1.rb instructions/*.rb Natalie stdlib rubygems.org* Applications*
  46. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  47. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  48. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  49. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  50. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  51. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  52. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  53. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  54. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  55. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  56. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  57. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  58. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  59. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  60. Ruby source parse.y compile.c YARV RubyParser.y IRBuilder.java JRuby IR RubyParser.y

    Translator.java Tru ffl e natalie_parser pass1.rb instructions/*.rb
  61. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer
  62. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby
  63. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree
  64. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt
  65. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb
  66. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo
  67. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo prettier
  68. Ruby source parser gem ruby_parser gem tree-sitter-ruby ripper rubocop solargraph

    unparser ruby-next standard fl og fl ay debride fasterer vscode-ruby syntax_tree rubyfmt irb rufo prettier
  69. Different representations • Every tool represents Ruby di ff erently

    • Every tool requires contributors to relearn
  70. Different representations • Every tool represents Ruby di ff erently

    • Every tool requires contributors to relearn • None of this knowledge translates to contributing to CRuby
  71. Maintenance burden • Every parser has to update every time

    new syntax is introduced • Very minor changes have very major consequences
  72. Maintenance burden • Every parser has to update every time

    new syntax is introduced • Very minor changes have very major consequences • Helping Ruby can inadvertently hurt the Ruby ecosystem
  73. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers
  74. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby
  75. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby
  76. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby
  77. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby
  78. Fractured ecosystem • ruby/ruby/parse.y • ruby/ruby/ext/ripper/parse.y • lib-ruby-parser/lib-ruby-parser • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet • whitequark/parser • seattlerb/ruby_parser Ruby p a rsers • jruby/jruby • oracle/tru ff l eruby • tree-sitter/tree-sitter-ruby • sisshiki1969/ruruby • natalie-lang/natalie_parser
  79. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet Synt a x trees
  80. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees
  81. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor
  82. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast
  83. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e
  84. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter
  85. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby
  86. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser
  87. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser
  88. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  89. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  90. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  91. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  92. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  93. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  94. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  95. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  96. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  97. Fractured ecosystem • NODE, RubyVM::AbstractSyntaxTree • Ripper::SexpBuilder{,PP} • ruby-syntax-tree/syntax_tree •

    lib-ruby-parser/lib-ruby-parser • mruby/mruby • typedruby/typedruby, sorbet/sorbet • whitequark/parser, rubocop/rubocop-ast Synt a x trees • seattlerb/sexp_processor • org.jruby.ast • Tru ff l e • tree-sitter/tree-sitter • sisshiki1969/ruruby • natalie-lang/natalie_parser • ruby/prism
  98. Fractured ecosystem • lib-ruby-parser/lib-ruby-parser • mruby/mruby Synt a x trees

    • tree-sitter/tree-sitter • sisshiki1969/ruruby • ruby/prism
  99. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit 👶
  100. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit 👶
  101. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit 👶
  102. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit 👶
  103. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit 👶
  104. Timeline 2022/01/01 2023/01/01 2024/01/01 2022/07/01 2023/07/01 Prototype fi rst commit

    First call with CRuby team YARP fi rst commit Tru ff l eRuby fi rst commit JRuby fi rst commit CRuby fi rst commit Syntax Tree fi rst commit Parser fi rst commit Rename to prism 👶
  105. Design • Comparative analysis with existing trees • "Semantic" versus

    "syntactic" trees • Provide enough value for all consumers Appro a ch
  106. Design Implement a tions • Make compilation as simple as

    possible • Never have to check child to compile current node
  107. Design Implement a tions • Make compilation as simple as

    possible • Never have to check child to compile current node • Cover as much common functionality as possible
  108. Design Tooling • Named fi elds and known types •

    Nodes with enough information to round-trip
  109. Design Tooling • Named fi elds and known types •

    Nodes with enough information to round-trip • Location information for as much as possible
  110. Design M a inten a nce • Template as much

    as possible • Provide as much documentation as possible
  111. Design M a inten a nce • Template as much

    as possible • Provide as much documentation as possible • Build up as big of a test suite as possible
  112. Challenges Encoding • # -*- encoding: Shift_JIS -*- • 90

    possible encodings for source fi les • 154 aliases for those possible encodings
  113. Challenges Encoding • # -*- encoding: Shift_JIS -*- • 90

    possible encodings for source fi les • 154 aliases for those possible encodings • => we must support them all
  114. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s))
  115. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal)
  116. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding)
  117. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding)
  118. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding) • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x
  119. Challenges Esc a ping • \\, \a, \b, \e, \f,

    \n, \r, \r\n, \s, \t, \v (character(s)) • \000 (octal) • \x00 (hex, can change encoding) • \u0000 \u{00 00} (hex, can change encoding) • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x • => brute force testing
  120. Challenges Regul a r Expressions • You must know local

    variables to parse correctly • /(?<foo>\d+)/ can make local variables
  121. Challenges Regul a r Expressions • You must know local

    variables to parse correctly • /(?<foo>\d+)/ can make local variables • => we need a regex parser
  122. Challenges Heredocs • Declarations function as any other expression •

    => every newline, you must check for heredoc contents
  123. Current status • C API • CRuby, Natalie, Rust tooling

    • Sorbet, Artichoke, C/C++/Rust/Zig tools
  124. Current status class Natalie::Parser def ast Prism .parse(@code_str, filepath: @path)

    .value .accept(PathVisitor.new(@path)) end end N a t a lie
  125. Current status • C API • CRuby, Natalie, Rust tooling

    • Sorbet, Artichoke, C/C++/Rust/Zig tools
  126. Current status • C API • CRuby, Natalie, Rust tooling

    • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools
  127. Current status • C API • CRuby, Natalie, Rust tooling

    • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools
  128. Current status • C API • CRuby, Natalie, Rust tooling

    • Sorbet, Artichoke, C/C++/Rust/Zig tools • Ruby API • Natalie, Ruby tools • Serialization API • JRuby, Tru ffl eRuby, JavaScript
  129. public interface ParserBindingPrism { void pm_serialize_parse( Buffer buffer, @In byte[]

    source, int size, @In byte[] metadata ); } Current status JRuby
  130. Current status J a v a Script export async function

    loadPrism() { const fileURL = new URL("prism.wasm", import.meta.url); const wasm = await WebAssembly.compile( await readFile(fileURLToPath(fileURL)) ); const wasi = new WASI({ version: "preview1" }); const imports = wasi.getImportObject(); const instance = await WebAssembly.instantiate(wasm, imports); wasi.initialize(instance); return function parse(source) { return parsePrism(instance.exports, source); } }
  131. What’s next? • Improved error tolerance and messaging • Performance

    enhancements • Multi-version support • Further integrations into the ecosystem
  132. What’s next? • Improved error tolerance and messaging • Performance

    enhancements • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools
  133. What’s next? • Improved error tolerance and messaging • Performance

    enhancements • Multi-version support • Further integrations into the ecosystem • Improved Ruby library for external tools • Contributor community around a single tool
  134. • An ecosystem of runtimes, that get to work on

    what makes each one unique. The Future of Understanding Ruby
  135. • An ecosystem of runtimes, that get to work on

    what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. The Future of Understanding Ruby
  136. • An ecosystem of runtimes, that get to work on

    what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. • Lower barriers to entry for contributions around the ecosystem. The Future of Understanding Ruby
  137. • An ecosystem of runtimes, that get to work on

    what makes each one unique. • An ecosystem of tools, that all agree on Ruby representations. • Lower barriers to entry for contributions around the ecosystem. • A community that works together to empower every tool and everyone. The Future of Understanding Ruby