Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Future of Understanding Ruby

Kevin Newton
November 14, 2023

The Future of Understanding Ruby

For decades, the Ruby community has been fractured in the way that it parses and understands Ruby code. After countless tools have been developed and person-hours have been invested, we still don't have a common language for understanding Ruby code. No longer! Starting in Ruby 3.3, we will have a single API for parsing and understanding Ruby code. This talk will cover the history of how we got here, what is getting built today, and what you can expect from this exciting future.

Kevin Newton

November 14, 2023
Tweet

More Decks by Kevin Newton

Other Decks in Technology

Transcript

  1. match /1+2 /


    View full-size slide

  2. match /1+2 #/


    View full-size slide

  3. /\d+/ =~ "3"


    match /1+2 #/


    View full-size slide

  4. /(?\d+)/ =~ "3"


    match /1+2 #/


    View full-size slide

  5. * = <<~EOF, /(?\d+)/ =~ "3"


    matchy match


    EOF


    match /1+2 #/


    View full-size slide

  6. * = <<~EOF, /(?

    matchy match


    EOF


    atch>\d+)/ =~ "3"


    match /1+2 #/


    View full-size slide

  7. What does this code do?

    View full-size slide

  8. What does this code do?

    View full-size slide

  9. What does this code do?


    How would you
    fi
    nd out?

    View full-size slide

  10. Dependencies

    View full-size slide

  11. Applications

    View full-size slide

  12. External libraries
    Applications

    View full-size slide

  13. Standard libraries
    External libraries
    Applications

    View full-size slide

  14. Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  15. Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  16. Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  17. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  18. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  19. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  20. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications def ❨╯°□°❩╯︵┻━┻


    puts "Calm down, yo."


    end


    View full-size slide

  21. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications def ❨╯°□°❩╯︵┻━┻


    puts "Calm down, yo."


    end


    View full-size slide

  22. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    [[KEYWORD_DEF, "def"],


    [IDENTIFIER, "❨╯°□°❩╯︵┻━┻"],


    [NEWLINE, "\n"],


    [IDENTIFIER, "puts"],


    [STRING_BEGIN, "\""],


    [STRING_CONTENT, "Calm down, yo."],


    [STRING_END, "\""],


    [NEWLINE, "\n"],


    [KEYWORD_END, "end"],


    [NEWLINE, "\n"],


    [EOF, ""]]


    View full-size slide

  23. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    View full-size slide

  24. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    View full-size slide

  25. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    View full-size slide

  26. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself

    View full-size slide

  27. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself

    View full-size slide

  28. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself


    putstring "Calm down, yo.”

    View full-size slide

  29. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself


    putstring "Calm down, yo.”

    View full-size slide

  30. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself


    putstring "Calm down, yo.”


    send :puts

    View full-size slide

  31. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself


    putstring "Calm down, yo.”


    send :puts

    View full-size slide

  32. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications @ DefNode (location: (1,0)-(3,3))


    ├── name: :❨╯°□°❩╯︵┻━┻


    └── body:


    @ StatementsNode (location: (2,2)-(2,23))


    └── body: (length: 1)


    └── @ CallNode (location: (2,2)-(2,23))


    ├── arguments:


    │ @ ArgumentsNode (location: (2,7)-(2,23))


    │ └── arguments: (length: 1)


    │ └── @ StringNode (location: (2,7)-(2,23))


    │ └── unescaped: "Calm down, yo."


    └── name: :puts


    putself


    putstring "Calm down, yo.”


    send :puts


    leave

    View full-size slide

  33. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    putself


    putstring "Calm down, yo.”


    send :puts


    leave

    View full-size slide

  34. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    putself


    putstring "Calm down, yo.”


    send :puts


    leave

    View full-size slide

  35. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  36. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  37. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  38. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  39. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  40. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications == disasm: #


    0000 putself


    0001 putstring "Calm down, yo."


    0003 send


    0005 leave

    View full-size slide

  41. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  42. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  43. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    Sidekiq.❨╯°□°❩╯︵┻━┻

    View full-size slide

  44. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  45. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications

    View full-size slide

  46. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y

    View full-size slide

  47. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c

    View full-size slide

  48. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV

    View full-size slide

  49. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib

    View full-size slide

  50. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org

    View full-size slide

  51. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org

    View full-size slide

  52. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org

    View full-size slide

  53. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y

    View full-size slide

  54. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    IRBuilder.java

    View full-size slide

  55. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    IRBuilder.java
    JRuby IR

    View full-size slide

  56. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    IRBuilder.java
    JRuby IR
    JRuby stdlib

    View full-size slide

  57. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    IRBuilder.java
    JRuby IR
    JRuby stdlib
    rubygems.org*

    View full-size slide

  58. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    IRBuilder.java
    JRuby IR
    JRuby stdlib
    rubygems.org*
    Applications*

    View full-size slide

  59. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org

    View full-size slide

  60. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y

    View full-size slide

  61. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    Translator.java

    View full-size slide

  62. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    Translator.java
    Tru
    ff
    l
    e

    View full-size slide

  63. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    Translator.java
    Tru
    ff
    l
    e
    TRuby stdlib

    View full-size slide

  64. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    Translator.java
    Tru
    ff
    l
    e
    TRuby stdlib
    rubygems.org*

    View full-size slide

  65. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    RubyParser.y
    Translator.java
    Tru
    ff
    l
    e
    TRuby stdlib
    rubygems.org*
    Applications*

    View full-size slide

  66. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org

    View full-size slide

  67. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser

    View full-size slide

  68. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser
    pass1.rb

    View full-size slide

  69. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  70. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser
    pass1.rb
    instructions/*.rb
    Natalie stdlib

    View full-size slide

  71. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser
    pass1.rb
    instructions/*.rb
    Natalie stdlib
    rubygems.org*

    View full-size slide

  72. Ruby source
    Parser
    Compiler
    Byte code/IR
    Standard libraries
    External libraries
    Applications
    parse.y
    compile.c
    YARV
    CRuby stdlib
    rubygems.org
    natalie_parser
    pass1.rb
    instructions/*.rb
    Natalie stdlib
    rubygems.org*
    Applications*

    View full-size slide

  73. Ruby source
    parse.y
    compile.c
    YARV

    View full-size slide

  74. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR

    View full-size slide

  75. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e

    View full-size slide

  76. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  77. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  78. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  79. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  80. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  81. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  82. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  83. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  84. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  85. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  86. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  87. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  88. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  89. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  90. Ruby source
    parse.y
    compile.c
    YARV
    RubyParser.y
    IRBuilder.java
    JRuby IR
    RubyParser.y
    Translator.java
    Tru
    ffl
    e
    natalie_parser
    pass1.rb
    instructions/*.rb

    View full-size slide

  91. Ruby source
    Parser

    View full-size slide

  92. Ruby source
    Parser
    Compiler

    View full-size slide

  93. Ruby source
    Parser
    Compiler
    Byte code/IR

    View full-size slide

  94. Ruby source
    Parser
    Compiler
    Byte code/IR

    View full-size slide

  95. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y

    View full-size slide

  96. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c

    View full-size slide

  97. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  98. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  99. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    parser gem

    View full-size slide

  100. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    parser gem
    rubocop

    View full-size slide

  101. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  102. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    ruby_parser gem

    View full-size slide

  103. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    ruby_parser gem
    fasterer

    View full-size slide

  104. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  105. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    tree-sitter-ruby

    View full-size slide

  106. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    tree-sitter-ruby
    vscode-ruby

    View full-size slide

  107. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  108. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    ripper

    View full-size slide

  109. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV
    ripper
    syntax_tree

    View full-size slide

  110. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  111. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV

    View full-size slide

  112. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV RubyVM::ISeq

    View full-size slide

  113. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV RubyVM::ISeq
    typeprof

    View full-size slide

  114. Ruby source
    Parser
    Compiler
    Byte code/IR
    parse.y
    compile.c
    YARV RubyVM::ISeq
    tenderjit

    View full-size slide

  115. Ruby source
    parser gem

    View full-size slide

  116. Ruby source
    parser gem ruby_parser gem

    View full-size slide

  117. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby

    View full-size slide

  118. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper

    View full-size slide

  119. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop

    View full-size slide

  120. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph

    View full-size slide

  121. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser

    View full-size slide

  122. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next

    View full-size slide

  123. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard

    View full-size slide

  124. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fasterer

    View full-size slide

  125. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fasterer

    View full-size slide

  126. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    fasterer

    View full-size slide

  127. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer

    View full-size slide

  128. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby

    View full-size slide

  129. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree

    View full-size slide

  130. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree
    rubyfmt

    View full-size slide

  131. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree
    rubyfmt
    irb

    View full-size slide

  132. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree
    rubyfmt
    irb
    rufo

    View full-size slide

  133. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree
    rubyfmt
    irb
    rufo
    prettier

    View full-size slide

  134. Ruby source
    parser gem ruby_parser gem tree-sitter-ruby ripper
    rubocop
    solargraph
    unparser
    ruby-next
    standard
    fl
    og
    fl
    ay
    debride
    fasterer
    vscode-ruby
    syntax_tree
    rubyfmt
    irb
    rufo
    prettier

    View full-size slide

  135. Di
    ff
    erent representations


    lead to di
    ff
    erent assumptions.

    View full-size slide

  136. Different representations

    View full-size slide

  137. Different representations
    • Every tool represents Ruby di
    ff
    erently

    View full-size slide

  138. Different representations
    • Every tool represents Ruby di
    ff
    erently

    • Every tool requires contributors to relearn

    View full-size slide

  139. Different representations
    • Every tool represents Ruby di
    ff
    erently

    • Every tool requires contributors to relearn

    • None of this knowledge translates to contributing to CRuby

    View full-size slide

  140. Maintaining a Ruby parser is a
    massive maintenance burden.

    View full-size slide

  141. Maintenance burden

    View full-size slide

  142. Maintenance burden
    • Every parser has to update every time new syntax is introduced

    View full-size slide

  143. Maintenance burden
    • Every parser has to update every time new syntax is introduced

    • Very minor changes have very major consequences

    View full-size slide

  144. Maintenance burden
    • Every parser has to update every time new syntax is introduced

    • Very minor changes have very major consequences

    • Helping Ruby can inadvertently hurt the Ruby ecosystem

    View full-size slide

  145. Fractured ecosystem

    View full-size slide

  146. Fractured ecosystem

    View full-size slide

  147. Fractured ecosystem
    Ruby p
    a
    rsers

    View full-size slide

  148. Fractured ecosystem
    • ruby/ruby/parse.y
    Ruby p
    a
    rsers

    View full-size slide

  149. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y
    Ruby p
    a
    rsers

    View full-size slide

  150. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser
    Ruby p
    a
    rsers

    View full-size slide

  151. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby
    Ruby p
    a
    rsers

    View full-size slide

  152. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet
    Ruby p
    a
    rsers

    View full-size slide

  153. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser
    Ruby p
    a
    rsers

    View full-size slide

  154. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers

    View full-size slide

  155. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers
    • jruby/jruby

    View full-size slide

  156. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers
    • jruby/jruby

    • oracle/tru
    ff l
    eruby

    View full-size slide

  157. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers
    • jruby/jruby

    • oracle/tru
    ff l
    eruby

    • tree-sitter/tree-sitter-ruby

    View full-size slide

  158. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers
    • jruby/jruby

    • oracle/tru
    ff l
    eruby

    • tree-sitter/tree-sitter-ruby

    • sisshiki1969/ruruby

    View full-size slide

  159. Fractured ecosystem
    • ruby/ruby/parse.y

    • ruby/ruby/ext/ripper/parse.y

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser

    • seattlerb/ruby_parser
    Ruby p
    a
    rsers
    • jruby/jruby

    • oracle/tru
    ff l
    eruby

    • tree-sitter/tree-sitter-ruby

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    View full-size slide

  160. Fractured ecosystem
    Synt
    a
    x trees

    View full-size slide

  161. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree
    Synt
    a
    x trees

    View full-size slide

  162. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}
    Synt
    a
    x trees

    View full-size slide

  163. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree
    Synt
    a
    x trees

    View full-size slide

  164. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser
    Synt
    a
    x trees

    View full-size slide

  165. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby
    Synt
    a
    x trees

    View full-size slide

  166. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet
    Synt
    a
    x trees

    View full-size slide

  167. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees

    View full-size slide

  168. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    View full-size slide

  169. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    View full-size slide

  170. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    View full-size slide

  171. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    View full-size slide

  172. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    View full-size slide

  173. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    View full-size slide

  174. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    View full-size slide

  175. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  176. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  177. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  178. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  179. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  180. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  181. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  182. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  183. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  184. Fractured ecosystem
    • NODE, RubyVM::AbstractSyntaxTree

    • Ripper::SexpBuilder{,PP}

    • ruby-syntax-tree/syntax_tree

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • typedruby/typedruby, sorbet/sorbet

    • whitequark/parser, rubocop/rubocop-ast
    Synt
    a
    x trees
    • seattlerb/sexp_processor

    • org.jruby.ast

    • Tru
    ff
    l
    e

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • natalie-lang/natalie_parser

    • ruby/prism

    View full-size slide

  185. Fractured ecosystem
    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby
    Synt
    a
    x trees
    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby

    • ruby/prism

    View full-size slide

  186. Fractured ecosystem
    • ruby/prism

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby
    Synt
    a
    x trees

    View full-size slide

  187. Fractured ecosystem
    • ruby/prism

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby
    Synt
    a
    x trees

    View full-size slide

  188. ecosystem
    • ruby/prism

    • lib-ruby-parser/lib-ruby-parser

    • mruby/mruby

    • tree-sitter/tree-sitter

    • sisshiki1969/ruruby
    Synt
    a
    x trees

    View full-size slide

  189. Prism
    • Compatibility

    View full-size slide

  190. Prism
    • Compatibility

    • Maintainability

    View full-size slide

  191. Prism
    • Compatibility

    • Maintainability

    • Error tolerance

    View full-size slide

  192. Prism
    • Compatibility

    • Maintainability

    • Error tolerance

    • Portability

    View full-size slide

  193. Prism
    • Compatibility

    • Maintainability

    • Error tolerance

    • Portability

    • Performance

    View full-size slide

  194. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01

    View full-size slide

  195. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit

    View full-size slide

  196. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team

    View full-size slide

  197. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit

    View full-size slide

  198. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    👶

    View full-size slide

  199. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    👶

    View full-size slide

  200. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    JRuby
    fi
    rst commit
    👶

    View full-size slide

  201. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    JRuby
    fi
    rst commit
    CRuby
    fi
    rst commit
    👶

    View full-size slide

  202. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    JRuby
    fi
    rst commit
    CRuby
    fi
    rst commit
    Syntax Tree
    fi
    rst commit
    👶

    View full-size slide

  203. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    JRuby
    fi
    rst commit
    CRuby
    fi
    rst commit
    Syntax Tree
    fi
    rst commit
    Parser
    fi
    rst commit
    👶

    View full-size slide

  204. Timeline
    2022/01/01 2023/01/01 2024/01/01
    2022/07/01 2023/07/01
    Prototype
    fi
    rst commit
    First call with CRuby team
    YARP
    fi
    rst commit
    Tru
    ff
    l
    eRuby
    fi
    rst commit
    JRuby
    fi
    rst commit
    CRuby
    fi
    rst commit
    Syntax Tree
    fi
    rst commit
    Parser
    fi
    rst commit
    Rename to prism
    👶

    View full-size slide

  205. How was it built?

    View full-size slide

  206. Design
    Appro
    a
    ch

    View full-size slide

  207. Design
    • Comparative analysis with existing trees
    Appro
    a
    ch

    View full-size slide

  208. Design
    • Comparative analysis with existing trees

    • "Semantic" versus "syntactic" trees
    Appro
    a
    ch

    View full-size slide

  209. Design
    • Comparative analysis with existing trees

    • "Semantic" versus "syntactic" trees

    • Provide enough value for all consumers
    Appro
    a
    ch

    View full-size slide

  210. Design
    Implement
    a
    tions

    View full-size slide

  211. Design
    Implement
    a
    tions
    • Make compilation as simple as possible

    View full-size slide

  212. Design
    Implement
    a
    tions
    • Make compilation as simple as possible

    • Never have to check child to compile current node

    View full-size slide

  213. Design
    Implement
    a
    tions
    • Make compilation as simple as possible

    • Never have to check child to compile current node

    • Cover as much common functionality as possible

    View full-size slide

  214. Design
    Tooling

    View full-size slide

  215. Design
    Tooling
    • Named
    fi
    elds and known types

    View full-size slide

  216. Design
    Tooling
    • Named
    fi
    elds and known types

    • Nodes with enough information to round-trip

    View full-size slide

  217. Design
    Tooling
    • Named
    fi
    elds and known types

    • Nodes with enough information to round-trip

    • Location information for as much as possible

    View full-size slide

  218. Design
    M
    a
    inten
    a
    nce

    View full-size slide

  219. Design
    M
    a
    inten
    a
    nce
    • Template as much as possible

    View full-size slide

  220. Design
    M
    a
    inten
    a
    nce
    • Template as much as possible

    • Provide as much documentation as possible

    View full-size slide

  221. Design
    M
    a
    inten
    a
    nce
    • Template as much as possible

    • Provide as much documentation as possible

    • Build up as big of a test suite as possible

    View full-size slide

  222. Challenges
    Encoding

    View full-size slide

  223. Challenges
    Encoding
    • # -*- encoding: Shift_JIS -*-

    View full-size slide

  224. Challenges
    Encoding
    • # -*- encoding: Shift_JIS -*-

    • 90 possible encodings for source
    fi
    les

    View full-size slide

  225. Challenges
    Encoding
    • # -*- encoding: Shift_JIS -*-

    • 90 possible encodings for source
    fi
    les

    • 154 aliases for those possible encodings

    View full-size slide

  226. Challenges
    Encoding
    • # -*- encoding: Shift_JIS -*-

    • 90 possible encodings for source
    fi
    les

    • 154 aliases for those possible encodings

    • => we must support them all

    View full-size slide

  227. Challenges
    Esc
    a
    ping

    View full-size slide

  228. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    View full-size slide

  229. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    • \000 (octal)

    View full-size slide

  230. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    • \000 (octal)

    • \x00 (hex, can change encoding)

    View full-size slide

  231. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    • \000 (octal)

    • \x00 (hex, can change encoding)

    • \u0000 \u{00 00} (hex, can change encoding)

    View full-size slide

  232. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    • \000 (octal)

    • \x00 (hex, can change encoding)

    • \u0000 \u{00 00} (hex, can change encoding)

    • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x

    View full-size slide

  233. Challenges
    Esc
    a
    ping
    • \\, \a, \b, \e, \f, \n, \r, \r\n, \s, \t, \v (character(s))

    • \000 (octal)

    • \x00 (hex, can change encoding)

    • \u0000 \u{00 00} (hex, can change encoding)

    • \cx, \C-x, \M-x, \C-\M-x, \M-\C-x

    • => brute force testing

    View full-size slide

  234. Challenges
    Regul
    a
    r Expressions

    View full-size slide

  235. Challenges
    Regul
    a
    r Expressions
    • You must know local variables to parse correctly

    View full-size slide

  236. Challenges
    Regul
    a
    r Expressions
    • You must know local variables to parse correctly

    • /(?\d+)/ can make local variables

    View full-size slide

  237. Challenges
    Regul
    a
    r Expressions
    • You must know local variables to parse correctly

    • /(?\d+)/ can make local variables

    • => we need a regex parser

    View full-size slide

  238. Challenges
    Heredocs

    View full-size slide

  239. Challenges
    Heredocs
    • Declarations function as any other expression

    View full-size slide

  240. Challenges
    Heredocs
    • Declarations function as any other expression

    • => every newline, you must check for heredoc contents

    View full-size slide

  241. What’s the current status?

    View full-size slide

  242. Current status

    View full-size slide

  243. Current status
    • C API


    • CRuby, Natalie, Rust tooling


    • Sorbet, Artichoke, C/C++/Rust/Zig tools

    View full-size slide

  244. Current status
    void


    Init_ISeq(void)


    {


    rb_define_singleton_method(


    rb_cISeq,


    "compile_prism",


    iseqw_s_compile_prism,


    -1


    );


    }


    CRuby

    View full-size slide

  245. Current status
    class Natalie::Parser


    def ast


    Prism


    .parse(@code_str, filepath: @path)


    .value


    .accept(PathVisitor.new(@path))


    end


    end


    N
    a
    t
    a
    lie

    View full-size slide

  246. Current status
    • C API


    • CRuby, Natalie, Rust tooling


    • Sorbet, Artichoke, C/C++/Rust/Zig tools

    View full-size slide

  247. Current status
    • C API


    • CRuby, Natalie, Rust tooling


    • Sorbet, Artichoke, C/C++/Rust/Zig tools


    • Ruby API


    • Natalie, Ruby tools

    View full-size slide

  248. Current status
    def parse(source_buffer)


    @source_buffer = source_buffer


    build_ast(


    ::Prism.parse(


    source_buffer.source,


    source_buffer.name


    ).value


    )


    end


    p
    a
    rser-prism

    View full-size slide

  249. Current status
    • C API


    • CRuby, Natalie, Rust tooling


    • Sorbet, Artichoke, C/C++/Rust/Zig tools


    • Ruby API


    • Natalie, Ruby tools

    View full-size slide

  250. Current status
    • C API


    • CRuby, Natalie, Rust tooling


    • Sorbet, Artichoke, C/C++/Rust/Zig tools


    • Ruby API


    • Natalie, Ruby tools


    • Serialization API


    • JRuby, Tru
    ffl
    eRuby, JavaScript

    View full-size slide

  251. public interface ParserBindingPrism {


    void pm_serialize_parse(


    Buffer buffer,


    @In byte[] source,


    int size,


    @In byte[] metadata


    );


    }


    Current status
    JRuby

    View full-size slide

  252. JNIEXPORT jbyteArray JNICALL


    Java_org_prism_Parser_parseAndSerialize(...) {


    ...


    pm_parse_serialize((uint8_t*) bytes, size,
    &buffer, NULL);


    ...


    }


    Current status
    Truf
    f
    leRuby

    View full-size slide

  253. Current status
    J
    a
    v
    a
    Script
    export async function loadPrism() {


    const fileURL = new URL("prism.wasm", import.meta.url);


    const wasm = await WebAssembly.compile(


    await readFile(fileURLToPath(fileURL))


    );


    const wasi = new WASI({ version: "preview1" });


    const imports = wasi.getImportObject();


    const instance = await WebAssembly.instantiate(wasm, imports);


    wasi.initialize(instance);


    return function parse(source) {


    return parsePrism(instance.exports, source);


    }


    }


    View full-size slide

  254. What’s next?

    View full-size slide

  255. What’s next?

    View full-size slide

  256. What’s next?
    • Improved error tolerance and messaging

    View full-size slide

  257. What’s next?
    • Improved error tolerance and messaging

    • Performance enhancements

    View full-size slide

  258. What’s next?
    • Improved error tolerance and messaging

    • Performance enhancements

    • Multi-version support

    View full-size slide

  259. What’s next?
    • Improved error tolerance and messaging

    • Performance enhancements

    • Multi-version support

    • Further integrations into the ecosystem

    View full-size slide

  260. What’s next?
    • Improved error tolerance and messaging

    • Performance enhancements

    • Multi-version support

    • Further integrations into the ecosystem

    • Improved Ruby library for external tools

    View full-size slide

  261. What’s next?
    • Improved error tolerance and messaging

    • Performance enhancements

    • Multi-version support

    • Further integrations into the ecosystem

    • Improved Ruby library for external tools

    • Contributor community around a single tool

    View full-size slide

  262. I have worked on parsing
    Ruby for 6 years.

    View full-size slide

  263. Parsing Ruby is not fun.

    View full-size slide

  264. Building Ruby tools is


    very fun.

    View full-size slide

  265. We can have all of these tools,


    and more.

    View full-size slide

  266. * = <<~EOF, /(?

    matchy match


    EOF


    atch>\d+)/ =~ "3"


    match /1+2 #/


    View full-size slide

  267. The Future of Understanding Ruby

    View full-size slide

  268. • An ecosystem of runtimes, that get to work on what makes each one unique.
    The Future of Understanding Ruby

    View full-size slide

  269. • An ecosystem of runtimes, that get to work on what makes each one unique.


    • An ecosystem of tools, that all agree on Ruby representations.
    The Future of Understanding Ruby

    View full-size slide

  270. • An ecosystem of runtimes, that get to work on what makes each one unique.


    • An ecosystem of tools, that all agree on Ruby representations.


    • Lower barriers to entry for contributions around the ecosystem.
    The Future of Understanding Ruby

    View full-size slide

  271. • An ecosystem of runtimes, that get to work on what makes each one unique.


    • An ecosystem of tools, that all agree on Ruby representations.


    • Lower barriers to entry for contributions around the ecosystem.


    • A community that works together to empower every tool and everyone.
    The Future of Understanding Ruby

    View full-size slide

  272. Let’s build it together.

    View full-size slide