Overview
Prism is a new Ruby parser built from scratch with the following goals:
Slide 5
Slide 5 text
Overview
Prism is a new Ruby parser built from scratch with the following goals:
• Compatibility - must match CRuby semantics exactly in order to merge
Slide 6
Slide 6 text
Overview
Prism is a new Ruby parser built from scratch with the following goals:
• Compatibility - must match CRuby semantics exactly in order to merge
• Maintainability - must be as maintainable as possible
Slide 7
Slide 7 text
Overview
Prism is a new Ruby parser built from scratch with the following goals:
• Compatibility - must match CRuby semantics exactly in order to merge
• Maintainability - must be as maintainable as possible
• Error tolerance - must be able to recover from errors and continue parsing
Slide 8
Slide 8 text
Overview
Prism is a new Ruby parser built from scratch with the following goals:
• Compatibility - must match CRuby semantics exactly in order to merge
• Maintainability - must be as maintainable as possible
• Error tolerance - must be able to recover from errors and continue parsing
• Portability - must be able to be run from any ruby platform
Slide 9
Slide 9 text
Overview
Prism is a new Ruby parser built from scratch with the following goals:
• Compatibility - must match CRuby semantics exactly in order to merge
• Maintainability - must be as maintainable as possible
• Error tolerance - must be able to recover from errors and continue parsing
• Portability - must be able to be run from any ruby platform
• Performance - must be as fast and memory e
ffi
cient as possible
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
Slide 12
Slide 12 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
Slide 13
Slide 13 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Slide 14
Slide 14 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
👶
Slide 15
Slide 15 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
👶
Slide 16
Slide 16 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
JRuby
fi
rst commit
👶
Slide 17
Slide 17 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
JRuby
fi
rst commit
CRuby
fi
rst commit
👶
Slide 18
Slide 18 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
JRuby
fi
rst commit
CRuby
fi
rst commit
Syntax Tree
fi
rst commit
👶
Slide 19
Slide 19 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
JRuby
fi
rst commit
CRuby
fi
rst commit
Syntax Tree
fi
rst commit
Parser
fi
rst commit
👶
Slide 20
Slide 20 text
Timeline
2022/01/01 2023/01/01 2024/01/01
2022/07/01 2023/07/01
Prototype
fi
rst commit
First call with CRuby team
YARP
fi
rst commit
Tru
ff
l
eRuby
fi
rst commit
JRuby
fi
rst commit
CRuby
fi
rst commit
Syntax Tree
fi
rst commit
Parser
fi
rst commit
Rename to prism
👶
Slide 21
Slide 21 text
How was it built?
Slide 22
Slide 22 text
Design
Slide 23
Slide 23 text
Design
Approach
Slide 24
Slide 24 text
• Comparative analysis with existing trees
Approach
Design
Slide 25
Slide 25 text
• Comparative analysis with existing trees
• "Semantic" versus "syntactic" trees
Approach
Design
Slide 26
Slide 26 text
• Comparative analysis with existing trees
• "Semantic" versus "syntactic" trees
• Provide for all consumers
Approach
Design
Slide 27
Slide 27 text
Implementations
Design
Slide 28
Slide 28 text
• Make compilation as simple as possible
Implementations
Design
Slide 29
Slide 29 text
• Make compilation as simple as possible
• Never have to check child to compile current node
Implementations
Design
Slide 30
Slide 30 text
• Make compilation as simple as possible
• Never have to check child to compile current node
• Cover as much common functionality as possible
Implementations
Design
Slide 31
Slide 31 text
Tooling
Design
Slide 32
Slide 32 text
• Named
fi
elds and known types
Tooling
Design
Slide 33
Slide 33 text
• Named
fi
elds and known types
• Nodes with enough information to round-trip
Tooling
Design
Slide 34
Slide 34 text
• Named
fi
elds and known types
• Nodes with enough information to round-trip
• Location information for as much as possible
Tooling
Design
Slide 35
Slide 35 text
Maintenance
Design
Slide 36
Slide 36 text
• Template as much as possible
Maintenance
Design
Slide 37
Slide 37 text
• Template as much as possible
• Provide as much documentation as possible
Maintenance
Design
Slide 38
Slide 38 text
• Template as much as possible
• Provide as much documentation as possible
• Build up as big of a test suite as possible
Maintenance
Design
Slide 39
Slide 39 text
Challenges
Slide 40
Slide 40 text
Challenges
• Escaping - required to properly
fi
nd string-like terminators
Slide 41
Slide 41 text
Challenges
• Escaping - required to properly
fi
nd string-like terminators
• Regular expressions - required because of named capture groups
Slide 42
Slide 42 text
Challenges
• Escaping - required to properly
fi
nd string-like terminators
• Regular expressions - required because of named capture groups
• Encoding support - required to properly
fi
nd string-like terminators
Slide 43
Slide 43 text
Challenges
• Escaping - required to properly
fi
nd string-like terminators
• Regular expressions - required because of named capture groups
• Encoding support - required to properly
fi
nd string-like terminators
• Heredocs - very complicated semantics, can be almost anywhere in source
Slide 44
Slide 44 text
Testing
Slide 45
Slide 45 text
Testing
• Lexer testing - testing against ripper lex
Slide 46
Slide 46 text
Testing
• Lexer testing - testing against ripper lex
• Snapshot testing - protect from regressions, check parser structure
Slide 47
Slide 47 text
Testing
• Lexer testing - testing against ripper lex
• Snapshot testing - protect from regressions, check parser structure
• Brute-force testing - all escape sequences in all contexts
Slide 48
Slide 48 text
Testing
• Lexer testing - testing against ripper lex
• Snapshot testing - protect from regressions, check parser structure
• Brute-force testing - all escape sequences in all contexts
• Fuzzing - random input, check for crashes
Slide 49
Slide 49 text
Testing
• Lexer testing - testing against ripper lex
• Snapshot testing - protect from regressions, check parser structure
• Brute-force testing - all escape sequences in all contexts
• Fuzzing - random input, check for crashes
• Grammar-based fuzzing - random input, check for crashes and parse errors
Slide 50
Slide 50 text
Integrations
Slide 51
Slide 51 text
Integrations
• C - CRuby, Sorbet, Artichoke, Natalie, C/C++/Rust/Zig tooling
Separate understandings
• Every tool represents Ruby di
ff
erently
Slide 101
Slide 101 text
Separate understandings
• Every tool represents Ruby di
ff
erently
• Every tool requires contributors to relearn
Slide 102
Slide 102 text
Separate understandings
• Every tool represents Ruby di
ff
erently
• Every tool requires contributors to relearn
• None of this knowledge translates to contributing to CRuby
Slide 103
Slide 103 text
Maintenance burden
Slide 104
Slide 104 text
Maintenance burden
• Every parser has to update every time new syntax is introduced
Slide 105
Slide 105 text
Maintenance burden
• Every parser has to update every time new syntax is introduced
• Very minor changes have very major consequences
Slide 106
Slide 106 text
Maintenance burden
• Every parser has to update every time new syntax is introduced
• Very minor changes have very major consequences
• Helping Ruby can inadvertently hurt Ruby tooling
Slide 107
Slide 107 text
What is the future of prism?
Slide 108
Slide 108 text
The future of prism
Slide 109
Slide 109 text
The future of prism
• Improved error tolerance and messaging
Slide 110
Slide 110 text
The future of prism
• Improved error tolerance and messaging
• Performance enhancements (SIMD instructions, arena allocation, etc.)
Slide 111
Slide 111 text
The future of prism
• Improved error tolerance and messaging
• Performance enhancements (SIMD instructions, arena allocation, etc.)
• Multi-version support
Slide 112
Slide 112 text
The future of prism
• Improved error tolerance and messaging
• Performance enhancements (SIMD instructions, arena allocation, etc.)
• Multi-version support
• Further integrations into the ecosystem
Slide 113
Slide 113 text
The future of prism
• Improved error tolerance and messaging
• Performance enhancements (SIMD instructions, arena allocation, etc.)
• Multi-version support
• Further integrations into the ecosystem
• Improved Ruby library for external tools
Slide 114
Slide 114 text
The future of prism
• Improved error tolerance and messaging
• Performance enhancements (SIMD instructions, arena allocation, etc.)
• Multi-version support
• Further integrations into the ecosystem
• Improved Ruby library for external tools
• Contributor community around a single tool