Slide 1
Slide 1 text
Kevin Newton, Shopify
Why Prism?
Fukuok
a
RubyistK
a
igi 04
Slide 2
Slide 2 text
Disclaimer 免責事項
I have Japanese translations on these slides that were done
by DeepL. I am sorry if they are bad.
このスライドはDeepLが
日
本語に翻訳したものです。悪
かったらごめんなさい。
Slide 3
Slide 3 text
What is Prism?
Prismとは？
Slide 4
Slide 4 text
What is Prism?
C Libr
a
ry (C99+)
Slide 5
Slide 5 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
C Libr
a
ry (C99+)
Slide 6
Slide 6 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
C Libr
a
ry (C99+)
Slide 7
Slide 7 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
C Libr
a
ry (C99+)
Slide 8
Slide 8 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
C Libr
a
ry (C99+)
Slide 9
Slide 9 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
C Libr
a
ry (C99+)
Slide 10
Slide 10 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
C Libr
a
ry (C99+)
Slide 11
Slide 11 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
C Libr
a
ry (C99+)
Slide 12
Slide 12 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
• Rust bindings → bindgen
C Libr
a
ry (C99+)
Slide 13
Slide 13 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
• Rust bindings → bindgen
C Libr
a
ry (C99+)
Slide 14
Slide 14 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
• Rust bindings → bindgen
• C++ bindings → extern C
C Libr
a
ry (C99+)
Slide 15
Slide 15 text
What is Prism?
Ruby language parser written in C99
C99で書かれたRuby
言
語パーサー
• CRuby bindings → native extension
• Java bindings → serialization and FFI
• JavaScript/TypeScript bindings → WASM
• Rust bindings → bindgen
• C++ bindings → extern C
C Libr
a
ry (C99+)
Slide 16
Slide 16 text
What is Prism?
Ruby Gem (CRuby 2.7+)
Slide 17
Slide 17 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
Ruby Gem (CRuby 2.7+)
Slide 18
Slide 18 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
Ruby Gem (CRuby 2.7+)
irb(main):001> Prism.parse("1 + 2")
=>
#,
@value=
@ ProgramNode (location: (1,0)-(1,5))
├── flags: ∅
├── locals: []
└── statements:
@ StatementsNode (location: (1,0)-(1,5))
├── flags: ∅
└── body: (length: 1)
└── @ CallNode (location: (1,0)-(1,5))
├── flags: newline
├── receiver:
│ @ IntegerNode (location: (1,0)-(1,1))
│ ├── flags: static_literal, decimal
│ └── value: 1
├── call_operator_loc: ∅
├── name: :+
├── message_loc: (1,2)-(1,3) = "+"
├── opening_loc: ∅
├── arguments:
│ @ ArgumentsNode (location: (1,4)-(1,5))
│ ├── flags: ∅
│ └── arguments: (length: 1)
│ └── @ IntegerNode (location: (1,4)-(1,5))
│ ├── flags: static_literal, decimal
│ └── value: 2
├── closing_loc: ∅
└── block: ∅,
@warnings=
[# @level=:verbose>]>
irb(main):002>
Slide 19
Slide 19 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
Ruby Gem (CRuby 2.7+)
Slide 20
Slide 20 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
Ruby Gem (CRuby 2.7+)
Slide 21
Slide 21 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
Ruby Gem (CRuby 2.7+)
irb(main):001> Ripper.sexp_raw("1 + 2")
=>
[:program,
[:stmts_add,
[:stmts_new],
[:binary, [:@int, "1", [1, 0]], :+, [:@int, "2", [1, 4]]]]]
irb(main):002> Prism::Translation::Ripper.sexp_raw("1 + 2")
=>
[:program,
[:stmts_add,
[:stmts_new],
[:binary, [:@int, "1", [1, 0]], :+, [:@int, "2", [1, 4]]]]]
irb(main):003>
Slide 22
Slide 22 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
Ruby Gem (CRuby 2.7+)
Slide 23
Slide 23 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
Ruby Gem (CRuby 2.7+)
irb(main):001> RubyParser.new.parse("1 + 2")
=> s(:call, s(:lit, 1), :+, s(:lit, 2))
irb(main):002> Prism::Translation::RubyParser.new.parse("1 + 2")
=> s(:call, s(:lit, 1), :+, s(:lit, 2))
irb(main):003>
Slide 24
Slide 24 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
Ruby Gem (CRuby 2.7+)
Slide 25
Slide 25 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
Ruby Gem (CRuby 2.7+)
irb(main):001> Parser::CurrentRuby.parse("1 + 2")
=>
s(:send,
s(:int, 1), :+,
s(:int, 2))
irb(main):002> Prism::Translation::Parser.parse("1 + 2")
=>
s(:send,
s(:int, 1), :+,
s(:int, 2))
irb(main):003>
Slide 26
Slide 26 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
• APIs for pattern matching, searching, etc.
パターンマッチング、検索などのためのAPI。
Ruby Gem (CRuby 2.7+)
Slide 27
Slide 27 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
• APIs for pattern matching, searching, etc.
パターンマッチング、検索などのためのAPI。
Ruby Gem (CRuby 2.7+)
irb(main):001> value = Prism.parse("[1, 2, 3, 4, 5]").value
=>
@ ProgramNode (location: (1,0)-(1,15))
...
irb(main):002* value.breadth_first_search do |node|
irb(main):003* node in Prism::IntegerNode[value: 5..]
irb(main):004> end
=>
@ IntegerNode (location: (1,13)-(1,14))
├── flags: static_literal, decimal
└── value: 5
irb(main):005>
Slide 28
Slide 28 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
• APIs for pattern matching, searching, etc.
パターンマッチング、検索などのためのAPI。
• RBS + Steep, RBI + Sorbet
Ruby Gem (CRuby 2.7+)
Slide 29
Slide 29 text
What is Prism?
• Syntax tree with rich location information
豊富な位置情報を持つ構
文
ツリー
• Translation 翻訳
• Ripper
• seattlerb/ruby_parser
• whitequark/parser
• APIs for pattern matching, searching, etc.
パターンマッチング、検索などのためのAPI。
• RBS + Steep, RBI + Sorbet
Ruby Gem (CRuby 2.7+)
Slide 30
Slide 30 text
What is Prism?
CRuby compiler
Slide 31
Slide 31 text
What is Prism?
• A new compiler for CRuby 3.3+
CRuby 3.3+
用
の新しいコンパイラ
CRuby compiler
Slide 32
Slide 32 text
What is Prism?
• A new compiler for CRuby 3.3+
CRuby 3.3+
用
の新しいコンパイラ
• Generates the same YARV instruction sequences as compile.c
compile.cと同じYARV命令シーケンスを
生
成する。
CRuby compiler
Slide 33
Slide 33 text
What is Prism?
• A new compiler for CRuby 3.3+
CRuby 3.3+
用
の新しいコンパイラ
• Generates the same YARV instruction sequences as compile.c
compile.cと同じYARV命令シーケンスを
生
成する。
• A richer AST simpli
fi
es compilation
リッチなASTはコンパイルを単純化する
CRuby compiler
Slide 34
Slide 34 text
What is Prism?
CRuby compiler
Slide 35
Slide 35 text
What is Prism?
CRuby compiler
Slide 36
Slide 36 text
What is Prism?
CRuby compiler
Slide 37
Slide 37 text
What is Prism?
CRuby compiler
Same optimizations, less code.
同じ最適化、より少ないコード。
Slide 38
Slide 38 text
Requirements
必要条件
Slide 39
Slide 39 text
Requirements
Criteri
a
Slide 40
Slide 40 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
Criteri
a
Slide 41
Slide 41 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
Criteri
a
Slide 42
Slide 42 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
Criteri
a
Slide 43
Slide 43 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
Criteri
a
Slide 44
Slide 44 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
Criteri
a
Slide 45
Slide 45 text
Requirements
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 46
Slide 46 text
Requirements
Fe
a
tures
Slide 47
Slide 47 text
Requirements
• Written in C99, supporting same platforms as CRuby
C99で書かれ、CRubyと同じプラットフォームをサポートする。
Fe
a
tures
Slide 48
Slide 48 text
Requirements
• Written in C99, supporting same platforms as CRuby
C99で書かれ、CRubyと同じプラットフォームをサポートする。
• Bindings to Java, C++, Rust, and WASM
Java、C++、Rust、WASMへのバインディング
Fe
a
tures
Slide 49
Slide 49 text
Requirements
• Written in C99, supporting same platforms as CRuby
C99で書かれ、CRubyと同じプラットフォームをサポートする。
• Bindings to Java, C++, Rust, and WASM
Java、C++、Rust、WASMへのバインディング
• FFI bindings can only require one function call
FFIバインディングは1つの関数呼び出ししか必要としない
Fe
a
tures
Slide 50
Slide 50 text
Requirements
• Written in C99, supporting same platforms as CRuby
C99で書かれ、CRubyと同じプラットフォームをサポートする。
• Bindings to Java, C++, Rust, and WASM
Java、C++、Rust、WASMへのバインディング
• FFI bindings can only require one function call
FFIバインディングは1つの関数呼び出ししか必要としない
• Small memory footprint to support embedded targets
組み込みターゲットをサポートする
小
さなメモリフットプリント
Fe
a
tures
Slide 51
Slide 51 text
Requirements
P
a
rser intern
a
ls
Slide 52
Slide 52 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
P
a
rser intern
a
ls
Slide 53
Slide 53 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
P
a
rser intern
a
ls
if (parser->encoding->isupper_char(start, length)) {
return PM_TOKEN_CONSTANT;
} else {
return PM_TOKEN_IDENTIFIER;
}
Slide 54
Slide 54 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
P
a
rser intern
a
ls
Slide 55
Slide 55 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
P
a
rser intern
a
ls
/(?\w)/ =~ "b"
a # => "b"
Slide 56
Slide 56 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
P
a
rser intern
a
ls
Slide 57
Slide 57 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
P
a
rser intern
a
ls
"a #{"b"} c " "d e " "f"
== disasm: #@test.rb:1 (1,0)-(1,24)>
0000 putstring "a b c d e f" ( 1)
[Li]
0002 leave
Slide 58
Slide 58 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
• RActor pragmas RActor のコメント
P
a
rser intern
a
ls
Slide 59
Slide 59 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
• RActor pragmas RActor のコメント
P
a
rser intern
a
ls
# shareable_constant_value: literal
C = { a: b }
Slide 60
Slide 60 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
• RActor pragmas RActor のコメント
P
a
rser intern
a
ls
== disasm: #@test.rb:1 (1,0)-(1,12)>
0000 putobject :a ( 1)[Li]
0002 putself
0003 opt_send_without_block
0005 newhash 2
0007 dup
0008 putspecialobject 3
0010 setconstant :C
0012 leave
Slide 61
Slide 61 text
Requirements
• Encoding support (90 encodings, 154 names) エンコーディングɾサポート
• Regular expression support (named captures, encoding, extended mode)
正規表現のサポート
• Static literals/string concatenation
文
字列の連結
• RActor pragmas RActor のコメント
P
a
rser intern
a
ls
== disasm: #@test.rb:1 (1,0)-(3,12)>
0000 putobject RubyVM::FrozenCore ( 3)[Li]
0002 putobject :a
0004 putobject RubyVM::FrozenCore
0006 putself
0007 opt_send_without_block
0009 putobject "C"
0011 opt_send_without_block
0013 newhash 2
0015 opt_send_without_block
0017 dup
0018 putspecialobject 3
0020 setconstant :C
0022 leave
Slide 62
Slide 62 text
Requirements
P
a
rser extern
a
ls
Slide 63
Slide 63 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
P
a
rser extern
a
ls
Slide 64
Slide 64 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
P
a
rser extern
a
ls
Prism.parse(source)
Prism.parse_file(filepath)
Prism.parse_file(File.mkfifo("tmp/fifo"))
Prism.parse_stream(STDIN)
Slide 65
Slide 65 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
P
a
rser extern
a
ls
Slide 66
Slide 66 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
P
a
rser extern
a
ls
SCRIPT_LINES__ = {}
require "prism"
SCRIPT_LINES__.keys.length # => 6
Slide 67
Slide 67 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
P
a
rser extern
a
ls
Slide 68
Slide 68 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
P
a
rser extern
a
ls
DATA.read # => "Hello, world!"
__END__
Hello, world!
Slide 69
Slide 69 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
P
a
rser extern
a
ls
Slide 70
Slide 70 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
P
a
rser extern
a
ls
@ ProgramNode (location: (1,0)-(1,1))
+-- locals: []
+-- statements:
@ StatementsNode (location: (1,0)-(1,1))
+-- body: (length: 1)
+-- @ IntegerNode (location: (1,0)-(1,1))
+-- IntegerBaseFlags: decimal
+-- value: 1
Slide 71
Slide 71 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
• shebang
fl
ags
P
a
rser extern
a
ls
Slide 72
Slide 72 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
• shebang
fl
ags
P
a
rser extern
a
ls
#! /usr/local/bin/ruby -s
Slide 73
Slide 73 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
• shebang
fl
ags
• eval scopes
P
a
rser extern
a
ls
Slide 74
Slide 74 text
Requirements
• String,
fi
le, FIFO
fi
le, IO
• RubyVM.keep_script_lines, SCRIPT_LINES__
• DATA, __END__
• --dump=parsetree
• shebang
fl
ags
• eval scopes
P
a
rser extern
a
ls
a = 1
eval("a + 2")
Slide 75
Slide 75 text
By-products
副産物
Slide 76
Slide 76 text
By-products
• Bug #19838 — Ripper nested heredocs
• Bug #19848 — Ripper BOM
• Bug #19924 — Character literal \xFF stops parsing
• Bug #20025 — Parsing identi
fi
ers is case-folding dependent
• Bug #20041 — Array restructuring and default values in parameters
• Bug #20043 — de
fi
ned? checks for method existence, but only sometimes
• Bug #20217 — void value not checked for begin/ensure/rescue
• Bug #20392 — delegate super calls with a block
P
a
rser/compiler bugs (p
a
ge 1/2)
Slide 77
Slide 77 text
By-products
P
a
rser/compiler bugs (p
a
ge 2/2)
• Bug #20417 — RubyVM::AbstractSyntaxTree.parse warns about block local variables
• Bug #20449 — Ripper issue in
fi
eld production on Ruby 3.1.5
• Bug #20474 — Heredoc common leading whitespace calculation
• Bug #20571 — Compile error on for loop
• Bug #20572 — Compilation hangs with nested until
Slide 78
Slide 78 text
By-products
• Bug #19835 — Parser leaks memory for incomplete tokens
• Bug #19836 — Parser leaks memory for incomplete lambdas
• Bug #20521 — Memory leak in Ripper parsing
Memory le
a
ks
Slide 79
Slide 79 text
By-products
• Bug #20219 — Segfault with circular parameter
• Bug #20339 — Parser segfault with ractor comment
• Bug #20341 — Parser segfault with shareable constant with array
• Bug #20468 — Segfault on safe navigation in for target
Segf
a
ults
Slide 80
Slide 80 text
By-products
• Bug #19904 — Deprecate or warn on multiple regular expression encodings
• Bug #19973 — Duplicate keyword argument names don't always warn
• Bug #20457 — Final `return` is eliminated from the AST
• Bug #20465 — parse.y adds an extra empty string to the AST
• Bug #20466 — Interpolated regular expressions have di
ff
erent encoding than
interpolated strings
Us
a
bility issues
Slide 81
Slide 81 text
Criticisms
批判
Slide 82
Slide 82 text
Criticisms
“L
a
ck of
a
form
a
l gr
a
mm
a
r will le
a
d to
a
synt
a
x combin
a
tion explosion”
正式な
文
法が
欠
如していると、構
文
の組み合わせが爆発してしまう。
Slide 83
Slide 83 text
Criticisms
b rescue d
b rescue d e
b rescue d(e)
b(c) rescue d
b(c) rescue d e
b(c) rescue d(e)
b c rescue d
b c rescue d e
b c rescue d(e)
a = b rescue d
a = b rescue d e
a = b rescue d(e)
a = b(c) rescue d
a = b(c) rescue d e
a = b(c) rescue d(e)
a = b c rescue d
a = b c rescue d e
a = b c rescue d(e)
Slide 84
Slide 84 text
Criticisms
for a in c do end
for a, b in c do end
for a, b, in c do end
for a, *, b in c do end
for (a, b) in c do end
for @a in c do end
for @@a in c do end
for $a in c do end
for A in c do end
for A::B in c do end
for a.b in c do end
for a::b in c do end
for a&.b in c do end
for a[:b] in c do end
for a[b:] in c do end
for a[&b] in c do end
Slide 85
Slide 85 text
Criticisms
for a in c do end
for a, b in c do end
for a, b, in c do end
for a, *, b in c do end
for (a, b) in c do end
for @a in c do end
for @@a in c do end
for $a in c do end
for A in c do end
for A::B in c do end
for a.b in c do end
for a::b in c do end
for a&.b in c do end
for a[:b] in c do end
for a[b:] in c do end
for a[&b] in c do end
💥💥 Segfault
Slide 86
Slide 86 text
Criticisms
for a in c do end
for a, b in c do end
for a, b, in c do end
for a, *, b in c do end
for (a, b) in c do end
for @a in c do end
for @@a in c do end
for $a in c do end
for A in c do end
for A::B in c do end
for a.b in c do end
for a::b in c do end
for a&.b in c do end
for a[:b] in c do end
for a[b:] in c do end
for a[&b] in c do end
If nobu doesn’t know about it, then no one knows about it.
nobu が知らなければ、誰も知らない。
Slide 87
Slide 87 text
Criticisms
“A h
a
nd-written p
a
rser is b
a
d for m
a
int
a
in
a
bility. LALR is better.”
手
書きのパーサーは保守性に悪い。LALRの
方
が良い。
Slide 88
Slide 88 text
Criticisms
Slide 89
Slide 89 text
Criticisms
Parser generator
• Ruby (CRuby)
• PostgreSQL/MySQL/SQLite
• PHP (Zend)
• Python (CPython)
Slide 90
Slide 90 text
Criticisms
Parser generator
• Ruby (CRuby)
• PostgreSQL/MySQL/SQLite
• PHP (Zend)
• Python (CPython)
Hand-written
• C/C++ (GCC)
• C/C++ (Clang)
• C# (Roslyn)
• CSS (Chromium)
• Go
• Java (OpenJDK)
• JavaScript (V8)
• Julia
• Lua
• Rust
• Swift
• TypeScript
• PHP (Microsoft)
Slide 91
Slide 91 text
Criticisms
Parser generator
• Ruby (CRuby)
• PostgreSQL/MySQL/SQLite
• PHP (Zend)
• Python (CPython)
Hand-written
• C/C++ (GCC)
• C/C++ (Clang)
• C# (Roslyn)
• CSS (Chromium)
• Go
• Java (OpenJDK)
• JavaScript (V8)
• Julia
• Lua
• Rust
• Swift
• TypeScript
• PHP (Microsoft)
Slide 92
Slide 92 text
Criticisms
“A h
a
nd-written p
a
rser requires more lines of code, it is not
a
s re
a
d
a
ble”
手
書きのパーサーは、より多くのコード
行
を必要とし、可読性に劣る。
Slide 93
Slide 93 text
Criticisms
It’s true, parse_conditional is very large.
確かにparse_conditionalは
非
常に
大
きい。
Slide 94
Slide 94 text
Criticisms
... but it does not have to be.
Now it is fewer lines than
parse.y.
…しかし、その必要はない。
今はparse.yより
行
数が少な
い。
Slide 95
Slide 95 text
Criticisms
“P
a
rser gener
a
tors
a
re b
a
sed on computer science.
H
a
nd-written p
a
rsers
a
re not.”
パーサー・ジェネレーターはコンピュータ科学に基づいている
手
書きのパーサーはそうではありません。
Slide 96
Slide 96 text
Criticisms
• Analysis of the Structure of Formula Translators (1961)
• Top Down Operator Precedence (1973)
• Evaluating and Improving Recursive Descent Parsers (1979)
• Incremental Recursive Descent Parsing (1990)
• Compact Recursive-Descent Parsing of Expressions (1985)
• Generalised Recursive Descent Parsing and Follow-Determinisms (1998)
• Parsing Expressions by Recursive Descent (1999)
• Recursive-descent Parsing (2012)
Slide 97
Slide 97 text
Status
ステータス
Slide 98
Slide 98 text
Status
Ruby implement
a
tions
Slide 99
Slide 99 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
Ruby implement
a
tions
Slide 100
Slide 100 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
Ruby implement
a
tions
Slide 101
Slide 101 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
Ruby implement
a
tions
Slide 102
Slide 102 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
• Natalie → serialization, FFI, C++, Ruby
Ruby implement
a
tions
Slide 103
Slide 103 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
• Natalie → serialization, FFI, C++, Ruby
• mruby → experimental
Ruby implement
a
tions
Slide 104
Slide 104 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
• Natalie → serialization, FFI, C++, Ruby
• mruby → experimental
• PicoRuby → experimental
Ruby implement
a
tions
Slide 105
Slide 105 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
• Natalie → serialization, FFI, C++, Ruby
• mruby → experimental
• PicoRuby → experimental
• Opal → experimental, Ruby, WASM
Ruby implement
a
tions
Slide 106
Slide 106 text
Status
• CRuby → C, optional in 3.3, default in next 3.4 preview
3.3ではオプション、次期3.4プレビューではデフォルト
• JRuby → serialization, FFI, Java, and WASM
• Tru
ffl
eRuby → serialization, FFI, Java
• Natalie → serialization, FFI, C++, Ruby
• mruby → experimental
• PicoRuby → experimental
• Opal → experimental, Ruby, WASM
• Garnet → WASM
Ruby implement
a
tions
Slide 107
Slide 107 text
Status
• minifyrb
• packwerk
• rbi
• rails
• repl_type_completer
• rubocop
Ruby libr
a
ries
a
nd tools
• ruby-lsp
• sorbet
• syntax_suggest
• syntax_tree
• typeprof
Slide 108
Slide 108 text
Status
• Involve the community of language developers from the start
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 109
Slide 109 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 110
Slide 110 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス ✅
• Same or better memory usage 同じかそれ以上のメモリ使
用
量
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 111
Slide 111 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス ✅
• Same or better memory usage 同じかそれ以上のメモリ使
用
量 ✅
• Error tolerant 誤差公差
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 112
Slide 112 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス ✅
• Same or better memory usage 同じかそれ以上のメモリ使
用
量 ✅
• Error tolerant 誤差公差 ✅
• Portable to other implementations and tools 他の実装やツールへの移植性
• Maintainable 維持可能
Criteri
a
Slide 113
Slide 113 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス ✅
• Same or better memory usage 同じかそれ以上のメモリ使
用
量 ✅
• Error tolerant 誤差公差 ✅
• Portable to other implementations and tools 他の実装やツールへの移植性 ✅
• Maintainable 維持可能
Criteri
a
Slide 114
Slide 114 text
Status
• Involve the community of language developers from the start ✅
言
語開発者のコミュニティを最初から巻き込む
• Same or better performance 同等以上のパフォーマンス ✅
• Same or better memory usage 同じかそれ以上のメモリ使
用
量 ✅
• Error tolerant 誤差公差 ✅
• Portable to other implementations and tools 他の実装やツールへの移植性 ✅
• Maintainable 維持可能 ✅
Criteri
a
Slide 115
Slide 115 text
Conclusions
結論
Slide 116
Slide 116 text
Having two parsers and compilers is bad for maintenance.
パーサーとコンパイラーを2つ持つことは
メンテナンス上良くない。
Slide 117
Slide 117 text
It will be a very large amount of work to
reimplement Prism internals.
Prismの内部を再実装するのは、
非
常に膨
大
な作業になるだろう。
Slide 118
Slide 118 text
If we work together on Prism,
we could have the best possible parser and compiler.
Prismで協
力
し合えば、
最
高
のパーサーとコンパイラができるだろう。
Slide 119
Slide 119 text
Kevin Newton, Shopify
Why Prism?
Fukuok
a
RubyistK
a
igi 04