Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Let’s create a programming language! [SoundCloud HQ edition]
Search
Denis Defreyne
June 07, 2016
Programming
0
170
Let’s create a programming language! [SoundCloud HQ edition]
Denis Defreyne
June 07, 2016
Tweet
Share
More Decks by Denis Defreyne
See All by Denis Defreyne
The importance of naming
ddfreyne
0
44
An introduction to fibers
ddfreyne
0
150
Code as data (RubyConfBY 2019 edition)
ddfreyne
0
87
Code as data
ddfreyne
0
140
How to memoize
ddfreyne
0
140
Clean & fast code with enumerators
ddfreyne
0
110
Fibers
ddfreyne
0
410
Let’s create a programming language! [RUG::B edition]
ddfreyne
1
210
Let’s write a parser! [SoundCloud HQ edition]
ddfreyne
0
200
Other Decks in Programming
See All in Programming
Polarsの成長: v0.14からv1.0までの変遷と今後の展望
zerebom
1
350
CSC307 Lecture 12
javiergs
PRO
0
220
iOSアプリでクリップボードにコピーしたことをユーザーに伝えるちょうど良いフィードバックを探す
ski
0
100
Cloudflare Workers x AWS Lambdaの組み合わせユースケース / Cloudflare Workers x AWS Lambda Combination Use Case
seike460
PRO
2
310
英語
s_shimotori
1
220
SRE チーム立ち上げ前に考えたこと・取り組んだこと / Considerations and Preparations Before Establishing an SRE Team
mackey0225
3
320
さきがけから振り返るアーキテクチャ刷新 / Reflecting on the Architectural Renewal from the Vanguard
nrslib
2
780
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
450
AHC035解説
terryu16
0
730
Temporalを取り巻く仕様を整理する
sajikix
0
120
ドメイン駆動設計の実践
masuda220
PRO
19
5.2k
AWS CDKにおける「再利用性」を考える / aws-cdk-reusability
gotok365
6
1.3k
Featured
See All Featured
Facilitating Awesome Meetings
lara
46
5.8k
Git: the NoSQL Database
bkeepers
PRO
423
64k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
26
1.8k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.5k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
12
3.8k
Rails Girls Zürich Keynote
gr2m
93
13k
Designing on Purpose - Digital PM Summit 2013
jponch
113
6.6k
The MySQL Ecosystem @ GitHub 2015
samlambert
248
12k
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Large-scale JavaScript Application Architecture
addyosmani
506
110k
Transcript
Let’s create a programming language! DENIS DEFREYNE SOUNDCLOUD, BERLIN JUNE
7, 2016
2 dynamic vs. static
3 dynamic vs. static
4 compiler interpreter vs.
5 compiler interpreter vs.
but why? 6
incremental 7
Language 1 8 Numbers
% 9 cat stuff.cke 4173 5 27 2016
% 10 cat stuff.cke | ./clarke 4173 5 27 2016
11 module Grammar extend DParse::DSL DIGIT = char_in('0'..'9') PROGRAM =
DIGIT end
12 res = Grammar::PROGRAM.apply($stdin.read) case res when DParse::Success handle_success(res)
when DParse::Failure handle_failure(res) end
13 def handle_success(success) p success.data end
14 DIGIT = char_in('0'..'9')
15 DIGIT = … NUMBER = repeat1(DIGIT)
16 DIGIT = … NUMBER = repeat1(DIGIT) .capture
17 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {
|d| d.to_i }
18 DIGIT = … NUMBER = … EXPRESSION = NUMBER
19 DIGIT = … NUMBER = … EXPRESSION = …
EXPRESSIONS = intersperse( EXPRESSION, char("\n").ignore, ).compact
20 DIGIT = … NUMBER = … EXPRESSION = …
EXPRESSIONS = … PROGRAM = seq(EXPRESSIONS, eof) .first
% 21 cat stuff.cke 4173 5 27 2016
% 22 cat stuff.cke | ./clarke [4173, 5, 27, 2016]
23 def handle_success(success) p success.data end
24 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end
25 def eval_expr(expr) expr end
% 26 cat stuff.cke | ./clarke 4173 5 27 2016
% 27 echo "aa" | ./clarke expected digit at line
1, column 1 aa ↑
Language 2 28 Function calls
% 29 cat stuff.cke 4173 add(2, 5)
% 30 cat stuff.cke | ./clarke 4173 7
31 LETTER = char_in('a'..'z') FUNCTION_NAME = repeat1(LETTER).capture
32 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION
}, char(','), ), char(')'), )
33 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION
}, char(',').ignore, ).compact, char(')'), )
34 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION
}, char(',').ignore, ).compact, char(')').ignore, ).compact
35 EXPRESSION = NUMBER
36 EXPRESSION = alt(NUMBER, FUNCTION_CALL)
% 37 cat stuff.cke 4173 add(2, 5)
% 38 cat stuff.cke | ./clarke 4173 ["add", [2, 5]]
39 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION
}, char(',').ignore, ).compact, char(')').ignore, ).compact
40 def eval_expr(expr) expr end
41 def eval_expr(expr) case expr when Integer expr when Array
eval_function_call(expr) end end
42 def eval_function_call(expr) case expr[0] when 'add' expr[1] .reduce(:+) when
'mul' … end end
43 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map {
|e| eval_expr(e) } .reduce(:+) when 'mul' … end end
% 44 cat stuff.cke 4173 add(2, add(1, 4))
% 45 cat stuff.cke | ./clarke 4173 7
Ugh, we forgot about whitespace. 46
Cleanup 1 47 Abstract syntax
48 class IntNode < Struct.new(:value) end class FunCallNode < Struct.new(:name,
:arguments) end
49 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {
|d| d.to_i }
50 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {
|d| IntNode.new(d.to_i) }
51 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact
52 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact.map do
|data| FunCallNode.new(data[0], data[1]) end
53 def eval_expr(expr) case expr when Integer expr when Array
eval_function_call(expr) end end
54 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode
eval_function_call(expr) end end
55 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map {
|e| eval_expr(e) } .reduce(:+) when 'mul' … end end
56 def eval_function_call(expr) case expr.name when 'add' expr.args .map {
|e| eval_expr(e) } .reduce(:+) when 'mul' … end end
Cleanup 2 57 Environment
58 $env = {} $env['add'] = FunDef.new( ['a', 'b'], ->(a,
b) { a + b }, )
59 class FunDef < Struct.new(:args, :body) end
60 def eval_function_call(expr) case expr.name when 'add' expr.args .map {
|e| eval_expr(e) } .reduce(:+) when 'mul' … end end
61 def eval_function_call(expr) fun = $env[expr.name] values = expr.args.map {
|e| eval_expr(e) } fun.body.call(*values) end
… with error handling, of course. 62
Language 3 63 Variables
% 64 cat stuff.cke 4173 add(2,amount)
% 65 cat stuff.cke | ./clarke 4173 7
66 class VarNode < Struct.new(:name) end
67 VARIABLE_NAME = repeat1(LETTER).capture
68 VARIABLE_REFERENCE = VARIABLE_NAME .map { |d| VarNode.new(d) }
69 EXPRESSION = alt( NUMBER, FUNCTION_CALL, )
70 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )
71 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode
eval_function_call(expr) end end
72 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode
eval_function_call(expr) when VarNode eval_var(expr) end end
73 def eval_var(expr) $env[expr.name] end
74 $env['amount'] = 5
Language 3.1 75 Variable assignment
% 76 cat stuff.cke 4173 amount=5 add(2,amount)
% 77 cat stuff.cke | ./clarke 4173 7
78 class AssignNode < Struct.new(:name, :rhs) end
79 ASSIGNMENT = seq( VARIABLE_NAME, char('='), lazy { EXPRESSION },
)
80 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION },
).compact
81 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION },
).compact.map do |data| AssignNode.new(data[0], data[1]) end
82 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )
83 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )
84 def eval_expr(expr) case expr when … … when AssignNode
eval_assign(expr) end end
85 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end
Cleanup 3 86 Print function
87 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end
88 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end
89 $env['print'] = FunDef.new( ['a'], ->(a) { p a },
)
% 90 cat stuff.cke 4173 amount=5 print(add(2,amount))
% 91 cat stuff.cke | ./clarke 7
Cleanup 4 92 Immutable env
93 def eval_expr(expr) case expr when … … when AssignNode
eval_assign(expr) end end
94 def eval_expr(expr, env) case expr when … … when
AssignNode eval_assign(expr, env) end end
95 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end
96 def eval_assign(expr, env) $env[expr.name] = eval_expr(expr.rhs) end
97 def eval_assign(expr, env) val = eval_expr(expr.rhs) new_env = env.merge(expr.name
=> val) ValAndEnv.new(val, new_env) end
98 class ValAndEnv < Struct.new(:val, :env) end
99 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end
100 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV)
101 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) success.data.reduce(init) do |result,
e| eval_expr(e, result.env) end end
102 INITIAL_ENV = { 'add' => FunDef.new( ['a', 'b'], ->(a,
b) { a + b }, ) }
Language 4 103 Scopes
% 104 cat stuff.cke 4173 amount = { a =
2 b = 3 add(a, b) } print(add(2, amount))
% 105 cat stuff.cke | ./clarke 7
106 class ScopeNode < Struct.new(:exprs) end
107 SCOPE = seq( char('{'), repeat1(lazy { EXPRESSION }), char('}'),
)
108 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore,
).compact
109 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore,
).compact.map do |data| ScopeNode.new(data[0]) end
110 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )
111 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, SCOPE, )
112 def eval_expr(expr, env) case expr when … … when
ScopeNode eval_scope(expr, env) end end
113 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result =
expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
114 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result =
expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
% 115 cat stuff.cke 4173 amount = { a =
2 b = 3 add(a, b) } print(add(2, amount))
Language 5 116 Conditionals
% 117 cat stuff.cke if (0) { print(100) } else
{ print(200) }
% 118 cat stuff.cke | ./clarke 200
SKIP 119
Language 6 120 Function definitions
% 121 cat stuff.cke fun multiply(a, b) { if (b)
{ add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
122 class FunDefNode < Struct.new( :name, :argument_names, :body) end
123 FUNCTION_DEF = seq( string('fun'), FUNCTION_NAME, char('('), intersperse( VARIABLE_NAME, char(','),
), char(')'), SCOPE, )
124 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore,
).compact, char(')').ignore, SCOPE, ).compact
125 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore,
).compact, char(')').ignore, SCOPE, ).compact.map do |data| FunDefNode.new(data[0], data[1], data[2]) end
126 class FunDef < Struct.new(:argument_names, :body) end
127
127 def eval_function_def(expr, env)
127 def eval_function_def(expr, env) fun_def = FunDef.new( expr.argument_names, expr.body)
127 def eval_function_def(expr, env) fun_def = FunDef.new( expr.argument_names, expr.body) ValAndEnv.new(
fun_def, env.merge(expr.name => fun_def), ) end
128 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map
{ |e| eval_expr(e) } fun.body.call(*values)
129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map
{ |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map
{ |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
130
130 when Scope new_env = env.merge( Hash[fun.argument_names.zip(values)])
130 when Scope new_env = env.merge( Hash[fun.argument_names.zip(values)]) eval_scope(fun.body, new_env) end
% 131 cat stuff.cke fun multiply(a, b) { if (b)
{ add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
Language 7 132 Operators
% 133 cat stuff.cke fun multiply(a, b) { if (b)
{ a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
134 10 - 2 - 5 * 2 + 2
^ 3 ^ 4
135 1. Precedence
136 4 + 5 * 2_
137 4 + (5 * 2)
138 (4 + 5) * 2 _
138 (4 + 5) * 2 _ WRONG!
139 2. Associativity
140 2 ^ 3 ^ 4_
141 2 ^ (3 ^ 4)
142 (2 ^ 3) ^ 4__
142 (2 ^ 3) ^ 4__ WRONG!
143 Shunting-yard algorithm (en.wikipedia.org/wiki/Shunting-yard_algorithm)
144
144 input: infix notation e.g. a + b * c
144 input: infix notation e.g. a + b * c
precedences e.g. a + b * c
144 input: infix notation e.g. a + b * c
precedences e.g. a + b * c associativities e.g. a + b * c
144 input: infix notation e.g. a + b * c
precedences e.g. a + b * c associativities e.g. a + b * c output: postfix notation e.g. a b c * +
145 tokens = "10 - 2 - 5 * 2
+ 2 ^ 3 ^ 4" .split(" ")
146 tokens = "10 - 2 - 5 * 2
+ 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, )
147 tokens = "10 - 2 - 5 * 2
+ 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, ) puts res.join(' ')
148 PRECEDENCES = { '^' => 3, '*' => 2,
'/' => 2, '+' => 1, '-' => 1, }
149 ASSOCIATIVITIES = { '^' => :right, '*' => :left,
'/' => :left, '+' => :left, '-' => :left, }
150 10 2 - 5 2 * - 2 3
4 ^ ^ +
151 10 2 - 5 2 * - 2 3
4 ^ ^ + 10
152 10 2 - 5 2 * - 2 3
4 ^ ^ + 2 10
153 10 2 - 5 2 * - 2 3
4 ^ ^ + 8
154 10 2 - 5 2 * - 2 3
4 ^ ^ + 5 8
155 10 2 - 5 2 * - 2 3
4 ^ ^ + 2 5 8
156 10 2 - 5 2 * - 2 3
4 ^ ^ + 10 8
157 10 2 - 5 2 * - 2 3
4 ^ ^ + -2
158 10 2 - 5 2 * - 2 3
4 ^ ^ + 2 -2
159 10 2 - 5 2 * - 2 3
4 ^ ^ + 3 2 -2
160 10 2 - 5 2 * - 2 3
4 ^ ^ + 4 3 2 -2
161 10 2 - 5 2 * - 2 3
4 ^ ^ + 81 2 -2
162 10 2 - 5 2 * - 2 3
4 ^ ^ + 241785163 -2
163 10 2 - 5 2 * - 2 3
4 ^ ^ + 241785163
164 OPERATOR = alt( char('^'), char('*'), char('/'), char('+'), char('-'), )
165 BIN_OP_EXPRESSION = intersperse( OPERAND, OPERATOR, )
166 postfix_tokens = shunting_yard( expr.tokens, PRECEDENCES, ASSOCIATIVITIES) stack = []
postfix_tokens.each do |e| if e.is_a?(Clarke::AST::Op) values = [stack.pop, stack.pop] stack.push(eval_op(e, values)) else stack.push(e) end end
167 def eval_op(e, values) case e.name when '^' values.reduce(:**) when
'+' values.reduce(:+) when … end end
% 168 cat stuff.cke fun multiply(a, b) { if (b)
{ a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
etc.
Types! Loops! Closures! Objects! Classes! Inheritance! Tuples! Records! Typechecking! Multimethods!
Enumerations! Proper lexical scoping! Pattern matching! Currying! Modules! 170
171
171 L1 Numbers
171 L1 Numbers L2 Function calls
171 L1 Numbers L2 Function calls C1 Abstract syntax
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function C4 Immutable environment
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions
171 L1 Numbers L2 Function calls C1 Abstract syntax C2
Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions L7 Operators
172 github.com/ddfreyne/clarke
172 github.com/ddfreyne/clarke SOON
173 Ready to interpret your questions.
[email protected]
@DENIS