Let’s create a programming language! [SoundCloud HQ edition]

Transcript

1. Let’s create a programming
 language! DENIS DEFREYNE SOUNDCLOUD, BERLIN JUNE 7, 2016
2. 2 dynamic vs. static
3. 3 dynamic vs. static
4. 4 compiler interpreter vs.
5. 5 compiler interpreter vs.
6. but why? 6
7. incremental 7
8. Language 1 8 Numbers
9. % 9 cat stuff.cke 4173 5 27 2016
10. % 10 cat stuff.cke | ./clarke 4173 5 27 2016
11. 11 module Grammar extend DParse::DSL DIGIT = char_in('0'..'9') PROGRAM = DIGIT end
12. 12 res = Grammar::PROGRAM.apply($stdin.read) 
 case res when DParse::Success handle_success(res) when DParse::Failure handle_failure(res) end
13. 13 def handle_success(success) p success.data end
14. 14 DIGIT = char_in('0'..'9') 
 
 
 
 
 
 

15. 15 DIGIT = … NUMBER = repeat1(DIGIT) 
 
 
 

16. 16 DIGIT = … NUMBER = repeat1(DIGIT) .capture 
 
 
 

17. 17 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i } 
 
 
 

18. 18 DIGIT = … NUMBER = … EXPRESSION = NUMBER 
 
 
 
 

19. 19 DIGIT = … NUMBER = … EXPRESSION = … EXPRESSIONS = intersperse( EXPRESSION, char("\n").ignore, ).compact
20. 20 DIGIT = … NUMBER = …
 EXPRESSION = … EXPRESSIONS = … PROGRAM = seq(EXPRESSIONS, eof) .first
 

21. % 21 cat stuff.cke 4173 5 27 2016
22. % 22 cat stuff.cke | ./clarke [4173, 5, 27, 2016]
23. 23 def handle_success(success) p success.data end
24. 24 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end
25. 25 def eval_expr(expr) expr end
26. % 26 cat stuff.cke | ./clarke 4173 5 27 2016
27. % 27 echo "aa" | ./clarke expected digit at line 1, column 1 aa ↑
28. Language 2 28 Function calls
29. % 29 cat stuff.cke 4173 add(2, 5)
30. % 30 cat stuff.cke | ./clarke 4173 7
31. 31 LETTER = char_in('a'..'z') FUNCTION_NAME = repeat1(LETTER).capture
32. 32 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(','), ), char(')'), )
33. 33 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')'), )
34. 34 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')').ignore, ).compact
35. 35 EXPRESSION = NUMBER
36. 36 EXPRESSION = alt(NUMBER, FUNCTION_CALL)
37. % 37 cat stuff.cke 4173 add(2, 5)
38. % 38 cat stuff.cke | ./clarke 4173 ["add", [2, 5]]
39. 39 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')').ignore, ).compact
40. 40 def eval_expr(expr) expr end
41. 41 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end
42. 42 def eval_function_call(expr) case expr[0] when 'add' expr[1] .reduce(:+) when 'mul' … end end
43. 43 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
44. % 44 cat stuff.cke 4173 add(2, add(1, 4))
45. % 45 cat stuff.cke | ./clarke 4173 7
46. Ugh, we forgot about whitespace. 46
47. Cleanup 1 47 Abstract syntax
48. 48 class IntNode < Struct.new(:value) end class FunCallNode < Struct.new(:name, :arguments) end
49. 49 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i } 
 
 
 

50. 50 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| IntNode.new(d.to_i) } 
 
 
 

51. 51 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact
52. 52 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact.map do |data| FunCallNode.new(data[0], data[1]) end
53. 53 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end
54. 54 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end
55. 55 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
56. 56 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
57. Cleanup 2 57 Environment
58. 58 $env = {} $env['add'] = FunDef.new( ['a', 'b'], ->(a, b) { a + b }, )
59. 59 class FunDef < Struct.new(:args, :body) end
60. 60 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
61. 61 def eval_function_call(expr) fun = $env[expr.name] values = expr.args.map { |e| eval_expr(e) } fun.body.call(*values) end
62. … with error handling, of course. 62
63. Language 3 63 Variables
64. % 64 cat stuff.cke 4173 add(2,amount)
65. % 65 cat stuff.cke | ./clarke 4173 7
66. 66 class VarNode < Struct.new(:name) end
67. 67 VARIABLE_NAME = repeat1(LETTER).capture
68. 68 VARIABLE_REFERENCE = VARIABLE_NAME .map { |d| VarNode.new(d) }
69. 69 EXPRESSION = alt(
 NUMBER,
 FUNCTION_CALL,
 )

70. 70 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )
71. 71 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end
 

72. 72 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) when VarNode eval_var(expr) end end
73. 73 def eval_var(expr) $env[expr.name] end
74. 74 $env['amount'] = 5
75. Language 3.1 75 Variable assignment
76. % 76 cat stuff.cke 4173 amount=5 add(2,amount)
77. % 77 cat stuff.cke | ./clarke 4173 7
78. 78 class AssignNode < Struct.new(:name, :rhs) end
79. 79 ASSIGNMENT = seq( VARIABLE_NAME, char('='), lazy { EXPRESSION }, )
80. 80 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact
81. 81 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact.map do |data| AssignNode.new(data[0], data[1]) end
82. 82 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )

83. 83 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )
84. 84 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end
85. 85 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end
86. Cleanup 3 86 Print function
87. 87 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end
88. 88 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end
89. 89 $env['print'] = FunDef.new( ['a'], ->(a) { p a }, )
90. % 90 cat stuff.cke 4173 amount=5 print(add(2,amount))
91. % 91 cat stuff.cke | ./clarke 7
92. Cleanup 4 92 Immutable env
93. 93 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end
94. 94 def eval_expr(expr, env) case expr when … … when AssignNode eval_assign(expr, env) end end
95. 95 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end
 

96. 96 def eval_assign(expr, env) $env[expr.name] = eval_expr(expr.rhs) end
 

97. 97 def eval_assign(expr, env) val = eval_expr(expr.rhs) new_env = env.merge(expr.name => val) ValAndEnv.new(val, new_env) end
98. 98 class ValAndEnv < Struct.new(:val, :env) end
99. 99 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end
100. 100 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) 
 
 

101. 101 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) success.data.reduce(init) do |result, e| eval_expr(e, result.env) end end
102. 102 INITIAL_ENV = { 'add' => FunDef.new( ['a', 'b'], ->(a, b) { a + b }, ) }
103. Language 4 103 Scopes
104. % 104 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))
105. % 105 cat stuff.cke | ./clarke 7
106. 106 class ScopeNode < Struct.new(:exprs) end
107. 107 SCOPE = seq( char('{'), repeat1(lazy { EXPRESSION }), char('}'), )
108. 108 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact
109. 109 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact.map do |data| ScopeNode.new(data[0]) end
110. 110 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )

111. 111 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, SCOPE, )
112. 112 def eval_expr(expr, env) case expr when … … when ScopeNode eval_scope(expr, env) end end
113. 113 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
114. 114 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
115. % 115 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))
116. Language 5 116 Conditionals
117. % 117 cat stuff.cke if (0) { print(100) } else { print(200) }
118. % 118 cat stuff.cke | ./clarke 200
119. SKIP 119
120. Language 6 120 Function definitions
121. % 121 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
122. 122 class FunDefNode < Struct.new( :name, :argument_names, :body) end
123. 123 FUNCTION_DEF = seq( string('fun'), FUNCTION_NAME, char('('), intersperse( VARIABLE_NAME, char(','), ), char(')'), SCOPE, )
 

124. 124 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact
 

125. 125 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact.map do |data| FunDefNode.new(data[0], data[1], data[2]) end
126. 126 class FunDef < Struct.new(:argument_names, :body) end
128. 127 def eval_function_def(expr, env)
129. 127 def eval_function_def(expr, env) fun_def = FunDef.new(
 expr.argument_names, expr.body)

130. 127 def eval_function_def(expr, env) fun_def = FunDef.new(
 expr.argument_names, expr.body)
 ValAndEnv.new(
 fun_def,
 env.merge(expr.name => fun_def),
 )
 end
131. 128 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } 
 
 
 
 fun.body.call(*values)
132. 129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
133. 129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
135. 130 when Scope
 new_env = env.merge(
 Hash[fun.argument_names.zip(values)])

136. 130 when Scope
 new_env = env.merge(
 Hash[fun.argument_names.zip(values)])
 eval_scope(fun.body, new_env)
 end
137. % 131 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
138. Language 7 132 Operators
139. % 133 cat stuff.cke fun multiply(a, b) { if (b) { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
140. 134 10 - 2 - 5 * 2 + 2 ^ 3 ^ 4
141. 135 1. Precedence
142. 136 4 + 5 * 2_
143. 137 4 + (5 * 2)
144. 138 (4 + 5) * 2 _
145. 138 (4 + 5) * 2 _ WRONG!
146. 139 2. Associativity
147. 140 2 ^ 3 ^ 4_
148. 141 2 ^ (3 ^ 4)
149. 142 (2 ^ 3) ^ 4__
150. 142 (2 ^ 3) ^ 4__ WRONG!
151. 143 Shunting-yard algorithm (en.wikipedia.org/wiki/Shunting-yard_algorithm)
153. 144 input: infix notation e.g. a + b * c
154. 144 input: infix notation e.g. a + b * c precedences e.g. a + b * c
155. 144 input: infix notation e.g. a + b * c precedences e.g. a + b * c associativities e.g. a + b * c
156. 144 input: infix notation e.g. a + b * c precedences e.g. a + b * c associativities e.g. a + b * c 
 output: postfix notation e.g. a b c * +
157. 145 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ") 
 
 
 
 
 
 

158. 146 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, )
159. 147 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, ) puts res.join(' ')
160. 148 PRECEDENCES = { '^' => 3, '*' => 2, '/' => 2, '+' => 1, '-' => 1, }
161. 149 ASSOCIATIVITIES = { '^' => :right, '*' => :left, '/' => :left, '+' => :left, '-' => :left, }
162. 150 10 2 - 5 2 * - 2 3 4 ^ ^ +
163. 151 10 2 - 5 2 * - 2 3 4 ^ ^ + 10
164. 152 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 10
165. 153 10 2 - 5 2 * - 2 3 4 ^ ^ + 8
166. 154 10 2 - 5 2 * - 2 3 4 ^ ^ + 5 8
167. 155 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 5 8
168. 156 10 2 - 5 2 * - 2 3 4 ^ ^ + 10 8
169. 157 10 2 - 5 2 * - 2 3 4 ^ ^ + -2
170. 158 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 -2
171. 159 10 2 - 5 2 * - 2 3 4 ^ ^ + 3 2 -2
172. 160 10 2 - 5 2 * - 2 3 4 ^ ^ + 4 3 2 -2
173. 161 10 2 - 5 2 * - 2 3 4 ^ ^ + 81 2 -2
174. 162 10 2 - 5 2 * - 2 3 4 ^ ^ + 241785163 -2
175. 163 10 2 - 5 2 * - 2 3 4 ^ ^ + 241785163
176. 164 OPERATOR = alt( char('^'), char('*'), char('/'), char('+'), char('-'), )
177. 165 BIN_OP_EXPRESSION = intersperse( OPERAND, OPERATOR, )
178. 166 postfix_tokens = shunting_yard(
 expr.tokens, PRECEDENCES, ASSOCIATIVITIES) stack = [] postfix_tokens.each do |e| if e.is_a?(Clarke::AST::Op) values = [stack.pop, stack.pop] stack.push(eval_op(e, values)) else stack.push(e) end end
179. 167 def eval_op(e, values) case e.name when '^' values.reduce(:**) when '+' values.reduce(:+) when … end end
180. % 168 cat stuff.cke fun multiply(a, b) { if (b) { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
181. etc.
182. Types! Loops! Closures! Objects!
 Classes! Inheritance! Tuples! Records! Typechecking! Multimethods! Enumerations! Proper lexical scoping! Pattern matching! Currying! Modules! 170
184. 171 L1 Numbers
185. 171 L1 Numbers L2 Function calls
186. 171 L1 Numbers L2 Function calls C1 Abstract syntax
187. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment
188. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables
189. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function
190. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment
191. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes
192. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals
193. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions
194. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions L7 Operators
195. 172 github.com/ddfreyne/clarke
196. 172 github.com/ddfreyne/clarke SOON
197. 173 Ready to interpret your questions. DENIS@SOUNDCLOUD.COM @DENIS