Let’s create a programming language! [SoundCloud HQ edition]

Slide 1

Slide 1 text

Let’s create a programming  language! DENIS DEFREYNE SOUNDCLOUD, BERLIN JUNE 7, 2016

Slide 2

Slide 2 text

2 dynamic vs. static

Slide 3

Slide 3 text

3 dynamic vs. static

Slide 4

Slide 4 text

4 compiler interpreter vs.

Slide 5

Slide 5 text

5 compiler interpreter vs.

Slide 6

Slide 6 text

but why? 6

Slide 7

Slide 7 text

incremental 7

Slide 8

Slide 8 text

Language 1 8 Numbers

Slide 9

Slide 9 text

% 9 cat stuff.cke 4173 5 27 2016

Slide 10

Slide 10 text

% 10 cat stuff.cke | ./clarke 4173 5 27 2016

Slide 11

Slide 11 text

11 module Grammar extend DParse::DSL DIGIT = char_in('0'..'9') PROGRAM = DIGIT end

Slide 12

Slide 12 text

12 res = Grammar::PROGRAM.apply($stdin.read)   case res when DParse::Success handle_success(res) when DParse::Failure handle_failure(res) end

Slide 13

Slide 13 text

13 def handle_success(success) p success.data end

Slide 14

Slide 14 text

14 DIGIT = char_in('0'..'9')              

Slide 15

Slide 15 text

15 DIGIT = … NUMBER = repeat1(DIGIT)        

Slide 16

Slide 16 text

16 DIGIT = … NUMBER = repeat1(DIGIT) .capture        

Slide 17

Slide 17 text

17 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i }        

Slide 18

Slide 18 text

18 DIGIT = … NUMBER = … EXPRESSION = NUMBER          

Slide 19

Slide 19 text

19 DIGIT = … NUMBER = … EXPRESSION = … EXPRESSIONS = intersperse( EXPRESSION, char("\n").ignore, ).compact

Slide 20

Slide 20 text

20 DIGIT = … NUMBER = …  EXPRESSION = … EXPRESSIONS = … PROGRAM = seq(EXPRESSIONS, eof) .first   

Slide 21

Slide 21 text

% 21 cat stuff.cke 4173 5 27 2016

Slide 22

Slide 22 text

% 22 cat stuff.cke | ./clarke [4173, 5, 27, 2016]

Slide 23

Slide 23 text

23 def handle_success(success) p success.data end

Slide 24

Slide 24 text

24 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

Slide 25

Slide 25 text

25 def eval_expr(expr) expr end

Slide 26

Slide 26 text

% 26 cat stuff.cke | ./clarke 4173 5 27 2016

Slide 27

Slide 27 text

% 27 echo "aa" | ./clarke expected digit at line 1, column 1 aa ↑

Slide 28

Slide 28 text

Language 2 28 Function calls

Slide 29

Slide 29 text

% 29 cat stuff.cke 4173 add(2, 5)

Slide 30

Slide 30 text

% 30 cat stuff.cke | ./clarke 4173 7

Slide 31

Slide 31 text

31 LETTER = char_in('a'..'z') FUNCTION_NAME = repeat1(LETTER).capture

Slide 32

Slide 32 text

32 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(','), ), char(')'), )

Slide 33

Slide 33 text

33 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')'), )

Slide 34

Slide 34 text

34 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')').ignore, ).compact

Slide 35

Slide 35 text

35 EXPRESSION = NUMBER

Slide 36

Slide 36 text

36 EXPRESSION = alt(NUMBER, FUNCTION_CALL)

Slide 37

Slide 37 text

% 37 cat stuff.cke 4173 add(2, 5)

Slide 38

Slide 38 text

% 38 cat stuff.cke | ./clarke 4173 ["add", [2, 5]]

Slide 39

Slide 39 text

39 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')').ignore, ).compact

Slide 40

Slide 40 text

40 def eval_expr(expr) expr end

Slide 41

Slide 41 text

41 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end

Slide 42

Slide 42 text

42 def eval_function_call(expr) case expr[0] when 'add' expr[1] .reduce(:+) when 'mul' … end end

Slide 43

Slide 43 text

43 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 44

Slide 44 text

% 44 cat stuff.cke 4173 add(2, add(1, 4))

Slide 45

Slide 45 text

% 45 cat stuff.cke | ./clarke 4173 7

Slide 46

Slide 46 text

Ugh, we forgot about whitespace. 46

Slide 47

Slide 47 text

Cleanup 1 47 Abstract syntax

Slide 48

Slide 48 text

48 class IntNode < Struct.new(:value) end class FunCallNode < Struct.new(:name, :arguments) end

Slide 49

Slide 49 text

49 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i }        

Slide 50

Slide 50 text

50 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| IntNode.new(d.to_i) }        

Slide 51

Slide 51 text

51 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact

Slide 52

Slide 52 text

52 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact.map do |data| FunCallNode.new(data[0], data[1]) end

Slide 53

Slide 53 text

53 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end

Slide 54

Slide 54 text

54 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end

Slide 55

Slide 55 text

55 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 56

Slide 56 text

56 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 57

Slide 57 text

Cleanup 2 57 Environment

Slide 58

Slide 58 text

58 $env = {} $env['add'] = FunDef.new( ['a', 'b'], ->(a, b) { a + b }, )

Slide 59

Slide 59 text

59 class FunDef < Struct.new(:args, :body) end

Slide 60

Slide 60 text

60 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 61

Slide 61 text

61 def eval_function_call(expr) fun = $env[expr.name] values = expr.args.map { |e| eval_expr(e) } fun.body.call(*values) end

Slide 62

Slide 62 text

… with error handling, of course. 62

Slide 63

Slide 63 text

Language 3 63 Variables

Slide 64

Slide 64 text

% 64 cat stuff.cke 4173 add(2,amount)

Slide 65

Slide 65 text

% 65 cat stuff.cke | ./clarke 4173 7

Slide 66

Slide 66 text

66 class VarNode < Struct.new(:name) end

Slide 67

Slide 67 text

67 VARIABLE_NAME = repeat1(LETTER).capture

Slide 68

Slide 68 text

68 VARIABLE_REFERENCE = VARIABLE_NAME .map { |d| VarNode.new(d) }

Slide 69

Slide 69 text

69 EXPRESSION = alt(  NUMBER,  FUNCTION_CALL,  ) 

Slide 70

Slide 70 text

70 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )

Slide 71

Slide 71 text

71 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end   

Slide 72

Slide 72 text

72 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) when VarNode eval_var(expr) end end

Slide 73

Slide 73 text

73 def eval_var(expr) $env[expr.name] end

Slide 74

Slide 74 text

74 $env['amount'] = 5

Slide 75

Slide 75 text

Language 3.1 75 Variable assignment

Slide 76

Slide 76 text

% 76 cat stuff.cke 4173 amount=5 add(2,amount)

Slide 77

Slide 77 text

% 77 cat stuff.cke | ./clarke 4173 7

Slide 78

Slide 78 text

78 class AssignNode < Struct.new(:name, :rhs) end

Slide 79

Slide 79 text

79 ASSIGNMENT = seq( VARIABLE_NAME, char('='), lazy { EXPRESSION }, )

Slide 80

Slide 80 text

80 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact

Slide 81

Slide 81 text

81 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact.map do |data| AssignNode.new(data[0], data[1]) end

Slide 82

Slide 82 text

82 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ) 

Slide 83

Slide 83 text

83 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )

Slide 84

Slide 84 text

84 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end

Slide 85

Slide 85 text

85 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end

Slide 86

Slide 86 text

Cleanup 3 86 Print function

Slide 87

Slide 87 text

87 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

Slide 88

Slide 88 text

88 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

Slide 89

Slide 89 text

89 $env['print'] = FunDef.new( ['a'], ->(a) { p a }, )

Slide 90

Slide 90 text

% 90 cat stuff.cke 4173 amount=5 print(add(2,amount))

Slide 91

Slide 91 text

% 91 cat stuff.cke | ./clarke 7

Slide 92

Slide 92 text

Cleanup 4 92 Immutable env

Slide 93

Slide 93 text

93 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end

Slide 94

Slide 94 text

94 def eval_expr(expr, env) case expr when … … when AssignNode eval_assign(expr, env) end end

Slide 95

Slide 95 text

95 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end   

Slide 96

Slide 96 text

96 def eval_assign(expr, env) $env[expr.name] = eval_expr(expr.rhs) end   

Slide 97

Slide 97 text

97 def eval_assign(expr, env) val = eval_expr(expr.rhs) new_env = env.merge(expr.name => val) ValAndEnv.new(val, new_env) end

Slide 98

Slide 98 text

98 class ValAndEnv < Struct.new(:val, :env) end

Slide 99

Slide 99 text

99 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

Slide 100

Slide 100 text

100 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV)      

Slide 101

Slide 101 text

101 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) success.data.reduce(init) do |result, e| eval_expr(e, result.env) end end

Slide 102

Slide 102 text

102 INITIAL_ENV = { 'add' => FunDef.new( ['a', 'b'], ->(a, b) { a + b }, ) }

Slide 103

Slide 103 text

Language 4 103 Scopes

Slide 104

Slide 104 text

% 104 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))

Slide 105

Slide 105 text

% 105 cat stuff.cke | ./clarke 7

Slide 106

Slide 106 text

106 class ScopeNode < Struct.new(:exprs) end

Slide 107

Slide 107 text

107 SCOPE = seq( char('{'), repeat1(lazy { EXPRESSION }), char('}'), )

Slide 108

Slide 108 text

108 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact

Slide 109

Slide 109 text

109 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact.map do |data| ScopeNode.new(data[0]) end

Slide 110

Slide 110 text

110 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, ) 

Slide 111

Slide 111 text

111 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, SCOPE, )

Slide 112

Slide 112 text

112 def eval_expr(expr, env) case expr when … … when ScopeNode eval_scope(expr, env) end end

Slide 113

Slide 113 text

113 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end

Slide 114

Slide 114 text

114 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end

Slide 115

Slide 115 text

% 115 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))

Slide 116

Slide 116 text

Language 5 116 Conditionals

Slide 117

Slide 117 text

% 117 cat stuff.cke if (0) { print(100) } else { print(200) }

Slide 118

Slide 118 text

% 118 cat stuff.cke | ./clarke 200

Slide 119

Slide 119 text

SKIP 119

Slide 120

Slide 120 text

Language 6 120 Function definitions

Slide 121

Slide 121 text

% 121 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))

Slide 122

Slide 122 text

122 class FunDefNode < Struct.new( :name, :argument_names, :body) end

Slide 123

Slide 123 text

123 FUNCTION_DEF = seq( string('fun'), FUNCTION_NAME, char('('), intersperse( VARIABLE_NAME, char(','), ), char(')'), SCOPE, )   

Slide 124

Slide 124 text

124 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact   

Slide 125

Slide 125 text

125 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact.map do |data| FunDefNode.new(data[0], data[1], data[2]) end

Slide 126

Slide 126 text

126 class FunDef < Struct.new(:argument_names, :body) end

Slide 127

Slide 127 text

127

Slide 128

Slide 128 text

127 def eval_function_def(expr, env)

Slide 129

Slide 129 text

127 def eval_function_def(expr, env) fun_def = FunDef.new(  expr.argument_names, expr.body) 

Slide 130

Slide 130 text

127 def eval_function_def(expr, env) fun_def = FunDef.new(  expr.argument_names, expr.body)  ValAndEnv.new(  fun_def,  env.merge(expr.name => fun_def),  )  end

Slide 131

Slide 131 text

128 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) }         fun.body.call(*values)

Slide 132

Slide 132 text

129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)

Slide 133

Slide 133 text

129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)

Slide 134

Slide 134 text

130

Slide 135

Slide 135 text

130 when Scope  new_env = env.merge(  Hash[fun.argument_names.zip(values)]) 

Slide 136

Slide 136 text

130 when Scope  new_env = env.merge(  Hash[fun.argument_names.zip(values)])  eval_scope(fun.body, new_env)  end

Slide 137

Slide 137 text

% 131 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))

Slide 138

Slide 138 text

Language 7 132 Operators

Slide 139

Slide 139 text

% 133 cat stuff.cke fun multiply(a, b) { if (b) { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))

Slide 140

Slide 140 text

134 10 - 2 - 5 * 2 + 2 ^ 3 ^ 4

Slide 141

Slide 141 text

135 1. Precedence

Slide 142

Slide 142 text

136 4 + 5 * 2_

Slide 143

Slide 143 text

137 4 + (5 * 2)

Slide 144

Slide 144 text

138 (4 + 5) * 2 _

Slide 145

Slide 145 text

138 (4 + 5) * 2 _ WRONG!

Slide 146

Slide 146 text

139 2. Associativity

Slide 147

Slide 147 text

140 2 ^ 3 ^ 4_

Slide 148

Slide 148 text

141 2 ^ (3 ^ 4)

Slide 149

Slide 149 text

142 (2 ^ 3) ^ 4__

Slide 150

Slide 150 text

142 (2 ^ 3) ^ 4__ WRONG!

Slide 151

Slide 151 text

143 Shunting-yard algorithm (en.wikipedia.org/wiki/Shunting-yard_algorithm)

Slide 152

Slide 152 text

144

Slide 153

Slide 153 text

144 input: infix notation e.g. a + b * c

Slide 154

Slide 154 text

144 input: infix notation e.g. a + b * c precedences e.g. a + b * c

Slide 155

Slide 155 text

144 input: infix notation e.g. a + b * c precedences e.g. a + b * c associativities e.g. a + b * c

Slide 156

Slide 156 text

144 input: infix notation e.g. a + b * c precedences e.g. a + b * c associativities e.g. a + b * c   output: postfix notation e.g. a b c * +

Slide 157

Slide 157 text

145 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ")              

Slide 158

Slide 158 text

146 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, )

Slide 159

Slide 159 text

147 tokens = "10 - 2 - 5 * 2 + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, ) puts res.join(' ')

Slide 160

Slide 160 text

148 PRECEDENCES = { '^' => 3, '*' => 2, '/' => 2, '+' => 1, '-' => 1, }

Slide 161

Slide 161 text

149 ASSOCIATIVITIES = { '^' => :right, '*' => :left, '/' => :left, '+' => :left, '-' => :left, }

Slide 162

Slide 162 text

150 10 2 - 5 2 * - 2 3 4 ^ ^ +

Slide 163

Slide 163 text

151 10 2 - 5 2 * - 2 3 4 ^ ^ + 10

Slide 164

Slide 164 text

152 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 10

Slide 165

Slide 165 text

153 10 2 - 5 2 * - 2 3 4 ^ ^ + 8

Slide 166

Slide 166 text

154 10 2 - 5 2 * - 2 3 4 ^ ^ + 5 8

Slide 167

Slide 167 text

155 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 5 8

Slide 168

Slide 168 text

156 10 2 - 5 2 * - 2 3 4 ^ ^ + 10 8

Slide 169

Slide 169 text

157 10 2 - 5 2 * - 2 3 4 ^ ^ + -2

Slide 170

Slide 170 text

158 10 2 - 5 2 * - 2 3 4 ^ ^ + 2 -2

Slide 171

Slide 171 text

159 10 2 - 5 2 * - 2 3 4 ^ ^ + 3 2 -2

Slide 172

Slide 172 text

160 10 2 - 5 2 * - 2 3 4 ^ ^ + 4 3 2 -2

Slide 173

Slide 173 text

161 10 2 - 5 2 * - 2 3 4 ^ ^ + 81 2 -2

Slide 174

Slide 174 text

162 10 2 - 5 2 * - 2 3 4 ^ ^ + 241785163 -2

Slide 175

Slide 175 text

163 10 2 - 5 2 * - 2 3 4 ^ ^ + 241785163

Slide 176

Slide 176 text

164 OPERATOR = alt( char('^'), char('*'), char('/'), char('+'), char('-'), )

Slide 177

Slide 177 text

165 BIN_OP_EXPRESSION = intersperse( OPERAND, OPERATOR, )

Slide 178

Slide 178 text

166 postfix_tokens = shunting_yard(  expr.tokens, PRECEDENCES, ASSOCIATIVITIES) stack = [] postfix_tokens.each do |e| if e.is_a?(Clarke::AST::Op) values = [stack.pop, stack.pop] stack.push(eval_op(e, values)) else stack.push(e) end end

Slide 179

Slide 179 text

167 def eval_op(e, values) case e.name when '^' values.reduce(:**) when '+' values.reduce(:+) when … end end

Slide 180

Slide 180 text

% 168 cat stuff.cke fun multiply(a, b) { if (b) { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))

Slide 181

Slide 181 text

etc.

Slide 182

Slide 182 text

Types! Loops! Closures! Objects!  Classes! Inheritance! Tuples! Records! Typechecking! Multimethods! Enumerations! Proper lexical scoping! Pattern matching! Currying! Modules! 170

Slide 183

Slide 183 text

171

Slide 184

Slide 184 text

171 L1 Numbers

Slide 185

Slide 185 text

171 L1 Numbers L2 Function calls

Slide 186

Slide 186 text

171 L1 Numbers L2 Function calls C1 Abstract syntax

Slide 187

Slide 187 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment

Slide 188

Slide 188 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables

Slide 189

Slide 189 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function

Slide 190

Slide 190 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment

Slide 191

Slide 191 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes

Slide 192

Slide 192 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals

Slide 193

Slide 193 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions

Slide 194

Slide 194 text

171 L1 Numbers L2 Function calls C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions L7 Operators

Slide 195

Slide 195 text

172 github.com/ddfreyne/clarke

Slide 196

Slide 196 text

172 github.com/ddfreyne/clarke SOON

Slide 197

Slide 197 text

173 Ready to interpret your questions. DENIS@SOUNDCLOUD.COM @DENIS