Let’s create a programming language! [SoundCloud HQ edition]

Let’s create a programming language! [SoundCloud HQ edition]

Be732ee41fd3038aa98a0a7e7b7be081?s=128

Denis Defreyne

June 07, 2016
Tweet

Transcript

  1. Let’s create a programming
 language! DENIS DEFREYNE SOUNDCLOUD, BERLIN JUNE

    7, 2016
  2. 2 dynamic vs. static

  3. 3 dynamic vs. static

  4. 4 compiler interpreter vs.

  5. 5 compiler interpreter vs.

  6. but why? 6

  7. incremental 7

  8. Language 1 8 Numbers

  9. % 9 cat stuff.cke 4173 5 27 2016

  10. % 10 cat stuff.cke | ./clarke 4173 5 27 2016

  11. 11 module Grammar extend DParse::DSL DIGIT = char_in('0'..'9') PROGRAM =

    DIGIT end
  12. 12 res = Grammar::PROGRAM.apply($stdin.read) 
 case res when DParse::Success handle_success(res)

    when DParse::Failure handle_failure(res) end
  13. 13 def handle_success(success) p success.data end

  14. 14 DIGIT = char_in('0'..'9') 
 
 
 
 
 


  15. 15 DIGIT = … NUMBER = repeat1(DIGIT) 
 
 


  16. 16 DIGIT = … NUMBER = repeat1(DIGIT) .capture 
 


    
 

  17. 17 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {

    |d| d.to_i } 
 
 
 

  18. 18 DIGIT = … NUMBER = … EXPRESSION = NUMBER

    
 
 
 
 

  19. 19 DIGIT = … NUMBER = … EXPRESSION = …

    EXPRESSIONS = intersperse( EXPRESSION, char("\n").ignore, ).compact
  20. 20 DIGIT = … NUMBER = …
 EXPRESSION = …

    EXPRESSIONS = … PROGRAM = seq(EXPRESSIONS, eof) .first
 

  21. % 21 cat stuff.cke 4173 5 27 2016

  22. % 22 cat stuff.cke | ./clarke [4173, 5, 27, 2016]

  23. 23 def handle_success(success) p success.data end

  24. 24 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

  25. 25 def eval_expr(expr) expr end

  26. % 26 cat stuff.cke | ./clarke 4173 5 27 2016

  27. % 27 echo "aa" | ./clarke expected digit at line

    1, column 1 aa ↑
  28. Language 2 28 Function calls

  29. % 29 cat stuff.cke 4173 add(2, 5)

  30. % 30 cat stuff.cke | ./clarke 4173 7

  31. 31 LETTER = char_in('a'..'z') FUNCTION_NAME = repeat1(LETTER).capture

  32. 32 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION

    }, char(','), ), char(')'), )
  33. 33 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION

    }, char(',').ignore, ).compact, char(')'), )
  34. 34 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION

    }, char(',').ignore, ).compact, char(')').ignore, ).compact
  35. 35 EXPRESSION = NUMBER

  36. 36 EXPRESSION = alt(NUMBER, FUNCTION_CALL)

  37. % 37 cat stuff.cke 4173 add(2, 5)

  38. % 38 cat stuff.cke | ./clarke 4173 ["add", [2, 5]]

  39. 39 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION

    }, char(',').ignore, ).compact, char(')').ignore, ).compact
  40. 40 def eval_expr(expr) expr end

  41. 41 def eval_expr(expr) case expr when Integer expr when Array

    eval_function_call(expr) end end
  42. 42 def eval_function_call(expr) case expr[0] when 'add' expr[1] .reduce(:+) when

    'mul' … end end
  43. 43 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map {

    |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
  44. % 44 cat stuff.cke 4173 add(2, add(1, 4))

  45. % 45 cat stuff.cke | ./clarke 4173 7

  46. Ugh, we forgot about whitespace. 46

  47. Cleanup 1 47 Abstract syntax

  48. 48 class IntNode < Struct.new(:value) end class FunCallNode < Struct.new(:name,

    :arguments) end
  49. 49 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {

    |d| d.to_i } 
 
 
 

  50. 50 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map {

    |d| IntNode.new(d.to_i) } 
 
 
 

  51. 51 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact

  52. 52 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact.map do

    |data| FunCallNode.new(data[0], data[1]) end
  53. 53 def eval_expr(expr) case expr when Integer expr when Array

    eval_function_call(expr) end end
  54. 54 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode

    eval_function_call(expr) end end
  55. 55 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map {

    |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
  56. 56 def eval_function_call(expr) case expr.name when 'add' expr.args .map {

    |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
  57. Cleanup 2 57 Environment

  58. 58 $env = {} $env['add'] = FunDef.new( ['a', 'b'], ->(a,

    b) { a + b }, )
  59. 59 class FunDef < Struct.new(:args, :body) end

  60. 60 def eval_function_call(expr) case expr.name when 'add' expr.args .map {

    |e| eval_expr(e) } .reduce(:+) when 'mul' … end end
  61. 61 def eval_function_call(expr) fun = $env[expr.name] values = expr.args.map {

    |e| eval_expr(e) } fun.body.call(*values) end
  62. … with error handling, of course. 62

  63. Language 3 63 Variables

  64. % 64 cat stuff.cke 4173 add(2,amount)

  65. % 65 cat stuff.cke | ./clarke 4173 7

  66. 66 class VarNode < Struct.new(:name) end

  67. 67 VARIABLE_NAME = repeat1(LETTER).capture

  68. 68 VARIABLE_REFERENCE = VARIABLE_NAME .map { |d| VarNode.new(d) }

  69. 69 EXPRESSION = alt(
 NUMBER,
 FUNCTION_CALL,
 )


  70. 70 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )

  71. 71 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode

    eval_function_call(expr) end end
 

  72. 72 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode

    eval_function_call(expr) when VarNode eval_var(expr) end end
  73. 73 def eval_var(expr) $env[expr.name] end

  74. 74 $env['amount'] = 5

  75. Language 3.1 75 Variable assignment

  76. % 76 cat stuff.cke 4173 amount=5 add(2,amount)

  77. % 77 cat stuff.cke | ./clarke 4173 7

  78. 78 class AssignNode < Struct.new(:name, :rhs) end

  79. 79 ASSIGNMENT = seq( VARIABLE_NAME, char('='), lazy { EXPRESSION },

    )
  80. 80 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION },

    ).compact
  81. 81 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION },

    ).compact.map do |data| AssignNode.new(data[0], data[1]) end
  82. 82 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )


  83. 83 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )

  84. 84 def eval_expr(expr) case expr when … … when AssignNode

    eval_assign(expr) end end
  85. 85 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end

  86. Cleanup 3 86 Print function

  87. 87 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

  88. 88 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

  89. 89 $env['print'] = FunDef.new( ['a'], ->(a) { p a },

    )
  90. % 90 cat stuff.cke 4173 amount=5 print(add(2,amount))

  91. % 91 cat stuff.cke | ./clarke 7

  92. Cleanup 4 92 Immutable env

  93. 93 def eval_expr(expr) case expr when … … when AssignNode

    eval_assign(expr) end end
  94. 94 def eval_expr(expr, env) case expr when … … when

    AssignNode eval_assign(expr, env) end end
  95. 95 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end
 


  96. 96 def eval_assign(expr, env) $env[expr.name] = eval_expr(expr.rhs) end
 


  97. 97 def eval_assign(expr, env) val = eval_expr(expr.rhs) new_env = env.merge(expr.name

    => val) ValAndEnv.new(val, new_env) end
  98. 98 class ValAndEnv < Struct.new(:val, :env) end

  99. 99 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

  100. 100 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) 
 
 


  101. 101 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) success.data.reduce(init) do |result,

    e| eval_expr(e, result.env) end end
  102. 102 INITIAL_ENV = { 'add' => FunDef.new( ['a', 'b'], ->(a,

    b) { a + b }, ) }
  103. Language 4 103 Scopes

  104. % 104 cat stuff.cke 4173 amount = { a =

    2 b = 3 add(a, b) } print(add(2, amount))
  105. % 105 cat stuff.cke | ./clarke 7

  106. 106 class ScopeNode < Struct.new(:exprs) end

  107. 107 SCOPE = seq( char('{'), repeat1(lazy { EXPRESSION }), char('}'),

    )
  108. 108 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore,

    ).compact
  109. 109 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore,

    ).compact.map do |data| ScopeNode.new(data[0]) end
  110. 110 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )


  111. 111 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, SCOPE, )

  112. 112 def eval_expr(expr, env) case expr when … … when

    ScopeNode eval_scope(expr, env) end end
  113. 113 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result =

    expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
  114. 114 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result =

    expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end
  115. % 115 cat stuff.cke 4173 amount = { a =

    2 b = 3 add(a, b) } print(add(2, amount))
  116. Language 5 116 Conditionals

  117. % 117 cat stuff.cke if (0) { print(100) } else

    { print(200) }
  118. % 118 cat stuff.cke | ./clarke 200

  119. SKIP 119

  120. Language 6 120 Function definitions

  121. % 121 cat stuff.cke fun multiply(a, b) { if (b)

    { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
  122. 122 class FunDefNode < Struct.new( :name, :argument_names, :body) end

  123. 123 FUNCTION_DEF = seq( string('fun'), FUNCTION_NAME, char('('), intersperse( VARIABLE_NAME, char(','),

    ), char(')'), SCOPE, )
 

  124. 124 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore,

    ).compact, char(')').ignore, SCOPE, ).compact
 

  125. 125 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore,

    ).compact, char(')').ignore, SCOPE, ).compact.map do |data| FunDefNode.new(data[0], data[1], data[2]) end
  126. 126 class FunDef < Struct.new(:argument_names, :body) end

  127. 127

  128. 127 def eval_function_def(expr, env)

  129. 127 def eval_function_def(expr, env) fun_def = FunDef.new(
 expr.argument_names, expr.body)


  130. 127 def eval_function_def(expr, env) fun_def = FunDef.new(
 expr.argument_names, expr.body)
 ValAndEnv.new(


    fun_def,
 env.merge(expr.name => fun_def),
 )
 end
  131. 128 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map

    { |e| eval_expr(e) } 
 
 
 
 fun.body.call(*values)
  132. 129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map

    { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
  133. 129 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map

    { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)
  134. 130

  135. 130 when Scope
 new_env = env.merge(
 Hash[fun.argument_names.zip(values)])


  136. 130 when Scope
 new_env = env.merge(
 Hash[fun.argument_names.zip(values)])
 eval_scope(fun.body, new_env)
 end

  137. % 131 cat stuff.cke fun multiply(a, b) { if (b)

    { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))
  138. Language 7 132 Operators

  139. % 133 cat stuff.cke fun multiply(a, b) { if (b)

    { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
  140. 134 10 - 2 - 5 * 2 + 2

    ^ 3 ^ 4
  141. 135 1. Precedence

  142. 136 4 + 5 * 2_

  143. 137 4 + (5 * 2)

  144. 138 (4 + 5) * 2 _

  145. 138 (4 + 5) * 2 _ WRONG!

  146. 139 2. Associativity

  147. 140 2 ^ 3 ^ 4_

  148. 141 2 ^ (3 ^ 4)

  149. 142 (2 ^ 3) ^ 4__

  150. 142 (2 ^ 3) ^ 4__ WRONG!

  151. 143 Shunting-yard algorithm (en.wikipedia.org/wiki/Shunting-yard_algorithm)

  152. 144

  153. 144 input: infix notation e.g. a + b * c

  154. 144 input: infix notation e.g. a + b * c

    precedences e.g. a + b * c
  155. 144 input: infix notation e.g. a + b * c

    precedences e.g. a + b * c associativities e.g. a + b * c
  156. 144 input: infix notation e.g. a + b * c

    precedences e.g. a + b * c associativities e.g. a + b * c 
 output: postfix notation e.g. a b c * +
  157. 145 tokens = "10 - 2 - 5 * 2

    + 2 ^ 3 ^ 4" .split(" ") 
 
 
 
 
 
 

  158. 146 tokens = "10 - 2 - 5 * 2

    + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, )
  159. 147 tokens = "10 - 2 - 5 * 2

    + 2 ^ 3 ^ 4" .split(" ") res = shunting_yard( tokens, PRECEDENCES, ASSOCIATIVITIES, ) puts res.join(' ')
  160. 148 PRECEDENCES = { '^' => 3, '*' => 2,

    '/' => 2, '+' => 1, '-' => 1, }
  161. 149 ASSOCIATIVITIES = { '^' => :right, '*' => :left,

    '/' => :left, '+' => :left, '-' => :left, }
  162. 150 10 2 - 5 2 * - 2 3

    4 ^ ^ +
  163. 151 10 2 - 5 2 * - 2 3

    4 ^ ^ + 10
  164. 152 10 2 - 5 2 * - 2 3

    4 ^ ^ + 2 10
  165. 153 10 2 - 5 2 * - 2 3

    4 ^ ^ + 8
  166. 154 10 2 - 5 2 * - 2 3

    4 ^ ^ + 5 8
  167. 155 10 2 - 5 2 * - 2 3

    4 ^ ^ + 2 5 8
  168. 156 10 2 - 5 2 * - 2 3

    4 ^ ^ + 10 8
  169. 157 10 2 - 5 2 * - 2 3

    4 ^ ^ + -2
  170. 158 10 2 - 5 2 * - 2 3

    4 ^ ^ + 2 -2
  171. 159 10 2 - 5 2 * - 2 3

    4 ^ ^ + 3 2 -2
  172. 160 10 2 - 5 2 * - 2 3

    4 ^ ^ + 4 3 2 -2
  173. 161 10 2 - 5 2 * - 2 3

    4 ^ ^ + 81 2 -2
  174. 162 10 2 - 5 2 * - 2 3

    4 ^ ^ + 241785163 -2
  175. 163 10 2 - 5 2 * - 2 3

    4 ^ ^ + 241785163
  176. 164 OPERATOR = alt( char('^'), char('*'), char('/'), char('+'), char('-'), )

  177. 165 BIN_OP_EXPRESSION = intersperse( OPERAND, OPERATOR, )

  178. 166 postfix_tokens = shunting_yard(
 expr.tokens, PRECEDENCES, ASSOCIATIVITIES) stack = []

    postfix_tokens.each do |e| if e.is_a?(Clarke::AST::Op) values = [stack.pop, stack.pop] stack.push(eval_op(e, values)) else stack.push(e) end end
  179. 167 def eval_op(e, values) case e.name when '^' values.reduce(:**) when

    '+' values.reduce(:+) when … end end
  180. % 168 cat stuff.cke fun multiply(a, b) { if (b)

    { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))
  181. etc.

  182. Types! Loops! Closures! Objects!
 Classes! Inheritance! Tuples! Records! Typechecking! Multimethods!

    Enumerations! Proper lexical scoping! Pattern matching! Currying! Modules! 170
  183. 171

  184. 171 L1 Numbers

  185. 171 L1 Numbers L2 Function calls

  186. 171 L1 Numbers L2 Function calls C1 Abstract syntax

  187. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment
  188. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables
  189. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function
  190. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function C4 Immutable environment
  191. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes
  192. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals
  193. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions
  194. 171 L1 Numbers L2 Function calls C1 Abstract syntax C2

    Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions L7 Operators
  195. 172 github.com/ddfreyne/clarke

  196. 172 github.com/ddfreyne/clarke SOON

  197. 173 Ready to interpret your questions. DENIS@SOUNDCLOUD.COM @DENIS