Let’s create a programming language! [RUG::B edition]

Slide 1

Slide 1 text

Let’s create a programming  language! DENIS DEFREYNE RUG : :B JUNE 2, 2016

Slide 2

Slide 2 text

2 dynamic vs. static

Slide 3

Slide 3 text

3 dynamic vs. static

Slide 4

Slide 4 text

4 compiler interpreter vs.

Slide 5

Slide 5 text

5 compiler interpreter vs.

Slide 6

Slide 6 text

but why? 6

Slide 7

Slide 7 text

incremental 7

Slide 8

Slide 8 text

Language 1 8 Numbers

Slide 9

Slide 9 text

% 9 cat stuff.cke 4173 5 27 2016

Slide 10

Slide 10 text

% 10 cat stuff.cke | ./clarke 4173 5 27 2016

Slide 11

Slide 11 text

11 module Grammar extend DParse::DSL DIGIT = char_in('0'..'9') PROGRAM = DIGIT end

Slide 12

Slide 12 text

12 res = Grammar::PROGRAM.apply($stdin.read)   case res when DParse::Success handle_success(res) when DParse::Failure handle_failure(res) end

Slide 13

Slide 13 text

13 def handle_success(success) p success.data end

Slide 14

Slide 14 text

14 DIGIT = char_in('0'..'9')              

Slide 15

Slide 15 text

15 DIGIT = … NUMBER = repeat1(DIGIT)        

Slide 16

Slide 16 text

16 DIGIT = … NUMBER = repeat1(DIGIT) .capture        

Slide 17

Slide 17 text

17 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i }        

Slide 18

Slide 18 text

18 DIGIT = … NUMBER = … EXPRESSION = NUMBER          

Slide 19

Slide 19 text

19 DIGIT = … NUMBER = … EXPRESSION = … EXPRESSIONS = intersperse( EXPRESSION, char("\n").ignore, ).compact

Slide 20

Slide 20 text

20 DIGIT = … NUMBER = …  EXPRESSION = … EXPRESSIONS = … PROGRAM = seq(EXPRESSIONS, eof) .first   

Slide 21

Slide 21 text

% 21 cat stuff.cke 4173 5 27 2016

Slide 22

Slide 22 text

% 22 cat stuff.cke | ./clarke [4173, 5, 27, 2016]

Slide 23

Slide 23 text

23 def handle_success(success) p success.data end

Slide 24

Slide 24 text

24 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

Slide 25

Slide 25 text

25 def eval_expr(expr) expr end

Slide 26

Slide 26 text

% 26 cat stuff.cke | ./clarke 4173 5 27 2016

Slide 27

Slide 27 text

% 27 echo "aa" | ./clarke expected digit at line 1, column 1 aa ↑

Slide 28

Slide 28 text

Language 2 28 Functions

Slide 29

Slide 29 text

% 29 cat stuff.cke 4173 add(2, 5)

Slide 30

Slide 30 text

% 30 cat stuff.cke | ./clarke 4173 7

Slide 31

Slide 31 text

31 LETTER = char_in('a'..'z') FUNCTION_NAME = repeat1(LETTER).capture

Slide 32

Slide 32 text

32 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(','), ), char(')'), )

Slide 33

Slide 33 text

33 FUNCTION_CALL = seq( FUNCTION_NAME, char('('), intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')'), )

Slide 34

Slide 34 text

34 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse( lazy { EXPRESSION }, char(',').ignore, ).compact, char(')').ignore, ).compact

Slide 35

Slide 35 text

35 EXPRESSION = NUMBER

Slide 36

Slide 36 text

36 EXPRESSION = alt(NUMBER, FUNCTION_CALL)

Slide 37

Slide 37 text

% 37 cat stuff.cke 4173 add(2, 5)

Slide 38

Slide 38 text

% 38 cat stuff.cke | ./clarke 4173 ["add", [2, 5]]

Slide 39

Slide 39 text

39 def eval_expr(expr) expr end

Slide 40

Slide 40 text

40 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end

Slide 41

Slide 41 text

41 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 42

Slide 42 text

% 42 cat stuff.cke 4173 add(2, 5)

Slide 43

Slide 43 text

% 43 cat stuff.cke | ./clarke 4173 7

Slide 44

Slide 44 text

Ugh, we forgot about whitespace. 44

Slide 45

Slide 45 text

Cleanup 1 45 Abstract syntax

Slide 46

Slide 46 text

46 class IntNode < Struct.new(:value) end class FunCallNode < Struct.new(:name, :arguments) end

Slide 47

Slide 47 text

47 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| d.to_i }        

Slide 48

Slide 48 text

48 DIGIT = … NUMBER = repeat1(DIGIT) .capture .map { |d| IntNode.new(d.to_i) }        

Slide 49

Slide 49 text

49 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact

Slide 50

Slide 50 text

50 FUNCTION_CALL = seq( FUNCTION_NAME, char('(').ignore, intersperse(…), char(')').ignore, ).compact.map do |data| FunCallNode.new(data[0], data[1]) end

Slide 51

Slide 51 text

51 def eval_expr(expr) case expr when Integer expr when Array eval_function_call(expr) end end

Slide 52

Slide 52 text

52 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end

Slide 53

Slide 53 text

53 def eval_function_call(expr) case expr[0] when 'add' expr[1] .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 54

Slide 54 text

54 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 55

Slide 55 text

Cleanup 2 55 Environment

Slide 56

Slide 56 text

56 class FunDef < Struct.new(:args, :body) end

Slide 57

Slide 57 text

57 $env = {} $env['add'] = FunDef.new( ['a', 'b'], ->(a, b) { a + b }, )

Slide 58

Slide 58 text

58 def eval_function_call(expr) case expr.name when 'add' expr.args .map { |e| eval_expr(e) } .reduce(:+) when 'mul' … end end

Slide 59

Slide 59 text

59 def eval_function_call(expr) fun = $env[expr.name] values = expr.args.map { |e| eval_expr(e) } fun.body.call(*values) end

Slide 60

Slide 60 text

… with error handling, of course. 60

Slide 61

Slide 61 text

Language 3 61 Variables

Slide 62

Slide 62 text

% 62 cat stuff.cke 4173 add(2,amount)

Slide 63

Slide 63 text

% 63 cat stuff.cke | ./clarke 4173 7

Slide 64

Slide 64 text

64 class VarNode < Struct.new(:name) end

Slide 65

Slide 65 text

65 VARIABLE_NAME = repeat1(LETTER).capture

Slide 66

Slide 66 text

66 VARIABLE_REFERENCE = VARIABLE_NAME .map { |d| VarNode.new(d) }

Slide 67

Slide 67 text

67 EXPRESSION = alt(  NUMBER,  FUNCTION_CALL,  ) 

Slide 68

Slide 68 text

68 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, )

Slide 69

Slide 69 text

69 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) end end   

Slide 70

Slide 70 text

70 def eval_expr(expr) case expr when IntNode expr.value when FunCallNode eval_function_call(expr) when VarNode eval_var(expr) end end

Slide 71

Slide 71 text

71 def eval_var(expr) $env[expr.name] end

Slide 72

Slide 72 text

72 $env['amount'] = 5

Slide 73

Slide 73 text

Language 3.1 73 Variable assignment

Slide 74

Slide 74 text

% 74 cat stuff.cke 4173 amount=5 add(2,amount)

Slide 75

Slide 75 text

% 75 cat stuff.cke | ./clarke 4173 7

Slide 76

Slide 76 text

76 class AssignNode < Struct.new(:name, :rhs) end

Slide 77

Slide 77 text

77 ASSIGNMENT = seq( VARIABLE_NAME, char('='), lazy { EXPRESSION }, )

Slide 78

Slide 78 text

78 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact

Slide 79

Slide 79 text

79 ASSIGNMENT = seq( VARIABLE_NAME, char('=').ignore, lazy { EXPRESSION }, ).compact.map do |data| AssignNode.new(data[0], data[1]) end

Slide 80

Slide 80 text

80 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ) 

Slide 81

Slide 81 text

81 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, )

Slide 82

Slide 82 text

82 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end

Slide 83

Slide 83 text

83 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end

Slide 84

Slide 84 text

Cleanup 3 84 Print function

Slide 85

Slide 85 text

85 def handle_success(success) success.data.each do |expr| p eval_expr(expr) end end

Slide 86

Slide 86 text

86 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

Slide 87

Slide 87 text

87 $env['print'] = FunDef.new( ['a'], ->(a) { p a }, )

Slide 88

Slide 88 text

% 88 cat stuff.cke 4173 amount=5 print(add(2,amount))

Slide 89

Slide 89 text

% 89 cat stuff.cke | ./clarke 7

Slide 90

Slide 90 text

Cleanup 4 90 Immutable env

Slide 91

Slide 91 text

91 def eval_expr(expr) case expr when … … when AssignNode eval_assign(expr) end end

Slide 92

Slide 92 text

92 def eval_expr(expr, env) case expr when … … when AssignNode eval_assign(expr, env) end end

Slide 93

Slide 93 text

93 def eval_assign(expr) $env[expr.name] = eval_expr(expr.rhs) end   

Slide 94

Slide 94 text

94 def eval_assign(expr, env) $env[expr.name] = eval_expr(expr.rhs) end   

Slide 95

Slide 95 text

95 def eval_assign(expr, env) val = eval_expr(expr.rhs) new_env = env.merge(expr.name => val) ValAndEnv.new(val, new_env) end

Slide 96

Slide 96 text

96 class ValAndEnv < Struct.new(:val, :env) end

Slide 97

Slide 97 text

97 def handle_success(success) success.data.each do |expr| eval_expr(expr) end end

Slide 98

Slide 98 text

98 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV)      

Slide 99

Slide 99 text

99 def handle_success(success) init = ValAndEv.new(0, INITIAL_ENV) success.data.reduce(init) do |result, e| eval_expr(e, result.env) end end

Slide 100

Slide 100 text

100 INITIAL_ENV = { 'add' => FunDef.new( ['a', 'b'], ->(a, b) { a + b }, ) }

Slide 101

Slide 101 text

Language 4 101 Scopes

Slide 102

Slide 102 text

% 102 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))

Slide 103

Slide 103 text

% 103 cat stuff.cke | ./clarke 7

Slide 104

Slide 104 text

104 class ScopeNode < Struct.new(:exprs) end

Slide 105

Slide 105 text

105 SCOPE = seq( char('{'), repeat1(lazy { EXPRESSION }), char('}'), )

Slide 106

Slide 106 text

106 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact

Slide 107

Slide 107 text

107 SCOPE = seq( char('{').ignore, repeat1(lazy { EXPRESSION }), char('}').ignore, ).compact.map do |data| ScopeNode.new(data[0]) end

Slide 108

Slide 108 text

108 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, ) 

Slide 109

Slide 109 text

109 EXPRESSION = alt( NUMBER, FUNCTION_CALL, VARIABLE_REFERENCE, ASSIGNMENT, SCOPE, )

Slide 110

Slide 110 text

110 def eval_expr(expr, env) case expr when … … when ScopeNode eval_scope(expr, env) end end

Slide 111

Slide 111 text

111 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end

Slide 112

Slide 112 text

112 def eval_scope(expr, env) init = ValAndEnv.new(0, env) result = expr.exprs.reduce(init) do |result, e| eval_expr(e, result.env) end ValAndEnv.new(result.val, env) end

Slide 113

Slide 113 text

% 113 cat stuff.cke 4173 amount = { a = 2 b = 3 add(a, b) } print(add(2, amount))

Slide 114

Slide 114 text

Language 5 114 Conditionals

Slide 115

Slide 115 text

% 115 cat stuff.cke if (0) { print(100) } else { print(200) }

Slide 116

Slide 116 text

% 116 cat stuff.cke | ./clarke 200

Slide 117

Slide 117 text

SKIP 117

Slide 118

Slide 118 text

Language 6 118 Function definitions

Slide 119

Slide 119 text

% 119 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))

Slide 120

Slide 120 text

120 class FunDefNode < Struct.new( :name, :argument_names, :body) end

Slide 121

Slide 121 text

121 FUNCTION_DEF = seq( string('fun'), FUNCTION_NAME, char('('), intersperse( VARIABLE_NAME, char(','), ), char(')'), SCOPE, )   

Slide 122

Slide 122 text

122 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact   

Slide 123

Slide 123 text

123 FUNCTION_DEF = seq( string('fun').ignore, FUNCTION_NAME, char('(').ignore, intersperse( VARIABLE_NAME, char(',').ignore, ).compact, char(')').ignore, SCOPE, ).compact.map do |data| FunDefNode.new(data[0], data[1], data[2]) end

Slide 124

Slide 124 text

124 class FunDef < Struct.new(:argument_names, :body) end

Slide 125

Slide 125 text

125

Slide 126

Slide 126 text

125 def eval_function_def(expr, env)

Slide 127

Slide 127 text

125 def eval_function_def(expr, env) fun_def = FunDef.new(  expr.argument_names, expr.body) 

Slide 128

Slide 128 text

125 def eval_function_def(expr, env) fun_def = FunDef.new(  expr.argument_names, expr.body)  ValAndEnv.new(  fun_def,  env.merge(expr.name => fun_def),  )  end

Slide 129

Slide 129 text

126 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) }         fun.body.call(*values)

Slide 130

Slide 130 text

127 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)

Slide 131

Slide 131 text

127 def eval_function_call(expr, env) fun = env[expr.name] values = expr.args.map { |e| eval_expr(e) } case fun.body when Proc when Scope … fun.body.call(*values)

Slide 132

Slide 132 text

128

Slide 133

Slide 133 text

128 when Scope  new_env = env.merge(  Hash[fun.argument_names.zip(values)]) 

Slide 134

Slide 134 text

128 when Scope  new_env = env.merge(  Hash[fun.argument_names.zip(values)])  eval_scope(fun.body, new_env)  end

Slide 135

Slide 135 text

% 129 cat stuff.cke fun multiply(a, b) { if (b) { add(a, multiply(a, sub(b, 1))) } else { 0 } } print(multiply(3, 5))

Slide 136

Slide 136 text

Language 7 130 Operators

Slide 137

Slide 137 text

% 131 cat stuff.cke fun multiply(a, b) { if (b) { a + multiply(a, b - 1) } else { 0 } } print(multiply(3, 5))

Slide 138

Slide 138 text

132 10 - 2 - 5 * 2 + 2 ^ 3 ^ 4

Slide 139

Slide 139 text

SKIP 133

Slide 140

Slide 140 text

etc.

Slide 141

Slide 141 text

Types! Loops! Closures! Objects!  Classes! Inheritance! Tuples! Records! Typechecking! Multimethods! Enumerations! Pattern matching! Currying! Modules! 135

Slide 142

Slide 142 text

136

Slide 143

Slide 143 text

136 L1 Numbers

Slide 144

Slide 144 text

136 L1 Numbers L2 Functions

Slide 145

Slide 145 text

136 L1 Numbers L2 Functions C1 Abstract syntax

Slide 146

Slide 146 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment

Slide 147

Slide 147 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables

Slide 148

Slide 148 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function

Slide 149

Slide 149 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment

Slide 150

Slide 150 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes

Slide 151

Slide 151 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals

Slide 152

Slide 152 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions

Slide 153

Slide 153 text

136 L1 Numbers L2 Functions C1 Abstract syntax C2 Environment L3 Variables C3 Print function C4 Immutable environment L4 Scopes L5 Conditionals L6 Function definitions L7 Operators

Slide 154

Slide 154 text

137 github.com/ddfreyne/clarke

Slide 155

Slide 155 text

137 github.com/ddfreyne/clarke SOON

Slide 156

Slide 156 text

138 Ready to interpret your questions. DENIS@STONESHIP.ORG @DDFREYNE