Slaying the Dragon

A talk that I gave at RedDotRubyConf 2016

Transcript

2. I <3 DUCK TYPING
7. HELLO!
8. @JASONYEOJS
9. JSYEO
11. SLAYING THE DRAGON
13. A GUIDE TO IMPLEMENTING YOUR OWN PROGRAMMING LANGUAGE
15. ¯\_(ツ)_/¯
16. FUN!
17. UNDERSTAND HOW PROGRAMMING LANGUAGES WORK
18. SystemStackError
19. PUT YOURSELF IN THE SHOES OF A LANGUAGE DESIGNER
21. PUT YOURSELF IN THE SHOES OF A LANGUAGE DESIGNER
22. APPRECIATE THINGS LIKE GARBAGE COLLECTION AND SCOPING RULES
23. LET'S MAKE A LISP
24. SUPER EASY TO IMPLEMENT
26. TONS OF GUIDES ONLINE
27. norvig.com/lispy.html
28. buildyourownlisp.com
29. github.com/kanaka/mal
30. •  EASIER TO UNDERSTAND BECAUSE…
31. DIAGRAMS!!
32. UNDERSTAND HOW LISP INFLUENCED RUBY
34. http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby- talk/179642
35. TO HELP RUBY FORWARD…
36. MatzLisp
38. matzlisp.org
39. LET'S GET STARTED!
40. github.com/kanaka/mal
41. MAL
42. TEACH YOU MAL IN < 5 MINUTES
43. mal>
44. mal> (+ 2 6)
45. mal> (+ 2 6) => 8 mal>
46. mal> (+ 2 6) => 8 mal> LISP FORM OR EXPRESSION
47. mal> (+ 2 6) => 8 mal> (- 4 8)
48. mal> (+ 2 6) => 8 mal> (- 4 8) => -4 mal>
49. mal> (+ 2 6) => 8 mal> (- 4 8) => -4 mal> (+ (- 4 8) 10)
50. mal> (+ 2 6) => 8 mal> (- 4 8) => -4 mal> (+ (- 4 8) 10) => 6 mal>
51. mal> (+ 2 6) => 8 mal> (- 4 8) => -4 mal> (+ (- 4 8) 10) => 6 mal> (< 4 10)
52. mal> (+ 2 6) => 8 mal> (- 4 8) => -4 mal> (+ (- 4 8) 10) => 6 mal> (< 4 10) true
53. mal>
54. mal> (if (< 4 10) 42 88)
55. mal> (if (< 4 10) 42 88) => 42
56. mal>
57. mal> (fn* [x] (* x x)) => #<function> mal>
58. mal> (fn* [x] (* x x)) => #<function> mal> PARAMS
59. mal> (fn* [x] (* x x)) => #<function> mal> FUNCTION BODY
60. irb> lambda { |x| x * x }
61. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8)
62. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) FUNCTION THAT YOU WANT TO CALL
63. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) ARGUMENTS
64. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8)
65. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) => 64
66. irb> lambda { |x| x * x }.call 8
67. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x)))
68. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function>
69. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function> mal> (sq 8)
70. mal> (fn* [x] (* x x)) => #<function> mal> ((fn* [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function> mal> (sq 8) => 64
71. •  NOW YOU KNOW (BASIC) LISP!
72. •  QUICK INTRO TO INTERPRETERS
73. (+ my_var 42)
74. Interpreter (+ my_var 42)
75. Interpreter (+ my_var 42) Machine
76. BUT THAT IS SO RUBY 1.8
77. (+ my_var 42)
78. Bytecode Compiler (+ my_var 42) push my_var push 42 send :+
79. Bytecode Compiler (+ my_var 42) push my_var push 42 send :+ Virtual Machine
80. Bytecode Compiler (+ my_var 42) push my_var push 42 send :+ Virtual Machine Machine
81. •  WHAT THE HECK IS A VIRTUAL MACHINE?
82. •  INTTERPRETS BYTECODE
83. •  AN ABSTRACTION LAYER BETWEEN BYTECODE AND NATIVE MACHINE
84. MRI?
85. •  RUBINIUS
86. •  A LANGAUGE PLATFORM
87. •  EASILY IMPLEMENT A LANGUAGE IN RUBINIUS
88. •  LET’S DIVE IN!
89. •  CUSTOMIZING RUBINIUS’ COMPILATION PIPELINE
90. •  PARSER
91. •  ABSTRACT SYNTAX TREE PARSER
92. •  MAL PARSER
93. TOKENIZER
94. TOKENIZER READER
95. TOKENIZER READER PARSING LOGIC
96. •  TOKENIZER
97. (+ my_var 42)
98. (+ my_var 42) ['(', '+', 'my_var', '42', ')']
99. /[\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"|;.*|[^\s\[\]{}('"`,;)]*)/
100. •  LEXING IN RUBY
101. •  LEXING IN MATZLISP
102. [1] pry(main)>
103. [1] pry(main)> require 'ripper' => true [2] pry(main)>
104. [1] pry(main)> require 'ripper' => true [2] pry(main)> Ripper.lex 'foo = 123'
105. [1] pry(main)> require 'ripper' => true [2] pry(main)> Ripper.lex 'foo = 123' => [[[1, 0], :on_ident, "foo"], [[1, 3], :on_sp, " "], [[1, 4], :on_op, "="], [[1, 5], :on_sp, " "], [[1, 6], :on_int, "123"]]
106. (+ my_var 42) ['(', '+', 'my_var', '42', ')']
107. •  READER
108. ['(', '+', 'my_var', '42', ')']
109. S EXPRESSIONS
110. ['(', '+', 'my_var', '42', ')'] [:list, [:symbol, '+'], [:symbol, 'my_var'], [:integer, 42]]
111. •  MAL GRAMMAR
112. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
113. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false 'IS A' OR 'CAN BE EXPANDED TO'
114. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false OR
115. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false ONE OR MORE
116. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false ZERO OR MORE
117. <sentence> ::= <subject> <verb> <object> <subject> ::= <noun> <object> ::= <noun> <verb> ::= eat | code | love <noun> ::= I | He | She | Ruby | Python
118. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
119. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
120. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
121. •  READER IMPLEMENTATION IS BASED ON THE GRAMMAR
122. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
123. <form> ::= <list> | <atom> def read_form(tokens) if tokens.first =~ /(\(|\[)/ read_list(tokens) else read_atom(tokens.shift) end end
124. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
125. <list> ::= '(' <form>+ ')' | '[' <form>* ']' def read_list(tokens) list = [:list] tokens.shift # pop our opening paren while tokens.first !~ /(\)|\])/ list << read_form(tokens) end tokens.shift # pop our closing paren list end
126. <form> ::= <list> | <atom> <list> ::= '(' <form>* ')' | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
127. <atom> ::= a-z+ | 0-9+ | true | false def read_atom(token) case token when /^-?\d+$/ [:integer, token.to_i] when 'true' [:boolean, :true] when 'false' [:boolean, :false] when /^\D+$/ [:symbol, token] else raise 'Reader error: Unknown token' end end
128. ['(', '+', 'my_var', '42', ')'] (+ my_var 42) [:list, [:symbol, '+'], [:symbol, 'my_var'], [:integer, 42]]
129. •  PARSING LOGIC
130. •  S EXPRESSIONS
131. •  ABSTRACT SYNTAX TREES
132. REPRESENTS THE SYNTACTIC STRUCTURE OF THE CODE
133. (+ my_var 42)
134. (+ my_var 42)
135. LEFT OPERAND (+ my_var 42)
136. LEFT OPERAND RIGHT OPERAND (+ my_var 42)
137. class IntegerNode < Node attr_reader :value def initialize(value) super @value = value end end THE INTEGER VALUE
138. class AddNode < Node attr_reader :left_operand, :right_operand def initialize(left_operand, right_operand) super @left_operand = left_operand @right_operand = right_operand end end BOTH ARE NODE OBJECTS
139. •  BYTECODE GENERATION
140. •  STACK BASED VM
141. STACK 1 + 2 INSTRUCTIONS
142. STACK 1 + 2 push 1 INSTRUCTIONS
143. STACK 1 + 2 push 1 INSTRUCTIONS 1
144. STACK 1 + 2 push 1 push 2 INSTRUCTIONS 1
145. STACK 1 + 2 push 1 push 2 INSTRUCTIONS 1 2
146. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS 1 2
147. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS 1 2 POPS OPERANDS AND PUSHES RESULT ON THE STACK
148. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS 3
149. STACK (1 + 2) * 3 push 1 push 2 add INSTRUCTIONS 3
150. STACK (1 + 2) * 3 push 1 push 2 add push 3 INSTRUCTIONS 3 3
151. STACK (1 + 2) * 3 push 1 push 2 add push 3 multiply INSTRUCTIONS 3 3
152. STACK (1 + 2) * 3 push 1 push 2 add push 3 multiply INSTRUCTIONS 9
153. •  BYTECODE COMPILATION IN MRI
154. RubyVM::InstructionSequence
155. [1] pry(main)> code = '(15 + 6) * 2'
156. [1] pry(main)> code = '(15 + 6) * 2' => '(15 + 6) * 2' [2] pry(main)>
157. RubyVM::InstructionSequence.compile(code) .disasm
158. [1] pry(main)> code = '(15 + 6) * 2' => '(15 + 6) * 2' [2] pry(main)> puts RubyVM::InstructionSequence.compile(code).disasm == disasm: #<ISeq:<compiled>@<compiled>>================================ 0000 trace 1 ( 1) 0002 putobject 15 0004 putobject 6 0006 opt_plus <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache> 0009 putobject 2 0011 opt_mult <callinfo!mid:*, argc:1, ARGS_SIMPLE>, <callcache> 0014 leave => nil
159. •  GENERATING BYTECODE
160. class Node def bytecode(g) # not implemented end end
161. •  PURE MATZLISP BYTECODE GENERATION DSL http://rubinius.com/doc/en/bytecode-compiler/generator/
162. class IntegerNode < Node attr_reader :value def initialize(value) @value = value end def bytecode(g) pos(g) g.push_int @value end end
163. class IntegerNode < Node attr_reader :value def initialize(value) @value = value end def bytecode(g) pos(g) g.push_int @value end end
164. class AddNode < Node attr_reader :left_operand, :right_operand def initialize(left_operand, right_operand) super @left_operand = left_operand @right_operand = right_operand end def bytecode(g) pos(g) @left_operand.bytecode(g) @right_operand.bytecode(g) g.send(:+, 1) end end
165. •  YOU HAVE A WORKING PROGRAMMING LANGUAGE!
166. •  WHERE TO GET HELP?
167. github.com/kanaka/mal
169. DIAGRAMS!!
170. github.com/queenfrankie/lani
171. github.com/jsyeo/malady
173. •  QUESTIONS?