Slaying the Dragon

A talk that I gave at RedDotRubyConf 2016

Transcript

1. View Slide

2. I <3 DUCK TYPING
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. HELLO!
   

   View Slide

8. @JASONYEOJS
   

   View Slide

9. JSYEO
   

   View Slide

10. View Slide

11. SLAYING THE DRAGON
    

    View Slide

12. View Slide

13. A GUIDE TO IMPLEMENTING YOUR OWN
    PROGRAMMING LANGUAGE
    

    View Slide

14. View Slide

15. ¯\_(ツ)_/¯
    

    View Slide

16. FUN!
    

    View Slide

17. UNDERSTAND HOW PROGRAMMING
    LANGUAGES WORK
    

    View Slide

18. SystemStackError
    

    View Slide

19. PUT YOURSELF IN THE SHOES OF A
    LANGUAGE DESIGNER
    

    View Slide

20. View Slide

21. PUT YOURSELF IN THE SHOES OF A
    LANGUAGE DESIGNER
    

    View Slide

22. APPRECIATE THINGS LIKE GARBAGE
    COLLECTION AND SCOPING RULES
    

    View Slide

23. LET'S MAKE A LISP
    

    View Slide

24. SUPER EASY TO IMPLEMENT
    

    View Slide

25. View Slide

26. TONS OF GUIDES ONLINE
    

    View Slide

27. norvig.com/lispy.html
    

    View Slide

28. buildyourownlisp.com
    

    View Slide

29. github.com/kanaka/mal
    

    View Slide

30. • 
    EASIER TO UNDERSTAND
    BECAUSE…
    

    View Slide

31. DIAGRAMS!!
    

    View Slide

32. UNDERSTAND HOW
    LISP INFLUENCED RUBY
    

    View Slide

33. View Slide

34. http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-
    talk/179642
    

    View Slide

35. TO HELP RUBY FORWARD…
    

    View Slide

36. MatzLisp
    

    View Slide

37. View Slide

38. matzlisp.org
    

    View Slide

39. LET'S GET STARTED!
    

    View Slide

40. github.com/kanaka/mal
    

    View Slide

41. MAL
    

    View Slide

42. TEACH YOU MAL IN
    < 5 MINUTES
    

    View Slide

43. mal>
    

    View Slide

44. mal> (+ 2 6)
    

    View Slide

45. mal> (+ 2 6)
    => 8
    mal>
    

    View Slide

46. mal> (+ 2 6)
    => 8
    mal>
    LISP FORM OR EXPRESSION
    

    View Slide

47. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    

    View Slide

48. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal>
    

    View Slide

49. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    

    View Slide

50. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal>
    

    View Slide

51. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal> (< 4 10)
    

    View Slide

52. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal> (< 4 10)
    true
    

    View Slide

53. mal>
    

    View Slide

54. mal> (if (< 4 10)
    42
    88)
    

    View Slide

55. mal> (if (< 4 10)
    42
    88)
    => 42
    

    View Slide

56. mal>
    

    View Slide

57. mal> (fn* [x] (* x x))
    => #
    mal>
    

    View Slide

58. mal> (fn* [x] (* x x))
    => #
    mal>
    PARAMS
    

    View Slide

59. mal> (fn* [x] (* x x))
    => #
    mal> FUNCTION BODY
    

    View Slide

60. irb> lambda { |x| x * x }
    

    View Slide

61. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    

    View Slide

62. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    FUNCTION THAT
    YOU WANT TO CALL
    

    View Slide

63. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    ARGUMENTS
    

    View Slide

64. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    

    View Slide

65. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    

    View Slide

66. irb> lambda { |x| x * x }.call 8
    

    View Slide

67. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    

    View Slide

68. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #
    

    View Slide

69. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #
    mal> (sq 8)
    

    View Slide

70. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #
    mal> (sq 8)
    => 64
    

    View Slide

71. • 
    NOW YOU KNOW (BASIC) LISP!
    

    View Slide

72. • 
    QUICK INTRO TO INTERPRETERS
    

    View Slide

73. (+ my_var 42)
    

    View Slide

74. Interpreter
    (+ my_var 42)
    

    View Slide

75. Interpreter
    (+ my_var 42)
    Machine
    

    View Slide

76. BUT THAT IS SO RUBY 1.8
    

    View Slide

77. (+ my_var 42)
    

    View Slide

78. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+
    

    View Slide

79. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+
    Virtual
    Machine
    

    View Slide

80. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+
    Virtual
    Machine
    Machine
    

    View Slide

81. • 
    WHAT THE HECK IS A VIRTUAL
    MACHINE?
    

    View Slide

82. • 
    INTTERPRETS BYTECODE
    

    View Slide

83. • 
    AN ABSTRACTION LAYER BETWEEN
    BYTECODE AND NATIVE MACHINE
    

    View Slide

84. MRI?
    

    View Slide

85. • 
    RUBINIUS
    

    View Slide

86. • 
    A LANGAUGE PLATFORM
    

    View Slide

87. • 
    EASILY IMPLEMENT A LANGUAGE
    IN RUBINIUS
    

    View Slide

88. • 
    LET’S DIVE IN!
    

    View Slide

89. • 
    CUSTOMIZING RUBINIUS’
    COMPILATION PIPELINE
    

    View Slide

90. • 
    PARSER
    

    View Slide

91. • 
    ABSTRACT SYNTAX TREE
    PARSER
    

    View Slide

92. • 
    MAL PARSER
    

    View Slide

93. TOKENIZER
    

    View Slide

94. TOKENIZER
    READER
    

    View Slide

95. TOKENIZER
    READER
    PARSING LOGIC
    

    View Slide

96. • 
    TOKENIZER
    

    View Slide

97. (+ my_var 42)
    

    View Slide

98. (+ my_var 42)
    ['(', '+', 'my_var', '42', ')']
    

    View Slide

99. /[\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"|;.*|[^\s\[\]{}('"`,;)]*)/
    

    View Slide

100. • 
     LEXING IN RUBY
     

     View Slide

101. • 
     LEXING IN MATZLISP
     

     View Slide

102. [1] pry(main)>
     

     View Slide

103. [1] pry(main)> require 'ripper'
     => true
     [2] pry(main)>
     

     View Slide

104. [1] pry(main)> require 'ripper'
     => true
     [2] pry(main)> Ripper.lex 'foo = 123'
     

     View Slide

105. [1] pry(main)> require 'ripper'
     => true
     [2] pry(main)> Ripper.lex 'foo = 123'
     => [[[1, 0], :on_ident, "foo"],
     [[1, 3], :on_sp, " "],
     [[1, 4], :on_op, "="],
     [[1, 5], :on_sp, " "],
     [[1, 6], :on_int, "123"]]
     

     View Slide

106. (+ my_var 42)
     ['(', '+', 'my_var', '42', ')']
     

     View Slide

107. • 
     READER
     

     View Slide

108. ['(', '+', 'my_var', '42', ')']
     

     View Slide

109. S EXPRESSIONS
     

     View Slide

110. ['(', '+', 'my_var', '42', ')']
     [:list,
     [:symbol, '+'],
     [:symbol, 'my_var'],
     [:integer, 42]]
     

     View Slide

111. • 
     MAL GRAMMAR
     

     View Slide

112. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

113. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     'IS A' OR 'CAN BE EXPANDED TO'
     

     View Slide

114. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     OR
     

     View Slide

115. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false ONE OR
     MORE
     

     View Slide

116. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     ZERO
     OR
     MORE
     

     View Slide

117. ::=
     
     ::=
     ::=
     ::= eat | code | love
     ::= I | He | She |
     Ruby | Python
     

     View Slide

118. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

119. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

120. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

121. • 
     READER IMPLEMENTATION IS
     BASED ON THE GRAMMAR
     

     View Slide

122. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

123. ::= |
     def read_form(tokens)
     if tokens.first =~ /(\(|\[)/
     read_list(tokens)
     else
     read_atom(tokens.shift)
     end
     end
     

     View Slide

124. ::= |
     ::= '(' + ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

125. ::= '(' + ')' |
     '[' * ']'
     def read_list(tokens)
     list = [:list]
     tokens.shift # pop our opening paren
     while tokens.first !~ /(\)|\])/
     list << read_form(tokens)
     end
     tokens.shift # pop our closing paren
     list
     end
     

     View Slide

126. ::= |
     ::= '(' * ')' |
     '[' * ']'
     ::= a-z+ | 0-9+ |
     true | false
     

     View Slide

127. ::= a-z+ | 0-9+ |
     true | false
     def read_atom(token)
     case token
     when /^-?\d+$/
     [:integer, token.to_i]
     when 'true'
     [:boolean, :true]
     when 'false'
     [:boolean, :false]
     when /^\D+$/
     [:symbol, token]
     else
     raise 'Reader error: Unknown token'
     end
     end
     

     View Slide

128. ['(', '+', 'my_var', '42', ')']
     (+ my_var 42)
     [:list,
     [:symbol, '+'],
     [:symbol, 'my_var'],
     [:integer, 42]]
     

     View Slide

129. • 
     PARSING LOGIC
     

     View Slide

130. • 
     S EXPRESSIONS
     

     View Slide

131. • 
     ABSTRACT SYNTAX TREES
     

     View Slide

132. REPRESENTS THE SYNTACTIC
     STRUCTURE OF THE CODE
     

     View Slide

133. (+ my_var 42)
     

     View Slide

134. (+ my_var 42)
     

     View Slide

135. LEFT OPERAND
     (+ my_var 42)
     

     View Slide

136. LEFT OPERAND RIGHT OPERAND
     (+ my_var 42)
     

     View Slide

137. class IntegerNode < Node
     attr_reader :value
     def initialize(value)
     super
     @value = value
     end
     end
     THE INTEGER
     VALUE
     

     View Slide

138. class AddNode < Node
     attr_reader :left_operand, :right_operand
     def initialize(left_operand, right_operand)
     super
     @left_operand = left_operand
     @right_operand = right_operand
     end
     end
     BOTH ARE
     NODE OBJECTS
     

     View Slide

139. • 
     BYTECODE GENERATION
     

     View Slide

140. • 
     STACK BASED VM
     

     View Slide

141. STACK
     1 + 2
     INSTRUCTIONS
     

     View Slide

142. STACK
     1 + 2
     push 1
     INSTRUCTIONS
     

     View Slide

143. STACK
     1 + 2
     push 1
     INSTRUCTIONS
     1
     

     View Slide

144. STACK
     1 + 2
     push 1
     push 2
     INSTRUCTIONS
     1
     

     View Slide

145. STACK
     1 + 2
     push 1
     push 2
     INSTRUCTIONS
     1
     2
     

     View Slide

146. STACK
     1 + 2
     push 1
     push 2
     add
     INSTRUCTIONS
     1
     2
     

     View Slide

147. STACK
     1 + 2
     push 1
     push 2
     add
     INSTRUCTIONS
     1
     2
     POPS OPERANDS AND
     PUSHES RESULT ON THE
     STACK
     

     View Slide

148. STACK
     1 + 2
     push 1
     push 2
     add
     INSTRUCTIONS
     3
     

     View Slide

149. STACK
     (1 + 2) * 3
     push 1
     push 2
     add
     INSTRUCTIONS
     3
     

     View Slide

150. STACK
     (1 + 2) * 3
     push 1
     push 2
     add
     push 3
     INSTRUCTIONS
     3
     3
     

     View Slide

151. STACK
     (1 + 2) * 3
     push 1
     push 2
     add
     push 3
     multiply
     INSTRUCTIONS
     3
     3
     

     View Slide

152. STACK
     (1 + 2) * 3
     push 1
     push 2
     add
     push 3
     multiply
     INSTRUCTIONS
     9
     

     View Slide

153. • 
     BYTECODE COMPILATION IN MRI
     

     View Slide

154. RubyVM::InstructionSequence
     

     View Slide

155. [1] pry(main)> code = '(15 + 6) * 2'
     

     View Slide

156. [1] pry(main)> code = '(15 + 6) * 2'
     => '(15 + 6) * 2'
     [2] pry(main)>
     

     View Slide

157. RubyVM::InstructionSequence.compile(code)
     .disasm
     

     View Slide

158. [1] pry(main)> code = '(15 + 6) * 2'
     => '(15 + 6) * 2'
     [2] pry(main)> puts RubyVM::InstructionSequence.compile(code).disasm
     == disasm: #@>================================
     0000 trace 1 ( 1)
     0002 putobject 15
     0004 putobject 6
     0006 opt_plus ,
     0009 putobject 2
     0011 opt_mult ,
     0014 leave
     => nil
     

     View Slide

159. • 
     GENERATING BYTECODE
     

     View Slide

160. class Node
     def bytecode(g)
     # not implemented
     end
     end
     

     View Slide

161. • 
     PURE MATZLISP BYTECODE
     GENERATION DSL
     http://rubinius.com/doc/en/bytecode-compiler/generator/
     

     View Slide

162. class IntegerNode < Node
     attr_reader :value
     def initialize(value)
     @value = value
     end
     def bytecode(g)
     pos(g)
     g.push_int @value
     end
     end
     

     View Slide

163. class IntegerNode < Node
     attr_reader :value
     def initialize(value)
     @value = value
     end
     def bytecode(g)
     pos(g)
     g.push_int @value
     end
     end
     

     View Slide

164. class AddNode < Node
     attr_reader :left_operand, :right_operand
     def initialize(left_operand, right_operand)
     super
     @left_operand = left_operand
     @right_operand = right_operand
     end
     def bytecode(g)
     pos(g)
     @left_operand.bytecode(g)
     @right_operand.bytecode(g)
     g.send(:+, 1)
     end
     end
     

     View Slide

165. • 
     YOU HAVE A WORKING
     PROGRAMMING LANGUAGE!
     

     View Slide

166. • 
     WHERE TO GET HELP?
     

     View Slide

167. github.com/kanaka/mal
     

     View Slide

168. View Slide

169. DIAGRAMS!!
     

     View Slide

170. github.com/queenfrankie/lani
     

     View Slide

171. github.com/jsyeo/malady
     

     View Slide

172. View Slide

173. • 
     QUESTIONS?
     

     View Slide