Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Slaying the Dragon

Slaying the Dragon

A talk that I gave at RedDotRubyConf 2016

Df4e511ca1ecf66a86a231d7bc728d07?s=128

Jason Yeo

June 23, 2016
Tweet

Transcript

  1. None
  2. I <3 DUCK TYPING

  3. None
  4. None
  5. None
  6. None
  7. HELLO!

  8. @JASONYEOJS

  9. JSYEO

  10. None
  11. SLAYING THE DRAGON

  12. None
  13. A GUIDE TO IMPLEMENTING YOUR OWN PROGRAMMING LANGUAGE

  14. None
  15. ¯\_(ツ)_/¯

  16. FUN!

  17. UNDERSTAND HOW PROGRAMMING LANGUAGES WORK

  18. SystemStackError

  19. PUT YOURSELF IN THE SHOES OF A LANGUAGE DESIGNER

  20. None
  21. PUT YOURSELF IN THE SHOES OF A LANGUAGE DESIGNER

  22. APPRECIATE THINGS LIKE GARBAGE COLLECTION AND SCOPING RULES

  23. LET'S MAKE A LISP

  24. SUPER EASY TO IMPLEMENT

  25. None
  26. TONS OF GUIDES ONLINE

  27. norvig.com/lispy.html

  28. buildyourownlisp.com

  29. github.com/kanaka/mal

  30. •  EASIER TO UNDERSTAND BECAUSE…

  31. DIAGRAMS!!

  32. UNDERSTAND HOW LISP INFLUENCED RUBY

  33. None
  34. http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby- talk/179642

  35. TO HELP RUBY FORWARD…

  36. MatzLisp

  37. None
  38. matzlisp.org

  39. LET'S GET STARTED!

  40. github.com/kanaka/mal

  41. MAL

  42. TEACH YOU MAL IN < 5 MINUTES

  43. mal>

  44. mal> (+ 2 6)

  45. mal> (+ 2 6) => 8 mal>

  46. mal> (+ 2 6) => 8 mal> LISP FORM OR

    EXPRESSION
  47. mal> (+ 2 6) => 8 mal> (- 4 8)

  48. mal> (+ 2 6) => 8 mal> (- 4 8)

    => -4 mal>
  49. mal> (+ 2 6) => 8 mal> (- 4 8)

    => -4 mal> (+ (- 4 8) 10)
  50. mal> (+ 2 6) => 8 mal> (- 4 8)

    => -4 mal> (+ (- 4 8) 10) => 6 mal>
  51. mal> (+ 2 6) => 8 mal> (- 4 8)

    => -4 mal> (+ (- 4 8) 10) => 6 mal> (< 4 10)
  52. mal> (+ 2 6) => 8 mal> (- 4 8)

    => -4 mal> (+ (- 4 8) 10) => 6 mal> (< 4 10) true
  53. mal>

  54. mal> (if (< 4 10) 42 88)

  55. mal> (if (< 4 10) 42 88) => 42

  56. mal>

  57. mal> (fn* [x] (* x x)) => #<function> mal>

  58. mal> (fn* [x] (* x x)) => #<function> mal> PARAMS

  59. mal> (fn* [x] (* x x)) => #<function> mal> FUNCTION

    BODY
  60. irb> lambda { |x| x * x }

  61. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8)
  62. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) FUNCTION THAT YOU WANT TO CALL
  63. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) ARGUMENTS
  64. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8)
  65. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) => 64
  66. irb> lambda { |x| x * x }.call 8

  67. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x)))
  68. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function>
  69. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function> mal> (sq 8)
  70. mal> (fn* [x] (* x x)) => #<function> mal> ((fn*

    [x] (* x x)) 8) => 64 mal> (def sq (fn* [x] (* x x))) => #<function> mal> (sq 8) => 64
  71. •  NOW YOU KNOW (BASIC) LISP!

  72. •  QUICK INTRO TO INTERPRETERS

  73. (+ my_var 42)

  74. Interpreter (+ my_var 42)

  75. Interpreter (+ my_var 42) Machine

  76. BUT THAT IS SO RUBY 1.8

  77. (+ my_var 42)

  78. Bytecode Compiler (+ my_var 42) push my_var push 42 send

    :+
  79. Bytecode Compiler (+ my_var 42) push my_var push 42 send

    :+ Virtual Machine
  80. Bytecode Compiler (+ my_var 42) push my_var push 42 send

    :+ Virtual Machine Machine
  81. •  WHAT THE HECK IS A VIRTUAL MACHINE?

  82. •  INTTERPRETS BYTECODE

  83. •  AN ABSTRACTION LAYER BETWEEN BYTECODE AND NATIVE MACHINE

  84. MRI?

  85. •  RUBINIUS

  86. •  A LANGAUGE PLATFORM

  87. •  EASILY IMPLEMENT A LANGUAGE IN RUBINIUS

  88. •  LET’S DIVE IN!

  89. •  CUSTOMIZING RUBINIUS’ COMPILATION PIPELINE

  90. •  PARSER

  91. •  ABSTRACT SYNTAX TREE PARSER

  92. •  MAL PARSER

  93. TOKENIZER

  94. TOKENIZER READER

  95. TOKENIZER READER PARSING LOGIC

  96. •  TOKENIZER

  97. (+ my_var 42)

  98. (+ my_var 42) ['(', '+', 'my_var', '42', ')']

  99. /[\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"|;.*|[^\s\[\]{}('"`,;)]*)/

  100. •  LEXING IN RUBY

  101. •  LEXING IN MATZLISP

  102. [1] pry(main)>

  103. [1] pry(main)> require 'ripper' => true [2] pry(main)>

  104. [1] pry(main)> require 'ripper' => true [2] pry(main)> Ripper.lex 'foo

    = 123'
  105. [1] pry(main)> require 'ripper' => true [2] pry(main)> Ripper.lex 'foo

    = 123' => [[[1, 0], :on_ident, "foo"], [[1, 3], :on_sp, " "], [[1, 4], :on_op, "="], [[1, 5], :on_sp, " "], [[1, 6], :on_int, "123"]]
  106. (+ my_var 42) ['(', '+', 'my_var', '42', ')']

  107. •  READER

  108. ['(', '+', 'my_var', '42', ')']

  109. S EXPRESSIONS

  110. ['(', '+', 'my_var', '42', ')'] [:list, [:symbol, '+'], [:symbol, 'my_var'],

    [:integer, 42]]
  111. •  MAL GRAMMAR

  112. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  113. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false 'IS A' OR 'CAN BE EXPANDED TO'
  114. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false OR
  115. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false ONE OR MORE
  116. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false ZERO OR MORE
  117. <sentence> ::= <subject> <verb> <object> <subject> ::= <noun> <object> ::=

    <noun> <verb> ::= eat | code | love <noun> ::= I | He | She | Ruby | Python
  118. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  119. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  120. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  121. •  READER IMPLEMENTATION IS BASED ON THE GRAMMAR

  122. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  123. <form> ::= <list> | <atom> def read_form(tokens) if tokens.first =~

    /(\(|\[)/ read_list(tokens) else read_atom(tokens.shift) end end
  124. <form> ::= <list> | <atom> <list> ::= '(' <form>+ ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  125. <list> ::= '(' <form>+ ')' | '[' <form>* ']' def

    read_list(tokens) list = [:list] tokens.shift # pop our opening paren while tokens.first !~ /(\)|\])/ list << read_form(tokens) end tokens.shift # pop our closing paren list end
  126. <form> ::= <list> | <atom> <list> ::= '(' <form>* ')'

    | '[' <form>* ']' <atom> ::= a-z+ | 0-9+ | true | false
  127. <atom> ::= a-z+ | 0-9+ | true | false def

    read_atom(token) case token when /^-?\d+$/ [:integer, token.to_i] when 'true' [:boolean, :true] when 'false' [:boolean, :false] when /^\D+$/ [:symbol, token] else raise 'Reader error: Unknown token' end end
  128. ['(', '+', 'my_var', '42', ')'] (+ my_var 42) [:list, [:symbol,

    '+'], [:symbol, 'my_var'], [:integer, 42]]
  129. •  PARSING LOGIC

  130. •  S EXPRESSIONS

  131. •  ABSTRACT SYNTAX TREES

  132. REPRESENTS THE SYNTACTIC STRUCTURE OF THE CODE

  133. (+ my_var 42)

  134. (+ my_var 42)

  135. LEFT OPERAND (+ my_var 42)

  136. LEFT OPERAND RIGHT OPERAND (+ my_var 42)

  137. class IntegerNode < Node attr_reader :value def initialize(value) super @value

    = value end end THE INTEGER VALUE
  138. class AddNode < Node attr_reader :left_operand, :right_operand def initialize(left_operand, right_operand)

    super @left_operand = left_operand @right_operand = right_operand end end BOTH ARE NODE OBJECTS
  139. •  BYTECODE GENERATION

  140. •  STACK BASED VM

  141. STACK 1 + 2 INSTRUCTIONS

  142. STACK 1 + 2 push 1 INSTRUCTIONS

  143. STACK 1 + 2 push 1 INSTRUCTIONS 1

  144. STACK 1 + 2 push 1 push 2 INSTRUCTIONS 1

  145. STACK 1 + 2 push 1 push 2 INSTRUCTIONS 1

    2
  146. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS

    1 2
  147. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS

    1 2 POPS OPERANDS AND PUSHES RESULT ON THE STACK
  148. STACK 1 + 2 push 1 push 2 add INSTRUCTIONS

    3
  149. STACK (1 + 2) * 3 push 1 push 2

    add INSTRUCTIONS 3
  150. STACK (1 + 2) * 3 push 1 push 2

    add push 3 INSTRUCTIONS 3 3
  151. STACK (1 + 2) * 3 push 1 push 2

    add push 3 multiply INSTRUCTIONS 3 3
  152. STACK (1 + 2) * 3 push 1 push 2

    add push 3 multiply INSTRUCTIONS 9
  153. •  BYTECODE COMPILATION IN MRI

  154. RubyVM::InstructionSequence

  155. [1] pry(main)> code = '(15 + 6) * 2'

  156. [1] pry(main)> code = '(15 + 6) * 2' =>

    '(15 + 6) * 2' [2] pry(main)>
  157. RubyVM::InstructionSequence.compile(code) .disasm

  158. [1] pry(main)> code = '(15 + 6) * 2' =>

    '(15 + 6) * 2' [2] pry(main)> puts RubyVM::InstructionSequence.compile(code).disasm == disasm: #<ISeq:<compiled>@<compiled>>================================ 0000 trace 1 ( 1) 0002 putobject 15 0004 putobject 6 0006 opt_plus <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache> 0009 putobject 2 0011 opt_mult <callinfo!mid:*, argc:1, ARGS_SIMPLE>, <callcache> 0014 leave => nil
  159. •  GENERATING BYTECODE

  160. class Node def bytecode(g) # not implemented end end

  161. •  PURE MATZLISP BYTECODE GENERATION DSL http://rubinius.com/doc/en/bytecode-compiler/generator/

  162. class IntegerNode < Node attr_reader :value def initialize(value) @value =

    value end def bytecode(g) pos(g) g.push_int @value end end
  163. class IntegerNode < Node attr_reader :value def initialize(value) @value =

    value end def bytecode(g) pos(g) g.push_int @value end end
  164. class AddNode < Node attr_reader :left_operand, :right_operand def initialize(left_operand, right_operand)

    super @left_operand = left_operand @right_operand = right_operand end def bytecode(g) pos(g) @left_operand.bytecode(g) @right_operand.bytecode(g) g.send(:+, 1) end end
  165. •  YOU HAVE A WORKING PROGRAMMING LANGUAGE!

  166. •  WHERE TO GET HELP?

  167. github.com/kanaka/mal

  168. None
  169. DIAGRAMS!!

  170. github.com/queenfrankie/lani

  171. github.com/jsyeo/malady

  172. None
  173. •  QUESTIONS?