Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Slaying the Dragon

Slaying the Dragon

A talk that I gave at RedDotRubyConf 2016

Jason Yeo

June 23, 2016
Tweet

Other Decks in Programming

Transcript

  1. I <3 DUCK TYPING

    View full-size slide

  2. SLAYING THE DRAGON

    View full-size slide

  3. A GUIDE TO IMPLEMENTING YOUR OWN
    PROGRAMMING LANGUAGE

    View full-size slide

  4. ¯\_(ツ)_/¯

    View full-size slide

  5. UNDERSTAND HOW PROGRAMMING
    LANGUAGES WORK

    View full-size slide

  6. SystemStackError

    View full-size slide

  7. PUT YOURSELF IN THE SHOES OF A
    LANGUAGE DESIGNER

    View full-size slide

  8. PUT YOURSELF IN THE SHOES OF A
    LANGUAGE DESIGNER

    View full-size slide

  9. APPRECIATE THINGS LIKE GARBAGE
    COLLECTION AND SCOPING RULES

    View full-size slide

  10. LET'S MAKE A LISP

    View full-size slide

  11. SUPER EASY TO IMPLEMENT

    View full-size slide

  12. TONS OF GUIDES ONLINE

    View full-size slide

  13. norvig.com/lispy.html

    View full-size slide

  14. buildyourownlisp.com

    View full-size slide

  15. github.com/kanaka/mal

    View full-size slide

  16. • 
    EASIER TO UNDERSTAND
    BECAUSE…

    View full-size slide

  17. UNDERSTAND HOW
    LISP INFLUENCED RUBY

    View full-size slide

  18. http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-
    talk/179642

    View full-size slide

  19. TO HELP RUBY FORWARD…

    View full-size slide

  20. matzlisp.org

    View full-size slide

  21. LET'S GET STARTED!

    View full-size slide

  22. github.com/kanaka/mal

    View full-size slide

  23. TEACH YOU MAL IN
    < 5 MINUTES

    View full-size slide

  24. mal> (+ 2 6)

    View full-size slide

  25. mal> (+ 2 6)
    => 8
    mal>

    View full-size slide

  26. mal> (+ 2 6)
    => 8
    mal>
    LISP FORM OR EXPRESSION

    View full-size slide

  27. mal> (+ 2 6)
    => 8
    mal> (- 4 8)

    View full-size slide

  28. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal>

    View full-size slide

  29. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)

    View full-size slide

  30. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal>

    View full-size slide

  31. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal> (< 4 10)

    View full-size slide

  32. mal> (+ 2 6)
    => 8
    mal> (- 4 8)
    => -4
    mal> (+ (- 4 8) 10)
    => 6
    mal> (< 4 10)
    true

    View full-size slide

  33. mal> (if (< 4 10)
    42
    88)

    View full-size slide

  34. mal> (if (< 4 10)
    42
    88)
    => 42

    View full-size slide

  35. mal> (fn* [x] (* x x))
    => #
    mal>

    View full-size slide

  36. mal> (fn* [x] (* x x))
    => #
    mal>
    PARAMS

    View full-size slide

  37. mal> (fn* [x] (* x x))
    => #
    mal> FUNCTION BODY

    View full-size slide

  38. irb> lambda { |x| x * x }

    View full-size slide

  39. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)

    View full-size slide

  40. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    FUNCTION THAT
    YOU WANT TO CALL

    View full-size slide

  41. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    ARGUMENTS

    View full-size slide

  42. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)

    View full-size slide

  43. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64

    View full-size slide

  44. irb> lambda { |x| x * x }.call 8

    View full-size slide

  45. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))

    View full-size slide

  46. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #

    View full-size slide

  47. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #
    mal> (sq 8)

    View full-size slide

  48. mal> (fn* [x] (* x x))
    => #
    mal> ((fn* [x] (* x x)) 8)
    => 64
    mal> (def sq (fn* [x] (* x x)))
    => #
    mal> (sq 8)
    => 64

    View full-size slide

  49. • 
    NOW YOU KNOW (BASIC) LISP!

    View full-size slide

  50. • 
    QUICK INTRO TO INTERPRETERS

    View full-size slide

  51. (+ my_var 42)

    View full-size slide

  52. Interpreter
    (+ my_var 42)

    View full-size slide

  53. Interpreter
    (+ my_var 42)
    Machine

    View full-size slide

  54. BUT THAT IS SO RUBY 1.8

    View full-size slide

  55. (+ my_var 42)

    View full-size slide

  56. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+

    View full-size slide

  57. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+
    Virtual
    Machine

    View full-size slide

  58. Bytecode
    Compiler
    (+ my_var 42)
    push my_var
    push 42
    send :+
    Virtual
    Machine
    Machine

    View full-size slide

  59. • 
    WHAT THE HECK IS A VIRTUAL
    MACHINE?

    View full-size slide

  60. • 
    INTTERPRETS BYTECODE

    View full-size slide

  61. • 
    AN ABSTRACTION LAYER BETWEEN
    BYTECODE AND NATIVE MACHINE

    View full-size slide

  62. • 
    RUBINIUS

    View full-size slide

  63. • 
    A LANGAUGE PLATFORM

    View full-size slide

  64. • 
    EASILY IMPLEMENT A LANGUAGE
    IN RUBINIUS

    View full-size slide

  65. • 
    LET’S DIVE IN!

    View full-size slide

  66. • 
    CUSTOMIZING RUBINIUS’
    COMPILATION PIPELINE

    View full-size slide

  67. • 
    PARSER

    View full-size slide

  68. • 
    ABSTRACT SYNTAX TREE
    PARSER

    View full-size slide

  69. • 
    MAL PARSER

    View full-size slide

  70. TOKENIZER
    READER

    View full-size slide

  71. TOKENIZER
    READER
    PARSING LOGIC

    View full-size slide

  72. • 
    TOKENIZER

    View full-size slide

  73. (+ my_var 42)

    View full-size slide

  74. (+ my_var 42)
    ['(', '+', 'my_var', '42', ')']

    View full-size slide

  75. /[\s,]*(~@|[\[\]{}()'`~^@]|"(?:\\.|[^\\"])*"|;.*|[^\s\[\]{}('"`,;)]*)/

    View full-size slide

  76. • 
    LEXING IN RUBY

    View full-size slide

  77. • 
    LEXING IN MATZLISP

    View full-size slide

  78. [1] pry(main)>

    View full-size slide

  79. [1] pry(main)> require 'ripper'
    => true
    [2] pry(main)>

    View full-size slide

  80. [1] pry(main)> require 'ripper'
    => true
    [2] pry(main)> Ripper.lex 'foo = 123'

    View full-size slide

  81. [1] pry(main)> require 'ripper'
    => true
    [2] pry(main)> Ripper.lex 'foo = 123'
    => [[[1, 0], :on_ident, "foo"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_op, "="],
    [[1, 5], :on_sp, " "],
    [[1, 6], :on_int, "123"]]

    View full-size slide

  82. (+ my_var 42)
    ['(', '+', 'my_var', '42', ')']

    View full-size slide

  83. • 
    READER

    View full-size slide

  84. ['(', '+', 'my_var', '42', ')']

    View full-size slide

  85. S EXPRESSIONS

    View full-size slide

  86. ['(', '+', 'my_var', '42', ')']
    [:list,
    [:symbol, '+'],
    [:symbol, 'my_var'],
    [:integer, 42]]

    View full-size slide

  87. • 
    MAL GRAMMAR

    View full-size slide

  88. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  89. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false
    'IS A' OR 'CAN BE EXPANDED TO'

    View full-size slide

  90. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false
    OR

    View full-size slide

  91. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false ONE OR
    MORE

    View full-size slide

  92. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false
    ZERO
    OR
    MORE

    View full-size slide

  93. ::=

    ::=
    ::=
    ::= eat | code | love
    ::= I | He | She |
    Ruby | Python

    View full-size slide

  94. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  95. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  96. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  97. • 
    READER IMPLEMENTATION IS
    BASED ON THE GRAMMAR

    View full-size slide

  98. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  99. ::= |
    def read_form(tokens)
    if tokens.first =~ /(\(|\[)/
    read_list(tokens)
    else
    read_atom(tokens.shift)
    end
    end

    View full-size slide

  100. ::= |
    ::= '(' + ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  101. ::= '(' + ')' |
    '[' * ']'
    def read_list(tokens)
    list = [:list]
    tokens.shift # pop our opening paren
    while tokens.first !~ /(\)|\])/
    list << read_form(tokens)
    end
    tokens.shift # pop our closing paren
    list
    end

    View full-size slide

  102. ::= |
    ::= '(' * ')' |
    '[' * ']'
    ::= a-z+ | 0-9+ |
    true | false

    View full-size slide

  103. ::= a-z+ | 0-9+ |
    true | false
    def read_atom(token)
    case token
    when /^-?\d+$/
    [:integer, token.to_i]
    when 'true'
    [:boolean, :true]
    when 'false'
    [:boolean, :false]
    when /^\D+$/
    [:symbol, token]
    else
    raise 'Reader error: Unknown token'
    end
    end

    View full-size slide

  104. ['(', '+', 'my_var', '42', ')']
    (+ my_var 42)
    [:list,
    [:symbol, '+'],
    [:symbol, 'my_var'],
    [:integer, 42]]

    View full-size slide

  105. • 
    PARSING LOGIC

    View full-size slide

  106. • 
    S EXPRESSIONS

    View full-size slide

  107. • 
    ABSTRACT SYNTAX TREES

    View full-size slide

  108. REPRESENTS THE SYNTACTIC
    STRUCTURE OF THE CODE

    View full-size slide

  109. (+ my_var 42)

    View full-size slide

  110. (+ my_var 42)

    View full-size slide

  111. LEFT OPERAND
    (+ my_var 42)

    View full-size slide

  112. LEFT OPERAND RIGHT OPERAND
    (+ my_var 42)

    View full-size slide

  113. class IntegerNode < Node
    attr_reader :value
    def initialize(value)
    super
    @value = value
    end
    end
    THE INTEGER
    VALUE

    View full-size slide

  114. class AddNode < Node
    attr_reader :left_operand, :right_operand
    def initialize(left_operand, right_operand)
    super
    @left_operand = left_operand
    @right_operand = right_operand
    end
    end
    BOTH ARE
    NODE OBJECTS

    View full-size slide

  115. • 
    BYTECODE GENERATION

    View full-size slide

  116. • 
    STACK BASED VM

    View full-size slide

  117. STACK
    1 + 2
    INSTRUCTIONS

    View full-size slide

  118. STACK
    1 + 2
    push 1
    INSTRUCTIONS

    View full-size slide

  119. STACK
    1 + 2
    push 1
    INSTRUCTIONS
    1

    View full-size slide

  120. STACK
    1 + 2
    push 1
    push 2
    INSTRUCTIONS
    1

    View full-size slide

  121. STACK
    1 + 2
    push 1
    push 2
    INSTRUCTIONS
    1
    2

    View full-size slide

  122. STACK
    1 + 2
    push 1
    push 2
    add
    INSTRUCTIONS
    1
    2

    View full-size slide

  123. STACK
    1 + 2
    push 1
    push 2
    add
    INSTRUCTIONS
    1
    2
    POPS OPERANDS AND
    PUSHES RESULT ON THE
    STACK

    View full-size slide

  124. STACK
    1 + 2
    push 1
    push 2
    add
    INSTRUCTIONS
    3

    View full-size slide

  125. STACK
    (1 + 2) * 3
    push 1
    push 2
    add
    INSTRUCTIONS
    3

    View full-size slide

  126. STACK
    (1 + 2) * 3
    push 1
    push 2
    add
    push 3
    INSTRUCTIONS
    3
    3

    View full-size slide

  127. STACK
    (1 + 2) * 3
    push 1
    push 2
    add
    push 3
    multiply
    INSTRUCTIONS
    3
    3

    View full-size slide

  128. STACK
    (1 + 2) * 3
    push 1
    push 2
    add
    push 3
    multiply
    INSTRUCTIONS
    9

    View full-size slide

  129. • 
    BYTECODE COMPILATION IN MRI

    View full-size slide

  130. RubyVM::InstructionSequence

    View full-size slide

  131. [1] pry(main)> code = '(15 + 6) * 2'

    View full-size slide

  132. [1] pry(main)> code = '(15 + 6) * 2'
    => '(15 + 6) * 2'
    [2] pry(main)>

    View full-size slide

  133. RubyVM::InstructionSequence.compile(code)
    .disasm

    View full-size slide

  134. [1] pry(main)> code = '(15 + 6) * 2'
    => '(15 + 6) * 2'
    [2] pry(main)> puts RubyVM::InstructionSequence.compile(code).disasm
    == disasm: #@>================================
    0000 trace 1 ( 1)
    0002 putobject 15
    0004 putobject 6
    0006 opt_plus ,
    0009 putobject 2
    0011 opt_mult ,
    0014 leave
    => nil

    View full-size slide

  135. • 
    GENERATING BYTECODE

    View full-size slide

  136. class Node
    def bytecode(g)
    # not implemented
    end
    end

    View full-size slide

  137. • 
    PURE MATZLISP BYTECODE
    GENERATION DSL
    http://rubinius.com/doc/en/bytecode-compiler/generator/

    View full-size slide

  138. class IntegerNode < Node
    attr_reader :value
    def initialize(value)
    @value = value
    end
    def bytecode(g)
    pos(g)
    g.push_int @value
    end
    end

    View full-size slide

  139. class IntegerNode < Node
    attr_reader :value
    def initialize(value)
    @value = value
    end
    def bytecode(g)
    pos(g)
    g.push_int @value
    end
    end

    View full-size slide

  140. class AddNode < Node
    attr_reader :left_operand, :right_operand
    def initialize(left_operand, right_operand)
    super
    @left_operand = left_operand
    @right_operand = right_operand
    end
    def bytecode(g)
    pos(g)
    @left_operand.bytecode(g)
    @right_operand.bytecode(g)
    g.send(:+, 1)
    end
    end

    View full-size slide

  141. • 
    YOU HAVE A WORKING
    PROGRAMMING LANGUAGE!

    View full-size slide

  142. • 
    WHERE TO GET HELP?

    View full-size slide

  143. github.com/kanaka/mal

    View full-size slide

  144. github.com/queenfrankie/lani

    View full-size slide

  145. github.com/jsyeo/malady

    View full-size slide

  146. • 
    QUESTIONS?

    View full-size slide