Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Creating a programming language for fun

Naren
September 21, 2019

Creating a programming language for fun

Creating an interpreter or a programming language can sound complicated but in reality getting started with it is not that difficult as it sounds – it is a bunch of switch statements and recursive logics. In this talk I am going to give an introduction to interpreter internals and we are going to walk through the high level blocks and the Golang code to create a simple interpreted programming language.
The key take away for the audience will be my lessons from creating a programming language, resources to get started to create one and most importantly having fun:)

Naren

September 21, 2019
Tweet

More Decks by Naren

Other Decks in Programming

Transcript

  1. Creating a programming language for fun
    Naren

    View Slide

  2. Naren
    Independent Solution Architect

    Privacy Enthusiast
    Amateur Systems Programmer

    @DudeWhoCode


    www.dudewho.codes

    View Slide

  3. How to write an interpreter?

    View Slide

  4. View Slide

  5. let a = 5 + 2 * 3

    View Slide

  6. Lexer Parser Evaluator
    AST Object System
    Interpreter

    View Slide

  7. Lexer
    Source Code Tokens
    // Declaring a var "a"
    let a = 5 + 2 * 3
    {LET, "let"},
    {IDENT, "a"},
    {ASSIGN, "="},
    {INT, "5"},
    {PLUS, "+"},
    {INT, "2"},
    {ASTERISK, "*"},
    {INT, "3"},
    Lexer

    View Slide

  8. let a = 5 + 2 * 3
    Keyword
    Identifier
    Operator
    Operator Operator
    Literal Literal Literal

    View Slide

  9. let a = 5 + 2 * 3;

    View Slide

  10. let a = 5 + 2 * 3;

    View Slide

  11. let a = 5 + 2 * 3;
    “let a = 5 + 2 * 3”
    0
    1
    ‘l’
    token = {LET, "let"}

    View Slide

  12. “let a = 5 + 2 * 3”
    3
    4
    ‘ ’
    token = {LET, "let"}
    let a = 5 + 2 * 3;

    View Slide

  13. “let a = 5 + 2 * 3”
    4
    5
    ‘a’
    token = {IDENT, "a"}
    let a = 5 + 2 * 3;

    View Slide

  14. “let a = 5 + 2 * 3”
    6
    7
    ‘=’
    token = {ASSIGN, "="}
    let a = 5 + 2 * 3;

    View Slide

  15. “let a = 5 + 2 * 3”
    8
    9
    ‘5’
    token = {INT, "5"}
    let a = 5 + 2 * 3;

    View Slide

  16. “let a = 5 + 2 * 3”
    10
    11
    ‘+’
    token = {PLUS, “+"}
    let a = 5 + 2 * 3;

    View Slide

  17. “let a = 5 + 2 * 3”
    12
    13
    ‘2’
    token = {INT, "2"}
    let a = 5 + 2 * 3;

    View Slide

  18. “let a = 5 + 2 * 3”
    14
    15
    ‘*’
    token = {ASTERISK, "*"}
    let a = 5 + 2 * 3;

    View Slide

  19. “let a = 5 + 2 * 3”
    16
    17
    ‘3’
    token = {INT, "3"}
    let a = 5 + 2 * 3;

    View Slide

  20. let a = 5 + 2 * 3;
    “let a = 5 + 2 * 3”
    17
    17
    ‘;’
    token = {SEMICOLON, ";"}

    View Slide

  21. Lexer
    Source Code Tokens
    let a = 5 + 2 * 3
    {LET, "let"},
    {IDENT, "a"},
    {ASSIGN, "="},
    {INT, "5"},
    {PLUS, "+"},
    {INT, "2"},
    {ASTERISK, "*"},
    {INT, "3"},

    View Slide

  22. Parser
    Parser
    Tokens AST

    View Slide

  23. Abstract Syntax Tree
    Abstract the syntax and store it in a tree data structure

    View Slide

  24. Why AST ?
    5 + 2 * 3 [5, +, 2, *, 3]

    View Slide

  25. let a = 5 + 2 * 3
    Expression
    Statement

    View Slide

  26. let a = 5 + 2 * 3
    Expression
    Statement
    Program

    View Slide

  27. let a = 5 + 2 * 3
    Expression
    Program
    Statement

    View Slide

  28. let a = 5 + 2 * 3
    Identifier Expression
    Statement

    View Slide

  29. Parsing
    Top down
    Pratt parser
    Bottom up

    View Slide

  30. Parser
    5 + 2 * 3
    +
    5 *
    2 3

    View Slide

  31. “To understand recursion, one must first understand recursion”
    –Stephen Hawking

    View Slide

  32. “To understand recursion, one must first understand recursion”
    –Stephen Hawking

    View Slide

  33. Pratt parser
    1. Null-Denotation(NUD) or prefix expression
    -5
    2. Left-Denotation(LED) or infix expression

    5 + 3

    5 * 3
    3. Precedences

    difference-1, sum-2, product-3

    View Slide

  34. 1. prefix expression
    2. infix expression

    View Slide

  35. 1. prefixFn()
    a, integers
    2. infixFn(left)


    +, *
    let a = 5 + 2 * 3

    View Slide

  36. let a = 5 + 2 * 3
    ast.LetStatement {
    name: “a”
    value:
    }

    View Slide

  37. 5 + 2 * 3;

    View Slide

  38. 5 + 2 * 3;

    View Slide

  39. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}

    View Slide

  40. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    1 2

    View Slide

  41. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    {5}
    +

    View Slide

  42. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    2

    View Slide

  43. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}


    View Slide

  44. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}

    View Slide

  45. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    2 3

    View Slide

  46. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {2}
    ast.Expression
    {operator: *}

    {2}
    *

    View Slide

  47. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    3

    View Slide

  48. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}

    View Slide

  49. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {3}

    View Slide

  50. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {3}
    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}

    View Slide

  51. ast.IntegerLiteral
    {3}
    5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}

    View Slide

  52. ast.IntegerLiteral
    {3}
    5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}

    View Slide

  53. Evaluator
    Evaluator
    AST
    Expected
    Output

    View Slide

  54. Virtual Machine
    Byte Code
    AST
    Expected
    Output
    Evaluator
    JVM, YARV

    View Slide

  55. Virtual Machine
    Byte Code
    AST
    Expected
    Output
    Machine
    Code
    Evaluator
    JIT

    View Slide

  56. Evaluator
    Tree walking
    interpreter
    AST
    Expected
    Output

    View Slide

  57. Object System
    A way to represent data and keep track of them
    >>> let a = 5 + 2 * 3
    >>> a + a

    View Slide

  58. let a = 5 + 2 * 3

    View Slide

  59. let a = 5 + 2 * 3

    View Slide

  60. let a = 5 + 2 * 3
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: +}

    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.LetStatement {
    name: “a”
    value:
    }

    View Slide

  61. ast.LetStatement {
    name: “a”
    value:
    }

    View Slide

  62. ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: +}

    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.LetStatement {
    name: “a”
    value:
    }

    View Slide

  63. ast.InfixExpression
    {operator: +}

    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {5}

    View Slide

  64. ast.InfixExpression
    {operator: +}

    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {5}

    View Slide

  65. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}

    View Slide

  66. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}

    View Slide

  67. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}

    View Slide

  68. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}

    View Slide

  69. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}

    View Slide

  70. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}

    View Slide

  71. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}

    View Slide

  72. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}

    View Slide

  73. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {6}

    View Slide

  74. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {6}

    View Slide

  75. object.Integer
    {11}

    View Slide

  76. ast.LetStatement {
    name: “a”
    value: obj.Int{5}
    }

    View Slide

  77. REPL
    (demo)

    View Slide

  78. let a = 5 + 2 * 3
    {LET, "let"},
    {IDENT, "a"},
    {ASSIGN, "="},
    {INT, "5"},
    {PLUS, "+"},
    {INT, "2"},
    {ASTERISK, "*"},
    {INT, "3"},
    +
    5 *
    2 3
    11
    Lexer Parser Evaluator

    View Slide

  79. Why create a programming language?

    View Slide

  80. "If you don’t know how compilers work, then you don’t know how
    computers work. If you’re not 100% sure whether you know how
    compilers work, then you don’t know how they work.”
    – Steve Yegge
    You learn more about computers

    View Slide

  81. Lexers and parsers are used
    everywhere
    DSL
    JSON SerDes
    Search engines

    View Slide

  82. You can see through the syntax

    View Slide

  83. Most importantly it’s fun

    View Slide

  84. Credit where credit’s due
    www.interpreterbook.com
    Thorsten Ball
    @thorstenball

    View Slide

  85. References
    Structure and Interpretation of Computer Programs
    https:/
    /dev.to/jrop/pratt-parsing
    https:/
    /tdop.github.io

    View Slide

  86. @DudeWhoCode

    [email protected]
    www.sushi-lang.org
    Thank you

    View Slide