Creating a programming language for fun

Creating an interpreter or a programming language can sound complicated but in reality getting started with it is not that difficult as it sounds – it is a bunch of switch statements and recursive logics. In this talk I am going to give an introduction to interpreter internals and we are going to walk through the high level blocks and the Golang code to create a simple interpreted programming language.
The key take away for the audience will be my lessons from creating a programming language, resources to get started to create one and most importantly having fun:)

Transcript

1. Creating a programming language for fun
   Naren
   

   View Slide

2. Naren
   Independent Solution Architect

   Privacy Enthusiast
   Amateur Systems Programmer

   @DudeWhoCode

   

   www.dudewho.codes
   

   View Slide

3. How to write an interpreter?
   

   View Slide

4. View Slide

5. let a = 5 + 2 * 3
   

   View Slide

6. Lexer Parser Evaluator
   AST Object System
   Interpreter
   

   View Slide

7. Lexer
   Source Code Tokens
   // Declaring a var "a"
   let a = 5 + 2 * 3
   {LET, "let"},
   {IDENT, "a"},
   {ASSIGN, "="},
   {INT, "5"},
   {PLUS, "+"},
   {INT, "2"},
   {ASTERISK, "*"},
   {INT, "3"},
   Lexer
   

   View Slide

8. let a = 5 + 2 * 3
   Keyword
   Identifier
   Operator
   Operator Operator
   Literal Literal Literal
   

   View Slide

9. let a = 5 + 2 * 3;
   

   View Slide

10. let a = 5 + 2 * 3;
    

    View Slide

11. let a = 5 + 2 * 3;
    “let a = 5 + 2 * 3”
    0
    1
    ‘l’
    token = {LET, "let"}
    

    View Slide

12. “let a = 5 + 2 * 3”
    3
    4
    ‘ ’
    token = {LET, "let"}
    let a = 5 + 2 * 3;
    

    View Slide

13. “let a = 5 + 2 * 3”
    4
    5
    ‘a’
    token = {IDENT, "a"}
    let a = 5 + 2 * 3;
    

    View Slide

14. “let a = 5 + 2 * 3”
    6
    7
    ‘=’
    token = {ASSIGN, "="}
    let a = 5 + 2 * 3;
    

    View Slide

15. “let a = 5 + 2 * 3”
    8
    9
    ‘5’
    token = {INT, "5"}
    let a = 5 + 2 * 3;
    

    View Slide

16. “let a = 5 + 2 * 3”
    10
    11
    ‘+’
    token = {PLUS, “+"}
    let a = 5 + 2 * 3;
    

    View Slide

17. “let a = 5 + 2 * 3”
    12
    13
    ‘2’
    token = {INT, "2"}
    let a = 5 + 2 * 3;
    

    View Slide

18. “let a = 5 + 2 * 3”
    14
    15
    ‘*’
    token = {ASTERISK, "*"}
    let a = 5 + 2 * 3;
    

    View Slide

19. “let a = 5 + 2 * 3”
    16
    17
    ‘3’
    token = {INT, "3"}
    let a = 5 + 2 * 3;
    

    View Slide

20. let a = 5 + 2 * 3;
    “let a = 5 + 2 * 3”
    17
    17
    ‘;’
    token = {SEMICOLON, ";"}
    

    View Slide

21. Lexer
    Source Code Tokens
    let a = 5 + 2 * 3
    {LET, "let"},
    {IDENT, "a"},
    {ASSIGN, "="},
    {INT, "5"},
    {PLUS, "+"},
    {INT, "2"},
    {ASTERISK, "*"},
    {INT, "3"},
    

    View Slide

22. Parser
    Parser
    Tokens AST
    

    View Slide

23. Abstract Syntax Tree
    Abstract the syntax and store it in a tree data structure
    

    View Slide

24. Why AST ?
    5 + 2 * 3 [5, +, 2, *, 3]
    

    View Slide

25. let a = 5 + 2 * 3
    Expression
    Statement
    

    View Slide

26. let a = 5 + 2 * 3
    Expression
    Statement
    Program
    

    View Slide

27. let a = 5 + 2 * 3
    Expression
    Program
    Statement
    

    View Slide

28. let a = 5 + 2 * 3
    Identifier Expression
    Statement
    

    View Slide

29. Parsing
    Top down
    Pratt parser
    Bottom up
    

    View Slide

30. Parser
    5 + 2 * 3
    +
    5 *
    2 3
    

    View Slide

31. “To understand recursion, one must first understand recursion”
    –Stephen Hawking
    

    View Slide

32. “To understand recursion, one must first understand recursion”
    –Stephen Hawking
    

    View Slide

33. Pratt parser
    1. Null-Denotation(NUD) or prefix expression
    -5
    2. Left-Denotation(LED) or infix expression

    5 + 3

    5 * 3
    3. Precedences

    difference-1, sum-2, product-3
    

    View Slide

34. 1. prefix expression
    2. infix expression
    

    View Slide

35. 1. prefixFn()
    a, integers
    2. infixFn(left)

    

    +, *
    let a = 5 + 2 * 3
    

    View Slide

36. let a = 5 + 2 * 3
    ast.LetStatement {
    name: “a”
    value:
    }
    

    View Slide

37. 5 + 2 * 3;
    

    View Slide

38. 5 + 2 * 3;
    

    View Slide

39. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    

    View Slide

40. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    1 2
    

    View Slide

41. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    {5}
    +
    

    View Slide

42. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    2
    

    View Slide

43. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    

    View Slide

44. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    

    View Slide

45. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    2 3
    

    View Slide

46. 5 + 2 * 3;
    ast.IntegerLiteral
    {5}
    ast.Expression
    {operator: +}

    ast.IntegerLiteral
    {2}
    ast.Expression
    {operator: *}

    {2}
    *
    

    View Slide

47. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    3
    

    View Slide

48. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    

    View Slide

49. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {3}
    

    View Slide

50. 5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {3}
    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    

    View Slide

51. ast.IntegerLiteral
    {3}
    5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    

    View Slide

52. ast.IntegerLiteral
    {3}
    5 + 2 * 3;
    ast.Expression
    {operator: +}

    ast.Expression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    

    View Slide

53. Evaluator
    Evaluator
    AST
    Expected
    Output
    

    View Slide

54. Virtual Machine
    Byte Code
    AST
    Expected
    Output
    Evaluator
    JVM, YARV
    

    View Slide

55. Virtual Machine
    Byte Code
    AST
    Expected
    Output
    Machine
    Code
    Evaluator
    JIT
    

    View Slide

56. Evaluator
    Tree walking
    interpreter
    AST
    Expected
    Output
    

    View Slide

57. Object System
    A way to represent data and keep track of them
    >>> let a = 5 + 2 * 3
    >>> a + a
    

    View Slide

58. let a = 5 + 2 * 3
    

    View Slide

59. let a = 5 + 2 * 3
    

    View Slide

60. let a = 5 + 2 * 3
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: +}

    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.LetStatement {
    name: “a”
    value:
    }
    

    View Slide

61. ast.LetStatement {
    name: “a”
    value:
    }
    

    View Slide

62. ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: +}

    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {5}
    ast.IntegerLiteral
    {2}
    ast.LetStatement {
    name: “a”
    value:
    }
    

    View Slide

63. ast.InfixExpression
    {operator: +}

    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {5}
    

    View Slide

64. ast.InfixExpression
    {operator: +}

    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    ast.IntegerLiteral
    {5}
    

    View Slide

65. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    

    View Slide

66. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    

    View Slide

67. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    

    View Slide

68. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    ast.IntegerLiteral
    {2}
    

    View Slide

69. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}
    

    View Slide

70. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    ast.IntegerLiteral
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}
    

    View Slide

71. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}
    

    View Slide

72. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {3}
    ast.InfixExpression
    {operator: *}

    object.Integer
    {2}
    

    View Slide

73. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {6}
    

    View Slide

74. ast.InfixExpression
    {operator: +}

    object.Integer
    {5}
    object.Integer
    {6}
    

    View Slide

75. object.Integer
    {11}
    

    View Slide

76. ast.LetStatement {
    name: “a”
    value: obj.Int{5}
    }
    

    View Slide

77. REPL
    (demo)
    

    View Slide

78. let a = 5 + 2 * 3
    {LET, "let"},
    {IDENT, "a"},
    {ASSIGN, "="},
    {INT, "5"},
    {PLUS, "+"},
    {INT, "2"},
    {ASTERISK, "*"},
    {INT, "3"},
    +
    5 *
    2 3
    11
    Lexer Parser Evaluator
    

    View Slide

79. Why create a programming language?
    

    View Slide

80. "If you don’t know how compilers work, then you don’t know how
    computers work. If you’re not 100% sure whether you know how
    compilers work, then you don’t know how they work.”
    – Steve Yegge
    You learn more about computers
    

    View Slide

81. Lexers and parsers are used
    everywhere
    DSL
    JSON SerDes
    Search engines
    

    View Slide

82. You can see through the syntax
    

    View Slide

83. Most importantly it’s fun
    

    View Slide

84. Credit where credit’s due
    www.interpreterbook.com
    Thorsten Ball
    @thorstenball
    

    View Slide

85. References
    Structure and Interpretation of Computer Programs
    https:/
    /dev.to/jrop/pratt-parsing
    https:/
    /tdop.github.io
    

    View Slide

86. @DudeWhoCode
    

    naren@coppermind.io
    www.sushi-lang.org
    Thank you
    

    View Slide