Creating a programming language for fun

Creating an interpreter or a programming language can sound complicated but in reality getting started with it is not that difficult as it sounds – it is a bunch of switch statements and recursive logics. In this talk I am going to give an introduction to interpreter internals and we are going to walk through the high level blocks and the Golang code to create a simple interpreted programming language.
The key take away for the audience will be my lessons from creating a programming language, resources to get started to create one and most importantly having fun:)

Transcript

1. Creating a programming language for fun Naren
2. Naren Independent Solution Architect
 Privacy Enthusiast Amateur Systems Programmer
 @DudeWhoCode
 
 www.dudewho.codes
3. How to write an interpreter?
5. let a = 5 + 2 * 3
6. Lexer Parser Evaluator AST Object System Interpreter
7. Lexer Source Code Tokens // Declaring a var "a" let a = 5 + 2 * 3 {LET, "let"}, {IDENT, "a"}, {ASSIGN, "="}, {INT, "5"}, {PLUS, "+"}, {INT, "2"}, {ASTERISK, "*"}, {INT, "3"}, Lexer
8. let a = 5 + 2 * 3 Keyword Identifier Operator Operator Operator Literal Literal Literal
9. let a = 5 + 2 * 3;
10. let a = 5 + 2 * 3;
11. let a = 5 + 2 * 3; “let a = 5 + 2 * 3” 0 1 ‘l’ token = {LET, "let"}
12. “let a = 5 + 2 * 3” 3 4 ‘ ’ token = {LET, "let"} let a = 5 + 2 * 3;
13. “let a = 5 + 2 * 3” 4 5 ‘a’ token = {IDENT, "a"} let a = 5 + 2 * 3;
14. “let a = 5 + 2 * 3” 6 7 ‘=’ token = {ASSIGN, "="} let a = 5 + 2 * 3;
15. “let a = 5 + 2 * 3” 8 9 ‘5’ token = {INT, "5"} let a = 5 + 2 * 3;
16. “let a = 5 + 2 * 3” 10 11 ‘+’ token = {PLUS, “+"} let a = 5 + 2 * 3;
17. “let a = 5 + 2 * 3” 12 13 ‘2’ token = {INT, "2"} let a = 5 + 2 * 3;
18. “let a = 5 + 2 * 3” 14 15 ‘*’ token = {ASTERISK, "*"} let a = 5 + 2 * 3;
19. “let a = 5 + 2 * 3” 16 17 ‘3’ token = {INT, "3"} let a = 5 + 2 * 3;
20. let a = 5 + 2 * 3; “let a = 5 + 2 * 3” 17 17 ‘;’ token = {SEMICOLON, ";"}
21. Lexer Source Code Tokens let a = 5 + 2 * 3 {LET, "let"}, {IDENT, "a"}, {ASSIGN, "="}, {INT, "5"}, {PLUS, "+"}, {INT, "2"}, {ASTERISK, "*"}, {INT, "3"},
22. Parser Parser Tokens AST
23. Abstract Syntax Tree Abstract the syntax and store it in a tree data structure
24. Why AST ? 5 + 2 * 3 [5, +, 2, *, 3]
25. let a = 5 + 2 * 3 Expression Statement
26. let a = 5 + 2 * 3 Expression Statement Program
27. let a = 5 + 2 * 3 Expression Program Statement
28. let a = 5 + 2 * 3 Identifier Expression Statement
29. Parsing Top down Pratt parser Bottom up
30. Parser 5 + 2 * 3 + 5 * 2 3
31. “To understand recursion, one must first understand recursion” –Stephen Hawking
32. “To understand recursion, one must first understand recursion” –Stephen Hawking
33. Pratt parser 1. Null-Denotation(NUD) or prefix expression -5 2. Left-Denotation(LED) or infix expression
 5 + 3
 5 * 3 3. Precedences
 difference-1, sum-2, product-3
34. 1. prefix expression 2. infix expression
35. 1. prefixFn() a, integers 2. infixFn(left)
 
 +, * let a = 5 + 2 * 3
36. let a = 5 + 2 * 3 ast.LetStatement { name: “a” value: }
37. 5 + 2 * 3;
38. 5 + 2 * 3;
39. 5 + 2 * 3; ast.IntegerLiteral {5}
40. 5 + 2 * 3; ast.IntegerLiteral {5} 1 2
41. 5 + 2 * 3; ast.IntegerLiteral {5} ast.Expression {operator: +}
 {5} +
42. 5 + 2 * 3; ast.IntegerLiteral {5} ast.Expression {operator: +}
 2
43. 5 + 2 * 3; ast.IntegerLiteral {5} ast.Expression {operator: +}

44. 5 + 2 * 3; ast.Expression {operator: +}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2}
45. 5 + 2 * 3; ast.Expression {operator: +}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2} 2 3
46. 5 + 2 * 3; ast.IntegerLiteral {5} ast.Expression {operator: +}
 ast.IntegerLiteral {2} ast.Expression {operator: *}
 {2} *
47. 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2} 3
48. 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2}
49. 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2} ast.IntegerLiteral {3}
50. 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {3} ast.IntegerLiteral {5} ast.IntegerLiteral {2}
51. ast.IntegerLiteral {3} 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2}
52. ast.IntegerLiteral {3} 5 + 2 * 3; ast.Expression {operator: +}
 ast.Expression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2}
53. Evaluator Evaluator AST Expected Output
54. Virtual Machine Byte Code AST Expected Output Evaluator JVM, YARV
55. Virtual Machine Byte Code AST Expected Output Machine Code Evaluator JIT
56. Evaluator Tree walking interpreter AST Expected Output
57. Object System A way to represent data and keep track of them >>> let a = 5 + 2 * 3 >>> a + a
58. let a = 5 + 2 * 3
59. let a = 5 + 2 * 3
60. let a = 5 + 2 * 3 ast.IntegerLiteral {3} ast.InfixExpression {operator: +}
 ast.InfixExpression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2} ast.LetStatement { name: “a” value: }
61. ast.LetStatement { name: “a” value: }
62. ast.IntegerLiteral {3} ast.InfixExpression {operator: +}
 ast.InfixExpression {operator: *}
 ast.IntegerLiteral {5} ast.IntegerLiteral {2} ast.LetStatement { name: “a” value: }
63. ast.InfixExpression {operator: +}
 ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2} ast.IntegerLiteral {5}
64. ast.InfixExpression {operator: +}
 ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2} ast.IntegerLiteral {5}
65. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2}
66. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2}
67. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2}
68. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 ast.IntegerLiteral {2}
69. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 object.Integer {2}
70. ast.InfixExpression {operator: +}
 object.Integer {5} ast.IntegerLiteral {3} ast.InfixExpression {operator: *}
 object.Integer {2}
71. ast.InfixExpression {operator: +}
 object.Integer {5} object.Integer {3} ast.InfixExpression {operator: *}
 object.Integer {2}
72. ast.InfixExpression {operator: +}
 object.Integer {5} object.Integer {3} ast.InfixExpression {operator: *}
 object.Integer {2}
73. ast.InfixExpression {operator: +}
 object.Integer {5} object.Integer {6}
74. ast.InfixExpression {operator: +}
 object.Integer {5} object.Integer {6}
75. object.Integer {11}
76. ast.LetStatement { name: “a” value: obj.Int{5} }
77. REPL (demo)
78. let a = 5 + 2 * 3 {LET, "let"}, {IDENT, "a"}, {ASSIGN, "="}, {INT, "5"}, {PLUS, "+"}, {INT, "2"}, {ASTERISK, "*"}, {INT, "3"}, + 5 * 2 3 11 Lexer Parser Evaluator
79. Why create a programming language?
80. "If you don’t know how compilers work, then you don’t know how computers work. If you’re not 100% sure whether you know how compilers work, then you don’t know how they work.” – Steve Yegge You learn more about computers
81. Lexers and parsers are used everywhere DSL JSON SerDes Search engines
82. You can see through the syntax
83. Most importantly it’s fun
84. Credit where credit’s due www.interpreterbook.com Thorsten Ball @thorstenball
85. References Structure and Interpretation of Computer Programs https:/ /dev.to/jrop/pratt-parsing https:/ /tdop.github.io
86. @DudeWhoCode 
 naren@coppermind.io www.sushi-lang.org Thank you