Slide 1
Slide 1 text
Creating a programming language for fun
Naren
Slide 2
Slide 2 text
Naren
Independent Solution Architect
Privacy Enthusiast
Amateur Systems Programmer
@DudeWhoCode
www.dudewho.codes
Slide 3
Slide 3 text
How to write an interpreter?
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
let a = 5 + 2 * 3
Slide 6
Slide 6 text
Lexer Parser Evaluator
AST Object System
Interpreter
Slide 7
Slide 7 text
Lexer
Source Code Tokens
// Declaring a var "a"
let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},
Lexer
Slide 8
Slide 8 text
let a = 5 + 2 * 3
Keyword
Identifier
Operator
Operator Operator
Literal Literal Literal
Slide 9
Slide 9 text
let a = 5 + 2 * 3;
Slide 10
Slide 10 text
let a = 5 + 2 * 3;
Slide 11
Slide 11 text
let a = 5 + 2 * 3;
“let a = 5 + 2 * 3”
0
1
‘l’
token = {LET, "let"}
Slide 12
Slide 12 text
“let a = 5 + 2 * 3”
3
4
‘ ’
token = {LET, "let"}
let a = 5 + 2 * 3;
Slide 13
Slide 13 text
“let a = 5 + 2 * 3”
4
5
‘a’
token = {IDENT, "a"}
let a = 5 + 2 * 3;
Slide 14
Slide 14 text
“let a = 5 + 2 * 3”
6
7
‘=’
token = {ASSIGN, "="}
let a = 5 + 2 * 3;
Slide 15
Slide 15 text
“let a = 5 + 2 * 3”
8
9
‘5’
token = {INT, "5"}
let a = 5 + 2 * 3;
Slide 16
Slide 16 text
“let a = 5 + 2 * 3”
10
11
‘+’
token = {PLUS, “+"}
let a = 5 + 2 * 3;
Slide 17
Slide 17 text
“let a = 5 + 2 * 3”
12
13
‘2’
token = {INT, "2"}
let a = 5 + 2 * 3;
Slide 18
Slide 18 text
“let a = 5 + 2 * 3”
14
15
‘*’
token = {ASTERISK, "*"}
let a = 5 + 2 * 3;
Slide 19
Slide 19 text
“let a = 5 + 2 * 3”
16
17
‘3’
token = {INT, "3"}
let a = 5 + 2 * 3;
Slide 20
Slide 20 text
let a = 5 + 2 * 3;
“let a = 5 + 2 * 3”
17
17
‘;’
token = {SEMICOLON, ";"}
Slide 21
Slide 21 text
Lexer
Source Code Tokens
let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},
Slide 22
Slide 22 text
Parser
Parser
Tokens AST
Slide 23
Slide 23 text
Abstract Syntax Tree
Abstract the syntax and store it in a tree data structure
Slide 24
Slide 24 text
Why AST ?
5 + 2 * 3 [5, +, 2, *, 3]
Slide 25
Slide 25 text
let a = 5 + 2 * 3
Expression
Statement
Slide 26
Slide 26 text
let a = 5 + 2 * 3
Expression
Statement
Program
Slide 27
Slide 27 text
let a = 5 + 2 * 3
Expression
Program
Statement
Slide 28
Slide 28 text
let a = 5 + 2 * 3
Identifier Expression
Statement
Slide 29
Slide 29 text
Parsing
Top down
Pratt parser
Bottom up
Slide 30
Slide 30 text
Parser
5 + 2 * 3
+
5 *
2 3
Slide 31
Slide 31 text
“To understand recursion, one must first understand recursion”
–Stephen Hawking
Slide 32
Slide 32 text
“To understand recursion, one must first understand recursion”
–Stephen Hawking
Slide 33
Slide 33 text
Pratt parser
1. Null-Denotation(NUD) or prefix expression
-5
2. Left-Denotation(LED) or infix expression
5 + 3
5 * 3
3. Precedences
difference-1, sum-2, product-3
Slide 34
Slide 34 text
1. prefix expression
2. infix expression
Slide 35
Slide 35 text
1. prefixFn()
a, integers
2. infixFn(left)
+, *
let a = 5 + 2 * 3
Slide 36
Slide 36 text
let a = 5 + 2 * 3
ast.LetStatement {
name: “a”
value:
}
Slide 37
Slide 37 text
5 + 2 * 3;
Slide 38
Slide 38 text
5 + 2 * 3;
Slide 39
Slide 39 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
Slide 40
Slide 40 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
1 2
Slide 41
Slide 41 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
{5}
+
Slide 42
Slide 42 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
2
Slide 43
Slide 43 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
Slide 44
Slide 44 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
Slide 45
Slide 45 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
2 3
Slide 46
Slide 46 text
5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
ast.IntegerLiteral
{2}
ast.Expression
{operator: *}
{2}
*
Slide 47
Slide 47 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
3
Slide 48
Slide 48 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
Slide 49
Slide 49 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{3}
Slide 50
Slide 50 text
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{3}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
Slide 51
Slide 51 text
ast.IntegerLiteral
{3}
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
Slide 52
Slide 52 text
ast.IntegerLiteral
{3}
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
Slide 53
Slide 53 text
Evaluator
Evaluator
AST
Expected
Output
Slide 54
Slide 54 text
Virtual Machine
Byte Code
AST
Expected
Output
Evaluator
JVM, YARV
Slide 55
Slide 55 text
Virtual Machine
Byte Code
AST
Expected
Output
Machine
Code
Evaluator
JIT
Slide 56
Slide 56 text
Evaluator
Tree walking
interpreter
AST
Expected
Output
Slide 57
Slide 57 text
Object System
A way to represent data and keep track of them
>>> let a = 5 + 2 * 3
>>> a + a
Slide 58
Slide 58 text
let a = 5 + 2 * 3
Slide 59
Slide 59 text
let a = 5 + 2 * 3
Slide 60
Slide 60 text
let a = 5 + 2 * 3
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: +}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.LetStatement {
name: “a”
value:
}
Slide 61
Slide 61 text
ast.LetStatement {
name: “a”
value:
}
Slide 62
Slide 62 text
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: +}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.LetStatement {
name: “a”
value:
}
Slide 63
Slide 63 text
ast.InfixExpression
{operator: +}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{5}
Slide 64
Slide 64 text
ast.InfixExpression
{operator: +}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{5}
Slide 65
Slide 65 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
Slide 66
Slide 66 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
Slide 67
Slide 67 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
Slide 68
Slide 68 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
Slide 69
Slide 69 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}
Slide 70
Slide 70 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}
Slide 71
Slide 71 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}
Slide 72
Slide 72 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}
Slide 73
Slide 73 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{6}
Slide 74
Slide 74 text
ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{6}
Slide 75
Slide 75 text
object.Integer
{11}
Slide 76
Slide 76 text
ast.LetStatement {
name: “a”
value: obj.Int{5}
}
Slide 77
Slide 77 text
REPL
(demo)
Slide 78
Slide 78 text
let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},
+
5 *
2 3
11
Lexer Parser Evaluator
Slide 79
Slide 79 text
Why create a programming language?
Slide 80
Slide 80 text
"If you don’t know how compilers work, then you don’t know how
computers work. If you’re not 100% sure whether you know how
compilers work, then you don’t know how they work.”
– Steve Yegge
You learn more about computers
Slide 81
Slide 81 text
Lexers and parsers are used
everywhere
DSL
JSON SerDes
Search engines
Slide 82
Slide 82 text
You can see through the syntax
Slide 83
Slide 83 text
Most importantly it’s fun
Slide 84
Slide 84 text
Credit where credit’s due
www.interpreterbook.com
Thorsten Ball
@thorstenball
Slide 85
Slide 85 text
References
Structure and Interpretation of Computer Programs
https:/
/dev.to/jrop/pratt-parsing
https:/
/tdop.github.io