Naren
September 21, 2019
640

# Creating a programming language for fun

Creating an interpreter or a programming language can sound complicated but in reality getting started with it is not that difficult as it sounds – it is a bunch of switch statements and recursive logics. In this talk I am going to give an introduction to interpreter internals and we are going to walk through the high level blocks and the Golang code to create a simple interpreted programming language.
The key take away for the audience will be my lessons from creating a programming language, resources to get started to create one and most importantly having fun:)

## Naren

September 21, 2019

## Transcript

1. Creating a programming language for fun
Naren

2. Naren
Independent Solution Architect
Privacy Enthusiast
Amateur Systems Programmer
@DudeWhoCode

www.dudewho.codes

3. How to write an interpreter?

4. let a = 5 + 2 * 3

5. Lexer Parser Evaluator
AST Object System
Interpreter

6. Lexer
Source Code Tokens
// Declaring a var "a"
let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},
Lexer

7. let a = 5 + 2 * 3
Keyword
Identiﬁer
Operator
Operator Operator
Literal Literal Literal

8. let a = 5 + 2 * 3;

9. let a = 5 + 2 * 3;

10. let a = 5 + 2 * 3;
“let a = 5 + 2 * 3”
0
1
‘l’
token = {LET, "let"}

11. “let a = 5 + 2 * 3”
3
4
‘ ’
token = {LET, "let"}
let a = 5 + 2 * 3;

12. “let a = 5 + 2 * 3”
4
5
‘a’
token = {IDENT, "a"}
let a = 5 + 2 * 3;

13. “let a = 5 + 2 * 3”
6
7
‘=’
token = {ASSIGN, "="}
let a = 5 + 2 * 3;

14. “let a = 5 + 2 * 3”
8
9
‘5’
token = {INT, "5"}
let a = 5 + 2 * 3;

15. “let a = 5 + 2 * 3”
10
11
‘+’
token = {PLUS, “+"}
let a = 5 + 2 * 3;

16. “let a = 5 + 2 * 3”
12
13
‘2’
token = {INT, "2"}
let a = 5 + 2 * 3;

17. “let a = 5 + 2 * 3”
14
15
‘*’
token = {ASTERISK, "*"}
let a = 5 + 2 * 3;

18. “let a = 5 + 2 * 3”
16
17
‘3’
token = {INT, "3"}
let a = 5 + 2 * 3;

19. let a = 5 + 2 * 3;
“let a = 5 + 2 * 3”
17
17
‘;’
token = {SEMICOLON, ";"}

20. Lexer
Source Code Tokens
let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},

21. Parser
Parser
Tokens AST

22. Abstract Syntax Tree
Abstract the syntax and store it in a tree data structure

23. Why AST ?
5 + 2 * 3 [5, +, 2, *, 3]

24. let a = 5 + 2 * 3
Expression
Statement

25. let a = 5 + 2 * 3
Expression
Statement
Program

26. let a = 5 + 2 * 3
Expression
Program
Statement

27. let a = 5 + 2 * 3
Identiﬁer Expression
Statement

28. Parsing
Top down
Pratt parser
Bottom up

29. Parser
5 + 2 * 3
+
5 *
2 3

30. “To understand recursion, one must ﬁrst understand recursion”
–Stephen Hawking

31. “To understand recursion, one must ﬁrst understand recursion”
–Stephen Hawking

32. Pratt parser
1. Null-Denotation(NUD) or prefix expression
-5
2. Left-Denotation(LED) or infix expression
5 + 3
5 * 3
3. Precedences
difference-1, sum-2, product-3

33. 1. prefix expression
2. infix expression

34. 1. prefixFn()
a, integers
2. infixFn(left)

+, *
let a = 5 + 2 * 3

35. let a = 5 + 2 * 3
ast.LetStatement {
name: “a”
value:
}

36. 5 + 2 * 3;

37. 5 + 2 * 3;

38. 5 + 2 * 3;
ast.IntegerLiteral
{5}

39. 5 + 2 * 3;
ast.IntegerLiteral
{5}
1 2

40. 5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
{5}
+

41. 5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
2

42. 5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}

43. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}

44. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
2 3

45. 5 + 2 * 3;
ast.IntegerLiteral
{5}
ast.Expression
{operator: +}
ast.IntegerLiteral
{2}
ast.Expression
{operator: *}
{2}
*

46. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
3

47. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}

48. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{3}

49. 5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{3}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}

50. ast.IntegerLiteral
{3}
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}

51. ast.IntegerLiteral
{3}
5 + 2 * 3;
ast.Expression
{operator: +}
ast.Expression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}

52. Evaluator
Evaluator
AST
Expected
Output

53. Virtual Machine
Byte Code
AST
Expected
Output
Evaluator
JVM, YARV

54. Virtual Machine
Byte Code
AST
Expected
Output
Machine
Code
Evaluator
JIT

55. Evaluator
Tree walking
interpreter
AST
Expected
Output

56. Object System
A way to represent data and keep track of them
>>> let a = 5 + 2 * 3
>>> a + a

57. let a = 5 + 2 * 3

58. let a = 5 + 2 * 3

59. let a = 5 + 2 * 3
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: +}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.LetStatement {
name: “a”
value:
}

60. ast.LetStatement {
name: “a”
value:
}

61. ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: +}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{5}
ast.IntegerLiteral
{2}
ast.LetStatement {
name: “a”
value:
}

62. ast.InfixExpression
{operator: +}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{5}

63. ast.InfixExpression
{operator: +}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}
ast.IntegerLiteral
{5}

64. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}

65. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}

66. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}

67. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
ast.IntegerLiteral
{2}

68. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}

69. ast.InfixExpression
{operator: +}
object.Integer
{5}
ast.IntegerLiteral
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}

70. ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}

71. ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{3}
ast.InfixExpression
{operator: *}
object.Integer
{2}

72. ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{6}

73. ast.InfixExpression
{operator: +}
object.Integer
{5}
object.Integer
{6}

74. object.Integer
{11}

75. ast.LetStatement {
name: “a”
value: obj.Int{5}
}

76. REPL
(demo)

77. let a = 5 + 2 * 3
{LET, "let"},
{IDENT, "a"},
{ASSIGN, "="},
{INT, "5"},
{PLUS, "+"},
{INT, "2"},
{ASTERISK, "*"},
{INT, "3"},
+
5 *
2 3
11
Lexer Parser Evaluator

78. Why create a programming language?

79. "If you don’t know how compilers work, then you don’t know how
computers work. If you’re not 100% sure whether you know how
compilers work, then you don’t know how they work.”
– Steve Yegge

80. Lexers and parsers are used
everywhere
DSL
JSON SerDes
Search engines

81. You can see through the syntax

82. Most importantly it’s fun

83. Credit where credit’s due
www.interpreterbook.com
Thorsten Ball
@thorstenball

84. References
Structure and Interpretation of Computer Programs
https:/
/dev.to/jrop/pratt-parsing
https:/
/tdop.github.io

85. @DudeWhoCode

[email protected]
www.sushi-lang.org
Thank you