Let's Write a Parser

1. Let's Write a Parser Ionuț G. Stan — I T.A.K.E. — May 2016
2. About Me
3. • Software Developer at Eloquentix About Me
4. • Software Developer at Eloquentix • I work mostly with Scala About Me
5. • Software Developer at Eloquentix • I work mostly with Scala • I like FP, programming languages, compilers About Me
6. • Software Developer at Eloquentix • I work mostly with Scala • I like FP, programming languages, compilers • I started the Bucharest FP meet-up group About Me
7. • Software Developer at Eloquentix • I work mostly with Scala • I like FP, programming languages, compilers • I started the Bucharest FP meet-up group • I occasionally blog on igstan.ro About Me
8. Plan
9. • Vehicle Language: µML Plan
10. • Vehicle Language: µML • Compilers Overview Plan
11. • Vehicle Language: µML • Compilers Overview • Parsing: Intuitions and Live Coding Plan
12. Vehicle Language: µML
13. 1. Integers: 1, 23, 456, etc. Vehicle Language: µML
14. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. Vehicle Language: µML
15. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false Vehicle Language: µML
16. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false 4. Single-argument anonymous functions: fn a => a Vehicle Language: µML
17. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false 4. Single-argument anonymous functions: fn a => a 5. Function application: inc 42 Vehicle Language: µML
18. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false 4. Single-argument anonymous functions: fn a => a 5. Function application: inc 42 6. If expressions: if cond then t else f Vehicle Language: µML
19. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false 4. Single-argument anonymous functions: fn a => a 5. Function application: inc 42 6. If expressions: if cond then t else f 7. Addition and subtraction: a + b, a - b Vehicle Language: µML
20. 1. Integers: 1, 23, 456, etc. 2. Identifiers (only letters): inc, cond, a, etc. 3. Booleans: true and false 4. Single-argument anonymous functions: fn a => a 5. Function application: inc 42 6. If expressions: if cond then t else f 7. Addition and subtraction: a + b, a - b 8. Parenthesized expressions: (a + b) Vehicle Language: µML
21. 9. Let blocks/expressions:
 
 let
 val name = ...
 in
 name
 end Vehicle Language: µML
22. Small Example let val inc = fn a => a + 1 in inc 42 end
23. Compilers Overview
24. Compiler Compilers Overview
25. Compiler Compilers Overview Source Language
26. Target Language Compiler Compilers Overview Source Language
27. Target Language Compiler (fn a => a) 2 Compilers Overview Source Language
28. Target Language Compiler (fn a => a) 2 (function(a){return a})(2) Compilers Overview Source Language
29. Compilers Overview uage T Compiler ) 2 (funct
30. Parsing T Compiler ) 2 Parser uage (funct
31. Abstract Syntax Tree T Compiler ) 2 Parser APP FUN a VAR a INT 2 Abstract Syntax Tree (AST) uage (funct
32. Code Generation T Compiler ) 2 Parser CodeGen APP FUN a VAR a INT 2 Abstract Syntax Tree (AST) uage (funct
33. Many Intermediate Phases de T Compiler ) 2 Parser CodeGen ... AST (funct
34. Type Checking T Compiler ) 2 Parser CodeGen Type Checker AST Typed AST ... uage (funct
35. Last Year's Talk T Compiler ) 2 Parser CodeGen Type Checker AST Typed AST Last Year ... uage (funct
36. Today's Talk T Compiler ) 2 Parser CodeGen Type Checker AST Typed AST Today ... uage (funct
37. Parsing Compiler ) 2 Parser uage
38. Lexing + Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
39. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
40. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
41. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
42. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
43. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
44. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
45. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage
46. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
47. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
48. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
49. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
50. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
51. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
52. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
53. Lexing
54. Lexing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser uage • Expects a stream of characters or bytes • Groups them into semantically atomic units: tokens! • These are the words of the language! • What are the rules for grouping them, though?
55. • Grouping can be thought of as "split by space" Lexing
56. • Grouping can be thought of as "split by space" • Why not exactly that, though? Consider: Lexing
57. • Grouping can be thought of as "split by space" • Why not exactly that, though? Consider: Lexing val sum = 1 + 2 ! val sum=1+2 ! val str = "spaces matter here"
58. • We need rules for grouping characters into tokens Lexing
59. • We need rules for grouping characters into tokens • These rules form the lexical grammar Lexing
60. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions Lexing
61. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions • Conducive to easy and efficient implementations Lexing
62. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions • Conducive to easy and efficient implementations • Using a RegExp library Lexing
63. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions • Conducive to easy and efficient implementations • Using a RegExp library • By hand isn't hard either, just a little cumbersome Lexing
64. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions • Conducive to easy and efficient implementations • Using a RegExp library • By hand isn't hard either, just a little cumbersome • Lexer generators: Lex, Flex, Alex, ANTLR, etc. Lexing
65. • We need rules for grouping characters into tokens • These rules form the lexical grammar • Can be defined using regular expressions • Conducive to easy and efficient implementations • Using a RegExp library • By hand isn't hard either, just a little cumbersome • Lexer generators: Lex, Flex, Alex, ANTLR, etc. • Lexing is what you need for syntax definition files Lexing
66. µML — Lexical Grammar integers 0|[1-9][0-9]* identifiers [a-zA-Z]+ symbols (, ), +, -, =, => keywords if, then, else, let, val, in, end, fn, true, false
67. integers 0|[1-9][0-9]* identifiers [a-zA-Z]+ symbols (, ), +, -, =, => keywords if, then, else, let, val, in, end, fn, true, false µML — Lexical Grammar
68. integers 0|[1-9][0-9]* identifiers [a-zA-Z]+ symbols (, ), +, -, =, => keywords if, then, else, let, val, in, end, fn, true, false µML — Lexical Grammar
69. integers 0|[1-9][0-9]* identifiers [a-zA-Z]+ symbols (, ), +, -, =, => keywords if, then, else, let, val, in, end, fn, true, false µML — Lexical Grammar
70. integers 0|[1-9][0-9]* identifiers [a-zA-Z]+ symbols (, ), +, -, =, => keywords if, then, else, let, val, in, end, fn, true, false µML — Lexical Grammar
71. Code
72. Parsing
73. Parsing Compiler ) 2 Parser Lexer Tokens ( fn a => a ) 2 Parser APP FUN a VAR a INT 2 AST uage
74. • The lexer recognizes valid words in the language Parsing
75. • The lexer recognizes valid words in the language • Not all combinations of valid words form valid phrases in a language Parsing
76. • The lexer recognizes valid words in the language • Not all combinations of valid words form valid phrases in a language • Syntactically correct: val a = 1 Parsing
77. • The lexer recognizes valid words in the language • Not all combinations of valid words form valid phrases in a language • Syntactically correct: val a = 1 • Syntactically incorrect: val val val Parsing
78. • The lexer recognizes valid words in the language • Not all combinations of valid words form valid phrases in a language • Syntactically correct: val a = 1 • Syntactically incorrect: val val val • We must define the structure of phrases Parsing
79. • The lexer recognizes valid words in the language • Not all combinations of valid words form valid phrases in a language • Syntactically correct: val a = 1 • Syntactically incorrect: val val val • We must define the structure of phrases • A syntactical grammar achieves that Parsing
80. • Regular expressions are not powerful enough Parsing
81. • Regular expressions are not powerful enough • REs can't recognize nested structures Parsing
82. • Regular expressions are not powerful enough • REs can't recognize nested structures • Because they use a finite amount of memory Parsing
83. • Regular expressions are not powerful enough • REs can't recognize nested structures • Because they use a finite amount of memory • Nesting needs a stack to remember the upper structures you're traversing Parsing
84. • Regular expressions are not powerful enough • REs can't recognize nested structures • Because they use a finite amount of memory • Nesting needs a stack to remember the upper structures you're traversing • Syntactical grammars express nesting using recursion Parsing
86. It's not weird-looking Unicode characters that make regexes unsuitable for parsing.
87. Syntactical Grammar
88. µML — Syntactical Grammar expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr oper = + | - bool = true | false
89. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr oper = + | - bool = true | false Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
90. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr oper = + | - bool = true | false Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
91. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
92. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
93. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
94. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
95. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
96. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
97. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
98. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
99. expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | - Here, blue symbols represent tokens coming from the lexer, not keywords. µML — Syntactical Grammar
100. • Function application has higher precedence over infix expressions in ML Introducing Precedence
101. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 Introducing Precedence
102. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 • double 1 + 2 ≠ double (1 + 2) Introducing Precedence
103. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 • double 1 + 2 ≠ double (1 + 2) • A rule's alternatives don't encode precedence Introducing Precedence
104. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 • double 1 + 2 ≠ double (1 + 2) • A rule's alternatives don't encode precedence • Grammars convey this by chaining rules in order of precedence Introducing Precedence
105. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 • double 1 + 2 ≠ double (1 + 2) • A rule's alternatives don't encode precedence • Grammars convey this by chaining rules in order of precedence • Doesn't scale with many infix operators Introducing Precedence
106. • Function application has higher precedence over infix expressions in ML • double 1 + 2 = (double 1) + 2 • double 1 + 2 ≠ double (1 + 2) • A rule's alternatives don't encode precedence • Grammars convey this by chaining rules in order of precedence • Doesn't scale with many infix operators • Use a special parser for that, e.g., the Shunting Yard algorithm Introducing Precedence
107. Introducing Precedence expr = int | var | bool | ( expr ) | fn var => expr | if expr then expr else expr | let val var = expr in expr end | expr oper expr | expr expr bool = true | false oper = + | -
108. Introducing Precedence expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
109. Introducing Precedence expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
110. Introducing Precedence expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
111. Introducing Precedence expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
112. Parsing Strategies
113. • Two styles: Parsing Strategies
114. • Two styles: • Top-down parsing: builds tree from the root Parsing Strategies
115. • Two styles: • Top-down parsing: builds tree from the root • Bottom-up parsing: builds tree from the leaves Parsing Strategies
116. • Two styles: • Top-down parsing: builds tree from the root • Bottom-up parsing: builds tree from the leaves • Top-down is easy to write by hand Parsing Strategies
117. • Two styles: • Top-down parsing: builds tree from the root • Bottom-up parsing: builds tree from the leaves • Top-down is easy to write by hand • Bottom-up is not, but it's used by generators Parsing Strategies
118. • Two styles: • Top-down parsing: builds tree from the root • Bottom-up parsing: builds tree from the leaves • Top-down is easy to write by hand • Bottom-up is not, but it's used by generators • Parser generators: YACC, ANTLR, Bison, etc. Parsing Strategies
119. • The simplest known parsing strategy; amenable to hand-coding Recursive Descent Parser
120. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent Recursive Descent Parser
121. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar Recursive Descent Parser
122. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar • Main idea: each grammar production becomes a function Recursive Descent Parser
123. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar • Main idea: each grammar production becomes a function • Recursion in the grammar translates to recursion in the code, hence Recursive Recursive Descent Parser
124. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar • Main idea: each grammar production becomes a function • Recursion in the grammar translates to recursion in the code, hence Recursive • Recursion is the main difference compared to regexes; it needs a stack Recursive Descent Parser
125. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar • Main idea: each grammar production becomes a function • Recursion in the grammar translates to recursion in the code, hence Recursive • Recursion is the main difference compared to regexes; it needs a stack • Very popular, e.g., Clang uses it for C/C++/Obj-C Recursive Descent Parser
126. • The simplest known parsing strategy; amenable to hand-coding • Builds the tree top to bottom, from root to leaves, hence Descent • Parallels the structure of the grammar • Main idea: each grammar production becomes a function • Recursion in the grammar translates to recursion in the code, hence Recursive • Recursion is the main difference compared to regexes; it needs a stack • Very popular, e.g., Clang uses it for C/C++/Obj-C • Parser combinators are an abstraction over this idea Recursive Descent Parser
127. Code
128. • The current grammar has a problem Removing Left-Recursion
129. • The current grammar has a problem • But, it's only a problem for our current parsing strategy; others can easily cope with it Removing Left-Recursion
130. • The current grammar has a problem • But, it's only a problem for our current parsing strategy; others can easily cope with it • The problem is that some rules are left-recursive, i.e., the rule itself appears as the first symbol on the left Removing Left-Recursion
131. • The current grammar has a problem • But, it's only a problem for our current parsing strategy; others can easily cope with it • The problem is that some rules are left-recursive, i.e., the rule itself appears as the first symbol on the left • This is problematic for a recursive descent parser because the structure of function calls follow the structure of rule definitions Removing Left-Recursion
132. • The current grammar has a problem • But, it's only a problem for our current parsing strategy; others can easily cope with it • The problem is that some rules are left-recursive, i.e., the rule itself appears as the first symbol on the left • This is problematic for a recursive descent parser because the structure of function calls follow the structure of rule definitions • That means infinite recursion in the parser, which isn't good Removing Left-Recursion
133. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | - Left-Recursive Grammar
134. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | - Left-Recursive Grammar
135. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | app atomic Left-Recursive Grammar
136. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | atomic atomic | atomic atomic atomic | atomic atomic atomic atomic ... Left-Recursive Grammar
137. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic | atomic atomic | atomic (atomic atomic) | atomic (atomic (atomic atomic)) ... Left-Recursive Grammar
138. expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic { app } Left-Recursive Grammar
139. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix ! app = atomic { app } ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
140. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | infix oper infix
141. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | app oper infix
142. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | app oper infix | app oper app oper infix
143. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | app oper infix | app oper app oper infix | app oper app oper app oper infix
144. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | app oper infix | app oper app oper infix | app oper app oper app oper infix ...
145. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app | app (oper infix) | app (oper app (oper infix)) | app (oper app (oper app (oper infix))) ...
146. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app { oper infix }
147. Removing Left-Recursion expr = infix | fn var => expr | if expr then expr else expr ! infix = app { oper infix } ! app = atomic { app } ! 12 14 13 (12 14) 13 ! atomic = int | var | bool | ( expr ) | let val var = expr in expr end bool = true | false oper = + | -
148. github.com / igstan / itake-2016
149. • Write a lexer for JSON • Write a recursive descent parser for JSON • It's way easier than today's vehicle language • I promise! • Specification: json.org Homework
150. Thank You!
151. Questions!