Slide 1

Slide 1 text

Faster, Practical GLL Parsing Ali Afroozeh and Anastasia Izmaylova The foundation of the Iguana parsing framework

Slide 2

Slide 2 text

Parsing technology Compiler construction DSLs Reverse Engineering Speed Natural grammars Modularity

Slide 3

Slide 3 text

Parsing technology Compiler construction DSLs Reverse Engineering Speed Natural grammars Modularity

Slide 4

Slide 4 text

Parsing technology Compiler construction DSLs Reverse Engineering Speed Natural grammars Modularity General parsing technology

Slide 5

Slide 5 text

Declarative parser generation EBNF context-free syntax Disambiguation rules Lexical definitions Correct and efficient parser

Slide 6

Slide 6 text

Declarative parser generation EBNF context-free syntax Disambiguation rules Lexical definitions Correct and efficient parser Single-phase parsing

Slide 7

Slide 7 text

Practical general parsing O(n) O(n3) Deterministic Highly Ambiguous

Slide 8

Slide 8 text

Contributions

Slide 9

Slide 9 text

• Performance improvement of GLL parsing Contributions

Slide 10

Slide 10 text

• Performance improvement of GLL parsing • Efficient implementation of GLL parsing Contributions

Slide 11

Slide 11 text

• Performance improvement of GLL parsing • Efficient implementation of GLL parsing • Implementation of lexical disambiguation filters Contributions

Slide 12

Slide 12 text

def A(i: Int, input: String): Int = { if (input.charAt(i) == 'a') { val j = A(i + 1, input) if (j != -1) if (input.charAt(j) == 'b') return 1 } if (input.charAt(i) == 'a') { val j = A(i + 1, input) if (j != -1) if (input.charAt(j) == 'c') return 1 } if (input.charAt(i) == 'a') return 1 return -1 } A ::= aAb | aAc | a Recursive-descent parsing

Slide 13

Slide 13 text

def } } } A ::= aAb | aAc | a A ::= Aa | a Recursive-descent parsing

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Recursive-descent parsing

Slide 16

Slide 16 text

Recursive-descent parsing Memoization in CFGs Norvig 91

Slide 17

Slide 17 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91

Slide 18

Slide 18 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 LR Parsing Knuth 65

Slide 19

Slide 19 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65

Slide 20

Slide 20 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85

Slide 21

Slide 21 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85 RNGLR Parsing Scott and Johnstone 06

Slide 22

Slide 22 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85 RNGLR Parsing Scott and Johnstone 06 GLR with reduced stack Aycock and Horspool 99

Slide 23

Slide 23 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85 RNGLR Parsing Scott and Johnstone 06 GLR with reduced stack Aycock and Horspool 99 RIGLR Scott and Johnstone 05

Slide 24

Slide 24 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85 RNGLR Parsing Scott and Johnstone 06 GLR with reduced stack Aycock and Horspool 99 RIGLR Scott and Johnstone 05 GLL Parsing Scott and Johnstone 13

Slide 25

Slide 25 text

Recursive-descent parsing Memoization in CFGs Norvig 91 Memoization in CPS Johnson 91 Recursive-ascent parsing Penello 86 LR Parsing Knuth 65 GLR Parsing Tomita 85 RNGLR Parsing Scott and Johnstone 06 GLR with reduced stack Aycock and Horspool 99 RIGLR Scott and Johnstone 05 GLL Parsing Scott and Johnstone 13 Our work

Slide 26

Slide 26 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 27

Slide 27 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 28

Slide 28 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 29

Slide 29 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 30

Slide 30 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 31

Slide 31 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 32

Slide 32 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 33

Slide 33 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 34

Slide 34 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 35

Slide 35 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 36

Slide 36 text

descriptor, and jumps to execute the code associated with the grammar slot of the descriptor. An example of a GLL parser is given below for the grammar 0 : A ::= aAb | aAc | a. R := ?; P := ?; U := ? cU := (L0, 0); cI := 0; cN := $ L0 : if (R 6= ?) LA : add (A ::= .aAb, cU , cI , $) remove (L, u, i, w) from R add (A ::= .aAc, cU , cI , $) cU := u; cI := i; cN := w; goto L add (A ::= .a, cU , cI , $) else if (there exists a node (A, 0, n)) goto L0 report success else report failure L·aAb : if (I[cI ] = a) L·aAc : if (I[cI ] = a) cN := getNodeT (a, cI , cI + 1) cN := getNodeT (a, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cU := create (A ::= aA · b, cU , cI , cN ) cU := create (A ::= aA · c, cU , cI , cN ) goto LA goto LA LaA·b : if (I[cI ] = b) LaA·c : if (I[cI ] = c) cR := getNodeT (b, cI , cI + 1) cR := getNodeT (c, cI , cI + 1) else goto L0 else goto L0 cI := cI + 1 cI := cI + 1 cN := getNodeP (A ::= aAb·, cN , cR ) cN := getNodeP (A ::= aAc·, cN , cR ) pop (cU , cI , cN ); goto L0 pop (cU , cI , cN ); goto L0 We describe the execution of a GLL parser by explaining the steps of the parser at di↵erent grammar slots. Here, and in the rest of the paper, we do not include A ::= aAb | aAc | a

Slide 37

Slide 37 text

A new GSS

Slide 38

Slide 38 text

A ::= aAb | aAc | a a a c R

Slide 39

Slide 39 text

u0 A ::= aAb | aAc | a a a c R

Slide 40

Slide 40 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R

Slide 41

Slide 41 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R

Slide 42

Slide 42 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R

Slide 43

Slide 43 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 44

Slide 44 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 45

Slide 45 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 46

Slide 46 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 47

Slide 47 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 48

Slide 48 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1

Slide 49

Slide 49 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1

Slide 50

Slide 50 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1

Slide 51

Slide 51 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1

Slide 52

Slide 52 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 (A ::= ·aAb, u2, 1, $) (A ::= ·aAc, u2, 1, $) (A ::= ·a, u2, 1, $)

Slide 53

Slide 53 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 (A ::= ·aAb, u2, 1, $) (A ::= ·aAc, u2, 1, $) (A ::= ·a, u2, 1, $)

Slide 54

Slide 54 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 (A ::= ·aAb, u2, 1, $) (A ::= ·aAc, u2, 1, $) (A ::= ·a, u2, 1, $)

Slide 55

Slide 55 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 (A ::= ·aAb, u2, 1, $) (A ::= ·aAc, u2, 1, $) (A ::= ·a, u2, 1, $) A ::= aA · b, 2 A ::= aA · c, 2 u3 u4

Slide 56

Slide 56 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) u0 A ::= aAb | aAc | a a a c R A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 (A ::= ·aAb, u2, 1, $) (A ::= ·aAc, u2, 1, $) (A ::= ·a, u2, 1, $) A ::= aA · b, 2 A ::= aA · c, 2 u3 u4 P = {(u1, (A, 1, 2)), (u2, (A, 1, 2))}

Slide 57

Slide 57 text

A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 A ::= aA · b, 2 A ::= aA · c, 2 u3 u4 u0

Slide 58

Slide 58 text

A ::= aA · b, 1 u1 u2 A ::= aA · c, 1 A ::= aA · b, 2 A ::= aA · c, 2 u3 u4 u0 A, 2 A, 1 A, 0 A ::= aA · b A ::= aA · b A ::= aA · c A ::= aA · c

Slide 59

Slide 59 text

A ::= aAb | aAc | a a a c R

Slide 60

Slide 60 text

A ::= aAb | aAc | a a a c R A, 0 u0

Slide 61

Slide 61 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) A ::= aAb | aAc | a a a c R A, 0 u0

Slide 62

Slide 62 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) A ::= aAb | aAc | a a a c R A, 0 u0

Slide 63

Slide 63 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) A ::= aAb | aAc | a a a c R A, 0 u0

Slide 64

Slide 64 text

(A ::= ·aAb, u0, 0, $) (A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0

Slide 65

Slide 65 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0

Slide 66

Slide 66 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0

Slide 67

Slide 67 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0

Slide 68

Slide 68 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0

Slide 69

Slide 69 text

(A ::= ·aAc, u0, 0, $) (A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0 A, 1 A, 0 A ::= aA · c

Slide 70

Slide 70 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0 A, 1 A, 0 A ::= aA · c

Slide 71

Slide 71 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0 A, 1 A, 0 A ::= aA · c

Slide 72

Slide 72 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 1 A, 0 A ::= aA · b u1 u0 A, 1 A, 0 A ::= aA · c

Slide 73

Slide 73 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 2 A, 1 A, 0 A ::= aA · b A ::= aA · b A ::= aA · c A ::= aA · c u2

Slide 74

Slide 74 text

(A ::= ·a, u0, 0, $) (A ::= ·a, u1, 1, $) (A ::= ·aAb, u1, 1, $) (A ::= ·aAc, u1, 1, $) A ::= aAb | aAc | a a a c R A, 2 A, 1 A, 0 A ::= aA · b A ::= aA · b A ::= aA · c A ::= aA · c u2 P = {(u1, (A, 1, 2)}

Slide 75

Slide 75 text

Acknowledgement Alex ten Brink proposed the same modification to the GSS in GLL recognizers.

Slide 76

Slide 76 text

Implementation

Slide 77

Slide 77 text

Hash tables local to GSS nodes • GSS node as a common element • Faster hash code calculation • Fewer hash collisions

Slide 78

Slide 78 text

Duplicate descriptor elimination (L, u, i, w)

Slide 79

Slide 79 text

Duplicate descriptor elimination (L, u, i, w) (A ::= ↵ · , (A, j), i, (A ::= ↵ · , j, i)) ( A ::= x · , ( A, j ) , i, ( x, j, i ))

Slide 80

Slide 80 text

Duplicate descriptor elimination (L, u, i, w) (L, A, i, j) (A ::= ↵ · , (A, j), i, (A ::= ↵ · , j, i)) ( A ::= x · , ( A, j ) , i, ( x, j, i ))

Slide 81

Slide 81 text

Duplicate descriptor elimination (L, u, i, w) (L, j) (L, A, i, j) (A ::= ↵ · , (A, j), i, (A ::= ↵ · , j, i)) ( A ::= x · , ( A, j ) , i, ( x, j, i ))

Slide 82

Slide 82 text

Parsing Results

Slide 83

Slide 83 text

Parsing Results (u, w)

Slide 84

Slide 84 text

Parsing Results (u, w) ((A, i), (A, i, j))

Slide 85

Slide 85 text

Parsing Results (u, w) ((A, i), (A, i, j)) (A, i, j)

Slide 86

Slide 86 text

Parsing Results (u, w) ((A, i), (A, i, j)) (A, i, j) j

Slide 87

Slide 87 text

Lexical disambiguation filters

Slide 88

Slide 88 text

adapted from [14]. This grammar defines a Term as either a sequence of two erms, an identifier, a number, or the keyword "int". Id is defined as one or more repetition of a single character, and WS defines a possibly empty blank. Term ::= Term WS Term | Id | Num | "int" Id ::= Chars Chars ::= Chars Char | Char Char ::= ' a ' | .. | ' z ' Num ::= ' 1 ' | .. | ' 9 ' WS ::= ' ' | ✏ his grammar is ambiguous. For example, the input string "hi" can be parsed s either Term(Id("hi")), or Term(Term(Id("h")),Term(Id("i"))). Follow- ng the longest match rule, the first derivation is the intended one, as in the sec- nd one "h" is recognized as an identifier, while it is followed by "i". We can use follow restriction ( / ) to disallow an identifier to be followed by another char- cter: Id ::= Chars -/- Char. Another ambiguity occurs in the input string intx" which can be parsed as either Term(Id("intx")) or Term(Term("int"), erm(Id("x"))). We can solve this problem by adding a precede restriction ( \ ) s follows: Id ::= Char -\- Chars, specifying that Id cannot be preceded by character. Finally, we should exclude the recognition of "int" as Id. For this, e use an exclusion rule: Id ::= Chars \"int". Below we formally define each of these restrictions and show how they can be van den Brand et al. 2002

Slide 89

Slide 89 text

adapted from [14]. This grammar defines a Term as either a sequence of two erms, an identifier, a number, or the keyword "int". Id is defined as one or more repetition of a single character, and WS defines a possibly empty blank. Term ::= Term WS Term | Id | Num | "int" Id ::= Chars Chars ::= Chars Char | Char Char ::= ' a ' | .. | ' z ' Num ::= ' 1 ' | .. | ' 9 ' WS ::= ' ' | ✏ his grammar is ambiguous. For example, the input string "hi" can be parsed s either Term(Id("hi")), or Term(Term(Id("h")),Term(Id("i"))). Follow- ng the longest match rule, the first derivation is the intended one, as in the sec- nd one "h" is recognized as an identifier, while it is followed by "i". We can use follow restriction ( / ) to disallow an identifier to be followed by another char- cter: Id ::= Chars -/- Char. Another ambiguity occurs in the input string intx" which can be parsed as either Term(Id("intx")) or Term(Term("int"), erm(Id("x"))). We can solve this problem by adding a precede restriction ( \ ) s follows: Id ::= Char -\- Chars, specifying that Id cannot be preceded by character. Finally, we should exclude the recognition of "int" as Id. For this, e use an exclusion rule: Id ::= Chars \"int". Below we formally define each of these restrictions and show how they can be “hi": Term(Term(Id("h")), Term(Id("i"))) Term(Id("hi")) van den Brand et al. 2002

Slide 90

Slide 90 text

adapted from [14]. This grammar defines a Term as either a sequence of two erms, an identifier, a number, or the keyword "int". Id is defined as one or more repetition of a single character, and WS defines a possibly empty blank. Term ::= Term WS Term | Id | Num | "int" Id ::= Chars Chars ::= Chars Char | Char Char ::= ' a ' | .. | ' z ' Num ::= ' 1 ' | .. | ' 9 ' WS ::= ' ' | ✏ his grammar is ambiguous. For example, the input string "hi" can be parsed s either Term(Id("hi")), or Term(Term(Id("h")),Term(Id("i"))). Follow- ng the longest match rule, the first derivation is the intended one, as in the sec- nd one "h" is recognized as an identifier, while it is followed by "i". We can use follow restriction ( / ) to disallow an identifier to be followed by another char- cter: Id ::= Chars -/- Char. Another ambiguity occurs in the input string intx" which can be parsed as either Term(Id("intx")) or Term(Term("int"), erm(Id("x"))). We can solve this problem by adding a precede restriction ( \ ) s follows: Id ::= Char -\- Chars, specifying that Id cannot be preceded by character. Finally, we should exclude the recognition of "int" as Id. For this, e use an exclusion rule: Id ::= Chars \"int". Below we formally define each of these restrictions and show how they can be “hi": Term(Term(Id("h")), Term(Id("i"))) Term(Id("hi")) “intx": Term(Id("intx")) Term(Term("int"), Term(Id("x"))) van den Brand et al. 2002

Slide 91

Slide 91 text

adapted from [14]. This grammar defines a Term as either a sequence of two erms, an identifier, a number, or the keyword "int". Id is defined as one or more repetition of a single character, and WS defines a possibly empty blank. Term ::= Term WS Term | Id | Num | "int" Id ::= Chars Chars ::= Chars Char | Char Char ::= ' a ' | .. | ' z ' Num ::= ' 1 ' | .. | ' 9 ' WS ::= ' ' | ✏ his grammar is ambiguous. For example, the input string "hi" can be parsed s either Term(Id("hi")), or Term(Term(Id("h")),Term(Id("i"))). Follow- ng the longest match rule, the first derivation is the intended one, as in the sec- nd one "h" is recognized as an identifier, while it is followed by "i". We can use follow restriction ( / ) to disallow an identifier to be followed by another char- cter: Id ::= Chars -/- Char. Another ambiguity occurs in the input string intx" which can be parsed as either Term(Id("intx")) or Term(Term("int"), erm(Id("x"))). We can solve this problem by adding a precede restriction ( \ ) s follows: Id ::= Char -\- Chars, specifying that Id cannot be preceded by character. Finally, we should exclude the recognition of "int" as Id. For this, e use an exclusion rule: Id ::= Chars \"int". Below we formally define each of these restrictions and show how they can be “hi": Term(Term(Id("h")), Term(Id("i"))) Term(Id("hi")) “intx": Term(Id("intx")) Term(Term("int"), Term(Id("x"))) "int" Term(Id(“int”)) Term(“int”) van den Brand et al. 2002

Slide 92

Slide 92 text

• Follow restriction • Precede restriction • Exclude Id ::= Chars -/- [a-z] Id ::= Chars -\- [a-z] Id ::= Chars \ "int"

Slide 93

Slide 93 text

A ::= a A -/- [x] b A, 1 A, 0 A ::= aA · b u1 u0 A, 1 A, 0 A ::= aA · c | a A c | a

Slide 94

Slide 94 text

Evaluation

Slide 95

Slide 95 text

0 100 200 300 400 0 20000 40000 Number of b's CPU user time (milliseconds) Original GSS New GSS S ::= SSS | SS | b

Slide 96

Slide 96 text

● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●●● ● ● ● ● ● ● ● ● ● ● ● Amb OCaml C# Java 1 2 3 4 5 6 7 8 9 10 11 12 13 14 • Java (7449 files from JDK 1.7) • C# (2764 files from Roslyn compiler build-preview) • OCaml (871 files from OCaml 4.0.1) • Highly ambiguous (inputs from 1 to 400)

Slide 97

Slide 97 text

“Invasive Green Iguana in Grand CaymanSascha” by Grabow www.saschagrabow.com - CC BY-SA 3.0 https://github.com/cwi-swat/iguana