One Parser to Rule Them All

Onward! 2015, SPLASH 2015 http://2015.splashcon.org/event/onward2015-papers-one-parser-to-rule-them-all

Transcript

1. One Parser to Rule Them All Ali Afroozeh Anastasia Izmaylova afruze IAnastassija
2. Isn’t parsing a solved problem?
3. Syntax Definition %token Num E : E '*' E | E '+' E | Num ;
4. Syntax Definition %token Num E : E '*' E | E '+' E | Num ; State 0 0 $accept: . E $end State 1 3 E: Num . Num State 2 0 $accept: E . $end 1 E: E . '*' E 2 | E . '+' E E R3 State 3 0 $accept: E $end . $end State 4 1 E: E '*' . E '*' State 5 2 E: E '+' . E '+' Acc Num State 6 1 E: E . '*' E 1 | E '*' E . 2 | E . '+' E E Num State 7 1 E: E . '*' E 2 | E . '+' E 2 | E '+' E . E '+' R1 R2
5. Syntax Definition E ::= E '*' E | E '+' E | Num
6. Syntax Definition + left * left E ::= E '*' E | E '+' E | Num
7. Syntax Definition %left '+' %left '*' E : E '*' E | E '+' E | Num ; YACC ANTLR E : E '*' E | E '+' E | Num ; E ::= E '*' E left > E '+' E left | Num ; GLR/GLL + left * left E ::= E '*' E | E '+' E | Num
8. Context sensitivity
9. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3
10. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3
11. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3
12. Layout sensitivity align g x = case x of 0 -> 1 _ -> x + 2 + 3
13. Layout sensitivity offside g x = case x of 0 -> 1 _ -> x + 2 + 3
14. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3 g 0 => 1
15. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3 g 0 => 4
16. Layout sensitivity g x = case x of 0 -> 1 _ -> x + 2 + 3 g 0 => 4
17. Typedefs in C typedef int T; main() { int n = (T)+1; }
18. Typedefs in C n => 1 typedef int T; main() { int n = (T)+1; }
19. Typedefs in C typedef int T; main() { int T = 1; int n = (T)+1; } n => 2
20. Conditional directives void test() { #if Debug System.Console.WriteLine(“Debug"); } #else } #endif
22. Operator Precedence
23. Operator Precedence Syntax1
24. Operator Precedence Syntax1 Impl1
25. Operator Precedence Indentation rules Syntax1 Impl1
26. Operator Precedence Indentation rules Syntax1 Syntax2 Impl1
27. Operator Precedence Indentation rules Syntax1 Syntax2 Impl1 Impl2
28. Impl3 Operator Precedence Indentation rules Conditional directives Lexical filters Syntax3 Syntax1 Syntax2 Impl1 Impl2 Impl4
29. Impl3 Operator Precedence Indentation rules Conditional directives Lexical filters Syntax3 Syntax1 Syntax2 Impl1 Impl2 Impl4 Parse table modification, GSS modification, custom lexer, custom preprocessor, constraint solving
30. Data-dependent grammars
31. Data-dependent grammars User-defined, declarative meta constructs
32. Data-dependent grammars Lexical filters User-defined, declarative meta constructs
33. Data-dependent grammars Operator Precedence Lexical filters User-defined, declarative meta constructs
34. Data-dependent grammars Operator Precedence Indentation rules Lexical filters User-defined, declarative meta constructs
35. Data-dependent grammars Operator Precedence Indentation rules Conditional directives Lexical filters User-defined, declarative meta constructs
36. Data-dependent grammars Operator Precedence Indentation rules Conditional directives Lexical filters User-defined, declarative meta constructs
37. Data-dependent grammars Modified GLL parsing Operator Precedence Indentation rules Conditional directives Lexical filters User-defined, declarative meta constructs
38. Data-dependent grammars Modified GLL parsing Operator Precedence Indentation rules Conditional directives Lexical filters User-defined, declarative meta constructs
39. Data-dependent Grammars
40. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets :
41. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ
42. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ ~{6}aaaaaa
43. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ L8 ::= '~{' nm:Number {n=toInt(nm.yield)} '}' Octets(n) ~{6}aaaaaa
44. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ L8 ::= '~{' nm:Number {n=toInt(nm.yield)} '}' Octets(n) ~{6}aaaaaa
45. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ L8 ::= '~{' nm:Number {n=toInt(nm.yield)} '}' Octets(n) ~{6}aaaaaa
46. Data-dependent Grammars Octets ::= Octets Octet | ϵ List of octets : Fixed List of octets : Octets(n) ::= [n > 0] Octets(n - 1) Octet | [n == 0] ϵ L8 ::= '~{' nm:Number {n=toInt(nm.yield)} '}' Octets(n) ~{6}aaaaaa
47. Operator precedence E ::= '-' E > E '*' E left > E '+' E left | 'if' E 'then' E 'else' E | 'a'
48. Operator precedence E ::= '-' E > E '*' E left > E '+' E left | 'if' E 'then' E 'else' E | 'a' E(l,r) ::= [4 >= l] '-' E(l,4) | [3 >= r,3 >= l] E(3,3) '*' E(l,4) | [2 >= r,2 >= l] E(2,2) '+' E(l,3) | [1 >= l] 'if' E(0,0) 'then' E(0,0) 'else' E(0,0) | 'a'
49. Haskell indentation rules Decls ::= align (offside Decl)* | ignore ('{' Decl (';' Decl)* '}')
 Decl ::= FunLHS RHS
 RHS ::= '=' Exp 'where' Decls
50. Haskell indentation rules Decls ::= align (offside Decl)* | ignore ('{' Decl (';' Decl)* '}')
 Decl ::= FunLHS RHS
 RHS ::= '=' Exp 'where' Decls Decls ::= a0:Star1(a0.l) | ignore ('{' Decl Star2 '}') Star1(v) ::= Plus1(v) | ϵ Plus1(v) ::= offside a1:Decl [col(a1.l) == col(v)] | Plus1(v) offside a1:Decl [col(a1.l) == col(v)]
51. C# conditional directives Skipped ::= Part+ Part ::= PpCond | ... PpCond ::= PpIf PpElif* PpElse? PpEndif PpIf ::= '#' 'if' PpExp PpNL Skipped? PpElse ::= '#' 'else' PpNL Skipped? PpEndif ::= '#' 'endif' PpNL
52. global ds = {}
 LAYOUT ::= (Whitespace | Comment | Decl | If | Gbg)* Decl ::= '#' 'define' id:Id {ds=put(ds,id.yield,true)} PpNL | '#' 'undef' id:Id {ds=put(ds,id.yield,false)} PpNL If ::= '#' 'if' v=Exp(ds) [v] ? LAYOUT : (Skipped (Elif|Else|PpEndif)) Else ::= '#' 'else' LAYOUT Gbg ::= GbgElif* GbgElse? '#' 'endif' GbgElse ::= '#' 'else' Skipped C# conditional directives
53. Current results Language Files Success Java 8067 100% C# 5839 99% Haskell 6457 72%
54. Current results C# 2 3 4 5 6 0 1 2 3 size (#characters) in log10 y = 1.098 x − 3 R2 = 0.9821 Regression line Haskell 0 1 2 3 4 5 −1 0 1 2 3 4 y = 0.95 x − 1.616 R2 = 0.95 Regression line Java 2 3 4 5 −1 0 1 2 3 CPU time (milliseconds) in log10 y = 1.212 x − 3.181 R2 = 0.9395 Regression line Language Files Success Java 8067 100% C# 5839 99% Haskell 6457 72%
55. https://github.com/iguana-parser/iguana