GText: A Language Workbench based on GLL and Term Rewriting

Final presentation of the Master's thesis at Eindhoven University of Technology

Transcript

1. A language workbench based on GLL and Term Rewriting E E + E + E E 1 E 2 3 by: Ali Afroozeh supervisor: Prof. Mark van den Brand
2. Introduction
3. “Language workbenches are, in essence, tools that help you build your own DSLs and provide tool support for them in the style of modern IDEs. The idea is that these tools don't just provide an IDE to help create DSLs; they support building IDEs for editing these DSLs.” 
 
 Martin Fowler Language Workbench
4. Current workbenches ASF+SDF Meta Environment Rascal Spoofax
5. A language workbench is more than just parsing,
   but..
6. Category 1 • ANTLR-based • No modularity • Eclipse-based • EMF mapping
7. Category 2 ASF+SDF Meta Environment Rascal Spoofax • SDF-based • SGLR-based? • No integration with EMF
8. A new parsing algorithm:
   
9. Previous Work Master Thesis mlBNF – A Syntax Formalism for Domain Specific Languages M.W. Manders BSc April 5, 2011 Master Thesis mlBNF – A Syntax Formalism for Domain Specific Languages M.W. Manders BSc April 5, 2011 Supervisors Prof. Dr. M.G.J (Mark) van den Brand Prof. A (Adrian) Johnstone
10. Previous Work GLL
11. Previous Work GLL BNF
12. Previous Work GLL BNF Annotations
13. Previous Work GLL BNF EMF Mapping Annotations
14. GLL EMF Mapping Annotations
15. GLL EMF Mapping Annotations Error Reporting
16. GLL EMF Mapping Annotations Error Reporting Disambiguation
17. GLL EMF Mapping Annotations Error Reporting Disambiguation EBNF
18. GLL EMF Mapping Annotations Error Reporting Disambiguation Eclipse Plugin EBNF
19. GLL EMF Mapping Annotations Error Reporting Disambiguation Eclipse Plugin EBNF
20. Gtext: Goals
21. Gtext: Goals • EMF mapping • Ambiguous grammars
22. Gtext: Goals • EMF mapping • Ambiguous grammars • Difficult Parsing problems • Language embeddings
23. Gtext: Goals • EMF mapping • Ambiguous grammars • Difficult Parsing problems • Language embeddings • IDEs for complex DSLs
24. Outline • The GLL parsing algorithm • Disambiguation • Disambiguation examples • Implementation • Live Demo
25. GLL
26. Generalized LL Parsing
27. Generalized LL Parsing • by Elizabeth Scott and 
 Adrian Johnstone
28. Generalized LL Parsing • by Elizabeth Scott and 
 Adrian Johnstone • Generalizes recursive-descent parsing
29. Generalized LL Parsing • by Elizabeth Scott and 
 Adrian Johnstone • Generalizes recursive-descent parsing • Supports the full class of context-free grammars
30. Generalized LL Parsing • by Elizabeth Scott and 
 Adrian Johnstone • Generalizes recursive-descent parsing • Supports the full class of context-free grammars • Supports grammars with left recursion
31. An Example S ::= A d | B A ::= a B ::= b
32. An Example S ::= A d | B A ::= a B ::= b Nonterminal
33. An Example S ::= A d | B A ::= a B ::= b Terminal
34. An Example S ::= A d | B A ::= a B ::= b Body Head
35. An Example S ::= A d | B A ::= a B ::= b Alternate
36. An Example S ::= A d | B A ::= a B ::= b Produced language: ad, b
37. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
38. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
39. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
40. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
41. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
42. main() { i = 0; if(I[i] in {a,b}) parseS(); else error(); if(I[i] == $) return success(); else error(); } // S ::= A d | B parseS() { if(I[i] in {a}) parseA(); if(I[i] == d) i++; else error(); else if(I[i] in {b}) B(); else error(); else error(); } // A ::= a parseA() { if(I(i) == a) i++; else error(); } // B ::= b parseB() { if(I(i) == b) i++; else error();}
43. non-LL(1) grammars S ::= A S d | B S | ε A ::= a | c B ::= a | b E ::= E + E | E - E | E * E | E / E | Digit
44. non-LL(1) grammars S ::= A S d | B S | ε A ::= a | c B ::= a | b E ::= E + E | E - E | E * E | E / E | Digit Multiple choices
45. non-LL(1) grammars S ::= A S d | B S | ε A ::= a | c B ::= a | b E ::= E + E | E - E | E * E | E / E | Digit Multiple choices Left recursion
46. Top-down Parsers O(n) O(n3) O(2n) LL(1) GLL Backtracking
47. Parse Forest
48. GLL Parser SPPF
49. SPPF • Shared Packed Parse Forest • Merges all derivations • space complexity O(n3)
50. SPPF Packed Node Intermediate Node Symbol Node
51. SPPF E ::= E “+” E | Digit Digit ::= [1-9]+ Input: 1 + 2 E E E + Digit: 1 Digit: 2
52. SPPF E ::= E “+” E | Digit Digit ::= [1-9]+ Input: 1 + 2 E E E + Digit: 1 Digit: 2
53. SPPF E ::= E “+” E | Digit Digit ::= [1-9]+ Input: 1 + 2 E E E + Digit: 1 Digit: 2
54. SPPF E ::= E “+” E | Digit Digit ::= [1-9]+ Input: 1 + 2 E E E + Digit: 1 Digit: 2
55. SPPF E ::= E “+” E | Digit Digit ::= [1-9]+ Input: 1 + 2 + 3 E E E E + Digit: 1 E + Digit: 2 Digit: 3 E
56. Removing unnecessary nodes E E + E + E E Digit: 1 E Digit: 2 Digit: 3 E E E E + Digit: 1 E + Digit: 2 Digit: 3 E
57. E E + E + E E Digit: 1 E Digit: 2 Digit: 3 Two Derivations
58. Two Derivations E E + E E + E Digit: 3 Digit: 1 Digit: 2 E E + E Digit: 1 E + E Digit: 2 Digit: 3
59. Disambiguation
60. Ambiguity
61. Remove Rule
62. Remove Rule Illegal Pattern
63. Prefer Rule
64. Prefer Rule Preferred Pattern
65. Disambiguation Rules • Remove rule, removing an illegal pattern.
 remove [pattern]
 • Prefer rule, preferring one pattern to another.
 prefer [pattern1], [pattern2]
66. Examples
67. Left Associativity remove [E, “+”, E(E, “+”, E)] E E + E Digit: 1 E + E Digit: 2 Digit: 3
68. The Dangling Else Ambiguity E::= "expr" S ::= "if" E "then" S+ | "if" E "then" S+ "else" S+ | "other"
69. The Dangling Else Ambiguity E::= "expr" S ::= "if" E "then" S+ | "if" E "then" S+ "else" S+ | "other" Input: if expr then if expr then other else other
70. The Dangling Else Ambiguity E::= "expr" S ::= "if" E "then" S+ | "if" E "then" S+ "else" S+ | "other" Input: if expr then if expr then other else other
71. The Dangling Else Ambiguity E::= "expr" S ::= "if" E "then" S+ | "if" E "then" S+ "else" S+ | "other" Input: if expr then if expr then other else other
72. The Dangling Else Ambiguity S if E then S else S S expr if E then S expr other other
73. The Dangling Else Ambiguity S if E then S else S expr if E then S other expr other S if E then S expr if E then S else S expr other other ["if", E, "then", S] ["if", E, "then", S, “else”, S]
74. The Dangling Else Ambiguity S if E then S else S expr if E then S other expr other S if E then S expr if E then S else S expr other other ["if", E, "then", S] ["if", E, "then", S, “else”, S] , prefer
75. The Island-Water Ambiguity public static void main(String[] args) { int x = 0; x = 10; if( x > 1) { x = 15; } }
76. The Island-Water Ambiguity public static void main(String[] args) { int x = 0; x = 10; if( x > 1) { x = 15; } }
77. The Island-Water Ambiguity Program ::= Chunk* Chunk ::= Island | Water Island ::= Id “=” Digit “;” Water ::= Id | Integer | String | Char | SpecialChar SpecialChar ::= [; + - * / = ...] public static void main(String[] args) { int x = 0; x = 10; if( x > 1) { x = 15; } }
78. The Island-Water Ambiguity Input: x = 10; Program Chunk_* Chunk Chunk Chunk Chunk Chunk Water: x Water: = Water: 10 Water: ; Island Id: x = Digit: 10 ;
79. The Island-Water Ambiguity Program Chunk_* Chunk Chunk Chunk Chunk Chunk Water: x Water: = Water: 10 Water: ; Island Id: x = Digit: 10 ; prefer [ Chunk(Island) ], [ Chunk(Water), _* ]
80. Implementation
81. SPPF as an Algebraic Type SPPFNode ::= SymbolNode(label, children:SPPFNodeList) | PackedNode(children:SPPFNodeList) | IntermediateNode(children:SPPFNodeList) SPPFNodeList ::= concNode(SPPFNode*)
82. SPPF as a Term SymbolNode("E", concNode( PackedNode(concNode(SymbolNode("E", concNode(SymbolNode("digit", concNode()))), SymbolNode("+", concNode()), SymbolNode("E", concNode(SymbolNode("E", concNode(SymbolNode("Digit"))), SymbolNode("+", concNode()), SymbolNode("E", concNode(SymbolNode("Digit", concNode()))))))), PackedNode(concNode(SymbolNode("E", concNode(SymbolNode("E", concNode(SymbolNode("Digit"))), SymbolNode("+", concNode()), SymbolNode("E", concNode(SymbolNode("Digit", concNode()))))), SymbolNode("E", concNode(SymbolNode("+", concNode()))), SymbolNode("E", concNode(SymbolNode("Digit", concNode()))))))) E E + E + E E Digit: 1 E Digit: 2 Digit: 3
83. Disambiguation rules as Rewrite Rules SymbolNode("E", concNode(z1*, PackedNode(concNode(SymbolNode("E", concNode(SymbolNode("digit", concNode()))), SymbolNode("+", concNode()), SymbolNode("E", concNode(SymbolNode("E", concNode(SymbolNode("digit"))), SymbolNode("+", concNode()), SymbolNode("E", concNode(SymbolNode("digit", concNode()))))))), z2*)) -> SymbolNode(z1*, z2*) E E + E Digit: 1 E + E Digit: 2 Digit: 3
84. Parsing Language Embeddings Island Grammar-based Parsing using GLL and Tom Ali Afroozeh1, Jean-Christophe Bach2 , 3, Mark van den Brand1, Adrian Johnstone4, Maarten Manders1, Pierre-Etienne Moreau2 , 3, and Elizabeth Scott4 1 Eindhoven University of Technology, NL-5612 AZ Eindhoven, The Netherlands 2 Inria, Villers-l` es-Nancy, 54600, France 3 Universit´ e de Lorraine, LORIA, Vandœuvre-l` es-Nancy, 54500, France 4 Department of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom Abstract. Extending a language by embedding within it another lan- guage presents significant parsing challenges, especially if the embedding is recursive. The composite grammar is likely to be nondeterministic as a result of tokens that are valid in both the host and the embed- ded language. In this paper we examine the challenges of embedding the Tom language into a variety of general-purpose high level languages. Tom provides syntax and semantics for advanced pattern matching and tree rewriting facilities. Embedded Tom constructs are translated into the host language by a preprocessor, the output of which is a compos- ite program written purely in the host language. Tom implementations exist for Java , C , C# , Python and Caml . The current parser is com- plex and di cult to maintain. In this paper, we describe how Tom can be parsed using island grammars implemented with the Generalised LL
85. Live Demo!
86. Thank You! Questions?