Dynamic Language Embedding With Homogeneous Tool Support

Domain-specific languages (DSLs) are increasingly used as embedded languages within general-purpose host languages. DSLs provide a compact, dedicated syntax for specifying parts of an application related to specialized domains. Unfortunately, such language extensions typically do not integrate well with existing development tools. Editors, compilers and debuggers are either unaware of the extensions, or must be adapted at a non-trivial cost. Furthermore, these embedded languages typically conflict with the grammar of the host language and make it difficult to write hybrid code; few mechanisms exist to control the scope and usage of multiple tightly interconnected embedded languages.

In this dissertation we present Helvetia, a novel approach to embed languages into an existing host language by leveraging the underlying representation of the host language used by these tools. We introduce Language Boxes, an approach that offers a simple, modular mechanism to encapsulate (i) compositional changes to the host language, (ii) transformations to address various concerns such as compilation and syntax highlighting, and (iii) scoping rules to control visibility of fine-grained language changes. We describe the design and implementation of Helvetia and Language Boxes, discuss the required infrastructure of a host language enabling language embedding, and validate our approach by case studies that demonstrate different ways to extend or adapt the host language syntax and semantics.

Transcript

1. Dynamic Language Embedding With Homogeneous Tool Support PhD Defense Lukas Renggli Advisor Oscar Nierstrasz 1
2. 2 SELECT  email  FROM  users WHERE  username  =  'lr'
3. 3 SELECT  email  FROM  users WHERE  username  =  'lr' Syntax
4. 4 SELECT  email  FROM  users WHERE  username  =  'lr' Semantics
5. General Purpose Host Language 5 + SELECT  email  FROM  users WHERE  username  =  'lr'
6. General Purpose Host Language 6 ? SELECT  email  FROM  users WHERE  username  =  'lr' SyntaxSQL SemanticsSQL SyntaxHost SemanticsHost ToolsHost
8. 8 Host Language External Language
10. 10 Host Language Internal Language
12. 12 Host Language Embedded Language
13. 13 Non-Standard Host Language Embedded Language
15. 15 !" #"$%&
16. Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de 16
17. Conventional Language 17
18. 18 Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de
19. Multiple Context Specific Languages 19
20. 20 Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de
21. Homogeneous Code and Data Abstraction 21
22. § § § 22 Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de
23. § § § Homogeneous Tool Support 23
24. Thesis To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation. 24
25. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 25
26. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 26
27. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 27
28. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 28
29. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 29
30. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 30
31. 31 !" #"$%&
32. 32 Types of Embedded Languages
33. 33 ◦ ◦ ◦ ◦ ◦ • ◦ • ◦ ◦ • • • ◦ ◦ • ◦ • • • ◦ • • • Syntax Vocabulary Semantics
34. 34 Host Language ◦ ◦ ◦ ◦ ◦ • ◦ • ◦ ◦ • • • ◦ ◦ • ◦ • • • ◦ • • • Syntax Vocabulary Semantics
35. 35 Host Language ◦ ◦ ◦ ◦ ◦ • Internal Language ◦ • ◦ ◦ • • • ◦ ◦ • ◦ • • • ◦ • • • Syntax Vocabulary Semantics
36. 36 Host Language ◦ ◦ ◦ ◦ ◦ • Internal Language ◦ • ◦ Pidgin ◦ • • • ◦ ◦ • ◦ • • • ◦ • • • Syntax Vocabulary Semantics
37. 37 Host Language ◦ ◦ ◦ ◦ ◦ • Internal Language ◦ • ◦ Pidgin ◦ • • • ◦ ◦ • ◦ • • • ◦ Creole • • • Syntax Vocabulary Semantics
38. 38 Host Language ◦ ◦ ◦ Argot ◦ ◦ • Internal Language ◦ • ◦ Pidgin ◦ • • • ◦ ◦ • ◦ • • • ◦ Creole • • • Syntax Vocabulary Semantics
39. 39 Host Language ◦ ◦ ◦ Argot ◦ ◦ • Internal Language ◦ • ◦ Pidgin ◦ • • — • ◦ ◦ — • ◦ • — • • ◦ Creole • • • Syntax Vocabulary Semantics
40. 40 Pidgin ◦ • • Creole • • • Argot ◦ ◦ • Syntax Vocabulary Semantics
41. 41   6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH
42. 42   ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH   
43. 43   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH   
44. 44   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH   
45. 45   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH     3LGJLQ  &UHROH  $UJRW
46. 46 Package Name x = 1 y = 1 (2, 1) (2, 2) (1, 2) x = 2 y = 2
47. Package Name x = 1 y = 1 (2, 1) (2, 2) (1, 2) x = 2 y = 2 47 aBuilder  row  grow. aBuilder  row  fill. aBuilder  column  grow. aBuilder  column  fill. aBuilder  x:  1  y:  1  add:  (LabelShape  new   text:  [  :each  |  each  name  ];   borderColor:  #black;   borderWidth:  1;   yourself). aBuilder  x:  1  y:  2  w:  2  h:  1  add:  (RectangleShape  new   borderColor:  #black;   borderWidth:  1;   width:  200;   height:  100;   yourself)
48. Package Name x = 1 y = 1 (2, 1) (2, 2) (1, 2) x = 2 y = 2 48 row  =  grow. row  =  fill. column  =  grow. column  =  fill. (1  ,  1)  =  label     text:  [  :each  |  each  name  ];     borderColor:  #black;     borderWidth:  1. (1  ,  2)  -­‐  (2  ,  1)  =  rectangle     borderColor:  #black;     borderWidth:  1;     width:  200;     height:  100.
49. 49   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH   
50. shape  {   cols:  #grow,  #fill;   rows:  #grow,  #fill; } label  {   position:  1  ,  1;     text:  [  :each  |  each  name  ];   borderColor:  #black;   borderWidth:  1; } rectangle  {     position:  1  ,  2;   colspan:  2;   borderColor:  #black;   borderWidth:  1;   width:  200;   height:  100; } Package Name x = 1 y = 1 (2, 1) (2, 2) (1, 2) x = 2 y = 2 50
51. 51   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH   
52.   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH    52 Conventional Language
53.   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH    53 Conventional Language Context Specific
54.   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH    54 Conventional Language Context Specific Homogeneous Code & Data
55.   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH    55 Conventional Language Context Specific Homogeneous Code & Data Homogeneous Tool Support
56.   5XOHV  ʳ’ƒ”•‡ʴ  ʳ–”ƒ•ˆ‘”ʴ  ʳƒ––”‹„—–‡ʴ  ʳŠ‹‰ŠŽ‹‰Š–ʴ  ʳ‡””‘”ʴ  ʳ”‡ˆƒ…–‘”ʴ  6RXUFH &RGH  6\QWDFWLF $QDO\VLV  6HPDQWLF $QDO\VLV  &RGH *HQHUDWLRQ  ([HFXWDEOH &RGH     3LGJLQ  &UHROH  $UJRW 56 Conventional Language Context Specific Homogeneous Code & Data Homogeneous Tool Support ʳ’ƒ”•‡ʴ 6RXUFH &RGH 3LGJLQ &UHROH $UJRW
57. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 57 Renggli et al. ECOOP 2010
58. 58 Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de
59. 59 Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de
60. Editor Compiler Debugger Syntax Language 1 60
61. Editor Compiler Debugger Syntax Language 1 61 Language 2
62. Editor Compiler Debugger Syntax Language 1 62 Language 2
63. Editor Compiler Debugger Syntax Language 1 63 Language 2
64. Language Boxes 64
65. 65 SELECT  *  FROM  users
66. 66 |  r  | r  :=   ^  User  fromRow:  r . SELECT  *  FROM  users
67. Language Scope Language Concern Language Change Language Box 67
68. Language Scope Active? 68
69. Language Scope 69
70. ‣ System ‣ Packages ‣ Classes ‣ Methods Language Scope 70
71. Language Concern Semantics 71
72. Language Concern Transformation 72
73. Language Concern 73 Context Menus Navigation Search Code Expansion Code Completion Error Correction Custom Inspector Refactorings Code Folding Highlighting
74. Language Change 74 Syntax
75. Language Change 75 Host Language
76. Language Change 76 Host Language + SQL Language
77. Language Change 77 Host Language + SQL Language + ...
78. Host Language + SQL Language + ... = Custom Host Language Language Change 78
80. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 80 Renggli et al. SLE 2009
81. Language Boxes Host Language Dynamic Grammars 81 Language and Tool Extensions
82. Language Change Language Concern Language Scope Language Box 82
83. Language Change = Host Language δ 83
84. Language Change = Host Language Grammar δ 84
85. Language Change = Grammar Transformation 85
86. scanIdentifier self step. ((currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ]) ifTrue: [ [ self recordMatch: #IDENTIFIER. self step. (currentCharacter between: $0 and: $9) or: [ (currentCharacter between: $A and: $Z) or: [ currentCharacter between: $a and: $z ] ] ] whileTrue. ^ self reportLastMatch ] 86
87. #( #[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41] #[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112] #[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34] #[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[0 1 210 0 76 0 81] #[0 1 214 0 76 0 81] #[1 0 173 0 76 0 177 0 81] #[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41] #[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69] #[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34] #[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76] #[0 2 50 0 21 0 25 0 26 0 76 0 79] #[1 1 13 0 76 2 85 0 124 1 21 0 125] #[1 2 89 0 17 2 30 0 21 2 30 0 82] #[1 2 93 0 21 2 97 0 82] ) 87
88. Scannerless Parser Combinator 88
89. a..z a..z 0..9 ID  ::=  letter  {  letter  |  digit  }  ; 89
90. letter letter digit sequence choice many ID  ::=  letter  {  letter  |  digit  }  ; 90
91. Grammar Transformation 91
92. letter letter digit sequence choice many letter    !    letter  |  "_" 92
93. letter digit sequence choice many choice _ letter choice _ letter    !    letter  |  "_"   93
94. letter digit sequence choice many choice _ Optimizations 94
95. Grammar Composition 95
96. Insert  grammar  fragment before/after  grammar  production as  a  choice/sequence/replacement. 96 Language Change
97. Insert  SQL  grammar after  expression  production as  an  additional  choice 97 Language Change
98. 98 Language Change Variable Literal Parens Expr:
99. 99 Language Change Variable Literal Parens Expr: SQL
100. Conflicts & Ambiguities 100
101. SELECT  *  FROM  users 101
102. SELECT  *  FROM  users 102 |  r  | r  :=   ^  User  fromRow:  r .
103. expr sql | 103
104. SELECT  *  FROM  users <SQL: > 104
105. Parsing Expression Grammars 105
106. expr sql / ordered 106
107. expr sql / no conflict 107
108. expr sql / 108 surprise
109. / expr sql 109 surprise
110. expr sql | 110 unordered
111. expr sql | 111 !sql  expr  /  !expr  sql
112. expr sql $ 112 !sql  expr  /  !expr  sql  /  ui
113. expr sql $ 113 !sql  expr  /  !expr  sql  /  ui
114. Language and Tool Extensions Language Boxes Host Language Dynamic Grammars Renggli et al. DYLA 2010 114
115. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 115
116. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes Mondrian Object Relationships Positional Arguments Program Checking Quasiquoting Regular Expression Roman Numbers SPath Expression SQL Schematic Tables String Interpolation Transactional Memory Tuple Space 116 [http://scg.unibe.ch/research/helvetia/examples]
117. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes ✓Mondrian Object Relationships Positional Arguments Program Checking Quasiquoting Regular Expression Roman Numbers SPath Expression ✓SQL Schematic Tables String Interpolation Transactional Memory Tuple Space 117 [http://scg.unibe.ch/research/helvetia/examples]
118. Assignments and Swapping Asynchronous Messages Automaton Brainfuck Language Functional Pattern Matching Grammar Definition Message Pipes ✓Mondrian Object Relationships Positional Arguments ‣ Program Checking ‣ Quasiquoting Regular Expression Roman Numbers SPath Expression ✓SQL Schematic Tables String Interpolation ‣ Transactional Memory Tuple Space 118 [http://scg.unibe.ch/research/helvetia/examples]
119. apply hasChanged hasConflict Change object * changes Process 0..1 currentTransaction do: aBlock retry: aBlock checkpoint abort: anObject escapeContext Transaction previousCopy workingCopy ObjectChange applyBlock conflictTestBlock CustomChange * Transactional Memory 119
120. Meta-Programming Facilities ``(`,(aString) asRegex) 120
121. Domain-Specific Program Checking 121
122. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions Renggli et al. CLSS 2009 Renggli et al. IWST 2009 Nierstrasz et al. LNCS 2009 Renggli et al. TOOLS 2010 122
123. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 123 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
124. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 124 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
125. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 125 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
126. Language Boxes Host Language Dynamic Grammars Language and Tool Extensions 126 To support seamless integration of context-dependent languages without breaking the tools, we need 1. a host-language grammar that can be changed by language extensions, 2. a first-class language description used by the development environment, and 3. a transformation mechanism of the embedded language into a common executable representation.
127. Language Boxes Host Language Dynamic Grammars 127 Language and Tool Extensions