YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

Transcript

1. YAJS.vim and Vim Syntax Highlight othree @ HKOSCon
2. me • othree @ flickr, github, twitter… • MozTW • F2E at Qihoo • Speeches in COSCUP, OSDC.tw, JSDC… • https://blog.othree.net/
3. outline • Why YAJS.vim • Vim Syntax Highlight • The Quests
4. Why YAJS.vim
5. YAJS.vim • Yet Another JavaScript Syntax for Vim • Supports ECMAScript 6 syntax • Remove legacy code support • and JSDoc, Web APIs, console, …. • javascript instead of javaScript
6. default
7. YAJS.vim
9. Why YAJS • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011 • People can’t wait to use new features • Transpilers: Traceur, 6to5(Babel)…
10. • People start to use ES6(and lots of ES.next feature) • JavaScript syntax for Vim doesn’t support ES6 • So I start to do it
11. Dev Target • Support all valid ES6 syntax • If possible, add some error hint • Performance is prioritize to second place
13. Why no Push to Upstream • Lots of incompatible code change during dev • Can’t wait for a PR review
14. Status Now • Support major ES6 syntax • Still in development • Trying to build auto test
15. • https://github.com/othree/yajs.vim
17. Vim Syntax Highlight
18. • Pretend there is a new filetype called ‘ls’ • Not livescript
19. Syntax Items • Keyword • Match • Region
20. Keyword syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
21. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None group name item type
22. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
23. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None target group name source group name link groups
24. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None color definition group name
25. var foo = function () { };
26. var foo = function () { }; TOP region
27. • At TOP region, looking for keywords • function • return
28. var foo = function () { }; match keyword: function
29. var foo = function () { }; highlight it
30. var foo = function () { return 1; }; keep looking
31. var foo = function () { return 1; }; match keyword: return
32. var foo = function () { return 1; }; return 1; match keyword: return
33. Match syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match lsParens /(\k*)/ syntax match lsContains /contains/
34. • Match uses Vim’s regular expression • http://vimdoc.sourceforge.net/htmldoc/pattern.html
35. var foo = function (arg) { return 1; }; return 1; TOP region
36. • Looking for keywords and match pattern • Keyword first, match second
37. var foo = function(arg) { return 1; }; return 1; Match keyword: function
38. var foo = function(arg) { return 1; }; return 1; Match pattern
39. var foo = function(arg) { return 1; }; return 1;
40. var foo = function(arg) { return 1; }; return 1; (arg);
41. contained syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
42. contained • Vim not find contained group at TOP region
43. var foo = function (arg) { return 1; }; return 1; (arg);
44. nextgroup syntax keyword lsKeyword function nextgroup=lsParens syntax keyword lsKeyword return syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
45. nextgroup • Vim will lookup groups listed in nextgroup first • Before normal keyword, match, region • Not limited by contained
46. var foo = function(arg) { return 1; }; return 1; (arg); Ending of keyword function
47. var foo = function(arg) { return 1; }; return 1; (arg); Looking for groups in nextgroup
48. var foo = function(arg) { return 1; }; return 1; (arg); Match pattern
49. var foo = function(arg) { return 1; }; return 1; (arg); Not match any pattern
50. var foo = function (arg) { return 1; }; return 1; (arg); space
51. skipwhite syntax keyword lsKeyword function nextgroup=lsParens skipwhite syntax keyword lsKeyword return syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
52. Vim Keywords syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match lsParens /(\k*)/ syntax keyword lsContains contains
54. syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match lsParens /(\k*)/ syntax match lsContains /contains/
55. syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match lsParens /(\k*)/ syntax match lsContains /contains/ syntax keyword lsKeyword con
56. contains Match keyword first
58. back to the issue var foo = function () { return 1; }; return 1; match keyword: return
59. Region syntax keyword lsKeyword function syntax keyword lsReturn return syntax match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/
60. syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/ contains=lsReturn
61. var foo = function () { return 1; }; return 1;
62. var foo = function () { var bar = function () { return 1; } };
63. Nesting Region syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn
64. var foo = function () { var bar = function () { return 1; } };
65. syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn syntax keyword lsDeclare var let
66. var foo = function () { var bar = function () { return 1; } };
67. TOP region syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=TOP syntax keyword lsDeclare var let
68. var foo = function () { var bar = function () { return 1; } };
69. So…
70. • Not real lexical analysis, have limitation • For performance • Best situation: • every token are independent • no nesting
72. The Quests
73. Quest? • Vim syntax highlight system has limitation • To support all ES2015 syntax is like solve quests
74. Quests • class/object block • function block • template string • arrow function • default parameter
75. blocks
76. blocks • class/object/function/normal block • All starts with `{` and ends with `}` • But have different content
77. normal block for (i = 0; i < arr.length; i++) { console.log(arr[i]); }
78. • Have everything outside of the block • TOP region groups • Function block region groups
79. function block var foo = function () { var bar = 1; return 1; };
80. • Have almost everything in TOP region • TOP region groups • And `return` • But no import/export related stuff
81. object var foo = { bar: 1, buz: function () { return 1; } };
82. class var foo = { bar: 1, buz: function () { return 1; } };
83. • Have all property name syntax and value(RHS) • Property name syntaxes • identifier, number, string • computed • And then RHS expression
84. Issues • These blocks have very different contents • I have to know what this region is at position `{`
85. Normal don’t need to deal Function after `function` keyword, method syntax Class always after `class` keyword Object in RHS expression
86. RHS expression • Right hand side of an operator • Inside a pair of paren `(`, `)`
87. syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/ nextgroup=@javascriptExpression syntax region javascriptParenExp matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptExpression
88. function block
89. function block • Have almost everything in TOP region • TOP region groups • And `return` • But no import/export related stuff
90. syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=TOP
91. TOP • A special group cluster • Contains every group without `contained` • I called these stuff belongs to TOP region
92. but
93. JS in HTML <html> <head> <script> function foo() { return 1; } </script> </head> </html>
94. in HTML Syntax syn include @htmlJavaScript syntax/javascript.vim unlet b:current_syntax syn region javaScript start=+<script\_[^>]*>+ keepend end=+</script\_[^>]*>+me=s-1 contains=@htmlJavaScript
95. in Jinja Syntax " Pull in the HTML syntax. if g:jinja_syntax_html if version < 600 so <sfile>:p:h/html.vim else runtime! syntax/html.vim unlet b:current_syntax endif endif
96. TOP is HTML’s TOP
97. Detect Filetype if &filetype =~ 'javascript' syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=TOP else syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=@htmlJavaScript endif
98. Template String
99. Template String `string text` `string text line 1 string text line 2` `string text ${expression} string text`
100. syntax region javascriptString start=/\z(["']\)/ skip=/\\\\\|\\\z1\|\\\n/ end=/\z1\|$/
101. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\/
102. `string text` `string text line 1 string text line 2` `string text ${expression} string text`
103. The Issue
104. `string text ${"another string, and accent `"} string text`
105. • The string ends when match an accent ` • Try to recognize inner content
106. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\|$/ contains=javascriptTemplateSubstitution syntax region javascriptTemplateSubstitution contained matchgroup=javascriptTemplateSB start=/\${/ end=/}/
107. `string text ${"another string, and accent `"} string text`
108. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\|$/ contains=javascriptTemplateSubstitution syntax region javascriptTemplateSubstitution contained matchgroup=javascriptTemplateSB start=/\${/ end=/}/ contains=@javascriptExpression
109. `string text ${"another string, and accent `"} string text`
110. Arrow Function
111. Main Target • Highlight arrow function and its parameters
112. var foo = function (arg1) {};
113. var foo = function (arg1) {}; var fooArrow = (arg1) => {};
114. var foo = function (arg1) {}; var fooArrow = (arg1) => {}; var fooParen = (arg1);
115. var foo = function (arg1) {}; var fooArrow = (arg1) => {}; var fooParen = (arg1); Can’t lookup following token When cursor is here
116. => syntax match javascriptArrow /=>/ nextgroup=javascriptBlock skipwhite
117. => var foo = function (arg1) {}; var fooArrow = (arg1) => {}; var fooParen = (arg1);
118. Parameter?
119. Problem • Don’t know what it is when meet `(` • Function parameters or • Expression in parentheses
120. Before Default Parameter • Only identifier and comma inside function argument parens • Identifier name possible characters: [a-zA-Z_0-9$] • Not contain right parentheses
121. Workaround Solution • Use match to match an arrow function • Put this match rule after parentheses region
 (higher priority)
122. syntax region javascriptParens ... syntax match javascriptArrowFuncDef /(\_[^)]*)\_s*=>/ contains=javascriptArrowFuncArg, javascriptComma nextgroup=javascriptBlock skipwhite skipempty
123. syntax region javascriptParens ... syntax match javascriptArrowFuncDef /(\_[^)]*)\_s*=>/ contains=javascriptArrowFuncArg, javascriptComma nextgroup=javascriptBlock skipwhite skipempty Match pair of paren
124. syntax match javascriptArrowFunc /=>/ syntax region javascriptArrowFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptArrowFunc skipwhite skipempty
125. syntax cluster javascriptFuncArgElements contains=javascriptFuncKeyword, javascriptComma, javascriptDefaultAssign, @javascriptComments, javascriptFuncArgArray, javascriptFuncArgObject, javascriptSpreadOp
126. var foo = function (arg1) {}; var fooArrow = (arg1) => {}; var fooParen = (arg1);
127. Default Parameter
128. function multiply(a, b) { var b = typeof b !== 'undefined' ? b : 1; return a*b; }
129. syntax keyword javascriptFuncKeyword function nextgroup=javascriptFuncName skipwhite syntax match javascriptFuncName contained /[a-zA-Z_$]\k*/ nextgroup=javascriptFuncArg skipwhite syntax match javascriptFuncArg contained /(\_[^)]*)/ nextgroup=javascriptBlock skipwhite
130. function multiply(a, b) { var b = typeof b !== 'undefined' ? b : 1; return a*b; }
131. function multiply(a, b) { var b = typeof b !== 'undefined' ? b : 1; return a*b; } FuncKeyword FuncName FuncArg
132. function multiply(a, b = 1) { return a*b; }
133. function multiply(a, b = "(1+1)") { return a*b; } ???
134. function multiply(a, b = "(1+1)") { return a*b; } Match ending ) Not match block
135. • Not able to really know which paren is ending one • Have to well know the content of parens • Use region
136. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
137. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
138. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
139. function multiply(a, b = "(1+1)") { return a*b; } Default Assign Expression, string
140. Arrow Function Default Parameter
141. var foo = function (arg1) {}; var fooArrow = (arg1) => {}; var fooParen = (arg1);
142. var foo = function (arg1) {}; var fooArrow = (arg1 = 1) => {}; var fooArrow = (arg1 = 1, arg2 = `string`) => {};
143. var foo = function (arg1) {}; var fooArrow = (arg1 = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
144. Matching fails here var foo = function (arg1) {}; var fooArrow = (arg1 = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
145. Problem • Expression in parentheses and arrow function parameter mixed up
146. var fooArrow1 = function (arg1 = () => {}) {};
147. var fooArrow1 = function (arg1 = () => {}) {}; var fooArrow2 = function (arg1 = () => { return function () {}; }) {};
148. var fooArrow1 = function (arg1 = () => {}) {}; var fooArrow2 = function (arg1 = () => { return function () {}; }) {}; var fooArrow3 = function (arg1 = (arg2 = () => {}) => { return function () {}; }) {};
149. Problem • Parens, arrow function parameter mixed • Arrow function use match instead of region
150. My Latest Quest
151. Still Solving
152. Patch Also Welcome
154. Thank You
155. Questions?