YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

Transcript

1. YAJS.vim and Vim Syntax
   Highlight
   othree @ HKOSCon
   

   View Slide

2. me
   • othree @ flickr, github, twitter…
   
   • MozTW
   
   • F2E at Qihoo
   
   • Speeches in COSCUP, OSDC.tw, JSDC…
   
   • https://blog.othree.net/
   

   View Slide

3. outline
   • Why YAJS.vim
   
   • Vim Syntax Highlight
   
   • The Quests
   

   View Slide

4. Why YAJS.vim
   

   View Slide

5. YAJS.vim
   • Yet Another JavaScript Syntax for Vim
   
   • Supports ECMAScript 6 syntax
   
   • Remove legacy code support
   
   • and JSDoc, Web APIs, console, ….
   
   • javascript instead of javaScript
   

   View Slide

6. default
   

   View Slide

7. YAJS.vim
   

   View Slide

8. View Slide

9. Why YAJS
   • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011
   
   • People can’t wait to use new features
   
   • Transpilers: Traceur, 6to5(Babel)…
   

   View Slide

10. • People start to use ES6(and lots of ES.next feature)
    
    • JavaScript syntax for Vim doesn’t support ES6
    
    • So I start to do it
    

    View Slide

11. Dev Target
    • Support all valid ES6 syntax
    
    • If possible, add some error hint
    
    • Performance is prioritize to second place
    

    View Slide

12. View Slide

13. Why no Push to Upstream
    • Lots of incompatible code change during dev
    
    • Can’t wait for a PR review
    

    View Slide

14. Status Now
    • Support major ES6 syntax
    
    • Still in development
    
    • Trying to build auto test
    

    View Slide

15. • https://github.com/othree/yajs.vim
    

    View Slide

16. View Slide

17. Vim Syntax Highlight
    

    View Slide

18. • Pretend there is a new filetype called ‘ls’
    
    • Not livescript
    

    View Slide

19. Syntax Items
    • Keyword
    
    • Match
    
    • Region
    

    View Slide

20. Keyword
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    

    View Slide

21. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    group name
    item type
    

    View Slide

22. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    

    View Slide

23. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    target group name
    source group name
    link groups
    

    View Slide

24. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    color definition
    group name
    

    View Slide

25. var foo = function () {
    };
    

    View Slide

26. var foo = function () {
    };
    TOP region
    

    View Slide

27. • At TOP region, looking for keywords
    
    • function
    
    • return
    

    View Slide

28. var foo = function () {
    };
    match keyword: function
    

    View Slide

29. var foo = function () {
    };
    highlight it
    

    View Slide

30. var foo = function () {
    return 1;
    };
    keep looking
    

    View Slide

31. var foo = function () {
    return 1;
    };
    match keyword: return
    

    View Slide

32. var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return
    

    View Slide

33. Match
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/
    

    View Slide

34. • Match uses Vim’s regular expression
    
    • http://vimdoc.sourceforge.net/htmldoc/pattern.html
    

    View Slide

35. var foo = function (arg) {
    return 1;
    };
    return 1;
    TOP region
    

    View Slide

36. • Looking for keywords and match pattern
    
    • Keyword first, match second
    

    View Slide

37. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match keyword: function
    

    View Slide

38. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match pattern
    

    View Slide

39. var foo = function(arg) {
    return 1;
    };
    return 1;
    

    View Slide

40. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    

    View Slide

41. contained
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/
    

    View Slide

42. contained
    • Vim not find contained group at TOP region
    

    View Slide

43. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);
    

    View Slide

44. nextgroup
    syntax keyword lsKeyword function nextgroup=lsParens
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/
    

    View Slide

45. nextgroup
    • Vim will lookup groups listed in nextgroup first
    
    • Before normal keyword, match, region
    
    • Not limited by contained
    

    View Slide

46. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Ending of keyword function
    

    View Slide

47. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Looking for groups in nextgroup
    

    View Slide

48. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Match pattern
    

    View Slide

49. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Not match any pattern
    

    View Slide

50. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);
    space
    

    View Slide

51. skipwhite
    syntax keyword lsKeyword function nextgroup=lsParens skipwhite
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/
    

    View Slide

52. Vim Keywords
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax keyword lsContains contains
    

    View Slide

53. View Slide

54. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/
    

    View Slide

55. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/
    syntax keyword lsKeyword con
    

    View Slide

56. contains
    Match keyword first
    

    View Slide

57. View Slide

58. back to the issue
    var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return
    

    View Slide

59. Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/
    

    View Slide

60. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/ contains=lsReturn
    

    View Slide

61. var foo = function () {
    return 1;
    };
    return 1;
    

    View Slide

62. var foo = function () {
    var bar = function () {
    return 1;
    }
    };
    

    View Slide

63. Nesting Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn
    

    View Slide

64. var foo = function () {
    var bar = function () {
    return 1;
    }
    };
    

    View Slide

65. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn
    syntax keyword lsDeclare var let
    

    View Slide

66. var foo = function () {
    var bar = function () {
    return 1;
    }
    };
    

    View Slide

67. TOP region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=TOP
    syntax keyword lsDeclare var let
    

    View Slide

68. var foo = function () {
    var bar = function () {
    return 1;
    }
    };
    

    View Slide

69. So…
    

    View Slide

70. • Not real lexical analysis, have limitation
    
    • For performance
    
    • Best situation:
    
    • every token are independent
    
    • no nesting
    

    View Slide

71. View Slide

72. The Quests
    

    View Slide

73. Quest?
    • Vim syntax highlight system has limitation
    
    • To support all ES2015 syntax is like solve quests
    

    View Slide

74. Quests
    • class/object block
    
    • function block
    
    • template string
    
    • arrow function
    
    • default parameter
    

    View Slide

75. blocks
    

    View Slide

76. blocks
    • class/object/function/normal block
    
    • All starts with `{` and ends with `}`
    
    • But have different content
    

    View Slide

77. normal block
    for (i = 0; i < arr.length; i++) {
    console.log(arr[i]);
    }
    

    View Slide

78. • Have everything outside of the block
    
    • TOP region groups
    
    • Function block region groups
    

    View Slide

79. function block
    var foo = function () {
    var bar = 1;
    return 1;
    };
    

    View Slide

80. • Have almost everything in TOP region
    
    • TOP region groups
    
    • And `return`
    
    • But no import/export related stuff
    

    View Slide

81. object
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };
    

    View Slide

82. class
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };
    

    View Slide

83. • Have all property name syntax and value(RHS)
    
    • Property name syntaxes
    
    • identifier, number, string
    
    • computed
    
    • And then RHS expression
    

    View Slide

84. Issues
    • These blocks have very different contents
    
    • I have to know what this region is at position `{`
    

    View Slide

85. Normal don’t need to deal
    Function after `function` keyword, method syntax
    Class always after `class` keyword
    Object in RHS expression
    

    View Slide

86. RHS expression
    • Right hand side of an operator
    
    • Inside a pair of paren `(`, `)`
    

    View Slide

87. syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/
    nextgroup=@javascriptExpression
    syntax region javascriptParenExp
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptExpression
    

    View Slide

88. function block
    

    View Slide

89. function block
    • Have almost everything in TOP region
    
    • TOP region groups
    
    • And `return`
    
    • But no import/export related stuff
    

    View Slide

90. syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP
    

    View Slide

91. TOP
    • A special group cluster
    
    • Contains every group without `contained`
    
    • I called these stuff belongs to TOP region
    

    View Slide

92. but
    

    View Slide

93. JS in HTML
    
    
    <br/>function foo() {<br/>return 1;<br/>}<br/>
    
    
    

    View Slide

94. in HTML Syntax
    syn include @htmlJavaScript syntax/javascript.vim
    unlet b:current_syntax
    syn region javaScript
    start=+]*>+ keepend<br/>end=+]*>+me=s-1
    contains=@htmlJavaScript
    

    View Slide

95. in Jinja Syntax
    " Pull in the HTML syntax.
    if g:jinja_syntax_html
    if version < 600
    so :p:h/html.vim
    else
    runtime! syntax/html.vim
    unlet b:current_syntax
    endif
    endif
    

    View Slide

96. TOP is HTML’s TOP
    

    View Slide

97. Detect Filetype
    if &filetype =~ 'javascript'
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP
    else
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=@htmlJavaScript
    endif
    

    View Slide

98. Template String
    

    View Slide

99. Template String
    `string text`
    `string text line 1
    string text line 2`
    `string text ${expression} string text`
    

    View Slide

100. syntax region javascriptString
     start=/\z(["']\)/
     skip=/\\\\\|\\\z1\|\\\n/
     end=/\z1\|$/
     

     View Slide

101. syntax region javascriptTemplate
     start=/`/
     skip=/\\\\\|\\`\|\n/
     end=/`\/
     

     View Slide

102. `string text`
     `string text line 1
     string text line 2`
     `string text ${expression} string text`
     

     View Slide

103. The Issue
     

     View Slide

104. `string text ${"another string, and accent `"} string text`
     

     View Slide

105. • The string ends when match an accent `
     
     • Try to recognize inner content
     

     View Slide

106. syntax region javascriptTemplate
     start=/`/
     skip=/\\\\\|\\`\|\n/
     end=/`\|$/
     contains=javascriptTemplateSubstitution
     syntax region javascriptTemplateSubstitution contained
     matchgroup=javascriptTemplateSB
     start=/\${/
     end=/}/
     

     View Slide

107. `string text ${"another string, and accent `"} string text`
     

     View Slide

108. syntax region javascriptTemplate
     start=/`/
     skip=/\\\\\|\\`\|\n/
     end=/`\|$/
     contains=javascriptTemplateSubstitution
     syntax region javascriptTemplateSubstitution contained
     matchgroup=javascriptTemplateSB
     start=/\${/
     end=/}/
     contains=@javascriptExpression
     

     View Slide

109. `string text ${"another string, and accent `"} string text`
     

     View Slide

110. Arrow Function
     

     View Slide

111. Main Target
     • Highlight arrow function and its parameters
     

     View Slide

112. var foo = function (arg1) {};
     

     View Slide

113. var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     

     View Slide

114. var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     var fooParen = (arg1);
     

     View Slide

115. var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     var fooParen = (arg1);
     Can’t lookup following token
     When cursor is here
     

     View Slide

116. =>
     syntax match javascriptArrow
     /=>/
     nextgroup=javascriptBlock
     skipwhite
     

     View Slide

117. =>
     var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     var fooParen = (arg1);
     

     View Slide

118. Parameter?
     

     View Slide

119. Problem
     • Don’t know what it is when meet `(`
     
     • Function parameters or
     
     • Expression in parentheses
     

     View Slide

120. Before Default Parameter
     • Only identifier and comma inside function argument
     parens
     
     • Identifier name possible characters: [a-zA-Z_0-9$]
     
     • Not contain right parentheses
     

     View Slide

121. Workaround Solution
     • Use match to match an arrow function
     
     • Put this match rule after parentheses region

     (higher priority)
     

     View Slide

122. syntax region javascriptParens ...
     syntax match javascriptArrowFuncDef
     /(\_[^)]*)\_s*=>/
     contains=javascriptArrowFuncArg,
     javascriptComma
     nextgroup=javascriptBlock
     skipwhite
     skipempty
     

     View Slide

123. syntax region javascriptParens ...
     syntax match javascriptArrowFuncDef
     /(\_[^)]*)\_s*=>/
     contains=javascriptArrowFuncArg,
     javascriptComma
     nextgroup=javascriptBlock
     skipwhite
     skipempty
     Match pair of paren
     

     View Slide

124. syntax match javascriptArrowFunc /=>/
     syntax region javascriptArrowFuncArg contained
     matchgroup=javascriptParens
     start=/(/
     end=/)/
     contains=@javascriptFuncArgElements
     nextgroup=javascriptArrowFunc
     skipwhite
     skipempty
     

     View Slide

125. syntax cluster javascriptFuncArgElements
     contains=javascriptFuncKeyword,
     javascriptComma,
     javascriptDefaultAssign,
     @javascriptComments,
     javascriptFuncArgArray,
     javascriptFuncArgObject,
     javascriptSpreadOp
     

     View Slide

126. var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     var fooParen = (arg1);
     

     View Slide

127. Default Parameter
     

     View Slide

128. function multiply(a, b) {
     var b = typeof b !== 'undefined' ? b : 1;
     return a*b;
     }
     

     View Slide

129. syntax keyword javascriptFuncKeyword function
     nextgroup=javascriptFuncName skipwhite
     syntax match javascriptFuncName contained /[a-zA-Z_$]\k*/
     nextgroup=javascriptFuncArg skipwhite
     syntax match javascriptFuncArg contained /(\_[^)]*)/
     nextgroup=javascriptBlock skipwhite
     

     View Slide

130. function multiply(a, b) {
     var b = typeof b !== 'undefined' ? b : 1;
     return a*b;
     }
     

     View Slide

131. function multiply(a, b) {
     var b = typeof b !== 'undefined' ? b : 1;
     return a*b;
     }
     FuncKeyword
     FuncName
     FuncArg
     

     View Slide

132. function multiply(a, b = 1) {
     return a*b;
     }
     

     View Slide

133. function multiply(a, b = "(1+1)") {
     return a*b;
     }
     ???
     

     View Slide

134. function multiply(a, b = "(1+1)") {
     return a*b;
     }
     Match ending )
     Not match block
     

     View Slide

135. • Not able to really know which paren is ending one
     
     • Have to well know the content of parens
     
     • Use region
     

     View Slide

136. syntax region javascriptFuncArg contained
     matchgroup=javascriptParens start=/(/ end=/)/
     contains=@javascriptFuncArgElements
     nextgroup=javascriptBlock
     skipwhite
     syntax cluster javascriptFuncArgElements
     contains=...
     javascriptComma,
     javascriptDefaultAssign,
     @javascriptComments...
     syntax match javascriptDefaultAssign contained
     /=/
     nextgroup=@javascriptExpression
     skipwhite
     

     View Slide

137. syntax region javascriptFuncArg contained
     matchgroup=javascriptParens start=/(/ end=/)/
     contains=@javascriptFuncArgElements
     nextgroup=javascriptBlock
     skipwhite
     syntax cluster javascriptFuncArgElements
     contains=...
     javascriptComma,
     javascriptDefaultAssign,
     @javascriptComments...
     syntax match javascriptDefaultAssign contained
     /=/
     nextgroup=@javascriptExpression
     skipwhite
     

     View Slide

138. syntax region javascriptFuncArg contained
     matchgroup=javascriptParens start=/(/ end=/)/
     contains=@javascriptFuncArgElements
     nextgroup=javascriptBlock
     skipwhite
     syntax cluster javascriptFuncArgElements
     contains=...
     javascriptComma,
     javascriptDefaultAssign,
     @javascriptComments...
     syntax match javascriptDefaultAssign contained
     /=/
     nextgroup=@javascriptExpression
     skipwhite
     

     View Slide

139. function multiply(a, b = "(1+1)") {
     return a*b;
     }
     Default Assign
     Expression, string
     

     View Slide

140. Arrow Function
     Default Parameter
     

     View Slide

141. var foo = function (arg1) {};
     var fooArrow = (arg1) => {};
     var fooParen = (arg1);
     

     View Slide

142. var foo = function (arg1) {};
     var fooArrow = (arg1 = 1) => {};
     var fooArrow = (arg1 = 1, arg2 = `string`) => {};
     

     View Slide

143. var foo = function (arg1) {};
     var fooArrow = (arg1 = 1) => {};
     var fooArrow = (arg1 = function () {}) => {};
     

     View Slide

144. Matching fails here
     var foo = function (arg1) {};
     var fooArrow = (arg1 = 1) => {};
     var fooArrow = (arg1 = function () {}) => {};
     

     View Slide

145. Problem
     • Expression in parentheses and arrow function
     parameter mixed up
     

     View Slide

146. var fooArrow1 = function (arg1 = () => {}) {};
     

     View Slide

147. var fooArrow1 = function (arg1 = () => {}) {};
     var fooArrow2 = function (arg1 = () => {
     return function () {};
     }) {};
     

     View Slide

148. var fooArrow1 = function (arg1 = () => {}) {};
     var fooArrow2 = function (arg1 = () => {
     return function () {};
     }) {};
     var fooArrow3 = function (arg1 = (arg2 = () => {}) => {
     return function () {};
     }) {};
     

     View Slide

149. Problem
     • Parens, arrow function parameter mixed
     
     • Arrow function use match instead of region
     

     View Slide

150. My Latest Quest
     

     View Slide

151. Still Solving
     

     View Slide

152. Patch Also Welcome
     

     View Slide

153. View Slide

154. Thank You
     

     View Slide

155. Questions?
     

     View Slide