YAJS.vim and Vim Syntax Highlight

C4ce16f549c450f4759eb37f5d5d1a63?s=47 othree
June 25, 2016

YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

C4ce16f549c450f4759eb37f5d5d1a63?s=128

othree

June 25, 2016
Tweet

Transcript

  1. YAJS.vim and Vim Syntax Highlight othree @ HKOSCon

  2. me • othree @ flickr, github, twitter… • MozTW •

    F2E at Qihoo • Speeches in COSCUP, OSDC.tw, JSDC… • https://blog.othree.net/
  3. outline • Why YAJS.vim • Vim Syntax Highlight • The

    Quests
  4. Why YAJS.vim

  5. YAJS.vim • Yet Another JavaScript Syntax for Vim • Supports

    ECMAScript 6 syntax • Remove legacy code support • and JSDoc, Web APIs, console, …. • javascript instead of javaScript
  6. default

  7. YAJS.vim

  8. None
  9. Why YAJS • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011

    • People can’t wait to use new features • Transpilers: Traceur, 6to5(Babel)…
  10. • People start to use ES6(and lots of ES.next feature)

    • JavaScript syntax for Vim doesn’t support ES6 • So I start to do it
  11. Dev Target • Support all valid ES6 syntax • If

    possible, add some error hint • Performance is prioritize to second place
  12. None
  13. Why no Push to Upstream • Lots of incompatible code

    change during dev • Can’t wait for a PR review
  14. Status Now • Support major ES6 syntax • Still in

    development • Trying to build auto test
  15. • https://github.com/othree/yajs.vim

  16. None
  17. Vim Syntax Highlight

  18. • Pretend there is a new filetype called ‘ls’ •

    Not livescript
  19. Syntax Items • Keyword • Match • Region

  20. Keyword syntax keyword lsKeyword function syntax keyword lsKeyword return highlight

    link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
  21. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None group name item type
  22. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
  23. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None target group name source group name link groups
  24. syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None color definition group name
  25. var foo = function () { };

  26. var foo = function () { }; TOP region

  27. • At TOP region, looking for keywords • function •

    return
  28. var foo = function () { }; match keyword: function

  29. var foo = function () { }; highlight it

  30. var foo = function () { return 1; }; keep

    looking
  31. var foo = function () { return 1; }; match

    keyword: return
  32. var foo = function () { return 1; }; return

    1; match keyword: return
  33. Match syntax keyword lsKeyword function syntax keyword lsKeyword return syntax

    match lsParens /(\k*)/ syntax match lsContains /contains/
  34. • Match uses Vim’s regular expression • http://vimdoc.sourceforge.net/htmldoc/pattern.html

  35. var foo = function (arg) { return 1; }; return

    1; TOP region
  36. • Looking for keywords and match pattern • Keyword first,

    match second
  37. var foo = function(arg) { return 1; }; return 1;

    Match keyword: function
  38. var foo = function(arg) { return 1; }; return 1;

    Match pattern
  39. var foo = function(arg) { return 1; }; return 1;

  40. var foo = function(arg) { return 1; }; return 1;

    (arg);
  41. contained syntax keyword lsKeyword function syntax keyword lsKeyword return syntax

    match lsParens contained /(\k*)/ syntax match lsContains /contains/
  42. contained • Vim not find contained group at TOP region

  43. var foo = function (arg) { return 1; }; return

    1; (arg);
  44. nextgroup syntax keyword lsKeyword function nextgroup=lsParens syntax keyword lsKeyword return

    syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
  45. nextgroup • Vim will lookup groups listed in nextgroup first

    • Before normal keyword, match, region • Not limited by contained
  46. var foo = function(arg) { return 1; }; return 1;

    (arg); Ending of keyword function
  47. var foo = function(arg) { return 1; }; return 1;

    (arg); Looking for groups in nextgroup
  48. var foo = function(arg) { return 1; }; return 1;

    (arg); Match pattern
  49. var foo = function(arg) { return 1; }; return 1;

    (arg); Not match any pattern
  50. var foo = function (arg) { return 1; }; return

    1; (arg); space
  51. skipwhite syntax keyword lsKeyword function nextgroup=lsParens skipwhite syntax keyword lsKeyword

    return syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
  52. Vim Keywords syntax keyword lsKeyword function syntax keyword lsKeyword return

    syntax match lsParens /(\k*)/ syntax keyword lsContains contains
  53. None
  54. syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match

    lsParens /(\k*)/ syntax match lsContains /contains/
  55. syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match

    lsParens /(\k*)/ syntax match lsContains /contains/ syntax keyword lsKeyword con
  56. contains Match keyword first

  57. None
  58. back to the issue var foo = function () {

    return 1; }; return 1; match keyword: return
  59. Region syntax keyword lsKeyword function syntax keyword lsReturn return syntax

    match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/
  60. syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax

    match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/ contains=lsReturn
  61. var foo = function () { return 1; }; return

    1;
  62. var foo = function () { var bar = function

    () { return 1; } };
  63. Nesting Region syntax keyword lsKeyword function syntax keyword lsReturn contained

    return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn
  64. var foo = function () { var bar = function

    () { return 1; } };
  65. syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax

    match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn syntax keyword lsDeclare var let
  66. var foo = function () { var bar = function

    () { return 1; } };
  67. TOP region syntax keyword lsKeyword function syntax keyword lsReturn contained

    return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=TOP syntax keyword lsDeclare var let
  68. var foo = function () { var bar = function

    () { return 1; } };
  69. So…

  70. • Not real lexical analysis, have limitation • For performance

    • Best situation: • every token are independent • no nesting
  71. None
  72. The Quests

  73. Quest? • Vim syntax highlight system has limitation • To

    support all ES2015 syntax is like solve quests
  74. Quests • class/object block • function block • template string

    • arrow function • default parameter
  75. blocks

  76. blocks • class/object/function/normal block • All starts with `{` and

    ends with `}` • But have different content
  77. normal block for (i = 0; i < arr.length; i++)

    { console.log(arr[i]); }
  78. • Have everything outside of the block • TOP region

    groups • Function block region groups
  79. function block var foo = function () { var bar

    = 1; return 1; };
  80. • Have almost everything in TOP region • TOP region

    groups • And `return` • But no import/export related stuff
  81. object var foo = { bar: 1, buz: function ()

    { return 1; } };
  82. class var foo = { bar: 1, buz: function ()

    { return 1; } };
  83. • Have all property name syntax and value(RHS) • Property

    name syntaxes • identifier, number, string • computed • And then RHS expression
  84. Issues • These blocks have very different contents • I

    have to know what this region is at position `{`
  85. Normal don’t need to deal Function after `function` keyword, method

    syntax Class always after `class` keyword Object in RHS expression
  86. RHS expression • Right hand side of an operator •

    Inside a pair of paren `(`, `)`
  87. syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/ nextgroup=@javascriptExpression syntax region javascriptParenExp matchgroup=javascriptParens start=/(/

    end=/)/ contains=@javascriptExpression
  88. function block

  89. function block • Have almost everything in TOP region •

    TOP region groups • And `return` • But no import/export related stuff
  90. syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=TOP

  91. TOP • A special group cluster • Contains every group

    without `contained` • I called these stuff belongs to TOP region
  92. but

  93. JS in HTML <html> <head> <script> function foo() { return

    1; } </script> </head> </html>
  94. in HTML Syntax syn include @htmlJavaScript syntax/javascript.vim unlet b:current_syntax syn

    region javaScript start=+<script\_[^>]*>+ keepend end=+</script\_[^>]*>+me=s-1 contains=@htmlJavaScript
  95. in Jinja Syntax " Pull in the HTML syntax. if

    g:jinja_syntax_html if version < 600 so <sfile>:p:h/html.vim else runtime! syntax/html.vim unlet b:current_syntax endif endif
  96. TOP is HTML’s TOP

  97. Detect Filetype if &filetype =~ 'javascript' syntax region javascriptBlock matchgroup=javascriptBraces

    start=/{/ end=/}/ contains=TOP else syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=@htmlJavaScript endif
  98. Template String

  99. Template String `string text` `string text line 1 string text

    line 2` `string text ${expression} string text`
  100. syntax region javascriptString start=/\z(["']\)/ skip=/\\\\\|\\\z1\|\\\n/ end=/\z1\|$/

  101. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\/

  102. `string text` `string text line 1 string text line 2`

    `string text ${expression} string text`
  103. The Issue

  104. `string text ${"another string, and accent `"} string text`

  105. • The string ends when match an accent ` •

    Try to recognize inner content
  106. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\|$/ contains=javascriptTemplateSubstitution syntax region javascriptTemplateSubstitution

    contained matchgroup=javascriptTemplateSB start=/\${/ end=/}/
  107. `string text ${"another string, and accent `"} string text`

  108. syntax region javascriptTemplate start=/`/ skip=/\\\\\|\\`\|\n/ end=/`\|$/ contains=javascriptTemplateSubstitution syntax region javascriptTemplateSubstitution

    contained matchgroup=javascriptTemplateSB start=/\${/ end=/}/ contains=@javascriptExpression
  109. `string text ${"another string, and accent `"} string text`

  110. Arrow Function

  111. Main Target • Highlight arrow function and its parameters

  112. var foo = function (arg1) {};

  113. var foo = function (arg1) {}; var fooArrow = (arg1)

    => {};
  114. var foo = function (arg1) {}; var fooArrow = (arg1)

    => {}; var fooParen = (arg1);
  115. var foo = function (arg1) {}; var fooArrow = (arg1)

    => {}; var fooParen = (arg1); Can’t lookup following token When cursor is here
  116. => syntax match javascriptArrow /=>/ nextgroup=javascriptBlock skipwhite

  117. => var foo = function (arg1) {}; var fooArrow =

    (arg1) => {}; var fooParen = (arg1);
  118. Parameter?

  119. Problem • Don’t know what it is when meet `(`

    • Function parameters or • Expression in parentheses
  120. Before Default Parameter • Only identifier and comma inside function

    argument parens • Identifier name possible characters: [a-zA-Z_0-9$] • Not contain right parentheses
  121. Workaround Solution • Use match to match an arrow function

    • Put this match rule after parentheses region
 (higher priority)
  122. syntax region javascriptParens ... syntax match javascriptArrowFuncDef /(\_[^)]*)\_s*=>/ contains=javascriptArrowFuncArg, javascriptComma

    nextgroup=javascriptBlock skipwhite skipempty
  123. syntax region javascriptParens ... syntax match javascriptArrowFuncDef /(\_[^)]*)\_s*=>/ contains=javascriptArrowFuncArg, javascriptComma

    nextgroup=javascriptBlock skipwhite skipempty Match pair of paren
  124. syntax match javascriptArrowFunc /=>/ syntax region javascriptArrowFuncArg contained matchgroup=javascriptParens start=/(/

    end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptArrowFunc skipwhite skipempty
  125. syntax cluster javascriptFuncArgElements contains=javascriptFuncKeyword, javascriptComma, javascriptDefaultAssign, @javascriptComments, javascriptFuncArgArray, javascriptFuncArgObject, javascriptSpreadOp

  126. var foo = function (arg1) {}; var fooArrow = (arg1)

    => {}; var fooParen = (arg1);
  127. Default Parameter

  128. function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; }
  129. syntax keyword javascriptFuncKeyword function nextgroup=javascriptFuncName skipwhite syntax match javascriptFuncName contained

    /[a-zA-Z_$]\k*/ nextgroup=javascriptFuncArg skipwhite syntax match javascriptFuncArg contained /(\_[^)]*)/ nextgroup=javascriptBlock skipwhite
  130. function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; }
  131. function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; } FuncKeyword FuncName FuncArg
  132. function multiply(a, b = 1) { return a*b; }

  133. function multiply(a, b = "(1+1)") { return a*b; } ???

  134. function multiply(a, b = "(1+1)") { return a*b; } Match

    ending ) Not match block
  135. • Not able to really know which paren is ending

    one • Have to well know the content of parens • Use region
  136. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  137. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  138. syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  139. function multiply(a, b = "(1+1)") { return a*b; } Default

    Assign Expression, string
  140. Arrow Function Default Parameter

  141. var foo = function (arg1) {}; var fooArrow = (arg1)

    => {}; var fooParen = (arg1);
  142. var foo = function (arg1) {}; var fooArrow = (arg1

    = 1) => {}; var fooArrow = (arg1 = 1, arg2 = `string`) => {};
  143. var foo = function (arg1) {}; var fooArrow = (arg1

    = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
  144. Matching fails here var foo = function (arg1) {}; var

    fooArrow = (arg1 = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
  145. Problem • Expression in parentheses and arrow function parameter mixed

    up
  146. var fooArrow1 = function (arg1 = () => {}) {};

  147. var fooArrow1 = function (arg1 = () => {}) {};

    var fooArrow2 = function (arg1 = () => { return function () {}; }) {};
  148. var fooArrow1 = function (arg1 = () => {}) {};

    var fooArrow2 = function (arg1 = () => { return function () {}; }) {}; var fooArrow3 = function (arg1 = (arg2 = () => {}) => { return function () {}; }) {};
  149. Problem • Parens, arrow function parameter mixed • Arrow function

    use match instead of region
  150. My Latest Quest

  151. Still Solving

  152. Patch Also Welcome

  153. None
  154. Thank You

  155. Questions?