YAJS.vim and Vim Syntax Highlight

C4ce16f549c450f4759eb37f5d5d1a63?s=47 othree
June 25, 2016

YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

C4ce16f549c450f4759eb37f5d5d1a63?s=128

othree

June 25, 2016
Tweet

Transcript

  1. 2.

    me • othree @ flickr, github, twitter… • MozTW •

    F2E at Qihoo • Speeches in COSCUP, OSDC.tw, JSDC… • https://blog.othree.net/
  2. 5.

    YAJS.vim • Yet Another JavaScript Syntax for Vim • Supports

    ECMAScript 6 syntax • Remove legacy code support • and JSDoc, Web APIs, console, …. • javascript instead of javaScript
  3. 6.
  4. 8.
  5. 9.

    Why YAJS • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011

    • People can’t wait to use new features • Transpilers: Traceur, 6to5(Babel)…
  6. 10.

    • People start to use ES6(and lots of ES.next feature)

    • JavaScript syntax for Vim doesn’t support ES6 • So I start to do it
  7. 11.

    Dev Target • Support all valid ES6 syntax • If

    possible, add some error hint • Performance is prioritize to second place
  8. 12.
  9. 13.

    Why no Push to Upstream • Lots of incompatible code

    change during dev • Can’t wait for a PR review
  10. 14.

    Status Now • Support major ES6 syntax • Still in

    development • Trying to build auto test
  11. 16.
  12. 20.

    Keyword syntax keyword lsKeyword function syntax keyword lsKeyword return highlight

    link lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
  13. 21.

    syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None group name item type
  14. 22.

    syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None
  15. 23.

    syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None target group name source group name link groups
  16. 24.

    syntax keyword lsKeyword function syntax keyword lsKeyword return highlight link

    lsKeyword Statement highlight Statement ctermfg=251 ctermbg=None color definition group name
  17. 32.
  18. 33.

    Match syntax keyword lsKeyword function syntax keyword lsKeyword return syntax

    match lsParens /(\k*)/ syntax match lsContains /contains/
  19. 41.

    contained syntax keyword lsKeyword function syntax keyword lsKeyword return syntax

    match lsParens contained /(\k*)/ syntax match lsContains /contains/
  20. 44.

    nextgroup syntax keyword lsKeyword function nextgroup=lsParens syntax keyword lsKeyword return

    syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
  21. 45.

    nextgroup • Vim will lookup groups listed in nextgroup first

    • Before normal keyword, match, region • Not limited by contained
  22. 46.

    var foo = function(arg) { return 1; }; return 1;

    (arg); Ending of keyword function
  23. 47.

    var foo = function(arg) { return 1; }; return 1;

    (arg); Looking for groups in nextgroup
  24. 49.
  25. 51.

    skipwhite syntax keyword lsKeyword function nextgroup=lsParens skipwhite syntax keyword lsKeyword

    return syntax match lsParens contained /(\k*)/ syntax match lsContains /contains/
  26. 52.

    Vim Keywords syntax keyword lsKeyword function syntax keyword lsKeyword return

    syntax match lsParens /(\k*)/ syntax keyword lsContains contains
  27. 53.
  28. 54.

    syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match

    lsParens /(\k*)/ syntax match lsContains /contains/
  29. 55.

    syntax keyword lsKeyword function syntax keyword lsKeyword return syntax match

    lsParens /(\k*)/ syntax match lsContains /contains/ syntax keyword lsKeyword con
  30. 57.
  31. 58.

    back to the issue var foo = function () {

    return 1; }; return 1; match keyword: return
  32. 59.

    Region syntax keyword lsKeyword function syntax keyword lsReturn return syntax

    match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/
  33. 60.

    syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax

    match lsParens /(\k*)/ syntax region lsBlock start=/{/ end=/}/ contains=lsReturn
  34. 63.

    Nesting Region syntax keyword lsKeyword function syntax keyword lsReturn contained

    return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn
  35. 65.

    syntax keyword lsKeyword function syntax keyword lsReturn contained return syntax

    match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=lsBlock,lsKeyword,lsReturn syntax keyword lsDeclare var let
  36. 67.

    TOP region syntax keyword lsKeyword function syntax keyword lsReturn contained

    return syntax match lsParens /(\k*)/ syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/ contains=TOP syntax keyword lsDeclare var let
  37. 69.
  38. 70.

    • Not real lexical analysis, have limitation • For performance

    • Best situation: • every token are independent • no nesting
  39. 71.
  40. 73.

    Quest? • Vim syntax highlight system has limitation • To

    support all ES2015 syntax is like solve quests
  41. 74.
  42. 75.
  43. 76.
  44. 77.
  45. 78.

    • Have everything outside of the block • TOP region

    groups • Function block region groups
  46. 80.

    • Have almost everything in TOP region • TOP region

    groups • And `return` • But no import/export related stuff
  47. 83.

    • Have all property name syntax and value(RHS) • Property

    name syntaxes • identifier, number, string • computed • And then RHS expression
  48. 84.

    Issues • These blocks have very different contents • I

    have to know what this region is at position `{`
  49. 85.

    Normal don’t need to deal Function after `function` keyword, method

    syntax Class always after `class` keyword Object in RHS expression
  50. 86.
  51. 89.

    function block • Have almost everything in TOP region •

    TOP region groups • And `return` • But no import/export related stuff
  52. 91.

    TOP • A special group cluster • Contains every group

    without `contained` • I called these stuff belongs to TOP region
  53. 92.

    but

  54. 94.

    in HTML Syntax syn include @htmlJavaScript syntax/javascript.vim unlet b:current_syntax syn

    region javaScript start=+<script\_[^>]*>+ keepend end=+</script\_[^>]*>+me=s-1 contains=@htmlJavaScript
  55. 95.

    in Jinja Syntax " Pull in the HTML syntax. if

    g:jinja_syntax_html if version < 600 so <sfile>:p:h/html.vim else runtime! syntax/html.vim unlet b:current_syntax endif endif
  56. 97.

    Detect Filetype if &filetype =~ 'javascript' syntax region javascriptBlock matchgroup=javascriptBraces

    start=/{/ end=/}/ contains=TOP else syntax region javascriptBlock matchgroup=javascriptBraces start=/{/ end=/}/ contains=@htmlJavaScript endif
  57. 99.

    Template String `string text` `string text line 1 string text

    line 2` `string text ${expression} string text`
  58. 102.

    `string text` `string text line 1 string text line 2`

    `string text ${expression} string text`
  59. 103.
  60. 105.

    • The string ends when match an accent ` •

    Try to recognize inner content
  61. 114.
  62. 115.

    var foo = function (arg1) {}; var fooArrow = (arg1)

    => {}; var fooParen = (arg1); Can’t lookup following token When cursor is here
  63. 117.

    => var foo = function (arg1) {}; var fooArrow =

    (arg1) => {}; var fooParen = (arg1);
  64. 118.
  65. 119.

    Problem • Don’t know what it is when meet `(`

    • Function parameters or • Expression in parentheses
  66. 120.

    Before Default Parameter • Only identifier and comma inside function

    argument parens • Identifier name possible characters: [a-zA-Z_0-9$] • Not contain right parentheses
  67. 121.

    Workaround Solution • Use match to match an arrow function

    • Put this match rule after parentheses region
 (higher priority)
  68. 124.

    syntax match javascriptArrowFunc /=>/ syntax region javascriptArrowFuncArg contained matchgroup=javascriptParens start=/(/

    end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptArrowFunc skipwhite skipempty
  69. 126.
  70. 128.

    function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; }
  71. 129.

    syntax keyword javascriptFuncKeyword function nextgroup=javascriptFuncName skipwhite syntax match javascriptFuncName contained

    /[a-zA-Z_$]\k*/ nextgroup=javascriptFuncArg skipwhite syntax match javascriptFuncArg contained /(\_[^)]*)/ nextgroup=javascriptBlock skipwhite
  72. 130.

    function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; }
  73. 131.

    function multiply(a, b) { var b = typeof b !==

    'undefined' ? b : 1; return a*b; } FuncKeyword FuncName FuncArg
  74. 135.

    • Not able to really know which paren is ending

    one • Have to well know the content of parens • Use region
  75. 136.

    syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  76. 137.

    syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  77. 138.

    syntax region javascriptFuncArg contained matchgroup=javascriptParens start=/(/ end=/)/ contains=@javascriptFuncArgElements nextgroup=javascriptBlock skipwhite

    syntax cluster javascriptFuncArgElements contains=... javascriptComma, javascriptDefaultAssign, @javascriptComments... syntax match javascriptDefaultAssign contained /=/ nextgroup=@javascriptExpression skipwhite
  78. 141.
  79. 142.

    var foo = function (arg1) {}; var fooArrow = (arg1

    = 1) => {}; var fooArrow = (arg1 = 1, arg2 = `string`) => {};
  80. 143.

    var foo = function (arg1) {}; var fooArrow = (arg1

    = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
  81. 144.

    Matching fails here var foo = function (arg1) {}; var

    fooArrow = (arg1 = 1) => {}; var fooArrow = (arg1 = function () {}) => {};
  82. 147.

    var fooArrow1 = function (arg1 = () => {}) {};

    var fooArrow2 = function (arg1 = () => { return function () {}; }) {};
  83. 148.

    var fooArrow1 = function (arg1 = () => {}) {};

    var fooArrow2 = function (arg1 = () => { return function () {}; }) {}; var fooArrow3 = function (arg1 = (arg2 = () => {}) => { return function () {}; }) {};
  84. 153.
  85. 154.
  86. 155.