Upgrade to Pro — share decks privately, control downloads, hide ads and more …

YAJS.vim and Vim Syntax Highlight

othree
June 25, 2016

YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

othree

June 25, 2016
Tweet

More Decks by othree

Other Decks in Programming

Transcript

  1. YAJS.vim and Vim Syntax
    Highlight
    othree @ HKOSCon

    View full-size slide

  2. me
    • othree @ flickr, github, twitter…

    • MozTW

    • F2E at Qihoo

    • Speeches in COSCUP, OSDC.tw, JSDC…

    • https://blog.othree.net/

    View full-size slide

  3. outline
    • Why YAJS.vim

    • Vim Syntax Highlight

    • The Quests

    View full-size slide

  4. Why YAJS.vim

    View full-size slide

  5. YAJS.vim
    • Yet Another JavaScript Syntax for Vim

    • Supports ECMAScript 6 syntax

    • Remove legacy code support

    • and JSDoc, Web APIs, console, ….

    • javascript instead of javaScript

    View full-size slide

  6. Why YAJS
    • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011

    • People can’t wait to use new features

    • Transpilers: Traceur, 6to5(Babel)…

    View full-size slide

  7. • People start to use ES6(and lots of ES.next feature)

    • JavaScript syntax for Vim doesn’t support ES6

    • So I start to do it

    View full-size slide

  8. Dev Target
    • Support all valid ES6 syntax

    • If possible, add some error hint

    • Performance is prioritize to second place

    View full-size slide

  9. Why no Push to Upstream
    • Lots of incompatible code change during dev

    • Can’t wait for a PR review

    View full-size slide

  10. Status Now
    • Support major ES6 syntax

    • Still in development

    • Trying to build auto test

    View full-size slide

  11. • https://github.com/othree/yajs.vim

    View full-size slide

  12. Vim Syntax Highlight

    View full-size slide

  13. • Pretend there is a new filetype called ‘ls’

    • Not livescript

    View full-size slide

  14. Syntax Items
    • Keyword

    • Match

    • Region

    View full-size slide

  15. Keyword
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None

    View full-size slide

  16. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    group name
    item type

    View full-size slide

  17. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None

    View full-size slide

  18. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    target group name
    source group name
    link groups

    View full-size slide

  19. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    color definition
    group name

    View full-size slide

  20. var foo = function () {
    };

    View full-size slide

  21. var foo = function () {
    };
    TOP region

    View full-size slide

  22. • At TOP region, looking for keywords

    • function

    • return

    View full-size slide

  23. var foo = function () {
    };
    match keyword: function

    View full-size slide

  24. var foo = function () {
    };
    highlight it

    View full-size slide

  25. var foo = function () {
    return 1;
    };
    keep looking

    View full-size slide

  26. var foo = function () {
    return 1;
    };
    match keyword: return

    View full-size slide

  27. var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return

    View full-size slide

  28. Match
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/

    View full-size slide

  29. • Match uses Vim’s regular expression

    • http://vimdoc.sourceforge.net/htmldoc/pattern.html

    View full-size slide

  30. var foo = function (arg) {
    return 1;
    };
    return 1;
    TOP region

    View full-size slide

  31. • Looking for keywords and match pattern

    • Keyword first, match second

    View full-size slide

  32. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match keyword: function

    View full-size slide

  33. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match pattern

    View full-size slide

  34. var foo = function(arg) {
    return 1;
    };
    return 1;

    View full-size slide

  35. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);

    View full-size slide

  36. contained
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View full-size slide

  37. contained
    • Vim not find contained group at TOP region

    View full-size slide

  38. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);

    View full-size slide

  39. nextgroup
    syntax keyword lsKeyword function nextgroup=lsParens
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View full-size slide

  40. nextgroup
    • Vim will lookup groups listed in nextgroup first

    • Before normal keyword, match, region

    • Not limited by contained

    View full-size slide

  41. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Ending of keyword function

    View full-size slide

  42. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Looking for groups in nextgroup

    View full-size slide

  43. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Match pattern

    View full-size slide

  44. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Not match any pattern

    View full-size slide

  45. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);
    space

    View full-size slide

  46. skipwhite
    syntax keyword lsKeyword function nextgroup=lsParens skipwhite
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View full-size slide

  47. Vim Keywords
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax keyword lsContains contains

    View full-size slide

  48. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/

    View full-size slide

  49. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/
    syntax keyword lsKeyword con

    View full-size slide

  50. contains
    Match keyword first

    View full-size slide

  51. back to the issue
    var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return

    View full-size slide

  52. Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/

    View full-size slide

  53. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/ contains=lsReturn

    View full-size slide

  54. var foo = function () {
    return 1;
    };
    return 1;

    View full-size slide

  55. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View full-size slide

  56. Nesting Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn

    View full-size slide

  57. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View full-size slide

  58. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn
    syntax keyword lsDeclare var let

    View full-size slide

  59. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View full-size slide

  60. TOP region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=TOP
    syntax keyword lsDeclare var let

    View full-size slide

  61. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View full-size slide

  62. • Not real lexical analysis, have limitation

    • For performance

    • Best situation:

    • every token are independent

    • no nesting

    View full-size slide

  63. Quest?
    • Vim syntax highlight system has limitation

    • To support all ES2015 syntax is like solve quests

    View full-size slide

  64. Quests
    • class/object block

    • function block

    • template string

    • arrow function

    • default parameter

    View full-size slide

  65. blocks
    • class/object/function/normal block

    • All starts with `{` and ends with `}`

    • But have different content

    View full-size slide

  66. normal block
    for (i = 0; i < arr.length; i++) {
    console.log(arr[i]);
    }

    View full-size slide

  67. • Have everything outside of the block

    • TOP region groups

    • Function block region groups

    View full-size slide

  68. function block
    var foo = function () {
    var bar = 1;
    return 1;
    };

    View full-size slide

  69. • Have almost everything in TOP region

    • TOP region groups

    • And `return`

    • But no import/export related stuff

    View full-size slide

  70. object
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };

    View full-size slide

  71. class
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };

    View full-size slide

  72. • Have all property name syntax and value(RHS)

    • Property name syntaxes

    • identifier, number, string

    • computed

    • And then RHS expression

    View full-size slide

  73. Issues
    • These blocks have very different contents

    • I have to know what this region is at position `{`

    View full-size slide

  74. Normal don’t need to deal
    Function after `function` keyword, method syntax
    Class always after `class` keyword
    Object in RHS expression

    View full-size slide

  75. RHS expression
    • Right hand side of an operator

    • Inside a pair of paren `(`, `)`

    View full-size slide

  76. syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/
    nextgroup=@javascriptExpression
    syntax region javascriptParenExp
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptExpression

    View full-size slide

  77. function block

    View full-size slide

  78. function block
    • Have almost everything in TOP region

    • TOP region groups

    • And `return`

    • But no import/export related stuff

    View full-size slide

  79. syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP

    View full-size slide

  80. TOP
    • A special group cluster

    • Contains every group without `contained`

    • I called these stuff belongs to TOP region

    View full-size slide

  81. JS in HTML


    <br/>function foo() {<br/>return 1;<br/>}<br/>


    View full-size slide

  82. in HTML Syntax
    syn include @htmlJavaScript syntax/javascript.vim
    unlet b:current_syntax
    syn region javaScript
    start=+]*>+ keepend<br/>end=+]*>+me=s-1
    contains=@htmlJavaScript

    View full-size slide

  83. in Jinja Syntax
    " Pull in the HTML syntax.
    if g:jinja_syntax_html
    if version < 600
    so :p:h/html.vim
    else
    runtime! syntax/html.vim
    unlet b:current_syntax
    endif
    endif

    View full-size slide

  84. TOP is HTML’s TOP

    View full-size slide

  85. Detect Filetype
    if &filetype =~ 'javascript'
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP
    else
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=@htmlJavaScript
    endif

    View full-size slide

  86. Template String

    View full-size slide

  87. Template String
    `string text`
    `string text line 1
    string text line 2`
    `string text ${expression} string text`

    View full-size slide

  88. syntax region javascriptString
    start=/\z(["']\)/
    skip=/\\\\\|\\\z1\|\\\n/
    end=/\z1\|$/

    View full-size slide

  89. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\/

    View full-size slide

  90. `string text`
    `string text line 1
    string text line 2`
    `string text ${expression} string text`

    View full-size slide

  91. `string text ${"another string, and accent `"} string text`

    View full-size slide

  92. • The string ends when match an accent `

    • Try to recognize inner content

    View full-size slide

  93. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\|$/
    contains=javascriptTemplateSubstitution
    syntax region javascriptTemplateSubstitution contained
    matchgroup=javascriptTemplateSB
    start=/\${/
    end=/}/

    View full-size slide

  94. `string text ${"another string, and accent `"} string text`

    View full-size slide

  95. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\|$/
    contains=javascriptTemplateSubstitution
    syntax region javascriptTemplateSubstitution contained
    matchgroup=javascriptTemplateSB
    start=/\${/
    end=/}/
    contains=@javascriptExpression

    View full-size slide

  96. `string text ${"another string, and accent `"} string text`

    View full-size slide

  97. Arrow Function

    View full-size slide

  98. Main Target
    • Highlight arrow function and its parameters

    View full-size slide

  99. var foo = function (arg1) {};

    View full-size slide

  100. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};

    View full-size slide

  101. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View full-size slide

  102. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);
    Can’t lookup following token
    When cursor is here

    View full-size slide

  103. =>
    syntax match javascriptArrow
    /=>/
    nextgroup=javascriptBlock
    skipwhite

    View full-size slide

  104. =>
    var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View full-size slide

  105. Problem
    • Don’t know what it is when meet `(`

    • Function parameters or

    • Expression in parentheses

    View full-size slide

  106. Before Default Parameter
    • Only identifier and comma inside function argument
    parens

    • Identifier name possible characters: [a-zA-Z_0-9$]

    • Not contain right parentheses

    View full-size slide

  107. Workaround Solution
    • Use match to match an arrow function

    • Put this match rule after parentheses region

    (higher priority)

    View full-size slide

  108. syntax region javascriptParens ...
    syntax match javascriptArrowFuncDef
    /(\_[^)]*)\_s*=>/
    contains=javascriptArrowFuncArg,
    javascriptComma
    nextgroup=javascriptBlock
    skipwhite
    skipempty

    View full-size slide

  109. syntax region javascriptParens ...
    syntax match javascriptArrowFuncDef
    /(\_[^)]*)\_s*=>/
    contains=javascriptArrowFuncArg,
    javascriptComma
    nextgroup=javascriptBlock
    skipwhite
    skipempty
    Match pair of paren

    View full-size slide

  110. syntax match javascriptArrowFunc /=>/
    syntax region javascriptArrowFuncArg contained
    matchgroup=javascriptParens
    start=/(/
    end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptArrowFunc
    skipwhite
    skipempty

    View full-size slide

  111. syntax cluster javascriptFuncArgElements
    contains=javascriptFuncKeyword,
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments,
    javascriptFuncArgArray,
    javascriptFuncArgObject,
    javascriptSpreadOp

    View full-size slide

  112. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View full-size slide

  113. Default Parameter

    View full-size slide

  114. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }

    View full-size slide

  115. syntax keyword javascriptFuncKeyword function
    nextgroup=javascriptFuncName skipwhite
    syntax match javascriptFuncName contained /[a-zA-Z_$]\k*/
    nextgroup=javascriptFuncArg skipwhite
    syntax match javascriptFuncArg contained /(\_[^)]*)/
    nextgroup=javascriptBlock skipwhite

    View full-size slide

  116. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }

    View full-size slide

  117. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }
    FuncKeyword
    FuncName
    FuncArg

    View full-size slide

  118. function multiply(a, b = 1) {
    return a*b;
    }

    View full-size slide

  119. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    ???

    View full-size slide

  120. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    Match ending )
    Not match block

    View full-size slide

  121. • Not able to really know which paren is ending one

    • Have to well know the content of parens

    • Use region

    View full-size slide

  122. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View full-size slide

  123. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View full-size slide

  124. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View full-size slide

  125. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    Default Assign
    Expression, string

    View full-size slide

  126. Arrow Function
    Default Parameter

    View full-size slide

  127. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View full-size slide

  128. var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = 1, arg2 = `string`) => {};

    View full-size slide

  129. var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = function () {}) => {};

    View full-size slide

  130. Matching fails here
    var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = function () {}) => {};

    View full-size slide

  131. Problem
    • Expression in parentheses and arrow function
    parameter mixed up

    View full-size slide

  132. var fooArrow1 = function (arg1 = () => {}) {};

    View full-size slide

  133. var fooArrow1 = function (arg1 = () => {}) {};
    var fooArrow2 = function (arg1 = () => {
    return function () {};
    }) {};

    View full-size slide

  134. var fooArrow1 = function (arg1 = () => {}) {};
    var fooArrow2 = function (arg1 = () => {
    return function () {};
    }) {};
    var fooArrow3 = function (arg1 = (arg2 = () => {}) => {
    return function () {};
    }) {};

    View full-size slide

  135. Problem
    • Parens, arrow function parameter mixed

    • Arrow function use match instead of region

    View full-size slide

  136. My Latest Quest

    View full-size slide

  137. Still Solving

    View full-size slide

  138. Patch Also Welcome

    View full-size slide