Upgrade to Pro — share decks privately, control downloads, hide ads and more …

YAJS.vim and Vim Syntax Highlight

othree
June 25, 2016

YAJS.vim and Vim Syntax Highlight

JavaScript is very important for the world wide web universe. But JavaScript support on Vim is not update very often. So I keep contribute to the Vim community. One of my contribution is yajs.vim(Yet Another JavaScript Syntax). One of the most powerful JavaScript syntax highlight plug-in for Vim now. It supports almost every new syntax from ECMAScript 2015, JSDoc, lots of new Web API and fixed several old issues. This talk will first explain Vim syntax system. The design, concept and limitation of it. Then will share the problems encountered during the development in this environment. And finally, how I solved these problems.

othree

June 25, 2016
Tweet

More Decks by othree

Other Decks in Programming

Transcript

  1. YAJS.vim and Vim Syntax
    Highlight
    othree @ HKOSCon

    View Slide

  2. me
    • othree @ flickr, github, twitter…

    • MozTW

    • F2E at Qihoo

    • Speeches in COSCUP, OSDC.tw, JSDC…

    • https://blog.othree.net/

    View Slide

  3. outline
    • Why YAJS.vim

    • Vim Syntax Highlight

    • The Quests

    View Slide

  4. Why YAJS.vim

    View Slide

  5. YAJS.vim
    • Yet Another JavaScript Syntax for Vim

    • Supports ECMAScript 6 syntax

    • Remove legacy code support

    • and JSDoc, Web APIs, console, ….

    • javascript instead of javaScript

    View Slide

  6. default

    View Slide

  7. YAJS.vim

    View Slide

  8. View Slide

  9. Why YAJS
    • ECMAScript 6(a.k.a ES6, ES2015) starts since 2011

    • People can’t wait to use new features

    • Transpilers: Traceur, 6to5(Babel)…

    View Slide

  10. • People start to use ES6(and lots of ES.next feature)

    • JavaScript syntax for Vim doesn’t support ES6

    • So I start to do it

    View Slide

  11. Dev Target
    • Support all valid ES6 syntax

    • If possible, add some error hint

    • Performance is prioritize to second place

    View Slide

  12. View Slide

  13. Why no Push to Upstream
    • Lots of incompatible code change during dev

    • Can’t wait for a PR review

    View Slide

  14. Status Now
    • Support major ES6 syntax

    • Still in development

    • Trying to build auto test

    View Slide

  15. • https://github.com/othree/yajs.vim

    View Slide

  16. View Slide

  17. Vim Syntax Highlight

    View Slide

  18. • Pretend there is a new filetype called ‘ls’

    • Not livescript

    View Slide

  19. Syntax Items
    • Keyword

    • Match

    • Region

    View Slide

  20. Keyword
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None

    View Slide

  21. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    group name
    item type

    View Slide

  22. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None

    View Slide

  23. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    target group name
    source group name
    link groups

    View Slide

  24. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    highlight link lsKeyword Statement
    highlight Statement ctermfg=251 ctermbg=None
    color definition
    group name

    View Slide

  25. var foo = function () {
    };

    View Slide

  26. var foo = function () {
    };
    TOP region

    View Slide

  27. • At TOP region, looking for keywords

    • function

    • return

    View Slide

  28. var foo = function () {
    };
    match keyword: function

    View Slide

  29. var foo = function () {
    };
    highlight it

    View Slide

  30. var foo = function () {
    return 1;
    };
    keep looking

    View Slide

  31. var foo = function () {
    return 1;
    };
    match keyword: return

    View Slide

  32. var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return

    View Slide

  33. Match
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/

    View Slide

  34. • Match uses Vim’s regular expression

    • http://vimdoc.sourceforge.net/htmldoc/pattern.html

    View Slide

  35. var foo = function (arg) {
    return 1;
    };
    return 1;
    TOP region

    View Slide

  36. • Looking for keywords and match pattern

    • Keyword first, match second

    View Slide

  37. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match keyword: function

    View Slide

  38. var foo = function(arg) {
    return 1;
    };
    return 1;
    Match pattern

    View Slide

  39. var foo = function(arg) {
    return 1;
    };
    return 1;

    View Slide

  40. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);

    View Slide

  41. contained
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View Slide

  42. contained
    • Vim not find contained group at TOP region

    View Slide

  43. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);

    View Slide

  44. nextgroup
    syntax keyword lsKeyword function nextgroup=lsParens
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View Slide

  45. nextgroup
    • Vim will lookup groups listed in nextgroup first

    • Before normal keyword, match, region

    • Not limited by contained

    View Slide

  46. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Ending of keyword function

    View Slide

  47. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Looking for groups in nextgroup

    View Slide

  48. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Match pattern

    View Slide

  49. var foo = function(arg) {
    return 1;
    };
    return 1;
    (arg);
    Not match any pattern

    View Slide

  50. var foo = function (arg) {
    return 1;
    };
    return 1;
    (arg);
    space

    View Slide

  51. skipwhite
    syntax keyword lsKeyword function nextgroup=lsParens skipwhite
    syntax keyword lsKeyword return
    syntax match lsParens contained /(\k*)/
    syntax match lsContains /contains/

    View Slide

  52. Vim Keywords
    syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax keyword lsContains contains

    View Slide

  53. View Slide

  54. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/

    View Slide

  55. syntax keyword lsKeyword function
    syntax keyword lsKeyword return
    syntax match lsParens /(\k*)/
    syntax match lsContains /contains/
    syntax keyword lsKeyword con

    View Slide

  56. contains
    Match keyword first

    View Slide

  57. View Slide

  58. back to the issue
    var foo = function () {
    return 1;
    };
    return 1;
    match keyword: return

    View Slide

  59. Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/

    View Slide

  60. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock start=/{/ end=/}/ contains=lsReturn

    View Slide

  61. var foo = function () {
    return 1;
    };
    return 1;

    View Slide

  62. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View Slide

  63. Nesting Region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn

    View Slide

  64. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View Slide

  65. syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=lsBlock,lsKeyword,lsReturn
    syntax keyword lsDeclare var let

    View Slide

  66. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View Slide

  67. TOP region
    syntax keyword lsKeyword function
    syntax keyword lsReturn contained return
    syntax match lsParens /(\k*)/
    syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
    contains=TOP
    syntax keyword lsDeclare var let

    View Slide

  68. var foo = function () {
    var bar = function () {
    return 1;
    }
    };

    View Slide

  69. So…

    View Slide

  70. • Not real lexical analysis, have limitation

    • For performance

    • Best situation:

    • every token are independent

    • no nesting

    View Slide

  71. View Slide

  72. The Quests

    View Slide

  73. Quest?
    • Vim syntax highlight system has limitation

    • To support all ES2015 syntax is like solve quests

    View Slide

  74. Quests
    • class/object block

    • function block

    • template string

    • arrow function

    • default parameter

    View Slide

  75. blocks

    View Slide

  76. blocks
    • class/object/function/normal block

    • All starts with `{` and ends with `}`

    • But have different content

    View Slide

  77. normal block
    for (i = 0; i < arr.length; i++) {
    console.log(arr[i]);
    }

    View Slide

  78. • Have everything outside of the block

    • TOP region groups

    • Function block region groups

    View Slide

  79. function block
    var foo = function () {
    var bar = 1;
    return 1;
    };

    View Slide

  80. • Have almost everything in TOP region

    • TOP region groups

    • And `return`

    • But no import/export related stuff

    View Slide

  81. object
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };

    View Slide

  82. class
    var foo = {
    bar: 1,
    buz: function () {
    return 1;
    }
    };

    View Slide

  83. • Have all property name syntax and value(RHS)

    • Property name syntaxes

    • identifier, number, string

    • computed

    • And then RHS expression

    View Slide

  84. Issues
    • These blocks have very different contents

    • I have to know what this region is at position `{`

    View Slide

  85. Normal don’t need to deal
    Function after `function` keyword, method syntax
    Class always after `class` keyword
    Object in RHS expression

    View Slide

  86. RHS expression
    • Right hand side of an operator

    • Inside a pair of paren `(`, `)`

    View Slide

  87. syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/
    nextgroup=@javascriptExpression
    syntax region javascriptParenExp
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptExpression

    View Slide

  88. function block

    View Slide

  89. function block
    • Have almost everything in TOP region

    • TOP region groups

    • And `return`

    • But no import/export related stuff

    View Slide

  90. syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP

    View Slide

  91. TOP
    • A special group cluster

    • Contains every group without `contained`

    • I called these stuff belongs to TOP region

    View Slide

  92. but

    View Slide

  93. JS in HTML


    <br/>function foo() {<br/>return 1;<br/>}<br/>


    View Slide

  94. in HTML Syntax
    syn include @htmlJavaScript syntax/javascript.vim
    unlet b:current_syntax
    syn region javaScript
    start=+]*>+ keepend<br/>end=+]*>+me=s-1
    contains=@htmlJavaScript

    View Slide

  95. in Jinja Syntax
    " Pull in the HTML syntax.
    if g:jinja_syntax_html
    if version < 600
    so :p:h/html.vim
    else
    runtime! syntax/html.vim
    unlet b:current_syntax
    endif
    endif

    View Slide

  96. TOP is HTML’s TOP

    View Slide

  97. Detect Filetype
    if &filetype =~ 'javascript'
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=TOP
    else
    syntax region javascriptBlock
    matchgroup=javascriptBraces start=/{/ end=/}/
    contains=@htmlJavaScript
    endif

    View Slide

  98. Template String

    View Slide

  99. Template String
    `string text`
    `string text line 1
    string text line 2`
    `string text ${expression} string text`

    View Slide

  100. syntax region javascriptString
    start=/\z(["']\)/
    skip=/\\\\\|\\\z1\|\\\n/
    end=/\z1\|$/

    View Slide

  101. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\/

    View Slide

  102. `string text`
    `string text line 1
    string text line 2`
    `string text ${expression} string text`

    View Slide

  103. The Issue

    View Slide

  104. `string text ${"another string, and accent `"} string text`

    View Slide

  105. • The string ends when match an accent `

    • Try to recognize inner content

    View Slide

  106. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\|$/
    contains=javascriptTemplateSubstitution
    syntax region javascriptTemplateSubstitution contained
    matchgroup=javascriptTemplateSB
    start=/\${/
    end=/}/

    View Slide

  107. `string text ${"another string, and accent `"} string text`

    View Slide

  108. syntax region javascriptTemplate
    start=/`/
    skip=/\\\\\|\\`\|\n/
    end=/`\|$/
    contains=javascriptTemplateSubstitution
    syntax region javascriptTemplateSubstitution contained
    matchgroup=javascriptTemplateSB
    start=/\${/
    end=/}/
    contains=@javascriptExpression

    View Slide

  109. `string text ${"another string, and accent `"} string text`

    View Slide

  110. Arrow Function

    View Slide

  111. Main Target
    • Highlight arrow function and its parameters

    View Slide

  112. var foo = function (arg1) {};

    View Slide

  113. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};

    View Slide

  114. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View Slide

  115. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);
    Can’t lookup following token
    When cursor is here

    View Slide

  116. =>
    syntax match javascriptArrow
    /=>/
    nextgroup=javascriptBlock
    skipwhite

    View Slide

  117. =>
    var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View Slide

  118. Parameter?

    View Slide

  119. Problem
    • Don’t know what it is when meet `(`

    • Function parameters or

    • Expression in parentheses

    View Slide

  120. Before Default Parameter
    • Only identifier and comma inside function argument
    parens

    • Identifier name possible characters: [a-zA-Z_0-9$]

    • Not contain right parentheses

    View Slide

  121. Workaround Solution
    • Use match to match an arrow function

    • Put this match rule after parentheses region

    (higher priority)

    View Slide

  122. syntax region javascriptParens ...
    syntax match javascriptArrowFuncDef
    /(\_[^)]*)\_s*=>/
    contains=javascriptArrowFuncArg,
    javascriptComma
    nextgroup=javascriptBlock
    skipwhite
    skipempty

    View Slide

  123. syntax region javascriptParens ...
    syntax match javascriptArrowFuncDef
    /(\_[^)]*)\_s*=>/
    contains=javascriptArrowFuncArg,
    javascriptComma
    nextgroup=javascriptBlock
    skipwhite
    skipempty
    Match pair of paren

    View Slide

  124. syntax match javascriptArrowFunc /=>/
    syntax region javascriptArrowFuncArg contained
    matchgroup=javascriptParens
    start=/(/
    end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptArrowFunc
    skipwhite
    skipempty

    View Slide

  125. syntax cluster javascriptFuncArgElements
    contains=javascriptFuncKeyword,
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments,
    javascriptFuncArgArray,
    javascriptFuncArgObject,
    javascriptSpreadOp

    View Slide

  126. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View Slide

  127. Default Parameter

    View Slide

  128. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }

    View Slide

  129. syntax keyword javascriptFuncKeyword function
    nextgroup=javascriptFuncName skipwhite
    syntax match javascriptFuncName contained /[a-zA-Z_$]\k*/
    nextgroup=javascriptFuncArg skipwhite
    syntax match javascriptFuncArg contained /(\_[^)]*)/
    nextgroup=javascriptBlock skipwhite

    View Slide

  130. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }

    View Slide

  131. function multiply(a, b) {
    var b = typeof b !== 'undefined' ? b : 1;
    return a*b;
    }
    FuncKeyword
    FuncName
    FuncArg

    View Slide

  132. function multiply(a, b = 1) {
    return a*b;
    }

    View Slide

  133. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    ???

    View Slide

  134. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    Match ending )
    Not match block

    View Slide

  135. • Not able to really know which paren is ending one

    • Have to well know the content of parens

    • Use region

    View Slide

  136. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View Slide

  137. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View Slide

  138. syntax region javascriptFuncArg contained
    matchgroup=javascriptParens start=/(/ end=/)/
    contains=@javascriptFuncArgElements
    nextgroup=javascriptBlock
    skipwhite
    syntax cluster javascriptFuncArgElements
    contains=...
    javascriptComma,
    javascriptDefaultAssign,
    @javascriptComments...
    syntax match javascriptDefaultAssign contained
    /=/
    nextgroup=@javascriptExpression
    skipwhite

    View Slide

  139. function multiply(a, b = "(1+1)") {
    return a*b;
    }
    Default Assign
    Expression, string

    View Slide

  140. Arrow Function
    Default Parameter

    View Slide

  141. var foo = function (arg1) {};
    var fooArrow = (arg1) => {};
    var fooParen = (arg1);

    View Slide

  142. var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = 1, arg2 = `string`) => {};

    View Slide

  143. var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = function () {}) => {};

    View Slide

  144. Matching fails here
    var foo = function (arg1) {};
    var fooArrow = (arg1 = 1) => {};
    var fooArrow = (arg1 = function () {}) => {};

    View Slide

  145. Problem
    • Expression in parentheses and arrow function
    parameter mixed up

    View Slide

  146. var fooArrow1 = function (arg1 = () => {}) {};

    View Slide

  147. var fooArrow1 = function (arg1 = () => {}) {};
    var fooArrow2 = function (arg1 = () => {
    return function () {};
    }) {};

    View Slide

  148. var fooArrow1 = function (arg1 = () => {}) {};
    var fooArrow2 = function (arg1 = () => {
    return function () {};
    }) {};
    var fooArrow3 = function (arg1 = (arg2 = () => {}) => {
    return function () {};
    }) {};

    View Slide

  149. Problem
    • Parens, arrow function parameter mixed

    • Arrow function use match instead of region

    View Slide

  150. My Latest Quest

    View Slide

  151. Still Solving

    View Slide

  152. Patch Also Welcome

    View Slide

  153. View Slide

  154. Thank You

    View Slide

  155. Questions?

    View Slide