YAJS.vim and Vim Syntax
Highlight
othree @ HKOSCon
Slide 2
Slide 2 text
me
• othree @ flickr, github, twitter…
• MozTW
• F2E at Qihoo
• Speeches in COSCUP, OSDC.tw, JSDC…
• https://blog.othree.net/
Slide 3
Slide 3 text
outline
• Why YAJS.vim
• Vim Syntax Highlight
• The Quests
Slide 4
Slide 4 text
Why YAJS.vim
Slide 5
Slide 5 text
YAJS.vim
• Yet Another JavaScript Syntax for Vim
• Supports ECMAScript 6 syntax
• Remove legacy code support
• and JSDoc, Web APIs, console, ….
• javascript instead of javaScript
Slide 6
Slide 6 text
default
Slide 7
Slide 7 text
YAJS.vim
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Why YAJS
• ECMAScript 6(a.k.a ES6, ES2015) starts since 2011
• People can’t wait to use new features
• Transpilers: Traceur, 6to5(Babel)…
Slide 10
Slide 10 text
• People start to use ES6(and lots of ES.next feature)
• JavaScript syntax for Vim doesn’t support ES6
• So I start to do it
Slide 11
Slide 11 text
Dev Target
• Support all valid ES6 syntax
• If possible, add some error hint
• Performance is prioritize to second place
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Why no Push to Upstream
• Lots of incompatible code change during dev
• Can’t wait for a PR review
Slide 14
Slide 14 text
Status Now
• Support major ES6 syntax
• Still in development
• Trying to build auto test
Slide 15
Slide 15 text
• https://github.com/othree/yajs.vim
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
Vim Syntax Highlight
Slide 18
Slide 18 text
• Pretend there is a new filetype called ‘ls’
• Not livescript
Slide 19
Slide 19 text
Syntax Items
• Keyword
• Match
• Region
Slide 20
Slide 20 text
Keyword
syntax keyword lsKeyword function
syntax keyword lsKeyword return
highlight link lsKeyword Statement
highlight Statement ctermfg=251 ctermbg=None
Slide 21
Slide 21 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
highlight link lsKeyword Statement
highlight Statement ctermfg=251 ctermbg=None
group name
item type
Slide 22
Slide 22 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
highlight link lsKeyword Statement
highlight Statement ctermfg=251 ctermbg=None
Slide 23
Slide 23 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
highlight link lsKeyword Statement
highlight Statement ctermfg=251 ctermbg=None
target group name
source group name
link groups
Slide 24
Slide 24 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
highlight link lsKeyword Statement
highlight Statement ctermfg=251 ctermbg=None
color definition
group name
Slide 25
Slide 25 text
var foo = function () {
};
Slide 26
Slide 26 text
var foo = function () {
};
TOP region
Slide 27
Slide 27 text
• At TOP region, looking for keywords
• function
• return
Slide 28
Slide 28 text
var foo = function () {
};
match keyword: function
Slide 29
Slide 29 text
var foo = function () {
};
highlight it
Slide 30
Slide 30 text
var foo = function () {
return 1;
};
keep looking
Slide 31
Slide 31 text
var foo = function () {
return 1;
};
match keyword: return
Slide 32
Slide 32 text
var foo = function () {
return 1;
};
return 1;
match keyword: return
Slide 33
Slide 33 text
Match
syntax keyword lsKeyword function
syntax keyword lsKeyword return
syntax match lsParens /(\k*)/
syntax match lsContains /contains/
Slide 34
Slide 34 text
• Match uses Vim’s regular expression
• http://vimdoc.sourceforge.net/htmldoc/pattern.html
Slide 35
Slide 35 text
var foo = function (arg) {
return 1;
};
return 1;
TOP region
Slide 36
Slide 36 text
• Looking for keywords and match pattern
• Keyword first, match second
Slide 37
Slide 37 text
var foo = function(arg) {
return 1;
};
return 1;
Match keyword: function
Slide 38
Slide 38 text
var foo = function(arg) {
return 1;
};
return 1;
Match pattern
contained
syntax keyword lsKeyword function
syntax keyword lsKeyword return
syntax match lsParens contained /(\k*)/
syntax match lsContains /contains/
Slide 42
Slide 42 text
contained
• Vim not find contained group at TOP region
Slide 43
Slide 43 text
var foo = function (arg) {
return 1;
};
return 1;
(arg);
Slide 44
Slide 44 text
nextgroup
syntax keyword lsKeyword function nextgroup=lsParens
syntax keyword lsKeyword return
syntax match lsParens contained /(\k*)/
syntax match lsContains /contains/
Slide 45
Slide 45 text
nextgroup
• Vim will lookup groups listed in nextgroup first
• Before normal keyword, match, region
• Not limited by contained
Slide 46
Slide 46 text
var foo = function(arg) {
return 1;
};
return 1;
(arg);
Ending of keyword function
Slide 47
Slide 47 text
var foo = function(arg) {
return 1;
};
return 1;
(arg);
Looking for groups in nextgroup
Slide 48
Slide 48 text
var foo = function(arg) {
return 1;
};
return 1;
(arg);
Match pattern
Slide 49
Slide 49 text
var foo = function(arg) {
return 1;
};
return 1;
(arg);
Not match any pattern
Slide 50
Slide 50 text
var foo = function (arg) {
return 1;
};
return 1;
(arg);
space
Slide 51
Slide 51 text
skipwhite
syntax keyword lsKeyword function nextgroup=lsParens skipwhite
syntax keyword lsKeyword return
syntax match lsParens contained /(\k*)/
syntax match lsContains /contains/
Slide 52
Slide 52 text
Vim Keywords
syntax keyword lsKeyword function
syntax keyword lsKeyword return
syntax match lsParens /(\k*)/
syntax keyword lsContains contains
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
syntax match lsParens /(\k*)/
syntax match lsContains /contains/
Slide 55
Slide 55 text
syntax keyword lsKeyword function
syntax keyword lsKeyword return
syntax match lsParens /(\k*)/
syntax match lsContains /contains/
syntax keyword lsKeyword con
Slide 56
Slide 56 text
contains
Match keyword first
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
back to the issue
var foo = function () {
return 1;
};
return 1;
match keyword: return
Slide 59
Slide 59 text
Region
syntax keyword lsKeyword function
syntax keyword lsReturn return
syntax match lsParens /(\k*)/
syntax region lsBlock start=/{/ end=/}/
Slide 60
Slide 60 text
syntax keyword lsKeyword function
syntax keyword lsReturn contained return
syntax match lsParens /(\k*)/
syntax region lsBlock start=/{/ end=/}/ contains=lsReturn
Slide 61
Slide 61 text
var foo = function () {
return 1;
};
return 1;
Slide 62
Slide 62 text
var foo = function () {
var bar = function () {
return 1;
}
};
Slide 63
Slide 63 text
Nesting Region
syntax keyword lsKeyword function
syntax keyword lsReturn contained return
syntax match lsParens /(\k*)/
syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
contains=lsBlock,lsKeyword,lsReturn
Slide 64
Slide 64 text
var foo = function () {
var bar = function () {
return 1;
}
};
Slide 65
Slide 65 text
syntax keyword lsKeyword function
syntax keyword lsReturn contained return
syntax match lsParens /(\k*)/
syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
contains=lsBlock,lsKeyword,lsReturn
syntax keyword lsDeclare var let
Slide 66
Slide 66 text
var foo = function () {
var bar = function () {
return 1;
}
};
Slide 67
Slide 67 text
TOP region
syntax keyword lsKeyword function
syntax keyword lsReturn contained return
syntax match lsParens /(\k*)/
syntax region lsBlock matchgroup=lsBrack start=/{/ end=/}/
contains=TOP
syntax keyword lsDeclare var let
Slide 68
Slide 68 text
var foo = function () {
var bar = function () {
return 1;
}
};
Slide 69
Slide 69 text
So…
Slide 70
Slide 70 text
• Not real lexical analysis, have limitation
• For performance
• Best situation:
• every token are independent
• no nesting
Slide 71
Slide 71 text
No content
Slide 72
Slide 72 text
The Quests
Slide 73
Slide 73 text
Quest?
• Vim syntax highlight system has limitation
• To support all ES2015 syntax is like solve quests
Slide 74
Slide 74 text
Quests
• class/object block
• function block
• template string
• arrow function
• default parameter
Slide 75
Slide 75 text
blocks
Slide 76
Slide 76 text
blocks
• class/object/function/normal block
• All starts with `{` and ends with `}`
• But have different content
Slide 77
Slide 77 text
normal block
for (i = 0; i < arr.length; i++) {
console.log(arr[i]);
}
Slide 78
Slide 78 text
• Have everything outside of the block
• TOP region groups
• Function block region groups
Slide 79
Slide 79 text
function block
var foo = function () {
var bar = 1;
return 1;
};
Slide 80
Slide 80 text
• Have almost everything in TOP region
• TOP region groups
• And `return`
• But no import/export related stuff
Slide 81
Slide 81 text
object
var foo = {
bar: 1,
buz: function () {
return 1;
}
};
Slide 82
Slide 82 text
class
var foo = {
bar: 1,
buz: function () {
return 1;
}
};
Slide 83
Slide 83 text
• Have all property name syntax and value(RHS)
• Property name syntaxes
• identifier, number, string
• computed
• And then RHS expression
Slide 84
Slide 84 text
Issues
• These blocks have very different contents
• I have to know what this region is at position `{`
Slide 85
Slide 85 text
Normal don’t need to deal
Function after `function` keyword, method syntax
Class always after `class` keyword
Object in RHS expression
Slide 86
Slide 86 text
RHS expression
• Right hand side of an operator
• Inside a pair of paren `(`, `)`
Slide 87
Slide 87 text
syntax match javascriptOpSymbols /[+\-*/%\^~=<>&|?]\+/
nextgroup=@javascriptExpression
syntax region javascriptParenExp
matchgroup=javascriptParens start=/(/ end=/)/
contains=@javascriptExpression
Slide 88
Slide 88 text
function block
Slide 89
Slide 89 text
function block
• Have almost everything in TOP region
• TOP region groups
• And `return`
• But no import/export related stuff
Slide 90
Slide 90 text
syntax region javascriptBlock
matchgroup=javascriptBraces start=/{/ end=/}/
contains=TOP
Slide 91
Slide 91 text
TOP
• A special group cluster
• Contains every group without `contained`
• I called these stuff belongs to TOP region
Slide 92
Slide 92 text
but
Slide 93
Slide 93 text
JS in HTML
function foo() {
return 1;
}
Slide 94
Slide 94 text
in HTML Syntax
syn include @htmlJavaScript syntax/javascript.vim
unlet b:current_syntax
syn region javaScript
start=+]*>+ keepend
end=+]*>+me=s-1
contains=@htmlJavaScript
Slide 95
Slide 95 text
in Jinja Syntax
" Pull in the HTML syntax.
if g:jinja_syntax_html
if version < 600
so :p:h/html.vim
else
runtime! syntax/html.vim
unlet b:current_syntax
endif
endif
Slide 96
Slide 96 text
TOP is HTML’s TOP
Slide 97
Slide 97 text
Detect Filetype
if &filetype =~ 'javascript'
syntax region javascriptBlock
matchgroup=javascriptBraces start=/{/ end=/}/
contains=TOP
else
syntax region javascriptBlock
matchgroup=javascriptBraces start=/{/ end=/}/
contains=@htmlJavaScript
endif
Slide 98
Slide 98 text
Template String
Slide 99
Slide 99 text
Template String
`string text`
`string text line 1
string text line 2`
`string text ${expression} string text`
Slide 100
Slide 100 text
syntax region javascriptString
start=/\z(["']\)/
skip=/\\\\\|\\\z1\|\\\n/
end=/\z1\|$/
Slide 101
Slide 101 text
syntax region javascriptTemplate
start=/`/
skip=/\\\\\|\\`\|\n/
end=/`\/
Slide 102
Slide 102 text
`string text`
`string text line 1
string text line 2`
`string text ${expression} string text`
Slide 103
Slide 103 text
The Issue
Slide 104
Slide 104 text
`string text ${"another string, and accent `"} string text`
Slide 105
Slide 105 text
• The string ends when match an accent `
• Try to recognize inner content
Slide 106
Slide 106 text
syntax region javascriptTemplate
start=/`/
skip=/\\\\\|\\`\|\n/
end=/`\|$/
contains=javascriptTemplateSubstitution
syntax region javascriptTemplateSubstitution contained
matchgroup=javascriptTemplateSB
start=/\${/
end=/}/
Slide 107
Slide 107 text
`string text ${"another string, and accent `"} string text`
Slide 108
Slide 108 text
syntax region javascriptTemplate
start=/`/
skip=/\\\\\|\\`\|\n/
end=/`\|$/
contains=javascriptTemplateSubstitution
syntax region javascriptTemplateSubstitution contained
matchgroup=javascriptTemplateSB
start=/\${/
end=/}/
contains=@javascriptExpression
Slide 109
Slide 109 text
`string text ${"another string, and accent `"} string text`
Slide 110
Slide 110 text
Arrow Function
Slide 111
Slide 111 text
Main Target
• Highlight arrow function and its parameters
Slide 112
Slide 112 text
var foo = function (arg1) {};
Slide 113
Slide 113 text
var foo = function (arg1) {};
var fooArrow = (arg1) => {};
Slide 114
Slide 114 text
var foo = function (arg1) {};
var fooArrow = (arg1) => {};
var fooParen = (arg1);
Slide 115
Slide 115 text
var foo = function (arg1) {};
var fooArrow = (arg1) => {};
var fooParen = (arg1);
Can’t lookup following token
When cursor is here
Slide 116
Slide 116 text
=>
syntax match javascriptArrow
/=>/
nextgroup=javascriptBlock
skipwhite
Slide 117
Slide 117 text
=>
var foo = function (arg1) {};
var fooArrow = (arg1) => {};
var fooParen = (arg1);
Slide 118
Slide 118 text
Parameter?
Slide 119
Slide 119 text
Problem
• Don’t know what it is when meet `(`
• Function parameters or
• Expression in parentheses
Slide 120
Slide 120 text
Before Default Parameter
• Only identifier and comma inside function argument
parens
• Identifier name possible characters: [a-zA-Z_0-9$]
• Not contain right parentheses
Slide 121
Slide 121 text
Workaround Solution
• Use match to match an arrow function
• Put this match rule after parentheses region
(higher priority)
Slide 122
Slide 122 text
syntax region javascriptParens ...
syntax match javascriptArrowFuncDef
/(\_[^)]*)\_s*=>/
contains=javascriptArrowFuncArg,
javascriptComma
nextgroup=javascriptBlock
skipwhite
skipempty
Slide 123
Slide 123 text
syntax region javascriptParens ...
syntax match javascriptArrowFuncDef
/(\_[^)]*)\_s*=>/
contains=javascriptArrowFuncArg,
javascriptComma
nextgroup=javascriptBlock
skipwhite
skipempty
Match pair of paren
Slide 124
Slide 124 text
syntax match javascriptArrowFunc /=>/
syntax region javascriptArrowFuncArg contained
matchgroup=javascriptParens
start=/(/
end=/)/
contains=@javascriptFuncArgElements
nextgroup=javascriptArrowFunc
skipwhite
skipempty