The new syntax highlighter for Vim

VimConf 2017

Transcript

1. The new syntax highlighter for Vim 4 Nov. VimConf 2017
2. hi
3. :help pocke
4. :help pocke • Masataka Kuwabara • Vimmer / Rubyist • Actcat, inc. / SideCI • RuboCop's core developer ◦ RuboCop is a static analyzer for Ruby.
5. My most starred Vim plug-in is ...
6. sushibar.vim
8. Back to the topic
9. Agenda
10. Agenda • Talk about syntax highlighters ◦ Introduce the current syntax highlighter. ◦ Introduce the new syntax highlighter. ▪ I implement it
11. The current syntax highlighter for Vim
12. What's the current syntax highlighter?
13. :help syntax
14. :help syntax Syntax highlighting enables Vim to show parts of the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
15. For example:
16. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name . '-san!' endfunction Only two colors... Understanding syntax of the code is difficult.
17. :syntax on
18. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name . '-san!' endfunction Colorful
19. Useful!
20. But… The current highlighter has problems
21. :help syntax Syntax highlighting enables Vim to show parts of the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
22. For example: synmaxcol • Vim gives up parsing very long line. ◦ The limitation is configurable by `synmaxcol`. • DEMO ◦ https://github.com/pocke/vimconf-2017/blob/mast er/synmaxcol.json
23. And… the highlighter has another problem
24. :help syntax Syntax highlighting enables Vim to show parts of the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
25. Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247
26. Regular expression is too difficult for humans.
27. For example: $ vim vim-ruby/syntax/ruby.vim • vim-ruby defines syntax with many regular expressions. • DEMO
28. Regexp has another problem
29. For example: complex code p(????::?:) How should the code be highlight?(DEMO)
30. Conclusion of the Current Syntax Highlighter • The syntax highlighter has limitations for performance. ◦ e.g.) On long long line, on large <script>. • The syntax highlighter uses keywords and regexp to highlight text. ◦ The regexp is very difficult, and does not perfect.
31. Commercial: SideCI
33. SideCI • Automated Code Review Platform. • Please access https://sideci.com ! • Sorry, we doesn't support Vim script…
34. Back to the topic
35. The new syntax highlighter for Vim
36. Iro.vim(いろ.びむ) https://github.com/pocke/iro.vim
37. Features of iro.vim • The new syntax highlight plugin for Vim. • Highlight perfectly with high performance • Understandable code • DEMO
38. Design of Iro.vim
39. Design • Use `matchaddpos()` instead of `:syntax`. • Use a parser that is implemented in the target language to parse perfectly.
40. How iro.vim to work: In Ruby case
41. Example: "Hello world" in Ruby def foo puts "Hello world!" end First, Iro.vim receives the code, and passes the code to `iro#ruby#tokens()`.
42. iro#ruby#tokens() code to tokens def foo puts "Hello world!" end [[[1, 0], :on_kw, "def"], [[1, 3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], Ripper.lex(code)
43. iro#ruby#tokens() tokens to highlights data [[[1, 0], :on_kw, "def"], [[1, 3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]] }
44. Iro.vim highlights with the data { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]] } def foo puts "Hello world!" end
45. Understandable code
46. A tour of iro.vim source code
47. autoload/iro.vim • Define iro#redraw() • load autoload/iro.vim.rb • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim
48. autoload/iro.vim.rb • call matchaddpos() each highlights. • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim.rb
49. autoload/iro/ruby.vim.rb • parse a ruby file, and generate highlight data • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro/ruby.vim.rb
50. High Performance?
51. I have good news and bad news.
52. Performance • Bad news: A performance issue exists ◦ Iro.vim is very slow on a ruby file that has 10,000+ lines(DEMO). • Good news: Parsing is not a bottleneck ◦ Vim is very slow when maybe many highlight is registered by `matchaddpos()`.
53. Current status of iro.vim
54. Current status • Still developing, but it works • Support few languages. ◦ Ruby, YAML and Python(experimental). • Welcome Pull-Requests
55. :endslide • Iro.vim is a new syntax highlighter. ◦ It highlights perfectly Thank you for listening