The new syntax highlighter for Vim

7bc6612fa20296bf652f6b0357db81c1?s=47 pocke
November 04, 2017

The new syntax highlighter for Vim

VimConf 2017

7bc6612fa20296bf652f6b0357db81c1?s=128

pocke

November 04, 2017
Tweet

Transcript

  1. The new syntax highlighter for Vim 4 Nov. VimConf 2017

  2. hi

  3. :help pocke

  4. :help pocke • Masataka Kuwabara • Vimmer / Rubyist •

    Actcat, inc. / SideCI • RuboCop's core developer ◦ RuboCop is a static analyzer for Ruby.
  5. My most starred Vim plug-in is ...

  6. sushibar.vim

  7. None
  8. Back to the topic

  9. Agenda

  10. Agenda • Talk about syntax highlighters ◦ Introduce the current

    syntax highlighter. ◦ Introduce the new syntax highlighter. ▪ I implement it
  11. The current syntax highlighter for Vim

  12. What's the current syntax highlighter?

  13. :help syntax

  14. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
  15. For example:

  16. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name .

    '-san!' endfunction Only two colors... Understanding syntax of the code is difficult.
  17. :syntax on

  18. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name .

    '-san!' endfunction Colorful
  19. Useful!

  20. But… The current highlighter has problems

  21. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
  22. For example: synmaxcol • Vim gives up parsing very long

    line. ◦ The limitation is configurable by `synmaxcol`. • DEMO ◦ https://github.com/pocke/vimconf-2017/blob/mast er/synmaxcol.json
  23. And… the highlighter has another problem

  24. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
  25. Some people, when confronted with a problem, think “I know,

    I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247
  26. Regular expression is too difficult for humans.

  27. For example: $ vim vim-ruby/syntax/ruby.vim • vim-ruby defines syntax with

    many regular expressions. • DEMO
  28. Regexp has another problem

  29. For example: complex code p(????::?:) How should the code be

    highlight?(DEMO)
  30. Conclusion of the Current Syntax Highlighter • The syntax highlighter

    has limitations for performance. ◦ e.g.) On long long line, on large <script>. • The syntax highlighter uses keywords and regexp to highlight text. ◦ The regexp is very difficult, and does not perfect.
  31. Commercial: SideCI

  32. None
  33. SideCI • Automated Code Review Platform. • Please access https://sideci.com

    ! • Sorry, we doesn't support Vim script…
  34. Back to the topic

  35. The new syntax highlighter for Vim

  36. Iro.vim(いろ.びむ) https://github.com/pocke/iro.vim

  37. Features of iro.vim • The new syntax highlight plugin for

    Vim. • Highlight perfectly with high performance • Understandable code • DEMO
  38. Design of Iro.vim

  39. Design • Use `matchaddpos()` instead of `:syntax`. • Use a

    parser that is implemented in the target language to parse perfectly.
  40. How iro.vim to work: In Ruby case

  41. Example: "Hello world" in Ruby def foo puts "Hello world!"

    end First, Iro.vim receives the code, and passes the code to `iro#ruby#tokens()`.
  42. iro#ruby#tokens() code to tokens def foo puts "Hello world!" end

    [[[1, 0], :on_kw, "def"], [[1, 3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], Ripper.lex(code)
  43. iro#ruby#tokens() tokens to highlights data [[[1, 0], :on_kw, "def"], [[1,

    3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]] }
  44. Iro.vim highlights with the data { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]]

    } def foo puts "Hello world!" end
  45. Understandable code

  46. A tour of iro.vim source code

  47. autoload/iro.vim • Define iro#redraw() • load autoload/iro.vim.rb • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim

  48. autoload/iro.vim.rb • call matchaddpos() each highlights. • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim.rb

  49. autoload/iro/ruby.vim.rb • parse a ruby file, and generate highlight data

    • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro/ruby.vim.rb
  50. High Performance?

  51. I have good news and bad news.

  52. Performance • Bad news: A performance issue exists ◦ Iro.vim

    is very slow on a ruby file that has 10,000+ lines(DEMO). • Good news: Parsing is not a bottleneck ◦ Vim is very slow when maybe many highlight is registered by `matchaddpos()`.
  53. Current status of iro.vim

  54. Current status • Still developing, but it works • Support

    few languages. ◦ Ruby, YAML and Python(experimental). • Welcome Pull-Requests
  55. :endslide • Iro.vim is a new syntax highlighter. ◦ It

    highlights perfectly Thank you for listening