Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The new syntax highlighter for Vim

pocke
November 04, 2017

The new syntax highlighter for Vim

VimConf 2017

pocke

November 04, 2017
Tweet

More Decks by pocke

Other Decks in Programming

Transcript

  1. The new syntax highlighter
    for Vim
    4 Nov.
    VimConf 2017

    View Slide

  2. hi

    View Slide

  3. :help pocke

    View Slide

  4. :help pocke
    ● Masataka Kuwabara
    ● Vimmer / Rubyist
    ● Actcat, inc. / SideCI
    ● RuboCop's core developer
    ○ RuboCop is a static analyzer for Ruby.

    View Slide

  5. My most starred Vim plug-in
    is ...

    View Slide

  6. sushibar.vim

    View Slide

  7. View Slide

  8. Back to the topic

    View Slide

  9. Agenda

    View Slide

  10. Agenda
    ● Talk about syntax highlighters
    ○ Introduce the current syntax highlighter.
    ○ Introduce the new syntax highlighter.
    ■ I implement it

    View Slide

  11. The current syntax highlighter
    for Vim

    View Slide

  12. What's the current syntax
    highlighter?

    View Slide

  13. :help syntax

    View Slide

  14. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View Slide

  15. For example:

    View Slide

  16. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Only two colors...
    Understanding syntax of the code is difficult.

    View Slide

  17. :syntax on

    View Slide

  18. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Colorful

    View Slide

  19. Useful!

    View Slide

  20. But… The current highlighter
    has problems

    View Slide

  21. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View Slide

  22. For example: synmaxcol
    ● Vim gives up parsing very long line.
    ○ The limitation is configurable by
    `synmaxcol`.
    ● DEMO
    ○ https://github.com/pocke/vimconf-2017/blob/mast
    er/synmaxcol.json

    View Slide

  23. And… the highlighter has
    another problem

    View Slide

  24. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View Slide

  25. Some people, when
    confronted with a problem,
    think “I know, I’ll use regular
    expressions.” Now they have
    two problems.
    Jamie Zawinski http://regex.info/blog/2006-09-15/247

    View Slide

  26. Regular expression is too
    difficult for humans.

    View Slide

  27. For example:
    $ vim vim-ruby/syntax/ruby.vim
    ● vim-ruby defines syntax with many regular
    expressions.
    ● DEMO

    View Slide

  28. Regexp has another problem

    View Slide

  29. For example: complex code
    p(????::?:)
    How should the code be highlight?(DEMO)

    View Slide

  30. Conclusion of the Current Syntax
    Highlighter
    ● The syntax highlighter has limitations for
    performance.
    ○ e.g.) On long long line, on large .<br/>● The syntax highlighter uses keywords and<br/>regexp to highlight text.<br/>○ The regexp is very difficult, and does not perfect.<br/>

    View Slide

  31. Commercial: SideCI

    View Slide

  32. View Slide

  33. SideCI
    ● Automated Code Review Platform.
    ● Please access https://sideci.com !
    ● Sorry, we doesn't support Vim script…

    View Slide

  34. Back to the topic

    View Slide

  35. The new syntax highlighter
    for Vim

    View Slide

  36. Iro.vim(いろ.びむ)
    https://github.com/pocke/iro.vim

    View Slide

  37. Features of iro.vim
    ● The new syntax highlight plugin for Vim.
    ● Highlight perfectly with high performance
    ● Understandable code
    ● DEMO

    View Slide

  38. Design of Iro.vim

    View Slide

  39. Design
    ● Use `matchaddpos()` instead of `:syntax`.
    ● Use a parser that is implemented in the
    target language to parse perfectly.

    View Slide

  40. How iro.vim to work:
    In Ruby case

    View Slide

  41. Example: "Hello world" in Ruby
    def foo
    puts "Hello world!"
    end
    First, Iro.vim receives the code, and passes the
    code to `iro#ruby#tokens()`.

    View Slide

  42. iro#ruby#tokens()
    code to tokens
    def foo
    puts "Hello world!"
    end
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    Ripper.lex(code)

    View Slide

  43. iro#ruby#tokens()
    tokens to highlights data
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }

    View Slide

  44. Iro.vim
    highlights with the data
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }
    def foo
    puts "Hello world!"
    end

    View Slide

  45. Understandable code

    View Slide

  46. A tour of iro.vim source code

    View Slide

  47. autoload/iro.vim
    ● Define iro#redraw()
    ● load autoload/iro.vim.rb
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim

    View Slide

  48. autoload/iro.vim.rb
    ● call matchaddpos() each highlights.
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim.rb

    View Slide

  49. autoload/iro/ruby.vim.rb
    ● parse a ruby file, and generate highlight data
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro/ruby.vim.rb

    View Slide

  50. High Performance?

    View Slide

  51. I have good news and bad
    news.

    View Slide

  52. Performance
    ● Bad news: A performance issue exists
    ○ Iro.vim is very slow on a ruby file that has 10,000+
    lines(DEMO).
    ● Good news: Parsing is not a bottleneck
    ○ Vim is very slow when maybe many highlight is
    registered by `matchaddpos()`.

    View Slide

  53. Current status of iro.vim

    View Slide

  54. Current status
    ● Still developing, but it works
    ● Support few languages.
    ○ Ruby, YAML and Python(experimental).
    ● Welcome Pull-Requests

    View Slide

  55. :endslide
    ● Iro.vim is a new syntax highlighter.
    ○ It highlights perfectly
    Thank you for listening

    View Slide