Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The new syntax highlighter for Vim

pocke
November 04, 2017

The new syntax highlighter for Vim

VimConf 2017

pocke

November 04, 2017
Tweet

More Decks by pocke

Other Decks in Programming

Transcript

  1. The new syntax highlighter
    for Vim
    4 Nov.
    VimConf 2017

    View full-size slide

  2. :help pocke
    ● Masataka Kuwabara
    ● Vimmer / Rubyist
    ● Actcat, inc. / SideCI
    ● RuboCop's core developer
    ○ RuboCop is a static analyzer for Ruby.

    View full-size slide

  3. My most starred Vim plug-in
    is ...

    View full-size slide

  4. sushibar.vim

    View full-size slide

  5. Back to the topic

    View full-size slide

  6. Agenda
    ● Talk about syntax highlighters
    ○ Introduce the current syntax highlighter.
    ○ Introduce the new syntax highlighter.
    ■ I implement it

    View full-size slide

  7. The current syntax highlighter
    for Vim

    View full-size slide

  8. What's the current syntax
    highlighter?

    View full-size slide

  9. :help syntax

    View full-size slide

  10. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View full-size slide

  11. For example:

    View full-size slide

  12. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Only two colors...
    Understanding syntax of the code is difficult.

    View full-size slide

  13. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Colorful

    View full-size slide

  14. But… The current highlighter
    has problems

    View full-size slide

  15. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View full-size slide

  16. For example: synmaxcol
    ● Vim gives up parsing very long line.
    ○ The limitation is configurable by
    `synmaxcol`.
    ● DEMO
    ○ https://github.com/pocke/vimconf-2017/blob/mast
    er/synmaxcol.json

    View full-size slide

  17. And… the highlighter has
    another problem

    View full-size slide

  18. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

    View full-size slide

  19. Some people, when
    confronted with a problem,
    think “I know, I’ll use regular
    expressions.” Now they have
    two problems.
    Jamie Zawinski http://regex.info/blog/2006-09-15/247

    View full-size slide

  20. Regular expression is too
    difficult for humans.

    View full-size slide

  21. For example:
    $ vim vim-ruby/syntax/ruby.vim
    ● vim-ruby defines syntax with many regular
    expressions.
    ● DEMO

    View full-size slide

  22. Regexp has another problem

    View full-size slide

  23. For example: complex code
    p(????::?:)
    How should the code be highlight?(DEMO)

    View full-size slide

  24. Conclusion of the Current Syntax
    Highlighter
    ● The syntax highlighter has limitations for
    performance.
    ○ e.g.) On long long line, on large .<br/>● The syntax highlighter uses keywords and<br/>regexp to highlight text.<br/>○ The regexp is very difficult, and does not perfect.<br/>

    View full-size slide

  25. Commercial: SideCI

    View full-size slide

  26. SideCI
    ● Automated Code Review Platform.
    ● Please access https://sideci.com !
    ● Sorry, we doesn't support Vim script…

    View full-size slide

  27. Back to the topic

    View full-size slide

  28. The new syntax highlighter
    for Vim

    View full-size slide

  29. Iro.vim(いろ.びむ)
    https://github.com/pocke/iro.vim

    View full-size slide

  30. Features of iro.vim
    ● The new syntax highlight plugin for Vim.
    ● Highlight perfectly with high performance
    ● Understandable code
    ● DEMO

    View full-size slide

  31. Design of Iro.vim

    View full-size slide

  32. Design
    ● Use `matchaddpos()` instead of `:syntax`.
    ● Use a parser that is implemented in the
    target language to parse perfectly.

    View full-size slide

  33. How iro.vim to work:
    In Ruby case

    View full-size slide

  34. Example: "Hello world" in Ruby
    def foo
    puts "Hello world!"
    end
    First, Iro.vim receives the code, and passes the
    code to `iro#ruby#tokens()`.

    View full-size slide

  35. iro#ruby#tokens()
    code to tokens
    def foo
    puts "Hello world!"
    end
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    Ripper.lex(code)

    View full-size slide

  36. iro#ruby#tokens()
    tokens to highlights data
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }

    View full-size slide

  37. Iro.vim
    highlights with the data
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }
    def foo
    puts "Hello world!"
    end

    View full-size slide

  38. Understandable code

    View full-size slide

  39. A tour of iro.vim source code

    View full-size slide

  40. autoload/iro.vim
    ● Define iro#redraw()
    ● load autoload/iro.vim.rb
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim

    View full-size slide

  41. autoload/iro.vim.rb
    ● call matchaddpos() each highlights.
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim.rb

    View full-size slide

  42. autoload/iro/ruby.vim.rb
    ● parse a ruby file, and generate highlight data
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro/ruby.vim.rb

    View full-size slide

  43. High Performance?

    View full-size slide

  44. I have good news and bad
    news.

    View full-size slide

  45. Performance
    ● Bad news: A performance issue exists
    ○ Iro.vim is very slow on a ruby file that has 10,000+
    lines(DEMO).
    ● Good news: Parsing is not a bottleneck
    ○ Vim is very slow when maybe many highlight is
    registered by `matchaddpos()`.

    View full-size slide

  46. Current status of iro.vim

    View full-size slide

  47. Current status
    ● Still developing, but it works
    ● Support few languages.
    ○ Ruby, YAML and Python(experimental).
    ● Welcome Pull-Requests

    View full-size slide

  48. :endslide
    ● Iro.vim is a new syntax highlighter.
    ○ It highlights perfectly
    Thank you for listening

    View full-size slide