The new syntax highlighter for Vim

VimConf 2017

Transcript

1. The new syntax highlighter
   for Vim
   4 Nov.
   VimConf 2017
   

   View Slide

2. hi
   

   View Slide

3. :help pocke
   

   View Slide

4. :help pocke
   ● Masataka Kuwabara
   ● Vimmer / Rubyist
   ● Actcat, inc. / SideCI
   ● RuboCop's core developer
   ○ RuboCop is a static analyzer for Ruby.
   

   View Slide

5. My most starred Vim plug-in
   is ...
   

   View Slide

6. sushibar.vim
   

   View Slide

7. View Slide

8. Back to the topic
   

   View Slide

9. Agenda
   

   View Slide

10. Agenda
    ● Talk about syntax highlighters
    ○ Introduce the current syntax highlighter.
    ○ Introduce the new syntax highlighter.
    ■ I implement it
    

    View Slide

11. The current syntax highlighter
    for Vim
    

    View Slide

12. What's the current syntax
    highlighter?
    

    View Slide

13. :help syntax
    

    View Slide

14. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
    

    View Slide

15. For example:
    

    View Slide

16. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Only two colors...
    Understanding syntax of the code is difficult.
    

    View Slide

17. :syntax on
    

    View Slide

18. hello.vim
    function! Hello(name) abort
    echom 'Hello, ' . a:name . '-san!'
    endfunction
    Colorful
    

    View Slide

19. Useful!
    

    View Slide

20. But… The current highlighter
    has problems
    

    View Slide

21. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
    

    View Slide

22. For example: synmaxcol
    ● Vim gives up parsing very long line.
    ○ The limitation is configurable by
    `synmaxcol`.
    ● DEMO
    ○ https://github.com/pocke/vimconf-2017/blob/mast
    er/synmaxcol.json
    

    View Slide

23. And… the highlighter has
    another problem
    

    View Slide

24. :help syntax
    Syntax highlighting enables Vim to show parts of the
    text in another font or color.
    Those parts can be specific keywords or text matching
    a pattern.
    Vim doesn't parse the whole file (to keep it fast), so the
    highlighting has its limitations.
    https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12
    

    View Slide

25. Some people, when
    confronted with a problem,
    think “I know, I’ll use regular
    expressions.” Now they have
    two problems.
    Jamie Zawinski http://regex.info/blog/2006-09-15/247
    

    View Slide

26. Regular expression is too
    difficult for humans.
    

    View Slide

27. For example:
    $ vim vim-ruby/syntax/ruby.vim
    ● vim-ruby defines syntax with many regular
    expressions.
    ● DEMO
    

    View Slide

28. Regexp has another problem
    

    View Slide

29. For example: complex code
    p(????::?:)
    How should the code be highlight?(DEMO)
    

    View Slide

30. Conclusion of the Current Syntax
    Highlighter
    ● The syntax highlighter has limitations for
    performance.
    ○ e.g.) On long long line, on large .<br/>● The syntax highlighter uses keywords and<br/>regexp to highlight text.<br/>○ The regexp is very difficult, and does not perfect.<br/>

    View Slide

31. Commercial: SideCI
    

    View Slide

32. View Slide

33. SideCI
    ● Automated Code Review Platform.
    ● Please access https://sideci.com !
    ● Sorry, we doesn't support Vim script…
    

    View Slide

34. Back to the topic
    

    View Slide

35. The new syntax highlighter
    for Vim
    

    View Slide

36. Iro.vim(いろ.びむ)
    https://github.com/pocke/iro.vim
    

    View Slide

37. Features of iro.vim
    ● The new syntax highlight plugin for Vim.
    ● Highlight perfectly with high performance
    ● Understandable code
    ● DEMO
    

    View Slide

38. Design of Iro.vim
    

    View Slide

39. Design
    ● Use `matchaddpos()` instead of `:syntax`.
    ● Use a parser that is implemented in the
    target language to parse perfectly.
    

    View Slide

40. How iro.vim to work:
    In Ruby case
    

    View Slide

41. Example: "Hello world" in Ruby
    def foo
    puts "Hello world!"
    end
    First, Iro.vim receives the code, and passes the
    code to `iro#ruby#tokens()`.
    

    View Slide

42. iro#ruby#tokens()
    code to tokens
    def foo
    puts "Hello world!"
    end
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    Ripper.lex(code)
    

    View Slide

43. iro#ruby#tokens()
    tokens to highlights data
    [[[1, 0], :on_kw, "def"],
    [[1, 3], :on_sp, " "],
    [[1, 4], :on_ident, "foo"],
    [[1, 7], :on_nl, "\n"],
    [[2, 0], :on_sp, " "],
    [[2, 2], :on_ident, "puts"],
    [[2, 6], :on_sp, " "],
    [[2, 7], :on_tstring_beg, "\""],
    [[2, 8], :on_tstring_content, "Hello
    world!"],
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }
    

    View Slide

44. Iro.vim
    highlights with the data
    {
    "Keyword":[[1,1,3],[3,1,3]],
    "String":[[2,9,12]],
    "Delimiter":[[2,8,1], [2,21,1]]
    }
    def foo
    puts "Hello world!"
    end
    

    View Slide

45. Understandable code
    

    View Slide

46. A tour of iro.vim source code
    

    View Slide

47. autoload/iro.vim
    ● Define iro#redraw()
    ● load autoload/iro.vim.rb
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim
    

    View Slide

48. autoload/iro.vim.rb
    ● call matchaddpos() each highlights.
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro.vim.rb
    

    View Slide

49. autoload/iro/ruby.vim.rb
    ● parse a ruby file, and generate highlight data
    ● https://github.com/pocke/iro.vim/blob/mast
    er/autoload/iro/ruby.vim.rb
    

    View Slide

50. High Performance?
    

    View Slide

51. I have good news and bad
    news.
    

    View Slide

52. Performance
    ● Bad news: A performance issue exists
    ○ Iro.vim is very slow on a ruby file that has 10,000+
    lines(DEMO).
    ● Good news: Parsing is not a bottleneck
    ○ Vim is very slow when maybe many highlight is
    registered by `matchaddpos()`.
    

    View Slide

53. Current status of iro.vim
    

    View Slide

54. Current status
    ● Still developing, but it works
    ● Support few languages.
    ○ Ruby, YAML and Python(experimental).
    ● Welcome Pull-Requests
    

    View Slide

55. :endslide
    ● Iro.vim is a new syntax highlighter.
    ○ It highlights perfectly
    Thank you for listening
    

    View Slide