The new syntax highlighter for Vim

Transcript

1. The new syntax highlighter for Vim 4 Nov. VimConf 2017

2. hi

3. :help pocke

4. :help pocke • Masataka Kuwabara • Vimmer / Rubyist •

   Actcat, inc. / SideCI • RuboCop's core developer ◦ RuboCop is a static analyzer for Ruby.

5. My most starred Vim plug-in is ...

6. sushibar.vim

7. None

8. Back to the topic

9. Agenda

10. Agenda • Talk about syntax highlighters ◦ Introduce the current

    syntax highlighter. ◦ Introduce the new syntax highlighter. ▪ I implement it

11. The current syntax highlighter for Vim

12. What's the current syntax highlighter?

13. :help syntax

14. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

15. For example:

16. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name .

    '-san!' endfunction Only two colors... Understanding syntax of the code is difficult.

17. :syntax on

18. hello.vim function! Hello(name) abort echom 'Hello, ' . a:name .

    '-san!' endfunction Colorful

19. Useful!

20. But… The current highlighter has problems

21. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

22. For example: synmaxcol • Vim gives up parsing very long

    line. ◦ The limitation is configurable by `synmaxcol`. • DEMO ◦ https://github.com/pocke/vimconf-2017/blob/mast er/synmaxcol.json

23. And… the highlighter has another problem

24. :help syntax Syntax highlighting enables Vim to show parts of

    the text in another font or color. Those parts can be specific keywords or text matching a pattern. Vim doesn't parse the whole file (to keep it fast), so the highlighting has its limitations. https://github.com/vim/vim/blob/1d68d9b2bd60d848552c08763e590edde16056c9/runtime/doc/syntax.txt#L9-L12

25. Some people, when confronted with a problem, think “I know,

    I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247

26. Regular expression is too difficult for humans.

27. For example: $ vim vim-ruby/syntax/ruby.vim • vim-ruby defines syntax with

    many regular expressions. • DEMO

28. Regexp has another problem

29. For example: complex code p(????::?:) How should the code be

    highlight?(DEMO)

30. Conclusion of the Current Syntax Highlighter • The syntax highlighter

    has limitations for performance. ◦ e.g.) On long long line, on large <script>. • The syntax highlighter uses keywords and regexp to highlight text. ◦ The regexp is very difficult, and does not perfect.

31. Commercial: SideCI

32. None

33. SideCI • Automated Code Review Platform. • Please access https://sideci.com

    ! • Sorry, we doesn't support Vim script…

34. Back to the topic

35. The new syntax highlighter for Vim

36. Iro.vim(いろ.びむ) https://github.com/pocke/iro.vim

37. Features of iro.vim • The new syntax highlight plugin for

    Vim. • Highlight perfectly with high performance • Understandable code • DEMO

38. Design of Iro.vim

39. Design • Use `matchaddpos()` instead of `:syntax`. • Use a

    parser that is implemented in the target language to parse perfectly.

40. How iro.vim to work: In Ruby case

41. Example: "Hello world" in Ruby def foo puts "Hello world!"

    end First, Iro.vim receives the code, and passes the code to `iro#ruby#tokens()`.

42. iro#ruby#tokens() code to tokens def foo puts "Hello world!" end

    [[[1, 0], :on_kw, "def"], [[1, 3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], Ripper.lex(code)

43. iro#ruby#tokens() tokens to highlights data [[[1, 0], :on_kw, "def"], [[1,

    3], :on_sp, " "], [[1, 4], :on_ident, "foo"], [[1, 7], :on_nl, "\n"], [[2, 0], :on_sp, " "], [[2, 2], :on_ident, "puts"], [[2, 6], :on_sp, " "], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello world!"], { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]] }

44. Iro.vim highlights with the data { "Keyword":[[1,1,3],[3,1,3]], "String":[[2,9,12]], "Delimiter":[[2,8,1], [2,21,1]]

    } def foo puts "Hello world!" end

45. Understandable code

46. A tour of iro.vim source code

47. autoload/iro.vim • Define iro#redraw() • load autoload/iro.vim.rb • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim

48. autoload/iro.vim.rb • call matchaddpos() each highlights. • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro.vim.rb

49. autoload/iro/ruby.vim.rb • parse a ruby file, and generate highlight data

    • https://github.com/pocke/iro.vim/blob/mast er/autoload/iro/ruby.vim.rb

50. High Performance?

51. I have good news and bad news.

52. Performance • Bad news: A performance issue exists ◦ Iro.vim

    is very slow on a ruby file that has 10,000+ lines(DEMO). • Good news: Parsing is not a bottleneck ◦ Vim is very slow when maybe many highlight is registered by `matchaddpos()`.

53. Current status of iro.vim

54. Current status • Still developing, but it works • Support

    few languages. ◦ Ruby, YAML and Python(experimental). • Welcome Pull-Requests

55. :endslide • Iro.vim is a new syntax highlighter. ◦ It

    highlights perfectly Thank you for listening