A parser based syntax highlighter

RubyKaigi 2018
May 31st

Transcript

1. A parser based syntax highlighter RubyKaigi 2018 May 31st
2. self.inspect • Masataka Pocke Kuwabara • Software Engineer for Cookpad Inc. • Technical Advisor for SideCi, Inc. • A Vimmer
3. Previous talks
5. I told about a syntax highlighter in VimConf 2017 https://speakerdeck.com/pocke/the-new-syntax-highlighter-for-vim
6. In the VimConf • I told “Iro.vim”. ◦ “Iro” means “color” in English • https://github.com/pocke/iro.vim • It is a Vim plugin for Syntax Highlighting. • It is written in Ruby mainly.
7. Regional Rubykaigi in Okinawa #2 http://ruby.okinawa/okrk02/
8. Regional Rubykaigi in Okinawa #2 • I told the same theme in the conference. ◦ VimConf: For Vim users. ◦ Okinawa RubyKaigi: For Ruby users.
9. Today
10. Today: RubyKaigi 2018 • I’m talking the same content (Iro) as Okinawa, but it is deeper.
11. Today’s Agenda • Existing syntax highlighters implementation. • Problems of Existing syntax highlighter. • Introduce Iro. • Advantage of Iro. • Implementation of Iro. • The future of Iro.
12. Existing Syntax Highlighters Implementation
13. Regular Explession • Many syntax highlighters are implemented with regular expressions. ◦ Vim, Emacs, Atom, etcetc...
14. Explore Highlighter definitions for Ruby • Look Atom's highlighter definitions for Ruby code. • Atom uses “cson” for highlighter definition. ◦ “cson” is “CoffeeScript-Object-Notation”
15. atom/language-ruby: method definition • https://github.com/atom/language-ruby
16. Problems of Existing syntax highlighters
17. Syntax highlighters have two problems • Hard to read code of highlighters. • They does not highlight correctly.
18. Problem: Hard to Read
19. Regular Expression is difficult. • Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247
20. Do you understand the definitions easily? • https://github.com/vim-ruby/vim-ruby/blob/m aster/syntax/ruby.vim • https://github.com/atom/language-ruby/blob/ master/grammars/ruby.cson
21. Problem: not correct
22. Existing highlighters mistakes highlight sometimes • If highlighting is broken, we cannot understand meaning of the code easily.
23. Example: broken highlighting • It is hard to understand.
24. Other examples • Complex here document ◦ e.g. Here document in a string interpolation in a here document • Tricky code ◦ e.g. ????::?:, % %s%% %%%% or def end(def:def def;end)end ◦ You can find them from my CFP. ▪ http://pocke.hatenablog.com/entry/2018/05/27/152708
25. Why do they mistake highlighting? • They re-implement parser. ◦ So the implementations have difference between highlighter and language parser. • Regexp is not enough to parse programing languages. ◦ Regexp: for Regular Language ◦ Many Programing Languages: Context Free Language ◦ But many editor's implementations extend regexp, so probably it's enough.
26. Introduce Iro
27. Introduce Iro • Iro is a gem that is Ripper based syntax highlighter. • https://github.com/pocke/iro • $ gem install iro • Live demo: https://ruby-highlight.herokuapp.com/
28. What's Ripper? • A Ruby standard library. ◦ So you can use Ripper without gem install. • A Parser of Ruby. • It shares parse.y with Ruby interpretor. ◦ So Ripper understands Ruby syntax correctly.
29. Difference of Iro and Iro.vim • Iro is a gem. ◦ Ruby code -> Iro -> highlighting information • Iro.vim is a Vim plugin. ◦ Iro.vim passes Ruby code to Iro, and gets the information from Iro. ◦ Iro.vim has Python and YAML support also.
30. Iro.vim When code is changed Highlight Iro Parse the code Ruby source code Highlight positions
31. Advantages of Iro
32. Advantages of Iro • Easy to read Iro's code. • Iro can highlight correctly. • Highlighting local variables. • One implementation for many editors.
33. Easy to read Iro’s code
34. Iro is written in Ruby • Iro uses Ripper instead of regexp. • So I do not need re-implement parser with regular expression. • Detachment of Parser from Syntax Highlighter.
35. Iro can highlight correctly
36. Iro can highlight correctly. • Iro uses Ripper. • Ripper is the CRuby parser. • So Iro can highlight code same as CRuby.
37. Highlighting local variables
38. Iro highlights local variables • Iro can highlight local variables. ◦ Vim, Atom and VSCode don’t have this feature, but RubyMine can highlight them. Iro Existing Highlighter
39. One implementation for many editors
40. One implementation for many editors • Each existing syntax highlighter has a different regexp evaluator. • So each highlighter should define different syntax definition.
41. One implementation for many editors • Iro is a gem, and Iro is a protocol. • So you can implement a highlighter for your editor with the gem! • But currently Vim implementation only exists.
42. Iro.vim and Iro Iro.vim When code is changed Highlight Iro Parse the code Ruby source code Highlight positions
43. Iro emacs and Iro Iro Emacs When code is changed Highlight Iro Parse the code Ruby source code Highlight positions
44. Iro VSCode and Iro Iro VSCode When code is changed Highlight Iro Parse the code Ruby source code Highlight positions
45. Iro.vim and Iro.py Iro VSCode When code is changed Highlight Iro.py Parse the code Python source code Highlight positions
46. Implementation of Iro
47. Highlighting hello.rb by Iro def hello(name) puts "hello, #{name}" end
48. Understanding Ripper behaviour • Let know Ripper behaviour with Ripper.lex and Ripper.sexp methods. ◦ Ripper.lex: Tokenize with meta information ◦ Ripper.sexp: Parser to S-Expression • And Event driven API.
49. Ripper.lex
50. Ripper.lex $ ruby -rripper -e 'pp Ripper.lex(File.read("hello.rb"))' [[[1, 0], :on_kw, "def", EXPR_FNAME], [[1, 3], :on_sp, " ", EXPR_FNAME], [[1, 4], :on_ident, "hello", EXPR_ENDFN], [[1, 9], :on_lparen, "(", EXPR_BEG|EXPR_LABEL], [[1, 10], :on_ident, "name", EXPR_ARG], [[1, 14], :on_rparen, ")", EXPR_ENDFN], [[1, 15], :on_ignored_nl, "\n", EXPR_BEG], [[2, 0], :on_sp, " ", EXPR_BEG], [[2, 2], :on_ident, "puts", EXPR_CMDARG], [[2, 6], :on_sp, " ", EXPR_CMDARG], [[2, 7], :on_tstring_beg, "\"", EXPR_CMDARG], [[2, 8], :on_tstring_content, "hello, ", EXPR_CMDARG],
51. Ripper.lex output [[[1, 0], :on_kw, "def", EXPR_FNAME], [[1, 3], :on_sp, " ", EXPR_FNAME], [[1, 4], :on_ident, "hello", EXPR_ENDFN], … • It is an array of ◦ Position ◦ Scanner event name ▪ pp Ripper::SCANNER_EVENTS ◦ source code ◦ lex state (since Ruby 2.5)
52. What does Iro uses? • Iro uses the position and source code for highlighting position. • Iro uses the event name for highlighting group. Example: [[1, 0], :on_kw, "def", EXPR_FNAME] :on_kw -> Iro highlight it as Keyword. [1, 0] and "def" -> Iro highlight line 1, column 0, size 3.
53. Ripper.sexp
54. Ripper.sexp $ ruby -rripper -e 'pp Ripper.sexp(File.read("test.rb"))' [:program, [[:def, [:@ident, "hello", [1, 4]], [:paren, [:params, [[:@ident, "name", [1, 10]]], nil, nil, nil, nil, nil, nil]], [:bodystmt, [[:command, [:@ident, "puts", [2, 2]], [:args_add_block, [[:string_literal, [:string_content, [:@tstring_content, "hello, ", [2, 8]],
55. Ripper.sexp • It is a S-expression. • [:TYPE, child_sexp1, child_sexp2, ...] • child_sexp is a S-expression or a lex output.
56. Event driven Ripper API
57. Iro does not use #sexp and #lex • I described them, but Iro does not use them. • Iro uses event driven API instead.
58. Event Driven Ripper API • Ripper provides event driven API. • Ripper calls on_TYPE method when it visits TYPE. ◦ e.g. Ripper call on_kw method when it reads kw token.
59. Example: scanner event require 'ripper' class MyRipper < Ripper def on_kw(tok) puts "type: kw, source: #{tok}, position: #{lineno}, #{column}" end end MyRipper.parse(ARGF.read)
60. Output $ ruby myripper.rb hello.rb type: kw, source: def, position: 1, 0 type: kw, source: end, position: 3, 0
61. Example: parser event require 'ripper' class MyRipper < Ripper::SexpBuilderPP def on_def(name, params, body) p name p params p body end end MyRipper.parse(ARGF.read)
62. Output $ ruby test.rb hello.rb [:@ident, "hello", [1, 4]] [:paren, [:params, [[:@ident, "name", [1, 10]]], nil, nil, nil, nil, nil, nil]] [:bodystmt, [[:command, [:@ident, "puts", [2, 2]], [:args_add_block, [[:string_literal, [:string_content, [:@tstring_content, "hello, ", ...
63. Implementation of Iro
64. See pocke/iro/lib/iro/ruby/parser.rb https://github.com/pocke/iro/blob/master/lib/iro/ ruby/parser.rb
65. The future of Iro
66. Inline highlighting • Highlight code in code ◦ For example: # Here is Ruby code <<~SQL # Here is SQL SELECT * FROM users; SQL
67. More languages support • Currently Iro.vim supports Ruby, YAML and Python. ◦ “Iro.vim” has python and YAML support, so I’d like to extract the implementation to a gem or something. • I’d like to add support Slim, Markdown. ◦ Because Slim is a difficult language. ◦ Markdown has inline code block.
68. More editors support • Currently Iro supports Vim and HTML only. • I believe we can use Iro in other editors. ◦ e.g. Emacs, Atom or something.
69. FAQ
70. How about performance? • I can use Iro.vim comfortable. ◦ ⭕ 3,000 lines ◦ ❌ 30,000 lines • But I haven't compare performance with other implementations. ◦ Because I'm not sure how to compare highlighter's performance.
71. Does Iro work on broken Ruby file? • Code has syntax errors while editing. • So syntax highlighters should be able to highlight broken code. • Iro can highlight in almost cases.
72. Example • It has a syntax error, but Iro highlight it.
73. Conclusion
74. Conclusion • Iro is a Ripper based syntax highlighter. ◦ It can highlight correctly. • You can try using Iro now! ◦ For Vimmer: https://github.com/pocke/iro.vim ◦ Web demo: https://ruby-highlight.herokuapp.com Thank you for listening!
75. Events at our Booth 【Day 2】 12:00~13:00 Q&A with @pocke 15:20~15:50 Global Office Hours 【Day 3】 12:00~13:00 Q&A with @wyhaines 15:20~15:50 Ruby interpreter development live by @ko1 & @mame Cookpad X RubyKaigi 2018: Day 2 Party ⏰ June 1st, 19:30 - 21:30 (opens 19:00) Free (Registration required) Show up to this booth at 18:40 if you want to head with us!