A Ripper based syntax highlighter

Regional RubyKaigi in Okinawa 2
http://ruby.okinawa/okrk02/

Transcript

1. A ripper based syntax highlighter Regional RubyKaigi in Okinawa #2
2. self.inspect • Masataka Pocke Kuwabara • SideCI, Inc. • ”Rubyドキドキ調査隊” is serialized in WEB+DB PRESS (vol. 102~) • RuboCop’s core developer • Vimmer
3. I love... • Complex Ruby code (NOT FOR PRODUCTION!) ◦ Complex Ruby code can break RuboCop or Parser, it’s fun ◦ e.g. def __END__() end will break Parser. ▪ This bug has been fixed. • Vim. ◦ Vim is a powerful, useful and pluggable editor.
6. I told about a syntax highlighter in VimConf 2017 https://speakerdeck.com/pocke/the-new-syntax-highlighter-for-vim
7. In the VimConf • I told about “Iro.vim”. • It is a Vim plugin. • https://github.com/pocke/iro.vim • This is a next generation Syntax Highlighter for Vim. • It is written in Ruby mainly.
8. BTW… I’m talking about Vim, but today is not “VimConf”!
9. Today is...
10. So, I should talk about Ruby, “not Vim”.
11. In the VimConf • I told about “Iro.vim”. • It is a Vim plugin. • https://github.com/pocke/iro.vim • This is a next generation Syntax Highlighter for Vim. • It is written in Ruby mainly.
12. Agenda • Problems of Current syntax highlighter. • How does Iro solve the problems. • The feature of Iro.
13. Question • Do you know AST? ◦ Abstract Syntax Tree • Do you know Ripper? ◦ Parser for Ruby.
14. Problems of Current syntax highlighter
15. Syntax highlighters have two problems • Hard to read code of highlighters. • They does not highlight correctly.
16. Problem: Hard to Read
17. Highlighters are written in Regexp • Regular Expression is difficult. ◦ Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247
18. Do you understand this easily? vim-ruby’s syntax definition.
19. Problem: not correct
20. Problem: not correct • Some highlighters cannot highlight the following code correctly. ◦ ?????:?: ◦ % %s%% %%%% ◦ def def(def:def def;end)end
21. Examples: gist
22. Example: Vim(default)
23. How does Iro solve the problems.
24. What’s Iro? • A library for Ripper based syntax highlighter. • https://github.com/pocke/iro • It solves the problems. • Live demo: https://ruby-highlight.herokuapp.com/
25. Easy to read Iro’s code
26. Iro is written in Ruby • If you can read Ruby, you can read Iro. • It uses Ripper as a tokenizser and parser. Let’s read Iro code and understand how Iro work.
27. Tokenize
28. Definition of Ripper event name to highlight
29. Hook ripper event
30. Example def hello puts "Hello" end
31. Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], [[1, 4], :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello"], [[2, 21], :on_tstring_end, "\""], [[3, 0], :on_kw, "end"]]
32. Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], # Define [[1, 4], :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], # Delimiter [[2, 8], :on_tstring_content, "Hello"], # String [[2, 21], :on_tstring_end, "\""], # Delimiter [[3, 0], :on_kw, "end"]] # Keyword
33. Parse
34. Parse code with Ripper.sexp() • Tokenizing is good, but we need parsing to get more information. ◦ e.g. local variable, `def def() end`, symbol literal and so on.
35. Example: highlight :foo • Tokenize is not enough. It is just an ident. ◦ [[[1, 0], :on_symbeg, ":", EXPR_FNAME], [[1, 1], :on_ident, "foo", EXPR_ENDFN]] • Parsing is enough. It is a symbol. ◦ [:program, [[:symbol_literal, [:symbol, [:@ident, "foo", [1, 1]]]]]]
36. Hook ripper parser event
37. Iro can highlight correctly
38. Iro can highlight correctly. • Iro uses Ripper to tokenize and parse code. • Ripper is the MRI parser. • So Iro can highlight code same as MRI.
39. Before After
40. The feature of Iro
41. Inline highlighting • Highlight code in code ◦ For example: # Here is Ruby code <<~SQL # Here is SQL SELECT * FROM users; SQL
42. More languages support • Currently Iro.vim supports Ruby, YAML and Python. • I would like to add support Slim, Markdown.
43. More editors support • Currently Iro supports Vim and HTML only. • I believe we can use Iro in other editors. ◦ e.g. Emacs, Atom or something.
44. More token support • Currently Iro does not highlight symbol literals, instance variables, class variables and so on.
45. Conclusion
46. Conclusion • Iro is a Ripper based syntax highlighter. ◦ It can highlight correctly. • You can try using Iro now! ◦ For Vimmer: https://github.com/pocke/iro.vim ◦ Web demo: https://ruby-highlight.herokuapp.com Thank you for listening!