Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Ripper based syntax highlighter

7bc6612fa20296bf652f6b0357db81c1?s=47 pocke
March 10, 2018

A Ripper based syntax highlighter

Regional RubyKaigi in Okinawa 2
http://ruby.okinawa/okrk02/

7bc6612fa20296bf652f6b0357db81c1?s=128

pocke

March 10, 2018
Tweet

Transcript

  1. A ripper based syntax highlighter Regional RubyKaigi in Okinawa #2

  2. self.inspect • Masataka Pocke Kuwabara • SideCI, Inc. • ”Rubyドキドキ調査隊”

    is serialized in WEB+DB PRESS (vol. 102~) • RuboCop’s core developer • Vimmer
  3. I love... • Complex Ruby code (NOT FOR PRODUCTION!) ◦

    Complex Ruby code can break RuboCop or Parser, it’s fun ◦ e.g. def __END__() end will break Parser. ▪ This bug has been fixed. • Vim. ◦ Vim is a powerful, useful and pluggable editor.
  4. None
  5. None
  6. I told about a syntax highlighter in VimConf 2017 https://speakerdeck.com/pocke/the-new-syntax-highlighter-for-vim

  7. In the VimConf • I told about “Iro.vim”. • It

    is a Vim plugin. • https://github.com/pocke/iro.vim • This is a next generation Syntax Highlighter for Vim. • It is written in Ruby mainly.
  8. BTW… I’m talking about Vim, but today is not “VimConf”!

  9. Today is...

  10. So, I should talk about Ruby, “not Vim”.

  11. In the VimConf • I told about “Iro.vim”. • It

    is a Vim plugin. • https://github.com/pocke/iro.vim • This is a next generation Syntax Highlighter for Vim. • It is written in Ruby mainly.
  12. Agenda • Problems of Current syntax highlighter. • How does

    Iro solve the problems. • The feature of Iro.
  13. Question • Do you know AST? ◦ Abstract Syntax Tree

    • Do you know Ripper? ◦ Parser for Ruby.
  14. Problems of Current syntax highlighter

  15. Syntax highlighters have two problems • Hard to read code

    of highlighters. • They does not highlight correctly.
  16. Problem: Hard to Read

  17. Highlighters are written in Regexp • Regular Expression is difficult.

    ◦ Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247
  18. Do you understand this easily? vim-ruby’s syntax definition.

  19. Problem: not correct

  20. Problem: not correct • Some highlighters cannot highlight the following

    code correctly. ◦ ?????:?: ◦ % %s%% %%%% ◦ def def(def:def def;end)end
  21. Examples: gist

  22. Example: Vim(default)

  23. How does Iro solve the problems.

  24. What’s Iro? • A library for Ripper based syntax highlighter.

    • https://github.com/pocke/iro • It solves the problems. • Live demo: https://ruby-highlight.herokuapp.com/
  25. Easy to read Iro’s code

  26. Iro is written in Ruby • If you can read

    Ruby, you can read Iro. • It uses Ripper as a tokenizser and parser. Let’s read Iro code and understand how Iro work.
  27. Tokenize

  28. Definition of Ripper event name to highlight

  29. Hook ripper event

  30. Example def hello puts "Hello" end

  31. Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], [[1, 4],

    :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello"], [[2, 21], :on_tstring_end, "\""], [[3, 0], :on_kw, "end"]]
  32. Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], # Define

    [[1, 4], :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], # Delimiter [[2, 8], :on_tstring_content, "Hello"], # String [[2, 21], :on_tstring_end, "\""], # Delimiter [[3, 0], :on_kw, "end"]] # Keyword
  33. Parse

  34. Parse code with Ripper.sexp() • Tokenizing is good, but we

    need parsing to get more information. ◦ e.g. local variable, `def def() end`, symbol literal and so on.
  35. Example: highlight :foo • Tokenize is not enough. It is

    just an ident. ◦ [[[1, 0], :on_symbeg, ":", EXPR_FNAME], [[1, 1], :on_ident, "foo", EXPR_ENDFN]] • Parsing is enough. It is a symbol. ◦ [:program, [[:symbol_literal, [:symbol, [:@ident, "foo", [1, 1]]]]]]
  36. Hook ripper parser event

  37. Iro can highlight correctly

  38. Iro can highlight correctly. • Iro uses Ripper to tokenize

    and parse code. • Ripper is the MRI parser. • So Iro can highlight code same as MRI.
  39. Before After

  40. The feature of Iro

  41. Inline highlighting • Highlight code in code ◦ For example:

    # Here is Ruby code <<~SQL # Here is SQL SELECT * FROM users; SQL
  42. More languages support • Currently Iro.vim supports Ruby, YAML and

    Python. • I would like to add support Slim, Markdown.
  43. More editors support • Currently Iro supports Vim and HTML

    only. • I believe we can use Iro in other editors. ◦ e.g. Emacs, Atom or something.
  44. More token support • Currently Iro does not highlight symbol

    literals, instance variables, class variables and so on.
  45. Conclusion

  46. Conclusion • Iro is a Ripper based syntax highlighter. ◦

    It can highlight correctly. • You can try using Iro now! ◦ For Vimmer: https://github.com/pocke/iro.vim ◦ Web demo: https://ruby-highlight.herokuapp.com Thank you for listening!