Tweet
Tweet

Slide 1

Slide 1 text

A ripper based syntax highlighter Regional RubyKaigi in Okinawa #2

Slide 2

Slide 2 text

self.inspect ● Masataka Pocke Kuwabara ● SideCI, Inc. ● ”Rubyドキドキ調査隊” is serialized in WEB+DB PRESS (vol. 102~) ● RuboCop’s core developer ● Vimmer

Slide 3

Slide 3 text

I love... ● Complex Ruby code (NOT FOR PRODUCTION!) ○ Complex Ruby code can break RuboCop or Parser, it’s fun ○ e.g. def __END__() end will break Parser. ■ This bug has been fixed. ● Vim. ○ Vim is a powerful, useful and pluggable editor.

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

I told about a syntax highlighter in VimConf 2017 https://speakerdeck.com/pocke/the-new-syntax-highlighter-for-vim

Slide 7

Slide 7 text

In the VimConf ● I told about “Iro.vim”. ● It is a Vim plugin. ● https://github.com/pocke/iro.vim ● This is a next generation Syntax Highlighter for Vim. ● It is written in Ruby mainly.

Slide 8

Slide 8 text

BTW… I’m talking about Vim, but today is not “VimConf”!

Slide 9

Slide 9 text

Today is...

Slide 10

Slide 10 text

So, I should talk about Ruby, “not Vim”.

Slide 11

Slide 11 text

In the VimConf ● I told about “Iro.vim”. ● It is a Vim plugin. ● https://github.com/pocke/iro.vim ● This is a next generation Syntax Highlighter for Vim. ● It is written in Ruby mainly.

Slide 12

Slide 12 text

Agenda ● Problems of Current syntax highlighter. ● How does Iro solve the problems. ● The feature of Iro.

Slide 13

Slide 13 text

Question ● Do you know AST? ○ Abstract Syntax Tree ● Do you know Ripper? ○ Parser for Ruby.

Slide 14

Slide 14 text

Problems of Current syntax highlighter

Slide 15

Slide 15 text

Syntax highlighters have two problems ● Hard to read code of highlighters. ● They does not highlight correctly.

Slide 16

Slide 16 text

Problem: Hard to Read

Slide 17

Slide 17 text

Highlighters are written in Regexp ● Regular Expression is difficult. ○ Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. Jamie Zawinski http://regex.info/blog/2006-09-15/247

Slide 18

Slide 18 text

Do you understand this easily? vim-ruby’s syntax definition.

Slide 19

Slide 19 text

Problem: not correct

Slide 20

Slide 20 text

Problem: not correct ● Some highlighters cannot highlight the following code correctly. ○ ?????:?: ○ % %s%% %%%% ○ def def(def:def def;end)end

Slide 21

Slide 21 text

Examples: gist

Slide 22

Slide 22 text

Example: Vim(default)

Slide 23

Slide 23 text

How does Iro solve the problems.

Slide 24

Slide 24 text

What’s Iro? ● A library for Ripper based syntax highlighter. ● https://github.com/pocke/iro ● It solves the problems. ● Live demo: https://ruby-highlight.herokuapp.com/

Slide 25

Slide 25 text

Easy to read Iro’s code

Slide 26

Slide 26 text

Iro is written in Ruby ● If you can read Ruby, you can read Iro. ● It uses Ripper as a tokenizser and parser. Let’s read Iro code and understand how Iro work.

Slide 27

Slide 27 text

Tokenize

Slide 28

Slide 28 text

Definition of Ripper event name to highlight

Slide 29

Slide 29 text

Hook ripper event

Slide 30

Slide 30 text

Example def hello puts "Hello" end

Slide 31

Slide 31 text

Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], [[1, 4], :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], [[2, 8], :on_tstring_content, "Hello"], [[2, 21], :on_tstring_end, "\""], [[3, 0], :on_kw, "end"]]

Slide 32

Slide 32 text

Tokenize code with Ripper.lex() [[[1, 0], :on_kw, "def"], # Define [[1, 4], :on_ident, "hello"], [[2, 2], :on_ident, "puts"], [[2, 7], :on_tstring_beg, "\""], # Delimiter [[2, 8], :on_tstring_content, "Hello"], # String [[2, 21], :on_tstring_end, "\""], # Delimiter [[3, 0], :on_kw, "end"]] # Keyword

Slide 33

Slide 33 text

Parse

Slide 34

Slide 34 text

Parse code with Ripper.sexp() ● Tokenizing is good, but we need parsing to get more information. ○ e.g. local variable, `def def() end`, symbol literal and so on.

Slide 35

Slide 35 text

Example: highlight :foo ● Tokenize is not enough. It is just an ident. ○ [[[1, 0], :on_symbeg, ":", EXPR_FNAME], [[1, 1], :on_ident, "foo", EXPR_ENDFN]] ● Parsing is enough. It is a symbol. ○ [:program, [[:symbol_literal, [:symbol, [:@ident, "foo", [1, 1]]]]]]

Slide 36

Slide 36 text

Hook ripper parser event

Slide 37

Slide 37 text

Iro can highlight correctly

Slide 38

Slide 38 text

Iro can highlight correctly. ● Iro uses Ripper to tokenize and parse code. ● Ripper is the MRI parser. ● So Iro can highlight code same as MRI.

Slide 39

Slide 39 text

Before After

Slide 40

Slide 40 text

The feature of Iro

Slide 41

Slide 41 text

Inline highlighting ● Highlight code in code ○ For example: # Here is Ruby code <<~SQL # Here is SQL SELECT * FROM users; SQL

Slide 42

Slide 42 text

More languages support ● Currently Iro.vim supports Ruby, YAML and Python. ● I would like to add support Slim, Markdown.

Slide 43

Slide 43 text

More editors support ● Currently Iro supports Vim and HTML only. ● I believe we can use Iro in other editors. ○ e.g. Emacs, Atom or something.

Slide 44

Slide 44 text

More token support ● Currently Iro does not highlight symbol literals, instance variables, class variables and so on.

Slide 45

Slide 45 text

Conclusion

Slide 46

Slide 46 text

Conclusion ● Iro is a Ripper based syntax highlighter. ○ It can highlight correctly. ● You can try using Iro now! ○ For Vimmer: https://github.com/pocke/iro.vim ○ Web demo: https://ruby-highlight.herokuapp.com Thank you for listening!