Slide 1
Slide 1 text
ѱຐͷͷώΞυΩϡϝϯτ
ۚࢠ༤Ұ(@yui-knk)
2018/07/14
Rails Developers Meetup 2018 Day 3 Extreme
Slide 2
Slide 2 text
Thank you !!
Slide 3
Slide 3 text
ࣗݾհ
• ۚࢠ ༤Ұ
• Treasure Data ॴଐ
• APIνʔϜ (RailsΞϓϦΛॻ͍͍ͯ·͢)
• CRuby Committer 2015/12~
• GitHub (yui-knk)
Slide 4
Slide 4 text
"1*νʔϜ 3BJMTΞϓϦΛॻ͍
͍ͯ·͢
Slide 5
Slide 5 text
https://www.treasuredata.com/company/careers/jobs/?team=Engineering
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
RubyͷΛ͠Α͏
Slide 9
Slide 9 text
ࠓͷτϐοΫ
• Ruby 2.6ͷ
• ѱຐͷͷ
• RubyͷώΞυΩϡϝϯτ
• parser͔ΒΈͨώΞυΩϡϝϯτ
Slide 10
Slide 10 text
ࠓͷτϐοΫ
• Ruby 2.6ͷ
• ѱຐͷͷ
• RubyͷώΞυΩϡϝϯτ
• parser͔ΒΈͨώΞυΩϡϝϯτ
• Railsͷͳ͍Ͱ͢
Slide 11
Slide 11 text
ࠓͷඪ
• RubyͷώΞυΩϡϝϯτʹ͍ͭͯཧղΛਂΊΔ
• parserʹڵຯΛͭ
Slide 12
Slide 12 text
Ruby 2.6ͷ
Slide 13
Slide 13 text
3VCZQSFWJFX
• 2018/05/31 ϦϦʔε
• JIT
• RubyVM::AST
Slide 14
Slide 14 text
3VCZ7."45
• ͱͱRubyʹ `--dump=parsetree` ͱ͍͏Φϓγϣϯ
͕͋Δ
• Rubyͷ෦తͳσʔλΛDump͢Δͷ
Slide 15
Slide 15 text
$ ruby --dump=p -e '1 + 2'
###########################################################
## Do NOT use this node dump for any purpose other than ##
## debug and research. Compatibility is not guaranteed. ##
###########################################################
# @ NODE_SCOPE (line: 1, location: (1,0)-(1,5))
# +- nd_tbl: (empty)
# +- nd_args:
# | (null node)
# +- nd_body:
# @ NODE_OPCALL (line: 1, location: (1,0)-(1,5))*
# +- nd_mid: :+
# +- nd_recv:
# | @ NODE_LIT (line: 1, location: (1,0)-(1,1))
# | +- nd_lit: 1
# +- nd_args:
# @ NODE_ARRAY (line: 1, location: (1,4)-(1,5))
# +- nd_alen: 1
# +- nd_head:
# | @ NODE_LIT (line: 1, location: (1,4)-(1,5))
# | +- nd_lit: 2
# +- nd_next:
# (null node)
Slide 16
Slide 16 text
3VCZ7."45
• ෦తͳσʔλΛRubyͷੈքͰѻ͑ΔΑ͏ʹ͢Δͷ
• Ruby 2.6͔Βಋೖ͞ΕΔ༧ఆ
• ಛʹ require ͳͲෆཁ
• ASTͷߏʹ͍ͭͯޓੑ͕อূ͞Εͳ͍
Slide 17
Slide 17 text
3VCZ7."45
RubyVM::AST.parse("1 + 2")
# => #
RubyVM::AST.parse("1 + 2").children[1].children
# => [
# #,
# #
# ]
Slide 18
Slide 18 text
3VCZUSVOL
• Syntax Error͕ग़ΔΑ͏ʹमਖ਼͞Εͨ
• #children ͕֦ு͞Εͨ
Slide 19
Slide 19 text
4ZOUBY&SSPS
$ ruby -e 'RubyVM::AST.parse("1 + ")'
Traceback (most recent call last):
1: from -e:1:in `'
-e:1:in `parse': no file name:1: syntax error,
unexpected end-of-input (SyntaxError)
Slide 20
Slide 20 text
DIJMESFO
# 2.6.0-preview2
RubyVM::AST.parse("1 + 2").children[1].children
# => [
# #,
# #
# ]
# trunk
RubyVM::AST.parse("1 + 2").children[2].children
# => [
# #,
# :+,
# #
# ]
mid
Slide 21
Slide 21 text
DIJMESFO
# trunk
RubyVM::AST.parse("def a; end").children[2].children
# => [
# :a,
# #
# ]
Slide 22
Slide 22 text
*OTUBMM
$ rbenv install 2.6.0-dev
Slide 23
Slide 23 text
ѱຐͷͷ
Slide 24
Slide 24 text
͋Δ͍ͳͥࠓώΞυΩϡ
ϝϯτͷΛ͢Δ͔
Slide 25
Slide 25 text
ѱຐ
• ѱຐ parse.y ͱ͍ΘΕΔϑΝΠϧ͕͋Δ
• https://slide.rabbit-shocker.org/authors/nobu/
rubykaigi-2017/
• parse.y Rubyͷจ๏Λఆ͍ٛͯ͠ΔϑΝΠϧ
• Bison ͱ͍͏πʔϧͰ Cݴޠͷίʔυʹม͢Δ
op : '|' { ifndef_ripper($$ = '|'); }
| '^' { ifndef_ripper($$ = '^'); }
| '&' { ifndef_ripper($$ = '&'); }
Slide 26
Slide 26 text
QBSTFZ
$ git shortlog -s -n parse.y | head -10
912 nobu
362 matz
140 mame
91 yui-knk
55 ko1
38 aamine
37 akr
33 naruse
25 usa
8 normal
Slide 27
Slide 27 text
ͳͥѱຐʁ
• Large file (10000ߦҎ্͋Δ)
• ripper standard library ͱϑΝΠϧΛڞ༗͍ͯ͠Δ
• “#ifdef RIPPER” macro
• stateful lexer
• here document
Slide 28
Slide 28 text
Ґஔใ
# trunk
RubyVM::AST.parse("1 + 2").children[2].children
# => [
# #,
# :+,
# #
# ]
Slide 29
Slide 29 text
https://rubykaigi.org/2018/presentations/spikeolaf.html#jun01
Slide 30
Slide 30 text
https://twitter.com/tagomoris/status/1005018203313979392
Slide 31
Slide 31 text
https://twitter.com/n0kada/status/1005030519334006785
Slide 32
Slide 32 text
https://twitter.com/n0kada/status/1005030519334006785
proc do<
Slide 33
Slide 33 text
3VCZJTU)PUMJOLT
ʲୈճʳ
def foo; < " In foo!\n"
p baz
# => " In baz!\n"
https://magazine.rubyist.net/articles/0042/0042-Hotlinks.html#ruby
%E3%81%AE%E7%BF%92%E5%BE%97
Slide 34
Slide 34 text
https://magazine.rubyist.net/articles/0042/0042-index.html
Slide 35
Slide 35 text
ͳͥώΞυΩϡϝϯτͷʁ
• parse.y Λཧղ͢ΔҰॿʹͳΔ
• ΤοδέʔεʹͳΓ͍͢
Slide 36
Slide 36 text
RubyͷώΞυΩϡϝϯτ
Slide 37
Slide 37 text
Do you know here
document?
Slide 38
Slide 38 text
3VCZͷώΞυΩϡϝϯτ
str = <
Slide 39
Slide 39 text
3VCZͷώΞυΩϡϝϯτ
str = <
Slide 40
Slide 40 text
3VCZͷώΞυΩϡϝϯτ
str = <
Slide 41
Slide 41 text
3VCZͷώΞυΩϡϝϯτ
• <
Slide 42
Slide 42 text
453
• υΩϡϝϯτͷΠϯσϯτͦͷ··
• ऴߦͷΠϯσϯτ͕ڐՄ͞Εͳ͍
puts <
Slide 43
Slide 43 text
453
puts < syntax error, unexpected end-of-input
Slide 44
Slide 44 text
453
• υΩϡϝϯτͷΠϯσϯτͦͷ··
• ऴߦ͕ΠϯσϯτՄೳ
puts <<-STR
abcde
fghij
STR
abcde
fghij
Slide 45
Slide 45 text
d453
• υΩϡϝϯτͷΠϯσϯτ͕Ұ൪ઙ͍ͱ͜Ζʹௐ͞Ε
Δ
• ऴߦ͕ΠϯσϯτՄೳ
puts <<~STR
abcde
fghij
STR
abcde
fghij
Slide 46
Slide 46 text
3VCZͷώΞυΩϡϝϯτ
• <<"STR" (ࣜల։͢Δ)
• <<'STR' (ࣜల։͠ͳ͍)
• <<`STR` (ίϚϯυ࣮ߦ)
puts <<"STR"
#{1 + 2}
STR
# => 3
puts <<'STR'
#{1 + 2}
STR
# => #{1 + 2}
puts <<`STR`
uname
STR
# => Darwin
Slide 47
Slide 47 text
ώΞυΩϡϝϯτ͕ศརͳͱ͜
• ώΞυΩϡϝϯτΛϨγʔόʹͰ͖Δ
• ώΞυΩϡϝϯτͷͳ͔ʹώΞυΩϡϝϯτ͕ॻ͚Δ
• 1ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
Slide 48
Slide 48 text
• STR1 ͕Ϩγʔό
• #upcase ͕ϝιου
ϨγʔόʹͰ͖Δ
p <
Slide 49
Slide 49 text
ͳ͔ʹώΞυΩϡϝϯτ͕ॻ͚Δ
p <
Slide 50
Slide 50 text
• String#+ จࣈྻ݁߹
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (<
Slide 51
Slide 51 text
• p ( ·ͰಡΉ
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (
# "This is str1.\nThis is str2.\n"
Slide 52
Slide 52 text
• <
Slide 53
Slide 53 text
• STR1 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (<
Slide 54
Slide 54 text
• + ΛಡΉ
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (<
Slide 55
Slide 55 text
• <
Slide 56
Slide 56 text
• STR2 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (<
Slide 57
Slide 57 text
• ) ΛಡΉ
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
p (<
Slide 58
Slide 58 text
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
• ͋ΔߦΛॲཧ͍ͯ͠Δ్தͰ࣍ͷߦʹ͍͚Δ
Slide 59
Slide 59 text
# +- nd_body:
# @ NODE_DASGN_CURR (line: 1, location: (1,0)-
(1,10))*
# +- nd_vid: :s
# +- nd_value:
# @ NODE_STR (line: 1, location: (1,4)-
(1,10))*
# +- nd_lit: "abcde\n fghij\n"
Ґஔใ
s = <<~STR
abcde
fghij
STR
Slide 60
Slide 60 text
༨ஊ (1)
Slide 61
Slide 61 text
࣮༻ྫ
• 1ߦʹෳώΞυΩϡϝϯτ
• จࣈྻల։தʹώΞυΩϡϝϯτ
https://github.com/ruby/ruby/blob/trunk/test/ruby/test_array.rb#L2010
Slide 62
Slide 62 text
༨ஊ (2)
Slide 63
Slide 63 text
https://twitter.com/mametter/status/936197656824619008
Slide 64
Slide 64 text
• pi π (ԁप)
• e ωΠϐΞ
$ ruby pie.rb
Traceback (most recent call last):
11: from pie.rb:3:in `'
10: from pie.rb:1:in `_2_'
9: from pie.rb:4:in `_7_'
8: from pie.rb:1:in `_1_'
7: from pie.rb:5:in `_8_'
6: from pie.rb:9:in `_2_'
5: from pie.rb:2:in `_8_'
4: from pie.rb:6:in `_1_'
3: from pie.rb:5:in `_8_'
2: from pie.rb:3:in `_2_'
1: from pie.rb:5:in `_8_'
pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5)
| |
pi e
Slide 65
Slide 65 text
• “Look_Forward_Ruby_2_5” exception ͷbacktrace
$ ruby pie.rb
Traceback (most recent call last):
11: from pie.rb:3:in `'
10: from pie.rb:1:in `_2_'
9: from pie.rb:4:in `_7_'
8: from pie.rb:1:in `_1_'
7: from pie.rb:5:in `_8_'
6: from pie.rb:9:in `_2_'
5: from pie.rb:2:in `_8_'
4: from pie.rb:6:in `_1_'
3: from pie.rb:5:in `_8_'
2: from pie.rb:3:in `_2_'
1: from pie.rb:5:in `_8_'
pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5)
| |
pi e
Slide 66
Slide 66 text
• e method name (easy)
• pi backtrace (strange)
$ ruby pie.rb
Traceback (most recent call last):
11: from pie.rb:3:in `'
10: from pie.rb:1:in `_2_'
9: from pie.rb:4:in `_7_'
8: from pie.rb:1:in `_1_'
7: from pie.rb:5:in `_8_'
6: from pie.rb:9:in `_2_'
5: from pie.rb:2:in `_8_'
4: from pie.rb:6:in `_1_'
3: from pie.rb:5:in `_8_'
2: from pie.rb:3:in `_2_'
1: from pie.rb:5:in `_8_'
pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5)
| |
pi e
Slide 67
Slide 67 text
• e method name (easy)
• pi backtrace (strange)
$ ruby pie.rb
Traceback (most recent call last):
11: from pie.rb:3:in `'
10: from pie.rb:1:in `_2_'
9: from pie.rb:4:in `_7_'
8: from pie.rb:1:in `_1_'
7: from pie.rb:5:in `_8_'
6: from pie.rb:9:in `_2_'
5: from pie.rb:2:in `_8_'
4: from pie.rb:6:in `_1_'
3: from pie.rb:5:in `_8_'
2: from pie.rb:3:in `_2_'
1: from pie.rb:5:in `_8_'
pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5)
| |
pi e
Slide 68
Slide 68 text
ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ
• ͋ΔߦΛॲཧ͍ͯ͠Δ్தͰ࣍ͷߦʹ͍͚Δ
Slide 69
Slide 69 text
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_;
_7_;end;end
module M2;def self._8_;M6._1_;end;end
module M3;def self._2_; M53._8_;end;end; <
Slide 70
Slide 70 text
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_;
_7_;end;end
module M2;def self._8_;M6._1_;end;end
module M3;def self._2_; M53._8_;end;end; <
Slide 71
Slide 71 text
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_;
_7_;end;end
module M2;def self._8_;M6._1_;end;end
module M3;def self._2_; M53._8_;end;end; <
Slide 72
Slide 72 text
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_;
_7_;end;end
module M2;def self._8_;M6._1_;end;end
module M3;def self._2_; M53._8_;end;end; <
Slide 73
Slide 73 text
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_;
_7_;end;end
module M2;def self._8_;M6._1_;end;end
module M3;def self._2_; M53._8_;end;end; <
Slide 74
Slide 74 text
parser͔ΒΈͨ
ώΞυΩϡϝϯτ
Slide 75
Slide 75 text
IFSFEPDNPEFͱ
• lex.strtermͷঢ়ଶͷ͜ͱ
static enum yytokentype
parser_yylex(struct parser_params *p)
{
...
if (p->lex.strterm) {
if (p->lex.strterm->flags & STRTERM_HEREDOC) {
return here_document(p, &p->lex.strterm->u.heredoc);
}
else {
token_flush(p);
return parse_string(p, &p->lex.strterm->u.literal);
}
}
...
Slide 76
Slide 76 text
ॏཁͳؔ
• heredoc_identifier
• heredoc modeʹೖΔ
• heredoc_restore
• heredoc mode͔Βग़Δ
Slide 77
Slide 77 text
IFSFEPD@JEFOUJpFS
• func: IndentQuoteͷใ
• term_len: <
Slide 78
Slide 78 text
• func: 0x22 (INDENT & EXPAND)
p <<-STR1.upcase
This is str1.
STR1
Slide 79
Slide 79 text
• func: 0x22 (INDENT & EXPAND)
• term_len: 3
• term: STR1
p <<-STR1.upcase
This is str1.
STR1
term_len term
Slide 80
Slide 80 text
• func: 0x22 (INDENT & EXPAND)
• term_len: 3
• term: STR1
p <<-STR1.upcase
This is str1.
STR1
Ґஔใ = term_len + term
Slide 81
Slide 81 text
• func: 0x22 (INDENT & EXPAND)
• term_len: 3
• term: STR1
• lastidx: 9
• ruby_sourceline: 1
p <<-STR1.upcase
This is str1.
STR1
lastidx
lastline
Slide 82
Slide 82 text
• func: 0x22 (INDENT & EXPAND)
• term_len: 3
• term: STR1
• lastidx: 9
• ruby_sourceline: 1
p <<-STR1.upcase
This is str1.
STR1
lastidx
lastline
Slide 83
Slide 83 text
• ҎԼͷ3ͭ1ͭͷStringΦϒδΣΫτʹͳ͍ͬͯΔ
• term_len: 3
• func: 0x22 (INDENT & EXPAND)
• term: STR1
p <<-STR1.upcase
This is str1.
STR1
term_len term
3 0x22 STR1
·ΓͲ͜Ζ
String
Slide 84
Slide 84 text
·ͱΊ
• Ruby 2.6Ͱ RubyVM::AST ͕ಋೖ͞ΕΔ༧ఆ
• RubyͷώΞυΩϡϝϯτػೳ͕ͨ͘͞Μ͋Δ
• ώΞυΩϡϝϯτΤοδέʔεʹͳΓ͍͢
Slide 85
Slide 85 text
Thank you !!