Share
Embed
Heredoc In Demon Castle
yui-knk
July 14, 2018
Transcript
85 slides
-
ࣗݾհ • ۚࢠ ༤Ұ • Treasure Data ॴଐ • APIνʔϜ
(RailsΞϓϦΛॻ͍͍ͯ·͢) • CRuby Committer 2015/12~ • GitHub (yui-knk) -
$ ruby --dump=p -e '1 + 2' ########################################################### ## Do
NOT use this node dump for any purpose other than ## ## debug and research. Compatibility is not guaranteed. ## ########################################################### # @ NODE_SCOPE (line: 1, location: (1,0)-(1,5)) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_OPCALL (line: 1, location: (1,0)-(1,5))* # +- nd_mid: :+ # +- nd_recv: # | @ NODE_LIT (line: 1, location: (1,0)-(1,1)) # | +- nd_lit: 1 # +- nd_args: # @ NODE_ARRAY (line: 1, location: (1,4)-(1,5)) # +- nd_alen: 1 # +- nd_head: # | @ NODE_LIT (line: 1, location: (1,4)-(1,5)) # | +- nd_lit: 2 # +- nd_next: # (null node) -
3VCZ7."45 • ෦తͳσʔλΛRubyͷੈքͰѻ͑ΔΑ͏ʹ͢Δͷ • Ruby 2.6͔Βಋೖ͞ΕΔ༧ఆ • ಛʹ require ͳͲෆཁ
• ASTͷߏʹ͍ͭͯޓੑ͕อূ͞Εͳ͍ -
3VCZ7."45 RubyVM::AST.parse("1 + 2") # => #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:5): >
RubyVM::AST.parse("1 + 2").children[1].children # => [ # #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] -
4ZOUBY&SSPS $ ruby -e 'RubyVM::AST.parse("1 + ")' Traceback (most recent
call last): 1: from -e:1:in `<main>' -e:1:in `parse': no file name:1: syntax error, unexpected end-of-input (SyntaxError) -
DIJMESFO # 2.6.0-preview2 RubyVM::AST.parse("1 + 2").children[1].children # => [ #
#<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] # trunk RubyVM::AST.parse("1 + 2").children[2].children # => [ # #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # :+, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] mid -
DIJMESFO # trunk RubyVM::AST.parse("def a; end").children[2].children # => [ #
:a, # #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:10): > # ] -
ѱຐ • ѱຐ parse.y ͱ͍ΘΕΔϑΝΠϧ͕͋Δ • https://slide.rabbit-shocker.org/authors/nobu/ rubykaigi-2017/ • parse.y
Rubyͷจ๏Λఆ͍ٛͯ͠ΔϑΝΠϧ • Bison ͱ͍͏πʔϧͰ Cݴޠͷίʔυʹม͢Δ op : '|' { ifndef_ripper($$ = '|'); } | '^' { ifndef_ripper($$ = '^'); } | '&' { ifndef_ripper($$ = '&'); } -
QBSTFZ $ git shortlog -s -n parse.y | head -10
912 nobu 362 matz 140 mame 91 yui-knk 55 ko1 38 aamine 37 akr 33 naruse 25 usa 8 normal -
ͳͥѱຐʁ • Large file (10000ߦҎ্͋Δ) • ripper standard library ͱϑΝΠϧΛڞ༗͍ͯ͠Δ
• “#ifdef RIPPER” macro • stateful lexer • here document -
Ґஔใ # trunk RubyVM::AST.parse("1 + 2").children[2].children # => [ #
#<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # :+, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] -
3VCZJTU)PUMJOLT ʲୈճʳ def foo; <<bar end; def baz; <<quux end
In foo! bar In baz! quux p foo # => " In foo!\n" p baz # => " In baz!\n" https://magazine.rubyist.net/articles/0042/0042-Hotlinks.html#ruby %E3%81%AE%E7%BF%92%E5%BE%97 -
3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source
programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write. -
3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source
programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write. -
3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source
programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write. -
453 puts <<STR abcde fghij STR # => syntax error,
unexpected end-of-input -
3VCZͷώΞυΩϡϝϯτ • <<"STR" (ࣜల։͢Δ) • <<'STR' (ࣜల։͠ͳ͍) • <<`STR` (ίϚϯυ࣮ߦ)
puts <<"STR" #{1 + 2} STR # => 3 puts <<'STR' #{1 + 2} STR # => #{1 + 2} puts <<`STR` uname STR # => Darwin -
• STR1 ͕Ϩγʔό • #upcase ͕ϝιου ϨγʔόʹͰ͖Δ p <<STR1.upcase This
is str1. STR1 # "THIS IS STR1.\n" -
ͳ͔ʹώΞυΩϡϝϯτ͕ॻ͚Δ p <<STR1 This is str1. #{<<STR2}. 12345 This is
str2. STR2 ABCDE STR1 # "This is str1.\nThis is str2.\n. 12345\nABCDE\n" -
• String#+ จࣈྻ݁߹ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2) This is
str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n" -
• p ( ·ͰಡΉ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p ( # "This is
str1.\nThis is str2.\n" -
• STR1 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 This is str1. STR1
# "This is str1.\nThis is str2.\n" -
• + ΛಡΉ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + This is str1.
STR1 # "This is str1.\nThis is str2.\n" -
• <<STR2 ΛಡΉͱɺheredocΛಡΈͱΔঢ়ଶʹͳΔ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2 This is
str1. STR1 # "This is str1.\nThis is str2.\n" -
• STR2 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2 This is
str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n" -
• ) ΛಡΉ ߦʹෳͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2) This is
str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n" -
# +- nd_body: # @ NODE_DASGN_CURR (line: 1, location: (1,0)-
(1,10))* # +- nd_vid: :s # +- nd_value: # @ NODE_STR (line: 1, location: (1,4)- (1,10))* # +- nd_lit: "abcde\n fghij\n" Ґஔใ s = <<~STR abcde fghij STR -
• pi π (ԁप) • e ωΠϐΞ $ ruby
pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e -
• “Look_Forward_Ruby_2_5” exception ͷbacktrace $ ruby pie.rb Traceback (most recent
call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e -
• e method name (easy) • pi backtrace
(strange) $ ruby pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e -
• e method name (easy) • pi backtrace
(strange) $ ruby pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e -
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def
self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 -
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def
self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line -
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def
self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s) -
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def
self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s) Go back -
module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def
self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s) Go back Call entry point -
IFSFEPDNPEFͱ • lex.strtermͷঢ়ଶͷ͜ͱ static enum yytokentype parser_yylex(struct parser_params *p) {
... if (p->lex.strterm) { if (p->lex.strterm->flags & STRTERM_HEREDOC) { return here_document(p, &p->lex.strterm->u.heredoc); } else { token_flush(p); return parse_string(p, &p->lex.strterm->u.literal); } } ... -
IFSFEPD@JEFOUJpFS • func: IndentQuoteͷใ • term_len: <<STR ͷ͏ͪ STR Ҏ֎ͷ͞
(Ґஔܭࢉ༻) • term: STR ͱ͍͏String • lastline: ݱࡏͷߦͷϙΠϯλ • lastidx: <<STR ͷඌͷΧϥϜ൪߸ • ruby_sourceline: ݱࡏͷߦ൪߸ -
• func: 0x22 (INDENT & EXPAND) • term_len: 3 •
term: STR1 p <<-STR1.upcase This is str1. STR1 term_len term -
• func: 0x22 (INDENT & EXPAND) • term_len: 3 •
term: STR1 p <<-STR1.upcase This is str1. STR1 Ґஔใ = term_len + term -
• func: 0x22 (INDENT & EXPAND) • term_len: 3 •
term: STR1 • lastidx: 9 • ruby_sourceline: 1 p <<-STR1.upcase This is str1. STR1 lastidx lastline -
• func: 0x22 (INDENT & EXPAND) • term_len: 3 •
term: STR1 • lastidx: 9 • ruby_sourceline: 1 p <<-STR1.upcase This is str1. STR1 lastidx lastline -
• ҎԼͷ3ͭ1ͭͷStringΦϒδΣΫτʹͳ͍ͬͯΔ • term_len: 3 • func: 0x22 (INDENT &
EXPAND) • term: STR1 p <<-STR1.upcase This is str1. STR1 term_len term 3 0x22 STR1 ·ΓͲ͜Ζ String