Heredoc In Demon Castle

Transcript

1. 1.

   ѱຐͷ৓ͷώΞυΩϡϝϯτ ۚࢠ༤Ұ࿠(@yui-knk) 2018/07/14 Rails Developers Meetup 2018 Day 3 Extreme

2. 2.

   Thank you !!

3. 3.

   ࣗݾ঺հ • ۚࢠ ༤Ұ࿠ • Treasure Data ॴଐ • APIνʔϜ

   (RailsΞϓϦΛॻ͍͍ͯ·͢) • CRuby Committer 2015/12~ • GitHub (yui-knk)
4. 4.

   "1*νʔϜ
   3BJMTΞϓϦΛॻ͍ ͍ͯ·͢

5. 5.

   https://www.treasuredata.com/company/careers/jobs/?team=Engineering

6. 6.

   None
7. 7.

   None
8. 8.

   Rubyͷ࿩Λ͠Α͏

9. 9.

   ࠓ೔ͷτϐοΫ • Ruby 2.6ͷ࿩ • ѱຐͷ৓ͷ࿩ • RubyͷώΞυΩϡϝϯτ • parser͔ΒΈͨώΞυΩϡϝϯτ

10. 10.

    ࠓ೔ͷτϐοΫ • Ruby 2.6ͷ࿩ • ѱຐͷ৓ͷ࿩ • RubyͷώΞυΩϡϝϯτ • parser͔ΒΈͨώΞυΩϡϝϯτ

    • Railsͷ࿩͸ͳ͍Ͱ͢
11. 11.

    ࠓ೔ͷ໨ඪ • RubyͷώΞυΩϡϝϯτʹ͍ͭͯཧղΛਂΊΔ • parserʹڵຯΛ΋ͭ

12. 12.

    Ruby 2.6ͷ࿩

13. 13.

    3VCZ
    QSFWJFX • 2018/05/31 ϦϦʔε • JIT • RubyVM::AST

14. 14.

    3VCZ7."45
    • ΋ͱ΋ͱRubyʹ͸ `--dump=parsetree` ͱ͍͏Φϓγϣϯ ͕͋Δ • Rubyͷ಺෦తͳσʔλΛDump͢Δ΋ͷ

15. 15.

    $ ruby --dump=p -e '1 + 2' ########################################################### ## Do

    NOT use this node dump for any purpose other than ## ## debug and research. Compatibility is not guaranteed. ## ########################################################### # @ NODE_SCOPE (line: 1, location: (1,0)-(1,5)) # +- nd_tbl: (empty) # +- nd_args: # | (null node) # +- nd_body: # @ NODE_OPCALL (line: 1, location: (1,0)-(1,5))* # +- nd_mid: :+ # +- nd_recv: # | @ NODE_LIT (line: 1, location: (1,0)-(1,1)) # | +- nd_lit: 1 # +- nd_args: # @ NODE_ARRAY (line: 1, location: (1,4)-(1,5)) # +- nd_alen: 1 # +- nd_head: # | @ NODE_LIT (line: 1, location: (1,4)-(1,5)) # | +- nd_lit: 2 # +- nd_next: # (null node)
16. 16.

    3VCZ7."45
    • ಺෦తͳσʔλΛRubyͷੈքͰѻ͑ΔΑ͏ʹ͢Δ΋ͷ • Ruby 2.6͔Βಋೖ͞ΕΔ༧ఆ • ಛʹ require ͳͲ͸ෆཁ

    • ASTͷߏ଄ʹ͍ͭͯ͸ޓ׵ੑ͕อূ͞Εͳ͍
17. 17.

    3VCZ7."45
    RubyVM::AST.parse("1 + 2") # => #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:5): >

    RubyVM::AST.parse("1 + 2").children[1].children # => [ # #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ]
18. 18.

    3VCZ
    USVOL • Syntax Error͕ग़ΔΑ͏ʹमਖ਼͞Εͨ • #children ͕֦ு͞Εͨ

19. 19.

    4ZOUBY
    &SSPS $ ruby -e 'RubyVM::AST.parse("1 + ")' Traceback (most recent

    call last): 1: from -e:1:in `<main>' -e:1:in `parse': no file name:1: syntax error, unexpected end-of-input (SyntaxError)
20. 20.

    DIJMESFO # 2.6.0-preview2 RubyVM::AST.parse("1 + 2").children[1].children # => [ #

    #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] # trunk RubyVM::AST.parse("1 + 2").children[2].children # => [ # #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # :+, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ] mid
21. 21.

    DIJMESFO # trunk RubyVM::AST.parse("def a; end").children[2].children # => [ #

    :a, # #<RubyVM::AST::Node(NODE_SCOPE(0) 1:0, 1:10): > # ]
22. 22.

    *OTUBMM
     $ rbenv install 2.6.0-dev

23. 23.

    ѱຐͷ৓ͷ࿩

24. 24.

    ͋Δ͍͸ͳͥࠓ೔ώΞυΩϡ ϝϯτͷ࿩Λ͢Δ͔

25. 25.

    ѱຐ৓
    • ѱຐ৓ parse.y ͱ͍ΘΕΔϑΝΠϧ͕͋Δ • https://slide.rabbit-shocker.org/authors/nobu/ rubykaigi-2017/ • parse.y

    ͸Rubyͷจ๏Λఆ͍ٛͯ͠ΔϑΝΠϧ • Bison ͱ͍͏πʔϧͰ Cݴޠͷίʔυʹม׵͢Δ op : '|' { ifndef_ripper($$ = '|'); } | '^' { ifndef_ripper($$ = '^'); } | '&' { ifndef_ripper($$ = '&'); }
26. 26.

    QBSTFZ $ git shortlog -s -n parse.y | head -10

    912 nobu 362 matz 140 mame 91 yui-knk 55 ko1 38 aamine 37 akr 33 naruse 25 usa 8 normal
27. 27.

    ͳͥѱຐ৓
    ʁ • Large file (10000ߦҎ্͋Δ) • ripper standard library ͱϑΝΠϧΛڞ༗͍ͯ͠Δ

    • “#ifdef RIPPER” macro • stateful lexer • here document
28. 28.

    Ґஔ৘ใ # trunk RubyVM::AST.parse("1 + 2").children[2].children # => [ #

    #<RubyVM::AST::Node(NODE_LIT(59) 1:0, 1:1): >, # :+, # #<RubyVM::AST::Node(NODE_ARRAY(42) 1:4, 1:5): > # ]
29. 29.

    https://rubykaigi.org/2018/presentations/spikeolaf.html#jun01

30. 30.

    https://twitter.com/tagomoris/status/1005018203313979392

31. 31.

    https://twitter.com/n0kada/status/1005030519334006785

32. 32.

    https://twitter.com/n0kada/status/1005030519334006785 proc do<<S;end ͜͜͸endͷޙ S

33. 33.

    3VCZJTU
    )PUMJOLT
    
    ʲୈ
    
    ճʳ def foo; <<bar end; def baz; <<quux end

    In foo! bar In baz! quux p foo # => " In foo!\n" p baz # => " In baz!\n" https://magazine.rubyist.net/articles/0042/0042-Hotlinks.html#ruby %E3%81%AE%E7%BF%92%E5%BE%97
34. 34.

     https://magazine.rubyist.net/articles/0042/0042-index.html

35. 35.

    ͳͥώΞυΩϡϝϯτͷ࿩ʁ • parse.y Λཧղ͢ΔҰॿʹͳΔ • ΤοδέʔεʹͳΓ΍͍͢

36. 36.

    RubyͷώΞυΩϡϝϯτ

37. 37.

    Do you know here document?

38. 38.

    3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source

    programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write.
39. 39.

    3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source

    programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write.
40. 40.

    3VCZͷώΞυΩϡϝϯτ str = <<STR Ruby is... A dynamic, open source

    programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write. STR puts str # Ruby is... # A dynamic, open source programming language with a focus on simplicity and productivity. It # has an elegant syntax that is natural to read and easy to write.
41. 41.

    3VCZͷώΞυΩϡϝϯτ • <<STR • <<-STR • <<~STR

42. 42.

    453 • υΩϡϝϯτͷΠϯσϯτ͸ͦͷ·· • ऴ୺ߦͷΠϯσϯτ͕ڐՄ͞Εͳ͍ puts <<STR abcde fghij STR

    abcde fghij
43. 43.

    453 puts <<STR abcde fghij STR # => syntax error,

    unexpected end-of-input
44. 44.

    453 • υΩϡϝϯτͷΠϯσϯτ͸ͦͷ·· • ऴ୺ߦ͕ΠϯσϯτՄೳ puts <<-STR abcde fghij STR

    abcde fghij
45. 45.

    d453 • υΩϡϝϯτͷΠϯσϯτ͕Ұ൪ઙ͍ͱ͜Ζʹௐ੔͞Ε Δ • ऴ୺ߦ͕ΠϯσϯτՄೳ puts <<~STR abcde fghij

    STR abcde fghij
46. 46.

    3VCZͷώΞυΩϡϝϯτ • <<"STR" (ࣜల։͢Δ) • <<'STR' (ࣜల։͠ͳ͍) • <<`STR` (ίϚϯυ࣮ߦ)

    puts <<"STR" #{1 + 2} STR # => 3 puts <<'STR' #{1 + 2} STR # => #{1 + 2} puts <<`STR` uname STR # => Darwin
47. 47.

    ώΞυΩϡϝϯτ͕ศརͳͱ͜ • ώΞυΩϡϝϯτΛϨγʔόʹͰ͖Δ • ώΞυΩϡϝϯτͷͳ͔ʹώΞυΩϡϝϯτ͕ॻ͚Δ • 1ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ

48. 48.

    • STR1 ͕Ϩγʔό • #upcase ͕ϝιου ϨγʔόʹͰ͖Δ p <<STR1.upcase This

    is str1. STR1 # "THIS IS STR1.\n"
49. 49.

    ͳ͔ʹώΞυΩϡϝϯτ͕ॻ͚Δ p <<STR1 This is str1. #{<<STR2}. 12345 This is

    str2. STR2 ABCDE STR1 # "This is str1.\nThis is str2.\n. 12345\nABCDE\n"
50. 50.

    • String#+ ͸จࣈྻ݁߹ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2) This is

    str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n"
51. 51.

    • p ( ·ͰಡΉ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p ( # "This is

    str1.\nThis is str2.\n"
52. 52.

    • <<STR1 ΛಡΉͱɺheredocΛಡΈͱΔঢ়ଶʹͳΔ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 # "This is str1.\nThis

    is str2.\n"
53. 53.

    • STR1 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 This is str1. STR1

    # "This is str1.\nThis is str2.\n"
54. 54.

    • + ΛಡΉ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + This is str1.

    STR1 # "This is str1.\nThis is str2.\n"
55. 55.

    • <<STR2 ΛಡΉͱɺheredocΛಡΈͱΔঢ়ଶʹͳΔ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2 This is

    str1. STR1 # "This is str1.\nThis is str2.\n"
56. 56.

    • STR2 ·ͰಡΜͰɺheredocΛಡΈͱΔঢ়ଶΛ͵͚Δ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2 This is

    str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n"
57. 57.

    • ) ΛಡΉ ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ p (<<STR1 + <<STR2) This is

    str1. STR1 This is str2. STR2 # "This is str1.\nThis is str2.\n"
58. 58.

    ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ • ͋ΔߦΛॲཧ͍ͯ͠Δ్தͰ࣍ͷߦʹ͍͚Δ

59. 59.

    # +- nd_body: # @ NODE_DASGN_CURR (line: 1, location: (1,0)-

    (1,10))* # +- nd_vid: :s # +- nd_value: # @ NODE_STR (line: 1, location: (1,4)- (1,10))* # +- nd_lit: "abcde\n fghij\n" Ґஔ৘ใ s = <<~STR abcde fghij STR
60. 60.

    ༨ஊ (1)

61. 61.

    ࣮༻ྫ • 1ߦʹෳ਺ώΞυΩϡϝϯτ • จࣈྻల։தʹώΞυΩϡϝϯτ https://github.com/ruby/ruby/blob/trunk/test/ruby/test_array.rb#L2010

62. 62.

    ༨ஊ (2)

63. 63.

    https://twitter.com/mametter/status/936197656824619008

64. 64.

    • pi ͸ π (ԁप཰) • e ͸ωΠϐΞ਺ $ ruby

    pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e
65. 65.

    • “Look_Forward_Ruby_2_5” exception ͷbacktrace $ ruby pie.rb Traceback (most recent

    call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e
66. 66.

    • e ͸ method name (easy) • pi ͸ backtrace

    (strange) $ ruby pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e
67. 67.

    • e ͸ method name (easy) • pi ͸ backtrace

    (strange) $ ruby pie.rb Traceback (most recent call last): 11: from pie.rb:3:in `<main>' 10: from pie.rb:1:in `_2_' 9: from pie.rb:4:in `_7_' 8: from pie.rb:1:in `_1_' 7: from pie.rb:5:in `_8_' 6: from pie.rb:9:in `_2_' 5: from pie.rb:2:in `_8_' 4: from pie.rb:6:in `_1_' 3: from pie.rb:5:in `_8_' 2: from pie.rb:3:in `_2_' 1: from pie.rb:5:in `_8_' pie.rb:10:in `end': ^ ^ (Look_Forward_Ruby_2_5) | | pi e
68. 68.

    ߦʹෳ਺ͷώΞυΩϡϝϯτ͕ॻ͚Δ • ͋ΔߦΛॲཧ͍ͯ͠Δ్தͰ࣍ͷߦʹ͍͚Δ

69. 69.

    module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def

    self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1
70. 70.

    module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def

    self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line
71. 71.

    module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def

    self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s)
72. 72.

    module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def

    self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s) Go back
73. 73.

    module M1;def self._1_; M51._8_;end;end; module M1;def self._2_; _7_;end;end module M2;def

    self._8_;M6._1_;end;end module M3;def self._2_; M53._8_;end;end; <<STR1; M1._2_; #{def _7_; M1._1_; end} #{module M51;def self._8_;M9._2_;end;end; module M52;def self._8_;M3._2_;end;end; module M53;def self._8_; M10.end;end;end} #{module M6;def self._1_; M52._8_;end;end;} #{def _1_; raise; end} #{module M9;def self._2_; M2._8_;end;end} #{class Look_Forward_Ruby_2_5 < Exception;end; module M10; def self.end; raise Look_Forward_Ruby_2_5.new(" ^ ^\n | |\n pi e");end;end} STR1 Go to next line Define method(s) Go back Call entry point
74. 74.

    parser͔ΒΈͨ ώΞυΩϡϝϯτ

75. 75.

    IFSFEPD
    NPEF
    ͱ͸ • lex.strtermͷঢ়ଶͷ͜ͱ static enum yytokentype parser_yylex(struct parser_params *p) {

    ... if (p->lex.strterm) { if (p->lex.strterm->flags & STRTERM_HEREDOC) { return here_document(p, &p->lex.strterm->u.heredoc); } else { token_flush(p); return parse_string(p, &p->lex.strterm->u.literal); } } ...
76. 76.

    ॏཁͳؔ਺ • heredoc_identifier • heredoc modeʹೖΔ • heredoc_restore • heredoc

    mode͔Βग़Δ
77. 77.

    IFSFEPD@JEFOUJpFS • func: Indent΍Quoteͷ৘ใ • term_len: <<STR ͷ͏ͪ STR Ҏ֎ͷ௕͞

    (Ґஔܭࢉ༻) • term: STR ͱ͍͏String • lastline: ݱࡏͷߦ΁ͷϙΠϯλ • lastidx: <<STR ͷ຤ඌͷΧϥϜ൪߸ • ruby_sourceline: ݱࡏͷߦ൪߸
78. 78.

    • func: 0x22 (INDENT & EXPAND) p <<-STR1.upcase This is

    str1. STR1
79. 79.

    • func: 0x22 (INDENT & EXPAND) • term_len: 3 •

    term: STR1 p <<-STR1.upcase This is str1. STR1 term_len term
80. 80.

    • func: 0x22 (INDENT & EXPAND) • term_len: 3 •

    term: STR1 p <<-STR1.upcase This is str1. STR1 Ґஔ৘ใ = term_len + term
81. 81.

    • func: 0x22 (INDENT & EXPAND) • term_len: 3 •

    term: STR1 • lastidx: 9 • ruby_sourceline: 1 p <<-STR1.upcase This is str1. STR1 lastidx lastline
82. 82.

    • func: 0x22 (INDENT & EXPAND) • term_len: 3 •

    term: STR1 • lastidx: 9 • ruby_sourceline: 1 p <<-STR1.upcase This is str1. STR1 lastidx lastline
83. 83.

    • ҎԼͷ3ͭ͸1ͭͷStringΦϒδΣΫτʹͳ͍ͬͯΔ • term_len: 3 • func: 0x22 (INDENT &

    EXPAND) • term: STR1 p <<-STR1.upcase This is str1. STR1 term_len term 3 0x22 STR1 ͸·ΓͲ͜Ζ String
84. 84.

    ·ͱΊ • Ruby 2.6Ͱ͸ RubyVM::AST ͕ಋೖ͞ΕΔ༧ఆ • RubyͷώΞυΩϡϝϯτ͸ػೳ͕ͨ͘͞Μ͋Δ • ώΞυΩϡϝϯτ͸ΤοδέʔεʹͳΓ΍͍͢

85. 85.

    Thank you !!