Declarative parse.y - Speaker Deck

Declarative parse.y

by yui-knk

Slide 1

Slide 1 text

Declarative parse.y November 29, 2023 Fukuoka.rb #333 @yui-knk Yuichiro Kaneko

Slide 2

Slide 2 text

The Bison Slayer https://twitter.com/kakutani/status/1657762294431105025

Slide 3

Slide 3 text

About me • Yuichiro Kaneko • yui-knk (GitHub) / spikeolaf (Twitter) • The author of ruby/lrama LALR parser generator • CRuby committer, mainly develop parser related features • Code positions to RNode (2018, Ruby 2.6) • Coverage • Error reporting • RubyVM::AbstractSyntaxTree (2018, Ruby 2.6) • keep_tokens & error_tolerant option (2022, Ruby 3.2)

Slide 4

Slide 4 text

Fukuoka.rb #333 ͓ΊͰͱ͏͍͟͝·͢🎉

Slide 5

Slide 5 text

https://docs.google.com/presentation/d/1E4v9WPHBLjtvkN7QqulHPGJzKkwIweVfcaMsIQ984_Q/ edit#slide=id.g29612ae1b1e_0_0

Slide 6

Slide 6 text

Declarative • Declarative = “એݴత” • ྫ • SQL • ར఺ • ਺ֶత/࿦ཧֶͳߏ଄Λഎܠʹ࣋ͭ͜ͱ͕ଟ͍ • SQLͷ৔߹͸ؔ܎୅਺ • ڊਓͷݞʹ৐Δ͜ͱ͕Ͱ͖Δ • ղ͘΂͖໰୊ʹϑΥʔΧεͰ͖Δ

Slide 7

Slide 7 text

Declarativeͳจ๏ͱparser • BNF (Backus-Naur form) • ڞ௨ݴޠ(Ͱ͢ΑͶ) • ΦʔτϚτϯ • ܭࢉػͷ਺ֶతͳϞσϧ • ༗ݶΦʔτϚτϯ/ϓογϡμ΢ϯɾΦʔτϚτϯ/ઢܗ߆ଋΦʔτϚτϯ/ νϡʔϦϯάϚγϯ … • ܗࣜݴޠཧ࿦ • (ϓϩάϥϛϯά)ݴޠΛϞσϧԽͨ͠΋ͷ • ਖ਼نݴޠ/จ຺ࣗ༝ݴޠ/จ຺ґଘݴޠ/ؼೲతՄࢉݴޠ …

Slide 8

Slide 8 text

Declarativeʁ • arg_value, *arg_value, * ͷ͍ͣΕ͔͕ ‘,’ ۠੾ΓͰฒΜͰ͍Δͱ͍͏ϧʔ ϧ • declarative͚ͩͲprimitiveͳͷͰݟ௨͕͠ѱ͍ • ͜ͷޙͷ ydah͞Μͷൃදʹظ଴

Slide 9

Slide 9 text

parse.y͸declarative͔ʁ • ΋͜͠Μͳײ͡ͷϧʔϧͩͬͨΒɺͦΕ͸ͱͯ΋declarative

Slide 10

Slide 10 text

ݱ࣮͸Ͳ͏͔

Slide 11

Slide 11 text

ͳΜ͔ࢥ͍ͬͯͨͷͱҧ͏

Slide 12

Slide 12 text

ϧʔϧͱΞΫγϣϯ • ϧʔϧ: BNF • ΞΫγϣϯ: Cݴޠ ϧʔϧ ΞΫγϣϯ

Slide 13

Slide 13 text

ASTΛੜ੒͢ΔϩδοΫ • Node΍location৘ ใͳͲΛ͍͍ͬͯ͡ Δ

Slide 14

Slide 14 text

ίϝϯτʹજΉRipperͷӨ • ໨ΛڽΒ͢ͱίϝϯ τʹࠞͬͯ͟ripper ͷίʔυ͕͋Δ!!

Slide 15

Slide 15 text

ม਺ςʔϒϧΛ؅ཧ͢ΔϩδοΫ • localม਺ͷςʔϒ ϧΛpush, popͯ͠ ͍Δ

Slide 16

Slide 16 text

ঢ়ଶΛ؅ཧ͢ΔϩδοΫ • in_def΍in_classͱ ͍͏ม਺Λૢ࡞ͯ͠ ͍Δ • in_def΍in_class͸ def಺෦Ͱͷclassఆ ٛΛνΣοΫ͢Δͷ ʹར༻ͨ͠Γ͢Δ

Slide 17

Slide 17 text

parse.y for Under graduate • ΞΫγϣϯ(Cݴޠ)ͱ͍͏ͷ͸׬શʹCRubyݻ༗ͷϩδοΫ • parse.yͷ೉͠͞͸ΞΫγϣϯ(Cݴޠ)ʹ͋ΔͷͰ͸ͳ͍͔ • ΑΓdeclarativeʹ͍ͯ͘͜͠ͱͰɺڭՊॻͷ஌ࣝͱparse.yͷ࣮૷ͷဃ཭ Λͳ͍ͯ͘͘͠

Slide 18

Slide 18 text

ΞΫγϣϯͷத਎ • ASTΛߏங͢Δ • Ripper༻ͷϩδοΫ • ҙຯղੳ • def಺෦ͰͷclassఆٛͳͲͳͲ • https://i.loveruby.net/ja/rhg/book/syntree.html “ҙຯղੳ” ΋ࢀর • ͜ΕΛdeclarativeʹͰ͖ͳ͍͔?

Slide 19

Slide 19 text

ม਺ҰͭͰstackΛ؅ཧ͢Δ • k_classΛड͚ࡼʹ͠ ͯड͚౉͞ΕΔctxt (context) • ͜Ε͔Β্ॻ͖͞Ε Δ͜ͱʹඋ͑ͯݱ࣌ ఺ͷctxtΛk_classͷ semantic valueʹୀ ආ͢Δ 1. ctxtΛୀආ

Slide 20

Slide 20 text

ม਺ҰͭͰstackΛ؅ཧ͢Δ • classఆٛͷύʔεॲ ཧͷதͰctxtΛ্ॻ ͖͢Δ 1. ctxtΛୀආ 2. ctxtΛ্ॻ͖

Slide 21

Slide 21 text

ม਺ҰͭͰstackΛ؅ཧ͢Δ • classఆٛͷύʔεॲ ཧ͕ऴΘͬͨͷͰ k_classʹୀආ͓ͯ͠ ͍ͨctxtΛ෮چ͢Δ 1. ctxtΛୀආ 3. ctxtΛϦετΞ 2. ctxtΛ্ॻ͖

Slide 22

Slide 22 text

௨ৗͷstackͷ࢖͍ํ A A B A Push Pop

Slide 23

Slide 23 text

ม਺ҰͭͰstackΛ؅ཧ͢Δ A k_class A ctxt B k_class A k_class A A ctxt ctxt ctxt ctxt A A ୀආ ্ॻ͖ ϦετΞ ୀආઌ͕ফ͑Δ

Slide 24

Slide 24 text

ݱ࣮͸ݫ͍͠

Slide 25

Slide 25 text

ϧʔϧͱΞΫγϣϯ • ϧʔϧ: • BNF • Lrama͕ίϯτϩʔϧͰ͖Δ • ܗࣜݴޠཧ࿦΍ΦʔτϚτϯͷཧ࿦ʹै͏ඞཁ͕͋Δ • ΞΫγϣϯ • Cݴޠ • Lrama͕ίϯτϩʔϧͰ͖ͳ͍ • ࣮࣭ԿͰ΋Ͱ͖Δ

Slide 26

Slide 26 text

LR parserΛ৴͡Α

Slide 27

Slide 27 text

LR parser͸stackߏ଄ class A body end def m1 body end class B body end

Slide 28

Slide 28 text

StackΛ௥Ճͯ͠ΈΔ class A body end def m1 body end class B body end in_class in_def Ture Ture Ture False True False

Slide 29

Slide 29 text

PDAͷࣗવͳ֦ு • Pushdown automaton͸༗ݶΦʔτϚτϯʹελοΫΛ͚ͭͨ΋ͷ • LR parser͸Pushdown automatonͷߏ଄Λ͍ͯ͠Δ • ΦʔτϚτϯͱΦʔτϚτϯΛ૊Έ߹Θͤͨ΋ͷ͸ΦʔτϚτϯ

Slide 30

Slide 30 text

ΦʔτϚτϯͷ߹੒ x0 x1 x2 x3 a b c a d y0 y1 y2 y3 a a b a e ฏੴ๜඙ (2019) ܗࣜతϞσϧԽ ཭ࢄࣄ৅ʗ࣮࣌ؒʗϋΠϒϦουγεςϜͷϞσϧԽͱղੳ. ৿๺ग़൛. P.25 ਤ2.7Λࢀߟʹ࡞੒

Slide 31

Slide 31 text

ΦʔτϚτϯͷ߹੒ (x0, y0) (x0, y1) (x0, y3) (x0, y2) (x1, y0) (x1, y1) (x1, y3) (x1, y2) (x2, y0) (x2, y1) (x2, y3) (x2, y2) (x3, y0) (x3, y1) (x3, y3) (x3, y2) a c d c d c d e e e ฏੴ๜඙ (2019) ܗࣜతϞσϧԽ ཭ࢄࣄ৅ʗ࣮࣌ؒʗϋΠϒϦουγεςϜͷϞσϧԽͱղੳ. ৿๺ग़൛. P.25 ਤ2.9Λࢀߟʹ࡞੒

Slide 32

Slide 32 text

ruby/lrama #231 • Parser stateΛruleͷ Ұ෦ͱͯ͠هड़Ͱ͖Δ Α͏ʹ͢Δػೳ • ೥಺ʹೖΕΒΕΔ͔!? https://github.com/ruby/lrama/pull/231

Slide 33

Slide 33 text

Parser States (lex context) • ͜ͷล͸enum΍bool஋ͳͷͰදݱͰ͖ͦ͏

Slide 34

Slide 34 text

shareable_constant_value • Magic comment͸lexer͔Βparserʹ౉Βͳ͍ͷͰɺlexerଆ͔Βݱࡏͷ stack topͷ஋Λߋ৽Ͱ͖Δඞཁ͕͋Δ

Slide 35

Slide 35 text

in_rescue • retryΛॻ͍͍͍ͯͷ͸rescueͱ else (΋͘͠͸ ensure)ͷ͚ؒͩ • ͦͷͨΊࠓͷঢ়ଶΛ؅ཧ͢Δ

Slide 36

Slide 36 text

Nested in_rescue • ͜ΕΒ͸શͯ֎ଆͷrescueͷதʹ͍Δ ͷͰvalidʹͳΔ https://github.com/ruby/ruby/blob/v3_3_0_preview3/test/ruby/test_ast.rb#L287-L306

Slide 37

Slide 37 text

ಛผͳঢ়ଶΛ௥Ճ͢Δ • ಁաతʹͦͷલͷঢ়ଶΛࢀর͢Δঢ়ଶ(ε)Λ༻ҙ͢Δ • ௚લ͕after_resuceͳΒЏΛੵΉ • ݱࡏͷঢ়ଶΛධՁ͢Δͱ͖ʹ͸ЏΛແࢹͯ͠࠷ॳʹग़ݱ͢Δঢ়ଶΛΈΔΑ ͏ʹ͢Δ before_rescue after_rescue before_rescue after_rescue ε

Slide 38

Slide 38 text

next/break/redo • next/break/redoΛॻ͚Δ৔ॴʹ੍ݶ͕͋Δ

Slide 39

Slide 39 text

retryͱҟͳΔ࣮૷… • next/break/redoͷ؅ཧ͸in_def΍in_class ͱ͸ҧ͏࣮૷ʹͳ͍ͬͯΔ • ͜͜Ͱ͍͏node͸listͩͱࢥ͍ͬͯͩ͘͞

Slide 40

Slide 40 text

retryͷ৔߹ • retryͷ৔߹͸”retry”͕ग़͖ͯͨͱ͖ʹ in_rescue ΛνΣοΫͯ͠ɺ rescueͷ௚ޙग़ͳ͍৔߹͸Τϥʔʹ͍ͯ͠Δ

Slide 41

Slide 41 text

next/break/redoͷ৔߹ • stmt͕֬ఆͨ͠ஈ֊ͰνΣοΫ͍ͯ͠Δ • parse.yͷ࣮૷ʹແବ͕ͳ͍ͱԾఆͨ͠ͱ ͖ɺͳͥ͜͏ͳ͍ͬͯΔ͔෼͔Γ·͢ ͔ʁ

Slide 42

Slide 42 text

ޙஔͷwhile • ޙஔͷwhile/untilͷ৔߹ɺͦͷ௚લͷstmtͰ͸next/break/redoΛॻ͘ ͜ͱ͕Ͱ͖Δ

Slide 43

Slide 43 text

modi fi er_whileͷ࣮૷ • stmtͷ࣌఺Ͱ͸Τϥʔʹ͍͍͔ͯ͠ΛܾఆͰ͖ͳ͍ • ૺ۰ͨ͠next/break/redoΛͲ͔͜ʹ͓࣋ͬͯ͘ඞཁ͕͋Δ

Slide 44

Slide 44 text

node͸list

Slide 45

Slide 45 text

node͸list struct parser_params { rb_node_exits_t *exits; } struct RNode_EXITS { NODE *nd_chain; } struct RNode_BREAK { NODE *nd_chain; } struct RNode_BREAK { NODE *nd_chain; }

Slide 46

Slide 46 text

https://twitter.com/spikeolaf/status/1707031682648555756

Slide 47

Slide 47 text

ελοΫʹϦετΛࡌͤΔ • ཧղͯ͠͠·͑͹ϦετͱελοΫͷ૊Έ߹ΘͤͰ͔͠ͳ͍ • Lramaʹ૊ΈࠐΈͰϦετ(or ഑ྻ)Λ࣮૷ͯ͠ελοΫͰ؅ཧ͢Ε͹Α͍ k_whileʹ͸֎ଆͷ exits͕อଘ͞Ε͍ͯΔ whileશମ͕͓Θͬͨ Βrestore

Slide 48

Slide 48 text

ΑΓdeclarativeͳparse.y • ৽͍͠ঢ়ଶΛpush͢ΔՕॴ(allow_exits)ͱpop͢ΔՕॴ (restore_block_exit)͕1ͭͷϧʔϧ಺ʹด͍ͯ͡ΔͷͰɺ͜ͷϧʔϧ͚ͩ ΛݟΕ͹Α͍

Slide 49

Slide 49 text

ΑΓdeclarativeͳparse.y • push͢ΔՕॴ(allow_exits)ͱpop͢ΔՕॴ(restore_block_exit)͕2ͭͷ ϧʔϧʹ·͕͍ͨͬͯΔͷͰಡΈʹ͍͘ • ࣮ࡍ͸300ߦ͘Β͍཭Ε͍ͯΔ

Slide 50

Slide 50 text

ΑΓdeclarativeͳparse.y • ελοΫʹ৽͍͠ঢ়ଶΛpushͯ͠ɺ͜ͷϧʔϧͷύʔε͕͓ΘͬͨΒࣗಈ Ͱpopͯ͠΄͍͠ • Πϝʔδͱͯ͠͸around_actionΈ͍ͨͳ΍ͭ • ͍͍ײ͡ͷsyntaxΛ୳͠த • ΋ͱͷΦʔτϚτϯΛͲ͏มߋ͢Δͷ͔ਫ਼៛Խ͢Δඞཁ͕͋Γͦ͏

Slide 51

Slide 51 text

Parser States (local_vars) • parser stateքͷϘε • ԾҾ਺ɺlocalม਺ɺม਺͕ ࢖༻͞Ε͍ͯΔ͔൱͔ɺ numbered parameterʹؔ ͢Δ͋Ε͜ΕΛ1ͭʹ·ͱΊ ͨເͷΑ͏ͳߏ଄ମ • ελοΫͱू߹(Set)ͷ૊Έ ߹Θͤ

Slide 52

Slide 52 text

• Declarative • ਺ֶత/࿦ཧֶͳߏ଄Λഎܠʹ࣋ͭ͜ͱ͕ଟ͍ • ΦʔτϚτϯ΍ܗࣜݴޠཧ࿦ͷݞʹ৐Δ • ϧʔϧ(BNF)ͱΞΫγϣϯ(Cݴޠ) • ҙຯղੳʹ஫໨ ·ͱΊ

Slide 53

Slide 53 text

• ҙຯղੳ • ૉ௚ʹparserͱ౷߹Ͱ͖Δ (in_def΍in_class) • ͪΐͬͱ޻෉͕ඞཁ (shareable_constant_value΍in_rescue) • Ϧονͳσʔλߏ଄͕ඞཁ • Ϧετ (next/break/redo) • Set (local_vars) ·ͱΊ

Slide 54

Slide 54 text

Thank you!!