Rubyで書くParser (自力かライブラリか、それが問題だ）

Slide 1

Slide 1 text

RUBYͰॻ͘PARSER  (ࣗྗ͔ϥΠϒϥϦ͔ɺͦΕ͕໰୊ͩʣ 2017.Nov ෱ԬRubyձٞ02 ௗҪઇ

Slide 2

Slide 2 text

ௗҪઇ ࣗݾ঺հ w גࣜձࣾສ༿ۈ຿ w 3BJMTΞϓϦέʔγϣϯΤϯδχΞ w 3BJMT(JSMT5PLZPOEΦʔΨφΠβʔ w ༁ॻʹ w ʰϓϩάϥϛϯά&MJYJSʱ  %BWF5IPNBTɺ೥ΦʔϜࣾ  ࡫ాߞҰͱڞ༁ w ʰϧϏΟͷ΅͏͚ΜʱγϦʔζ  ϦϯμɾϦ΢Χεஶɹᠳӭࣾ

Slide 3

Slide 3 text

ௗҪઇ ࣗݾ঺հ ੜ·Ε͸෱Ԭ ౦۠ശ࡚খֶߍ ஜࢵঁֶԂதֶߍ ஜࢵঁֶԂߴ౳ֶߍ => େֶ͔Β౦ژ΁ ͍ͭͷؒʹ͔ϓϩάϥϚʹ 2012೥෱ԬRubyձٞ01 LT 2015 RailsGirls Fukuoka ίʔν ෱  Ԭ  ظ  ؒ

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

RUBY Ͱॻ͘PARSER ΋͘͡ • ͳͥParserΛॻ͘͜ͱʹͳͬͨͷ͔ • ઃܭ • Parser͸Ή͔͍ͣ͠ʂ • Treetop ͱ͍͏ gem • ͦΕͰ΋ָ͍ࣗ͠࡞ParserʢͨͿΜ΍Ίͨ΄͏͕͍͍ʣ

Slide 6

Slide 6 text

ͳͥPARSERΛॻ͘͜ͱʹͳͬͨͷ͔ ͦΕ͸ͪΐͬͱͨ͠ग़དྷ৺ͩͬͨ

Slide 7

Slide 7 text

RUBYʹELIXIRʢ஫̍ Έ͍ͨͳ  ύλʔϯϚονϯάʢ஫2 ΄͍͠… ஫1 Elixir: Erlang VM্Ͱಈ͘ϓϩάϥϛϯάݴޠ  ஫2 ύλʔϯϚονϯάɿElixirʹ͋Δ͔͍͍ͬ͜ػೳ

Slide 8

Slide 8 text

=~ ͰϚονͤͯ͞ύλʔϯม਺ͱͯ͠࢖͍͍ͨ RUBYͰॻ͖͍ͨύλʔϯϚον

Slide 9

Slide 9 text

=== Ͱ΋Ϛον͍ͤͨ͞ʢCASEจॻ͖͍ͨʣ RUBYͰॻ͖͍ͨύλʔϯϚον

Slide 10

Slide 10 text

͜Εʢ஫̍ ͕ಈ͘ͱ  આಘྗ͕͋ΔΜ͡Όͳ͍͔…ʢ஫2 ஫1 ࠷ॳ͸࢓༷͚ͩເݟͯͨ  ஫2 ݁ہઆಘྗ͕͔͋ͬͨ͸ෆ໌

Slide 11

Slide 11 text

΍ͬͯΈͨ  ʢRUBY KAIGI YOUTUBE  HTTPS://WWW.YOUTUBE.COM/WATCH? V=1M4IPJH0K0E&INDEX=19&T=6S&LIST=PL

Slide 12

Slide 12 text

RUBYΠϯλϓϦλͷCͷίʔυͷࠩ෼͸͜Ε͚ͩ ΍ͬͯΈͨ compile.c parse.y

Slide 13

Slide 13 text

ʢಈ͚͹ྑ͠ɺύϑΥʔϚϯεͳͲ͸ߟ͑ͳ͍΋ͷͱ͢Δʣ ઃܭ Ruby script Parse Compile Ruby byte code Evaluator

Slide 14

Slide 14 text

ʢಈ͚͹ྑ͠ɺύϑΥʔϚϯεͳͲ͸ߟ͑ͳ͍΋ͷͱ͢Δʣ ઃܭ Ruby script Parse Compile Ruby byte code PatternMatching %p([a, ‘bc’]) =~ [3, ‘bc’] “[a, ‘bc’]” ม਺Ϧετ[“a”] ม਺ͷఆٛ Evaluator pattern_match obj Parse pattern Binding ΛͱΔͨΊʹ͝ʹΐΔ ASTߏங Ϛον͢Δ͔νΣοΫ ม਺୅ೖ RubyͷClass

Slide 15

Slide 15 text

ʢಈ͚͹ྑ͠ɺύϑΥʔϚϯεͳͲ͸ߟ͑ͳ͍΋ͷͱ͢Δʣ ઃܭ Ruby script Parse Compile Ruby byte code PatternMatching %p([a, ‘bc’]) =~ [3, ‘bc’] “[a, ‘bc’]” ม਺Ϧετ[“a”] ม਺ͷఆٛ Evaluator pattern_match obj Parse pattern Binding ΛͱΔͨΊʹ͝ʹΐΔ RubyͷClass ࠓ೔ͷ࿩͸ίί‼︎ ASTߏங Ϛον͢Δ͔νΣοΫ ม਺୅ೖ

Slide 16

Slide 16 text

ʢPATTERN MATCHING ΫϥεͷʣPARSERͷ΍Δ͜ͱ ྫ͑͹ `%p ([a, ‘bc’])`ͱ͍͏ύλʔϯ͕ࢦఆ͞Εͨ৔߹ɺ  “[a, ‘bc’]” ͱ͍͏จࣈྻΛड͚औͬͯ… • छྨ:ʮ഑ྻʯͰ͋Δ • ཁૉͷҰ൪໨͕ม਺aͰ͋Δ • ཁૉͷೋ൪໨͕จࣈྻ ͷ ‘bc’ Ͱ͋Δ • ඞཁͳύλʔϯม਺ͷϦετɿ[a]Ͱ͋Δ ͜ͱΛղੳͯ͠ɺߏ଄ʹ͢Δ

Slide 17

Slide 17 text

%p([a, ‘bc’]) =~ [3, ‘bc’] PARSEͷྲྀΕ “[a, `bc`]” [ ͱ a ͱ , ͱ `bc` ͱ ] Tokenize จࣈྻ Tokens AST ASTߏங String  Node (‘bc’) Array   Node Variable  Node (a) ஫1 AST࡞Δͱ͖ʹࠓճ͸ύλʔϯม਺Ϧετ΋࡞Δ

Slide 18

Slide 18 text

PARSEͷྲྀΕ “{status: 200, users: [a, b] }” { ͱ status: ͱ 200 ͱ , ͱ   users: ͱ [ ͱ aͱ , ͱ b ͱ ] ͱ } Tokenize จࣈྻ Tokens ASTߏங %p({status: 200, users: [a, b] }) =~ {status: 200, users: [1, 3] } AST Variable  Node (b) val:Array   Node Variable  Node (a) Hash  Node val:   Integer  Node (200) key: Symbol  Node  (:status) key: Symbol  Node  (:users)

Slide 19

Slide 19 text

࠷ऴతʹཉ͍͠ͷ͸AST %p({status: 200, users: [a, b] }) =~ {status: 200, users: [1, 3] } AST Variable  Node (b) val:Array   Node Variable  Node (a) Hash  Node val:   Integer  Node (200) key: Symbole  Node  (:status) key: Symbole  Node  (:users) {status: 200, users: [a, b] } ɹASTΛḷͬͯɺ஋ͱύλʔϯͱϚον͢Δ͔Λௐ΂Δ Ϛονର৅ ͦ΋ͦ΋hash? key ͕ status: ͷ val ͸ 200? key ͕users: ͷ val ͸ ഑ྻʁ ഑ྻͷཁૉ਺͸2? ഑ྻͷཁૉ਺ͷ1൪໨Λม਺aʹ֨ೲ͠Αʔ ഑ྻͷཁૉ਺ͷ2൪໨Λม਺bʹ֨ೲ͠Αʔ

Slide 20

Slide 20 text

Ή͔͔ͣͬͨ͠  ʢͱ͘ʹTOKENIZE ʣ

Slide 21

Slide 21 text

Tokenize “[a, `bc`]” [ ͱ a ͱ , ͱ `bc` ͱ ] Tokenize จࣈྻ Tokens Tokens Token ͷλΠϓΛݟͯɺʮ͓ͬ഑ྻͷ։͖ه߸͕དྷ͔ͨΒɺ͜ͷޙ͸഑ྻ͕ด͡ Δ·Ͱ഑ྻͷத਎ͩͳʯΈ͍ͨʹASTΛ࡞ͬͯΏ͘ ஋ λΠϓ [ ഑ྻͷ։͖ه߸ a ม਺ , ΧϯϚ `bc` จࣈྻɹ ] ഑ྻͷด͡ه߸

Slide 22

Slide 22 text

࢖ͬͨͷ͸StringScanner#scan Tokenize • StringScanner#scan • จࣈྻΛ಄͔ΒεΩϟϯͯ͠ɺਖ਼نදݱʹϚονͨ͠ΒϚον෦෼ Λฦͯͦ͠ͷޙΖ·ͰindexΛ͢͢ΊΔ “[a, `bc`]” [ a , `bc` ] ਖ਼نදݱ λΠϓ /\[/ ഑ྻͷ։͖ ه߸ /[a-z_][a-z0-9_]*/ ม਺ /,/ ΧϯϚ /'.*?'/ จࣈྻɹ /\]/ ഑ྻͷด͡ ه߸ “a, `bc`]” “`bc`]” “]” “[a, `bc`]” จࣈྻ Tokens Scan

Slide 23

Slide 23 text

ίϛοτ೔࣌ʹ஫໨ ྫɿεϖʔε͕2ͭҎ্ʹͳΔͱࣦഊ͢Δόά “[a, `bc`]” “[a, `bc`]”

Slide 24

Slide 24 text

௚ͯ͠ͳ͍ʢ͕Μ͹Δʣ ྫɿࣗ͘͝વʹ{} Λলུͯ͠͏͔͝ͳ͍ϋογϡ %p({ user: 1, from: ‘Fukuoka’}) %p( user: 1, from: ‘Fukuoka’ )

Slide 25

Slide 25 text

TOKENIZEʹҰ཯ͷਖ਼نදݱηοτ͔͠దԠͰ͖ͳ͍ ྫɿ͋ΔλΠϓͷTOKENIZEಠࣗϧʔϧͳͲ͕ѻ͍͑ͯͳ͍ “Name is #{user.name}”

Slide 26

Slide 26 text

• %p( [x, :y, { "array" => [5, v] }] )   ͘Β͍·Ͱ͸ParseͰ͖ΔΑ͏ʹͳͬͨ • ࣗྗͰҰ͔ΒParserΛॻ͘ͷ͸͔ͳΓߝ౉Γ • ֦ுੑʹݶք΋͋ΔʢΘͨ͠ʹ͸ʣ

Slide 27

Slide 27 text

PARSERΛॻ͍ͯΈΔͱ… • ࠓ·Ͱࣗ͘͝વʹಡΈॻ͖͍ͯͨ͠`[1, 2, 3]` ΍`{status: 200, users: [1, 2] }`ͳͲ͕ɺ  ಥવʮ͜Ε͔Βղऍ͞ΕΔʢ·ͩҙຯΛ࣋ͨͳ͍ʣจࣈྻʯͱ ͯ͠໨ͷલʹݱΕΔ • εϖʔεɺΧϯϚɺ͢΂ͯʹҙຯ͕͋Δ • Rubyຊମͷparse͍͢͝ • ਓؒͷ໨΋͍͢͝

Slide 28

Slide 28 text

·͞ʹʮ΋͏Ұ౓ɺ RUBYͱग़ձ͏ʯମݧ

Slide 29

Slide 29 text

https://github.com/cjheath/treetop ͱ͜ΖͰTreetopͱ͍͏gem͕͋Γ·͢ • PEGϕʔεͷಠࣗͷهड़ํࣜͰਖ਼نදݱͳͲΛ࢖ͬͯจ๏ϧʔϧΛఆ ٛ͢Δ.treetopϑΝΠϧΛͭ͘Δ • ttίϚϯυʹͦͷϑΝΠϧΛ౉͢ͱɺͦΕΛݩʹrubyͷparserϑΝΠ ϧΛ࡞ͬͯ͘ΕΔ • ੜ੒͞ΕͨrubyϑΝΠϧΛrequire ͢Δ͜ͱͰɺsyntaxnode, ͍ΘΏ ΔASTΛߏங͢ΔParserΛ࢖͏͜ͱ͕Ͱ͖Δ • ϧʔϧͷωετͷهड़΋༰қ

Slide 30

Slide 30 text

࠷ॳ͔ΒTREETOPΛ  ࢖͑͹ྑ͔ͬͨͷͰ͸…

Slide 31

Slide 31 text

ࣗ࡞PARSERͱTREETOPൺֱද ࣗ࡞ Treetop هड़ͷચ࿅ ϧʔϧͷωετ όάͷग़ʹ͘͞ Rubyͱग़ձ͑Δ

Slide 32

Slide 32 text

ʢෛ͚੯͠Έ͚ͩͰ͸ͳ͍ʣ ͦΕͰ΋ָ͍ࣗ͠࡞PARSER • ͦ΋ͦ΋࡞Γ࢝Ίͨஈ֊ͰʮParserʯͱ͍͏΋ͷ͕΅Μ΍Γ͔͠ཧ ղͰ͖ͯͳ͔ͬͨ • ͜ͷஈ֊ͰTreetopΛ࢖ͬͯ΋ɺந৅౓ෛ͚ͯ͠࢖͍͜ͳͤͳ͔ͬͨ ͷͰ͸ͳ͍͔ • ͍·࢖͍ํ͕Θ͔Βͳͯ͘Treetopͷੜ੒ͨ͠Ruby ParserΛಡΉͱ ؾ͕࣋ͪΘ͔Δ • ࣗ෼ͷίʔυ͕શ෦จࣈྻʹݟ͑ΔମݧϓϥΠεϨε • ंྠͷ࠶ൃ໌Ͱ΋͍͍ɺंྠ͕৺ͷதʹ૊ΈཱͯΒΕΔͷେࣄ

Slide 33

Slide 33 text

ˎ͋Δఔ౓Ҏ্ෳࡶͳ͜ͱΛ ͠Α͏ͱ͢Δͱߦ͖٧·Δɺ ͦΖͦΖ৐Γ׵͑Δͷ͕٢ˎ

Slide 34

Slide 34 text

Կ౓Ͱ΋  RUBYʹग़ձ͍͖ͬͯ·͠ΐ͏ɺ  ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ