Share
Embed
erikuroda 4th ph.d defense
Feb.10 2025
Eri Kuroda@Ochamonizu University: 4th ph.d defense, public hearing
Eri KURODA
February 08, 2025
Transcript
95 slides
-
Real-world Understanding based on Predictions with Physical Properties ʢཧಛੑΛ;·͑ͨ༧ଌʹج࣮ͮ͘ੈքཧղʣ ͓ͷਫঁࢠେֶ
খྛҰݚڀࣨ ത࢜ޙظ՝ఔ3 ࠇా ክᣦ ത࢜จୈ4ճ৹ࠪձɾެௌձ 2025.2.10 -
96 จߏ • ݚڀഎܠ • ch1ɿΠϯτϩμΫγϣϯ • ຊݚڀʹ͓͚Δؔ࿈ݚڀ • ch2ɿχϡʔϥϧωοτϫʔΫ
• ch3ɿ࣮ੈքཧղɾݴޠϞσϧ • ఏҊݚڀ • ch4ɿώτͷ༧ଌූ߸ԽΛͨ͠ਂੜֶशϞσϧͷߏங • ch5ɿཧڥʹ͓͚ΔมԽநग़ͷऔΓΈ • ch6ɿ࣮ੈքʹ͓͚Δཧڥͷ༧ଌਪϞσϧ • ch7ɿཧతৗࣝΛ;·͑ͨମিಥ࣌ͷ༧ଌʹ͓͚Δݴޠදݱ • ݁ • ch8ɿ݁ 2 -
96 ࣍ • ݚڀഎܠ • ch1ɿΠϯτϩμΫγϣϯ • ຊݚڀʹ͓͚Δؔ࿈ݚڀ • ch2ɿχϡʔϥϧωοτϫʔΫ
• ch3ɿ࣮ੈքཧղɾݴޠϞσϧ • ఏҊݚڀ • ch4ɿώτͷ༧ଌූ߸ԽΛͨ͠ਂੜֶशϞσϧͷߏங • ch5ɿཧڥʹ͓͚ΔมԽநग़ͷऔΓΈ • ch6ɿ࣮ੈքʹ͓͚Δཧڥͷ༧ଌਪϞσϧ • ch7ɿཧతৗࣝΛ;·͑ͨମিಥ࣌ͷ༧ଌʹ͓͚Δݴޠදݱ • ݁ • ch8ɿ݁ 3 -
96 ώτʹΑΔ࣮ੈքཧղͱ༧ଌ ώτʹΑΔ࣮ੈքͷଊ͑ํ • ମ͕ͳʹͰ͋Δ͔ͷཧղ • ମͷಈ͖ͷཧղ • ಈ͖ͷ༧ଌ ྫ
• ಓ࿏ͷԣஅ • ࣗಈंͷӡస • ৗʹ༧ଌʢਪྔʣΛ͠ͳ͕Βੜ׆ ώτʹΑΔ༧ଌ • ॠؒతͳ༧ଌʢظؒʣ • ҼՌతͳ༧ଌʢظؒʣ 5 Έ߹Θͤͨ༧ଌΛৗʹߦ͍ͬͯΔ -
96 ػցʹΑΔ࣮ੈքͷཧղͱ༧ଌ ώτʹযΛ͋ͯͨݚڀ • AI, ػցֶशͷଆ໘ͰͷཧղɿχϡʔϥϧωοτϫʔΫ, BMIʢϒϨΠϯϚγϯΠϯλʔϑΣʔεʣ • ҩֶతͳཧղɿfMRI •
໘తͳཧղɿ৺ཧֶ ػցͱώτ͕ڞଘ͢Δੈք • ํͷཧղ • ώτ͕ओମͱͳͬͯ͏ͷ / ώτͷੜ׆Λࢧ͑Δͷ ػցʹ͓͚ΔώτΒ͠͞ͷදݱ • ۙLLMͷ։ൃ͕Ί͟·͍͠ • ݴޠੜ͕Ͱ͖Δͷ͔ΒɺώτΒ͠͞ώτͷࢥߟʹয͕ͯΒΕ͖͍ͯͯΔ 6 -
96 എܠɿػցʹΑΔώτΒ͍͠ਪɾ༧ଌΛࢦͯ͠ ػցֶशʹΑΔ࣮ੈքཧղɾ༧ଌݚڀͷॏཁੑ • ࣗಈӡసɺհޢ༻ϩϘοτͱ͍ͬͨਓؒͷੜ׆Λࢧԉ͢Δػೳ • ੈքϞσϧ [Ha+, 18] •
؍ଌͨ͠ڥͷܹ͔ΒώτͰͦͷڥΛϞσϧԽ͠ɺγϛϡϨʔγϣϯ͢Δ͜ͱͰཱ • AIݚڀ͕ࠓޙऔΓΉ͖3ͭͷ՝ͷ1ͭ [LeCun, 23] • ʮػց͍͔ʹͯ͠؍͔ΒੈքΛදݱͯ͠༧ଌΛ͠ɺߦಈ͢Δ͜ͱΛֶͿ͜ͱ͕Ͱ͖Δͷ͔ʯΛࣔࠦ • าߦऀͷΘ͔ͣͳಈ͖ͷҙຯ߹͍ΛஅͰ͖͍ͯͳ͍ ༧ଌݚڀ • ࣌ܥྻ༧ଌ • ҼՌਪ༧ଌ • ϚϧνϞʔμϧ༧ଌ • ʹͨ͠ڥΛɺώτͷΑ͏ʹཧղɾ༧ଌɾઆ໌͕Ͱ͖ΔΈ 7 -
96 తɿػցʹΑΔώτΒ͍͠ਪɾ༧ଌΛࢦͯ͠ 8 • ମͷ࣍ͷಈ͖Λ༧ଌ͠ɺߦಈΛܾఆ • গ͠ઌͷ༧ଌΛͯ͠ܭըΛཱͯΔ • ΓऔΓ؍͔ΒഎܠͳͲΛֶͿ •
ग़དྷࣄͷॏཁͳ͕େࣄ • ࣮ੈքͱݴޠͷ݁ͼ͖ͭ ώτͷ࣮ੈքཧղɾ༧ଌ • ஞ࣍తͳʢܾ·ִ͍ͬͨؒͷʣ༧ଌ • ը૾ʢࢹ֮ʹ૬ʣΛ༧ଌ • ը૾ಛྔͷ༧ଌ = ࣮ੈքͷ༧ଌ • ମͷཧ๏ଇΛͱʹͨ͠༧ଌ͕͍͠ • ମʹ͍ͭͯͷʮཧಛੑΛཧղ͠ɺ༧ଌ͢Δʯ ͜ͱͱɺݴޠ͕݁ͼ͍͍ͭͯͳ͍ ܭࢉػʹΑΔ༧ଌ ༷ʑͳ࣌ؒ෯Ͱ༧ଌΛߦ͑Δώτͷ֊ߏΛͨ͠ਂੜֶशϞσϧͷߏங ը૾ͷମͷཧಛੑͷมԽ͔Βɺಈ͖͕େ͖͘มΘΔมԽΛநग़Ͱ͖ΔΈ มԽʢিಥʣΛ༧ଌ͢ΔมԽ༧ଌϞσϧͷఏҊ ࣮ੈքͱݴޠΛ݁ͼ͚ͭΔͨΊʹɺ༧ଌ༰ΛݴޠͰදݱ ཧతৗࣝΛ༩͑ͨͱ͖ɺڥͷಛੑΛ;·͑ͯɺΑΓৄࡉͳจষͰදݱ త -
96 11 ch4: A Deep Generative Model Imitating Predictive Coding
in Human Brain ώτͷ༧ଌූ߸ԽΛͨ͠ ਂੜֶशϞσϧͷߏங -
96 എܠɾత 12 ώτʹ͓͚Δ࣌ؒೝࣝΛػցֶशϞσϧͰදݱ͍ͨ͠ PredNet [Lotter+, 2016] ώτͷΑ͏ʹ༷ʑͳ࣌ؒ෯Ͱ ະདྷΛ༧ଌͰ͖ͳ͍ ώτͷ֊ߏΛ฿
TD-VAE [Gregor+, 2018] ҙͷ࣌ؒ෯Ͱ ະདྷΛ༧ଌՄೳ ώτʹ͓͚Δใॲཧ ػߏΛө͍ͯ͠ͳ͍ ༧ଌΛରͱͨ͠දతͳઌߦݚڀ 1. ώτͷใॲཧػߏʹΑΓ͍ۙϞσϧΛߏங 2. ࣮ࡍͷώτͱͷ૬ؔͷௐࠪ ఏҊ -
96 શମਤ 13 fMRIσʔλ ׆ಈใ ࣗવಈը૾ ༧ଌ ग़ྗ ը૾ ༧ଌϞσϧ
ͷ ֊ త ͳ ߏ ॊ ೈ ͳ ࣌ ؒ ෯ Ͱ ͷ ༧ ଌ ॊೈͳ࣌ؒ෯Ͱকདྷͷ ग़དྷࣄΛ༧ଌՄೳʹ͢Δ -
96 • ਂֶशΛ༻͍ͨɺಈը૾͔Β࣍ͷը૾Λ༧ଌ͢Δݚڀ • େൽ࣭ʹ͓͚Δ༧ଌූ߸ԽͷॲཧΛ฿ • ͷใॲཧػߏΛදݱ PredNet [Lotter+, 16]
14 ༧ଌූ߸Խͱ • ༧ଌͱ؍ଌͷޡࠩΛࢉग़ • ޡࠩΛϘτϜΞοϓʹୡ • ޡࠩΛ࠷খԽ͢Δ༧ଌΛग़ྗ • ༧ଌΛτοϓμϯʹୡ ࣮ը૾ ༧ଌը૾ ࣌ࠁt → -
96 • Temporal Difference Variational Auto-Encoder • ಈը૾͔Βҙεςοϓઌͷը૾Λ༧ଌ͢Δݚڀ • ৴೦ঢ়ଶΛಋೖ
• POMDPʢ෦؍ଌϚϧίϑܾఆաఔʣͷ৴೦ঢ়ଶʹ૬ • ॊೈͳ࣌ؒ෯Ͱͷ༧ଌ TD-VAE [Gregor+, 18] 16 ࣌ࠁt → ༧ଌը૾2 ༧ଌը૾1 -
96 TD-VAE [Gregor+, 18] 17 !!! !!" !!! !!" "
" !! # # !!|!" $!! " % !" " " !" $!" " & !" Input Input Prediction Decoder Inference -
96 TD-VAE [Gregor+, 18] 18 !!! !!" !!! !!" "
" !! # # !!|!" $!! " % !" " " !" $!" " & !" Input Input Prediction Decoder Inference belief state(信念状態) 1. ώτͷใॲཧػߏ͕ө͞Ε͍ͯͳ͍ 2. fMRIデータとの相関が取れない -
96 !!! !"! ,ℓ "̂!!,ℓ "̂!!,ℓ$% !"!,ℓ%& "!!,ℓ$% "#"!,ℓ !"!!,ℓ$%
$!' !!' Observation input input Error representation Belief state Representation layer Belief state 0th module Belief state 1st module $!! Observation "!',ℓ !"",ℓ "!!,ℓ !"!,ℓ%& ఏҊϞσϧ [Kuroda+, 21] 19 ɿ؍ଌ ɿ৴೦ͷਪҠ ɿະདྷ͔Βͷਪ ɿࠩͷ ɿ༧ଌϞσϧͷߋ৽ ɿ༧ଌͷ ɿະདྷͷ༧ଌ ɿ༧ଌੜ -
96 త • ༧ଌִؒΛ1ඵʹͨ͠ͱ͖ɺࣗવಈը૾ͷ༧ଌ͕Մೳ͔Λݕূ ઃఆ • σʔληοτɿThe KITTI dataset •
࣌ࠁt=6Ҏ߱Λ༧ଌ ࣮ݧ1ɿ͍࣌ؒ෯Ͱͷ༧ଌ 20 ֶशઃఆ train data ʢ= 9:1ʣ 28,297 test data 426 #Layer 4 Size of convolutional filter 3 ×3 (for all convolutions) # Channels From lower module 3, 48, 96, 192 Optimization algorithm Adam [Kingma+, 15] -
96 • Vision meets Robotics: The KITTI Dataset • ΧʔϧεϧʔΤՊେֶɺ๛ాۀେֶγΧΰߍʹΑΔڞಉ࡞
• ੈք࠷େنͷंࡌ༻ϕϯνϚʔΫςετ ࣮ݧ1ɿσʔληοτ 21 ઃఆ ΧςΰϦʔ 6छྨ fps 10 (ϑϨʔϜ / ඵ) Ξϊςʔγϣϯ 15छྨ ը૾αΠζ 1242×375 Train data 28,297 Test data 426 -
96 • ը૾ධՁࢦඪSSIMʹΑΔ༧ଌը૾ͷҰக • Structure similarity • 𝑆𝑆𝐼𝑀 𝑥, 𝑦
= ("#!#"$%#)("'!"$%$) (#! $$#" $$%#)('! $$'" $$%$) • xɿݩը૾, yɿ༧ଌը૾, μɿฏۉ, σɿࢄ ࣮ݧ1ɿ݁Ռ ਫ਼ 23 SSIM↑ ఏҊϞσϧ 0.85 PredNet 0.65 -
96 త • ఏҊϞσϧʹΑΔਪ͕ɺ࣮ࡍͷώτͱ૬͕ؔ͋Δͷ͔ ઃఆ • σʔληοτɿࣗવಈը૾σʔληοτ [Nishimoto+, 11] •
Ϟσϧͷಛදݱͱώτ͔Βਪఆͨ͠ಛදݱΛൺֱ ࣮ݧ2ɿώτͱͷ૬ؔؔͷௐࠪ 24 -
96 ࣮ݧ2ɿશମਤ 25 ׆ಈใ ϖΞσʔλ Ϟσϧͷ ಛදݱ ਪఆ͞Εͨ ಛදݱ ૬ؔ
ճؼϞσϧ ʢϦοδճؼʣ ࣗવಈը૾ ༧ଌϞσϧ [Kay+, 08][Nishimoto+, 11] [Schoenmakers+, 13] -
96 PredNetʹ͓͚ΔϦοδճؼ ఏҊϞσϧʹ͓͚ΔϦοδճؼ ࣮ݧ2ɿ֓ཁਤ 26 R1 R0 R3 R2 Error
representation 3rd module Input layer Representation layer Ridge regression Brain activity watch movie, 10min cut 10 frames/sec 2nd module 1st module 0th module Stimulus image Belief state 0th module 2nd module R1 R0 R3 R2 Ridge regression Brain activity watch movie, 10min Stimulus image 3rd module 1st module cut 10 frames/sec Stimulus image Error representatio Belief state Representatio layer R1 R0 R3 R2 Error representation 3rd module Input layer Representation layer Ridge regression Brain activity watch movie, 10min cut 10 frames/sec 2nd module 1st module 0th module Stimulus image Belief state 0th module 2nd module R1 R0 R3 R2 Ridge regression Brain activity watch movie, 10min Stimulus image 3rd module 1st module cut 10 frames/sec Stimulus image Error representation Belief state Representation layer -
96 ࣮ݧ2ɿ݁Ռ 27 ૬ؔˢ α PredNet TD-VAE ఏҊϞσϧ 0.5 1,000
25,000 0.5 1,000 25,000 0.5 1,000 25,000 R0 ࠷Լ 0.2623 0.2971 0.3207 0.2636 0.2983 0.3285 0.2637 0.2983 0.3291 R2 0.0925 0.1459 0.1955 - - - 0.0003 0.0012 0.0016 R3 0.0254 0.1217 0.1871 - - - 0.0004 0.0009 0.0012 αɿϞσϧύϥϝʔλ R1ɿϦιʔεͷ੍ͰਪఆෆՄ -
96 • ώτʹ͓͚Δ࣌ؒೝࣝΛػցֶशϞσϧͰදݱ͍ͨ͠ • ώτͷ֊ߏΛͨ͠ɺॊೈͳ࣌ؒ෯Ͱͷ༧ଌ͕Ͱ͖Δ༧ଌϞσϧͷߏங ࣮ݧ1 • PredNetΑΓ༧ଌִ͕͍ؒը૾͕ੜͰ͖ͨ ࣮ݧ2 •
࠷ԼʢR0ʣͰɺώτͱ૬͕ؔݟΒΕΔ͜ͱ͕Θ͔ͬͨ • Ϟσϧ͕ώτͷ෦දݱͱ͍ۙ → ώτͷ࡞ۀϞσϧʹͳΓ͏Δ • ্ҐʹͳΔͱ૬͕ؔݟΒΕͳ͔ͬͨ • ਪΛR0ͷΈͰߦ͍ͬͯΔͨΊ • ׆ಈใ͕ը૾ͷڥΛཧղͨ͠σʔλͱݴ͑ͳ͍ • ͋͘·Ͱ ώτ͕ݟͨͷͷ৴߸ͱɺϞσϧͷ෦දݱʹ૬ؔ ͕ݟΒΕͨ ͱ͍͏͜ͱʹͱͲ·Δ ·ͱΊɾߟ 28 -
96 29 ch5: Extraction of Motion Change Points based on
the Physical Characteristics of Objects ը૾ͷମʹணͨ͠ ಈ͖ͷมԽநग़ͷऔΓΈ -
96 ͜͜·Ͱͷ༧ଌ • ೖྗɿը૾ • ը૾ಛྔͷ༧ଌʹͱͲ·Δ • ώτΒ͍͠༧ଌ • ೖྗશ෦Λࡉ͔͘༧ଌ͍ͯ͠ΔΘ͚Ͱ
ͳ͍ • ॏཁͳ෦ΛϐοΫΞοϓͯ͠༧ଌ ՝ • ը૾ͷঢ়گ͕ཧղͰ͖͍ͯͳ͍ • ॏཁͳ෦Λऔ͖ͬͯͯɺͦͷใ ʹରͯ͠ͷ༧ଌ͕Ͱ͖͍ͯͳ͍ • LLMͱͷੑೳͷൺֱ എܠɾత 30 -
96 ώτʹΑΔڥͷཧղ • ڥʢexɿը૾, ಈըͳͲʣʹ͋Δମ͕ԿͰ͋Δ͔Λೝࣝ • ೝࣝͨ͠ମͷಈ͖ʢexɿ, ଞͷମͱͷؔੑʣΛཧղ ݚڀ1Ͱͷڥͷཧղ •
কདྷى͜Γͦ͏ͳը૾ʢঢ়گͷඳࣸʣΛ༧ଌͨ͠ʹ͗͢ͳ͍ • ڥʹԿ͕͋Δ͔ʢମͷཧղʣɺମ͕ͲͷΑ͏ʹಈ͍͍ͯΔ͔ʢཧಛੑͷཧղʣ͕ Ͱ͖͍ͯͳ͍ • ڥʹ͋Δମಈ͖Λཧղ͢ΔΈͷ։ൃ ཧಛੑ ͱ 31 -
96 എܠɾత 32 • Variational Temporal Abstraction (VTA) [Kim+, 2019]
• ࢹ֮ใ͔ΒڥͷજࡏߏΛऔಘ • ڥ͕มԽ͢ΔλΠϛϯάΛநग़ • ϐΫηϧͷมԽʹண • ମͷཧಛੑʢͳͲʣΛ ߟྀͰ͖͍ͯͳ͍ 1. ମؒͷؔΛάϥϑϕʔεͰදݱ 2. άϥϑͷมԽΛͱʹڥͷมԽͷநग़ ఏҊ -
96 શମਤ 33 ै དྷ ͷ ख ๏ 3D໎࿏ ը૾ಛྔͷΈ
࣮ੈքʹ͍ͭͯ ཧղΛ͍ͯ͠ͳ͍ CLEVRER [Yi+, 19] άϥϑߏ ఏ Ҋ ख ๏ ମݕग़ ɾՃ ը૾ಛྔͳͲ มԽͷ ϑϥά Λநग़ VTA ! ! ! ! ! " " " " " ! ! ! ! # # # # # 0 1 0 0 ! ! ! " " # # # ! ! ! ! ! " " " " " ! ! ! ! # # # # # 0 1 0 0 ! ! ! ! ! " " # # # # # observation (input) observation abstraction boundry indicator temporal abstraction -
96 • Temporal Abstraction • ࣌ؒํʹ͓͍ͯɼ֊Խ͞Ε֤ͨϨϕϧͷΞΫγϣϯΛ౷߹ Temporal Abstraction 34 ྫɿྉཧΛ͢Δ
Ϩγϐͷબ ങ͍ ௐཧ ುΛ༻ҙ εʔύʔʹߦ͘ ುΛ͔͖ࠞͥΔ ۩ࡐΛΔ ɾख ɾख ɾख ϝϞாΛ༻ҙ ख ௐཧ ख ߴ த ϖϯΛ࣋ͭ ങ͍Ϧετ ͷ࡞ ങ͍Ϧετ ͷ࡞ ಉҰঢ়ଶ ಉҰঢ়ଶ -
96 • Temporal Abstraction • ࣌ؒํʹ͓͍ͯɼ֊Խ͞Ε֤ͨϨϕϧͷΞΫγϣϯΛ౷߹ Temporal Abstraction 35 Ϩγϐͷબ
ങ͍ ௐཧ ುΛ༻ҙ εʔύʔʹߦ͘ ುΛ͔͖ࠞͥΔ ۩ࡐΛΔ ɾख ɾख ɾख ϝϞாΛ༻ҙ ख ख ߴ த ϖϯΛ࣋ͭ ങ͍Ϧετͷ࡞ ಉҰঢ়ଶ ಉҰঢ়ଶ ྫɿྉཧΛ͢Δ -
96 Variational Temporal Abstraction [Kim+, 19] 36 ੨͍ಓΛา͍ͨͱ͖ ͍ಓΛา͍ͨͱ͖ શΠϕϯτ
େࣄͳՕॴ ʢมԽʣ શΠϕϯτ େࣄͳՕॴ ʢมԽʣ -
96 Variational Temporal Abstraction [Kim+, 19] 37 ! ! !
! ! " " " " " # # # # # ! ! ! ! ! " " # # # # # observation (input) observation abstraction boundry indicator temporal abstraction -
96 Variational Temporal Abstraction [Kim+, 19] 38 ϑϥάmͷಋೖ ! !
! ! ! " " " " " ! ! ! ! # # # # # 0 1 0 0 ! ! ! ! ! " " # # # # # observation (input) observation abstraction boundry indicator temporal abstraction -
96 • CLEVRER [Yi+, 2020] • CoLlision Events for Video
REpresentation and Reasoning σʔληοτ 39 ֓ཁ ಈը 20,000 ݸ (train:val:test=2:1:1) ϏσΦͷ͞ 5 ඵ ϑϨʔϜ 128ϑϨʔϜ ܗঢ় ཱํମɾٿɾԁப ૉࡐ ϝλϧɾϥόʔ ৭ փɼɼ੨ɼɼɼਫ৭ɼࢵɼԫ৭ Πϕϯτ ग़ݱɼফࣦɼিಥ Ξϊςʔγϣϯ object id, Ґஔ, , Ճ -
96 • ڥͷཧಛੑ͔Β࡞ͨ͠σʔληοτ Physical Training Dataset 40 ମೝࣝ ମͷ Ґஔใ
άϥϑߏ Ճ ମಉ࢜ͷ Ґஔํͷϑϥά ຒΊࠐΈ ϕΫτϧ ݁߹ -
96 ମೝࣝ • YOLO v3 [Redmon+, 18] • ମͷʨܗঢ়ʩ •
YOLACT [Bolya+,2019] • ମͷʨܗঢ়ɼ৭ʩ/ʨܗঢ়ɼ৭ɼૉࡐʩ Ґஔใ • औಘͨ͠όϯσΟϯάϘοΫεͷ ࠲ඪ͔Βମͷத৺࠲ඪΛࢉग़ Physical Training Dataset − ମೝࣝɾҐஔใ 41 ࣮ը૾ YOLO v3 YOLACT {ମ, ৭} YOLACT {ମ, ৭, ૉࡐ} (𝑥!, 𝑦!) (𝑥", 𝑦") 𝑐 = 𝑥, 𝑦 = ( 𝑥! + 𝑥" 2 , 𝑦! + 𝑦" 2 ) c -
96 ɾՃ • 𝑣#( = (𝑥$ − 𝑥$%! )/𝑒𝑡&'()*
• 𝑣+( = (𝑦$ − 𝑦$%! )/𝑒𝑡&'()* Ճ • 𝑎#( = (𝑣#( − 𝑣#) )/(𝑒𝑡&'()*×𝑡) • 𝑎+( = (𝑣+( − 𝑣+) )/(𝑒𝑡&'()*×𝑡) ମؒͷҐஔؔ • main object = (𝑥*+,- , 𝑦*+,- ) • others = (𝑥./012 , 𝑦./012 ) • 𝑥3,44 = 𝑥./012 − 𝑥*+,- • 𝑦3,44 = 𝑦./012 − 𝑦*+,- Physical Training Dataset − ɾՃɾମؒͷҐஔ 42 ϑϨʔϜؒͷܦա࣌ؒɿ𝑒𝑡!"#$% = 5/128 x main object others y ୈ2ݶ + − + ୈ1ݶ ୈ2ݶ − ୈ4ݶ ୈ3ݶ ୈ1ݶ ୈ3ݶ ୈ4ݶ x y -
96 άϥϑߏ • ϊʔυใ • ମͷܗঢ়ɼ৭ɼૉࡐ • Ͳͷମಉ͕͓࢜ޓ͍ͷಈ͖ʹؔ༩ͯ͠ ͍Δ͔͜ͷ࣌ͰΘ͔Βͳ͍ͨΊɺ શάϥϑͰ࡞
ຒΊࠐΈϕΫτϧ • node2vec [Grover+, 16] • graph2vec [Narayanan+, 17] Physical Training Dataset − άϥϑߏ 43 [[0.54, 0.29, 0.61…], [[0.82, 0.91, 0.15…], … [[0.14, 0.35, 0.69…]] 埋め込みベクトル -
96 ࣮ݧ1 • ఏҊ͢ΔมԽநग़ϞσϧΛ༻͍ͨͱ͖ɺিಥͷλΠϛϯάΛநग़Ͱ͖Δ͔ ࣮ݧ2 • LLMΛ༻͍ͨͱ͖ͷਫ਼ • GPT-4o •
GPT-4o-FT • GPT-4o mini • GPT-4 ࣮ݧɿিಥͷλΠϛϯάͷநग़ 44 -
96 త • ମͷಈ͖ͷมԽ͔Βɺিಥ໘ͷେ͖ͳมԽͷλΠϛϯάΛऔಘͰ͖Δ͔Λݕূ ઃఆ • ਖ਼ղͷিಥɿ30ϑϨʔϜ ਓʹΑΔিಥͷೝࣝɿ32ϑϨʔϜ • ਖ਼ղൣғɿ30ʙ32ϑϨʔϜ
• F1είΞͰࢉग़ ࣮ݧ1ɿมԽநग़ϞσϧʹΑΔিಥͷݕग़ 45 ֶशઃఆ σʔλʢ8:1:1ʣ 600,000 όοναΠζ 100 ग़ྗʢϑϥάʣ 80 ࠷దԽؔ Adam [Kingma+, 17] Τϥʔؔ KLμΠόʔδΣϯε -
96 • LLMʢGPTܥʣը૾ͷମΛࢹ֮తɾཧతʹଊ͑ΔೳྗΛ͍ͬͯΔͷ͔ • ਪʹ͓͚ΔڧΈɾऑΈ • Ϟσϧͷҧ͍ GPTͷछྨ • GPT-4o
• GPT-4o-FT • CLEVRERΛೝࣝͰ͖ΔΑ͏ʹϑΝΠϯνϡʔχϯάͨ͠Ϟσϧ • GPT-4o mini • GPT-4 ࣮ݧ2ɿGPTΛ༻͍ͨিಥͷݕग़ 47 -
96 ݕূઃఆ • 100িಥ • িಥલ10ϑϨʔϜɺিಥޙ9ϑϨʔϜͷ߹ܭ20ຕΛೖྗ • ਖ਼ղσʔλɿϑϨʔϜ19 ࢹ֮ɿϑϨʔϜ21 •
ग़ྗɿϑϨʔϜ ϓϩϯϓτ • simple • ԿϑϨʔϜͰମಉ͕࢜িಥ͢Δ͔આ໌͍ͯͩ͘͠͞ɻ • detail • #ମͷ৭, #ମͷܗ, #ମͷૉࡐ • #ঢ়گɿෳͷମ͕ಈ͍͓ͯΓɺ2ͭͷମ͕িಥ͠·͢ɻ • #࣭ɿͲͷମ͕ɺԿϑϨʔϜͰিಥ͍ͯ͠Δ͔Λઆ໌͍ͯͩ͘͠͞ɻ ࣮ݧ2ɿGPTΛ༻͍ͨিಥͷݕग़ 48 -
96 ࣮ݧ2ɿGPTΛ༻͍ͨিಥͷݕग़ − ݁Ռ 49 Model Prompt macro-F1 Accuracy ମ͕͍͋ͬͯΔ͔ʢ৭,
ܗ, ૉࡐʣ simple detail GPT-4o 㾎 75.4 2.53 㾎 75.3 2.65 GPT-4o-FT 㾎 75.8 2.65 㾎 76.1 2.63 GPT-4o mini 㾎 28.5 1.35 㾎 33.3 1.58 GPT-4 㾎 0 0 㾎 0 0 มԽநग़Ϟσϧ 63.5 2.75 macro-F1 • ֤িಥʹ͓͚ΔF1είΞͷฏۉ Accuracy • িಥʹؔ͢Δମͷछྨ͕͍͋ͬͯΔ͔Ͳ͏͔ max: 3 -
96 • ը૾ʹ͋Δମͷಈ͖͔ΒɺিಥফࣦͷλΠϛϯάΛଊ͑ΒΕΔΈͷߏங ࣮ݧ1ɿมԽநग़Ϟσϧ • ਖ਼ղσʔλͰ͋ΔΞϊςʔγϣϯϕʔεͷ݁Ռ͕࠷ਫ਼͕ߴ͍ • YOLACTͰݕͨ݁͠Ռͷํ͕ɺYOLO v3ͷͷΑΓਫ਼͕ߴ͍ •
node2vecͷํ͕graph2vecΑΓਫ਼͕ߴ͍ • ը૾ಛྔͱཧใΛ·ͱΊͯѻ͏ͱɺը૾ಛྔ͕ϊΠζʹͳΔ ࣮ݧ2ɿ(15 • GPT-4o-FTͷ݁Ռ͕࠷ߴ͍ • CLEVRERΛೝࣝͰ͖ΔΑ͏ʹfinetuning͔ͨ͠Β • ͱͱͷਪೳྗ͕ߴ͍΄Ͳɺmacro-F1ͷείΞߴ͍ • ͔͠͠GPTܥը૾ಛྔͰڥΛཧղ͍ͯ͠ΔͷͰɺώτͷΑ͏ʹڥΛଊ͑ͨͱݴ͍͍ • มԽநग़Ϟσϧମͷछྨͷཧղߴ͔͕ͬͨɺিಥΛ΄΅ଊ͑ΒΕΔΑ͏ʹͳͬͨΘ͚Ͱͳ͍ ·ͱΊɾߟ 50 -
96 51 ch6: Predictive Inference Model of the Physical Environment
࣮ੈքʹ͓͚Δ ཧڥͷ༧ଌਪϞσϧ -
96 ͜͜·Ͱͷݚڀ • ը૾ͷମͷಈ͖ʹண • িಥফࣦͳͲͷλΠϛϯάΛ ଊ͑Δ͜ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨ • ਓͷ࣮ੈքೝࣝͱ༧ଌ •
ݟͨͷΛࢹ֮తɾཧతʹ༧ଌ͢Δ ՝ • ࢹ֮ɾಈ͖ͷ྆ํͷ༧ଌͰ͖ͯ ͍ͳ͍ • LLMࣗମʹཧಛੑΛཧղ͢Δ ೳྗ͋Δͷ͔ എܠɾత 52 -
96 શମਤ 53 physical training data • άϥϑߏͷຒΊࠐΈϕΫτϧ • ֤ମͷɾՃ
• ମؒͷҐஔؔ CLEVRER [Yi+, 19] 2. ը૾ͷ༧ଌ (predicted image) • PredNet [Lotter+, 16] • PredRNN [Wang+, 17] • PredRNN v2 [Wang+, 21] • PreCNet [Straka+, 23] ༧ଌϞσϧͷߏங มԽ༧ଌ Ϟσϧ • VTA [Kim+, 19] (Variational Temporal Abstraction) 1. িಥͷλΠϛϯά ͷϑϥά Output Input Input -
96 PredNet [Lotter+, 16] • େൽ࣭ʹ͓͚Δ༧ଌූ߸Խͷ ॲཧΛ฿ • ΤϥʔΛ֊తʹਪ •
ݚڀ1Ͱ༻ͨ͠༧ଌϞσϧͱಉ͡ PreCNet [Straka+, 23] • PredNetΛվྑ • ೖྗใશମΛຖճਪ ج൫ʹͳΔ༧ଌϞσϧ 54 !ℓ"# !ℓ " # ℓ"# "ℓ"# " # ℓ "ℓ $ℓ"# $ℓ ⊝ ⊝ conv Prediction Target pool conv input Error +,-ReLU subtract %$! Input Representation conv LSTM !! ℓ ⊝ ⊝ "! ℓ#$ # $ ! ℓ#$ # $ ! ℓ "! ℓ %! upsample !! ℓ#$ ⊝ ⊝ upsample conv LSTM conv +,- ReLU subtract conv input conv LSTM Representation Pediction Error +,- ReLU subtract -
96 PredRNN [Wang+, 2017] • ConvLSTMΛ֊ʹͨ͠ܗͷ༧ଌϞσϧ • ۭؒɾ࣌ؒͷ྆ํʹHʢӅΕʣ͕ೖྗ PredRNN v2
[Wang+, 2022] • PredRNNΛվྑͨ͠৽ͨͳ༧ଌϞσϧ • HΛೖྗ͢ΔήʔτΛ૿ͨ͠ ج൫ʹͳΔ༧ଌϞσϧ 55 !!"# ℓ "! !! ℓ #! $! %! ℓ #′! '′! "′! %! ℓ"# '! !!"# ℓ "! !! ℓ ⨂ ⨂ ⨂ ⨂ ⨂ Input Gate Output Gate Input Modulation Gate Forget Gate Standard Temporal Memory Spatiotemporal Memory ! = !! + !" = 1 % & '# ( − '# + $ #%! 1 % & '# ( − '# " $ #%! ( %&'()*+& ℓ,! = cos(∆!! ℓ, ∆%! ℓ) !!"# ℓ "! !! ℓ #! $! %! ℓ #′! '′! "′! %! ℓ"# '! !!"# ℓ "! !! ℓ ⨂ ⨂ ⨂ ⨂ ⨂ Input Gate Output Gate Input Modulation Gate Forget Gate Standard Temporal Memory Spatiotemporal Memory PredRNN PredRNN v2 ΦϦδφϧConvLSTM ConvLSTMʹۭ࣌ؒهԱͷ ػߏΛՃ PredRNN, v2 ͷ෦ߏ -
96 PredNetϕʔεͷมԽ༧ଌϞσϧ 56 image data !!"_ℓ%& !!"_ℓ ⊝ ⊝ #!"_ℓ%&
$ % !"_ℓ%& $!"_ℓ%& $ % !"_ℓ $!"_ℓ !'"_ℓ%& !'"_ℓ ⊝ ⊝ #'"_ℓ%& $ % '"_ℓ%& $'"_ℓ%& $ % '"_ℓ $'"_ℓ img img output &'((!" #!"_ℓ &'(('" )( flag output )( = 0 ∶ &'(( < . )( = 1 ∶ &'(( > . &'(( = &'((!" + &'(('" 2!" Input Error Representation Prediction time t Physical data .: threshold Difference #'"_ℓ 2'" Input physical training data image data -
96 PredRNN, PredRNN v2ϕʔεͷมԽ༧ଌϞσϧ 57 !!_#$% ℓ'( , #!_#$% ℓ'(
!!_#$% !!_&'( ST-$%&'!"# ℓ%& ST-$%&'!"# ℓ%' ST-$%&'!"# ℓ%( ST-$%&'!"# ℓ%) ! " !)*_#$% ! " !)*_&'( $!)*_+,- ℓ'. $!_+,- ℓ'* $!_+,- ℓ'( $!_+,- ℓ'/ !!_+,- ℓ'* !!_+,- ℓ'( !!_+,- ℓ'/ $!_+,- ℓ'. $!_#$% ℓ'. %! = ' 0 ∶ +,--! < / 1 ∶ +,--! > / time ! $!)*_#$% ℓ'. image data ()***_,-. physical data ()***_!"# +,--! = +,--!_+,- + +,--!_#$% ST-$%&',-. ℓ%& ST- $%&',-. ℓ%' ST- $%&',-. ℓ%( ST- $%&',-. ℓ%) physical training data img output !!_+,- ℓ'( , #!_+,- ℓ'( !!_+,- ℓ'/ , #!_+,- ℓ'/ !!_+,- ℓ'. , #!_+,- ℓ'. !!_+,- ℓ'* , #!_+,- ℓ'* $!_#$% ℓ'* $!_#$% ℓ'( $!_#$% ℓ'/ !!_#$% ℓ'* !!_#$% ℓ'( !!_#$% ℓ'/ !!_#$% ℓ'/ , #!_#$% ℓ'/ !!_#$% ℓ'. , #!_#$% ℓ'. !!_#$% ℓ'* , #!_#$% ℓ'* -
96 PreCNetϕʔεͷมԽ༧ଌϞσϧ 58 !!_#$% ℓ'( !!_#$% ℓ ⊝ ⊝ "!_#$%
ℓ'( # $ !_#$% ℓ'( ! " !_#$% ℓ "!_#$% ℓ !!_#$% Input !!_)*+ ℓ'( !!_)*+ ℓ ⊝ ⊝ "!_)*+ ℓ'( ! " !_'() ℓ*+ ! " !_'() ℓ !!_&'( Input !"##!_#$% !"##!_&'( "!,(_#$% ℓ "!,(_)*+ ℓ "!_)*+ ℓ upsample upsample !! = # 0 ∶ '())! < + 1 ∶ '())! > + #$%%! = #$%%!_#$% + #$%%!_'() time ! image data physical data physical training data img img output pool pool Error Representation Prediction -
96 ࣮ݧ1 • ఏҊ͢ΔมԽ༧ଌϞσϧ4छྨΛ༻͍ͯɺিಥͷλΠϛϯάΛ༧ଌͰ͖Δ͔ ࣮ݧ2 • LLMΛ༻͍ͨͱ͖ͷਫ਼ • GPT-4o •
GPT-4o-FT • GPT-4o mini • GPT-4 ࣮ݧɿମͷিಥ༧ଌ 59 -
96 • ڥʹ͋ΔମͷিಥΛ༧ଌͰ͖Δ͔ ઃఆ • σʔληοτɿCLEVRER, Physical Training Dataset •
ਖ਼ղͷিಥɿ30ϑϨʔϜ ਓʹΑΔিಥͷೝࣝɿ32ϑϨʔϜ • ਖ਼ղൣғɿ30ʙ32ϑϨʔϜ • ରൣғɿ6ύλʔϯʢiʙviʣº10ϑϨʔϜ • F1είΞɺmacro-F1 ࣮ݧ1ɿมԽ༧ଌϞσϧʹΑΔিಥ༧ଌ 60 -
96 ࣮ݧ1ɿઃఆ 61 PredRNNɾPredRNN v2 ϕʔε PreCNetϕʔε ֶशσʔλ 600,000 600,000
ςετσʔλ 80,000 80,000 ΤϙοΫ 500,000 500,000 ϨΠϠʔ 4 4 νϟϯωϧ 128 3, 48, 96, 192 ΧʔωϧαΠζ 5×5 - ଛࣦؔ Adam [Kingma+, 17] Adam [Kingma+, 17] ֶशݮਰ 0.001 0.0001 αʢมԽఆͷᮢʣ 5 5 -
96 ࣮ݧ1ɿ݁Ռ − িಥϑϥάͷਫ਼ 62 ൣғ i ii iii iv
v vi macro- F1 PredNet ϕʔε 40.0 50.0 50.0 40.0 57.1 50.0 42.5 PredRNN ϕʔε 50.9 54.8 53.1 48.9 60.6 61.7 57.5 PredRNN v2ϕʔε 51.4 57.5 54.6 50.6 62.7 64.2 59.2 PreCNet ϕʔε 62.1 64.2 59.2 60.8 68.9 69.8 65.8 িಥ͢Δͱ༧ଌͨ͠ϑϥά͕ਖ਼͍͔͠Ͳ͏͔ -
96 • িಥલͷମͷಈ͖Λ༩͑ͨͱ͖ɺͦͷޙʹੜ͡ΔিಥΛ༧ଌͰ͖Δ͔ ݕূઃఆ • 100িಥ • িಥ͕ى͖Δ15ϑϨʔϜલʙ6ϑϨʔϜલͷܭ10ϑϨʔϜΛ༩͑Δ • ϑϨʔϜ14Ҏ߱Ͱੜ͡ΔিಥͷϑϨʔϜΛ༧ଌʢਖ਼ղ20ʙ22ʣ
ϓϩϯϓτ • simple • ԿϑϨʔϜͰମ͕িಥ͢Δ͔༧ଌ͍ͯͩ͘͠͞ɻ • detail • #ମͷ৭, #ମͷܗ, #ମͷૉࡐ • #ঢ়گɿෳͷମ͕ಈ͍͓ͯΓɺ2ͭͷମ͕িಥ͠·͢ɻ • #࣭ɿମ͕ԿϑϨʔϜޙʹিಥ͢Δ͔Λ༧ଌ͍ͯͩ͘͠͞ɻ ࣮ݧ2ɿGPTʹΑΔମͷিಥ༧ଌ 63 -
96 ࣮ݧ2ɿGPTʹΑΔମͷিಥ༧ଌ − ݁Ռ 64 Model Prompt macro-F1 Accuracy ମ͕͍͋ͬͯΔ͔
simple detail GPT-4o 㾎 55.2 1.95 㾎 54.7 1.95 GPT-4o-FT 㾎 61.5 2.11 㾎 62.0 2.10 GPT-4o mini 㾎 55.8 1.68 㾎 54.5 1.65 GPT-4 㾎 0 0 㾎 0 0 มԽ༧ଌϞσϧ 67.5 2.47 -
96 • ը૾ʹ͋ΔମͷিಥͷλΠϛϯάΛɺࢹ֮తɾཧతͳมԽ͔Β༧ଌͰ͖ΔϞσϧͷߏங ࣮ݧ1ɿมԽ༧ଌϞσϧ • ϕʔεͱͳΔ༧ଌϞσϧͷਫ਼͕ɺมԽ༧ଌϞσϧͷਫ਼ʹӨڹ • PreCNet͕࠷ਫ਼͕ߴ͍ • ༧ଌը૾ͷਫ਼ɺmacro-F1ͷείΞ
• F1είΞɿߴͯ͘65 • શʹিಥͷλΠϛϯάΛ͔ͭΊͨΘ͚Ͱͳ͍ • վળͷ༨͋Γ ࣮ݧ2ɿGPT • มԽ༧ଌϞσϧ͕࠷ਫ਼͕ߴ͔ͬͨ • GPTܥͰɺϑΝΠϯνϡʔχϯάͨ݁͠Ռ͕ߴ͔ͬͨ • ༧ଌʹͳΔͱҰؾʹείΞ͕Լ͕Δ • ϐΫηϧͷ༧ଌͱɺিಥ͢Δ͜ͱͭͳ͕͍ͬͯͳ͍ ·ͱΊɾߟ 65 -
96 67 ch7: Verbal Representation of Object Collision Prediction Based
on Physical CommonSense Knowledge ཧతৗࣝΛ;·͑ͨ ମিಥ࣌ͷ༧ଌʹ͓͚Δݴޠදݱ -
96 ͜͜·Ͱͷݚڀ • ը૾ͷମΛଊ͑ɺ ମ͝ͱͷཧಛੑΛଊ͑ͨ • ମಉ࢜ͷিಥͷλΠϛϯάΛ ༧ଌ͢ΔϞσϧͷߏங ՝ •
িಥʹ͍ͭͯΛݴޠͱͯ͠ൃ͠ɺ ଞऀʹ͑Δ͜ͱ͕Ͱ͖ͳ͍ • ࣮ੈքͷΑ͏ʹෳࡶͳڥͷಛΛ ౿·͑ͨཧղ͕Ͱ͖͍ͯͳ͍ • LLMͰͷੑೳͷݕূ എܠɾత 68 -
96 ཧಛੑΛදͨ͠ άϥϑߏͷ༧ଌ શମਤ 69 physical training data ݴޠੜ Ϟσϧ
• άϥϑຒΊࠐΈϕΫτϧ • • Ճ • ମؒͷҐஔؔ ༧ଌը૾ ੜจ ৭ͷԁப͕৭ͷٿʹͿ͔ͭΔ ମͷ৭ ✔ɼܗ ✔ ༧ଌͨ͠ঢ়گΛද͢จ Input ཧಛੑΛදͨ͠ άϥϑߏͷ༧ଌ ༧ଌͰ͖Δͷ มԽ ༧ଌϞσϧ ϕʔεͱͳΔϞσϧ • PredNet [Lotter+,16] • PredRNN [Wang+, 17] • PredRNN v2 [Wang+, 21] • PreCNet [Straka+, 23] ཧతৗࣝͷ݅ ʢྫʣ݅ 29 • ମAͷॏܰ͘͞ɼମBͷॏ͞ॏ͍ • ମAͷ͘͞ɼମBͷ͍͞ • চ͕βϥβϥ͍ͯ͠Δ ཧతৗࣝΛؚΉ จষੜϞσϧ ࠶ੜͨ͠จ ৭ͷԁப͕৭ͷٿʹ͍Α͘ িಥͯ͠ɺ৭ͷٿ͕ԕ͘ʹ͖ ඈ͞ΕΔɻ ৗࣝΛؚΉৄࡉͳจ ମAʹ৭ͷԁப ମBʹ৭ͷٿ ੜจ Ճ CLEVRER [Yi+, 19] -
96 ࣮ݧ1-1 • มԽ༧ଌϞσϧ4छྨͰ༧ଌͨ͠িಥ༰Λͱʹݴޠੜ ࣮ݧ1-2 • LLMΛ༻͍ͯিಥͷ༧ଌ͔Βݴޠੜ • GPT-4o •
GPT-4o-FT • GPT-4o mini • GPT-4 ࣮ݧ1ɿিಥ༧ଌͷݴޠੜ 70 -
96 త • ࣮ੈքͱݴޠΛ݁ͼ͚ͭͯཧղ͢ΔͨΊʹɺมԽ༧ଌϞσϧͰ༧ଌͨ͠༰Λੜ ઃఆ • TransformerͷDecoder෦Λ༻ ࣮ݧ1-1ɿ༧ଌ༰Λݴޠੜ 71 ֶशઃఆ
ϖΞσʔλ 219,303 ʢ9จ * 24,367ճͷিಥʣ ςετσʔλ 10,965 όοναΠζ 8 ӅΕ 512 ଛࣦؔ Adam [Kingma+, 17] -
96 • 9छྨͷςϯϓϨʔτ • িಥͨ͠ 2ͭͷମ • ʮʨփ, , ੨,
, , ਫ, ࢵ, ԫʩ৭ͷ ʨٿ, ԁப, ཱํମʩʯ ࣮ݧ1-1ɿςϯϓϨʔτ࡞ 72 • ʮAͱB͕ۙͮ͘ʯ • ʮA͕Bʹۙͮ͘ʯ • ʮB͕Aʹۙͮ͘ʯ িಥલʢ5ϑϨʔϜલʣ • ʮAͱB͕Ϳ͔ͭΔʯ • ʮA͕Bʹ͔͡ΕΔʯ • ʮB͕Aʹ͔͡ΕΔʯ িಥ • ʮAͱB͕ΕΔʯ • ʮA͔ΒB͕ΕΔʯ • ʮB͔ΒA͕ΕΔʯ িಥޙʢ5ϑϨʔϜޙʣ ʮ੨৭ͷٿͱփ৭ͷٿ͕Ϳ͔ͭΔʯ ʮ੨৭ͷٿ͕փ৭ͷٿʹ͔͡ΕΔʯ ʮփ৭ͷٿ͕੨৭ͷٿʹ͔͡ΕΔʯ িಥ িಥલ ʮ੨৭ͷٿͱփ৭ͷٿ͕ۙͮ͘ʯ ʮ੨৭ͷٿ͕փ৭ͷٿʹۙͮ͘ʯ ʮփ৭ͷٿ͕੨৭ͷٿʹۙͮ͘ʯ িಥޙ ʮ੨৭ͷٿͱփ৭ͷٿ͕ΕΔʯ ʮ੨৭ͷٿ͔Βփ৭ͷٿ͕ΕΔʯ ʮփ৭ͷٿ͔Β੨৭ͷٿ͕ΕΔʯ จষςϯϓϨʔτྫɿিಥ͢Δମʢ੨৭ͷٿɾփ৭ͷٿʣ 5ϑϨʔϜ 5ϑϨʔϜ -
96 ࣮ݧ1-1ɿݴޠੜϞσϧ 73 ֶशࡁΈ DecoderϞσϧ ༧ଌ༰Λ ࣔͨ͠ੜจ pred graph embedding
input Decoder Softmax <bos> w1 w2 wt <eos> … w1 w2 wt … Transformer DecoderֶशϞσϧ text ϖΞσʔλ Linear graph embedding train: 219,303 ϖΞ test: 10,965 ݸ -
96 ࣮ݧ1-1ɿ݁Ռ − ੜྫ1 74 ൣғ i ৭ ܗ ਖ਼ղจ
• ʮ৭ͷٿͱ৭ͷԁப͕Ϳ͔ͭΔʯ “Green sphere and red cylinder collide.” • ʮ৭ͷٿ͕৭ͷԁபʹ͔͡ΕΔʯ “Green sphere is repulsed by red cylinder.” • ʮ৭ͷԁப͕৭ͷٿʹ͔͡ΕΔʯ “Red cylinder is repulsed by green sphere.” PredNet ϕʔε ʮ৭ͷԁப͕৭ͷԁபʹ͔͡ΕΔʯ “Green cylinder is repulsed by red cylinder.” ✔ ✘ PredRNN ϕʔε ʮ৭ͷԁபͱ৭ͷԁப͕Ϳ͔ͭΔʯ “Green cylinder and red cylinder collide.” ✔ ✘ PredRNN v2ϕʔε ʮ৭ͷٿ͕৭ͷԁபʹ͔͡ΕΔʯ “Red cylinder is repulsed by green sphere.” ✔ ✔ PreCNet ϕʔε ʮ৭ͷٿ͕৭ͷԁபʹ͔͡ΕΔʯ “Red cylinder is repulsed by green sphere.” ✔ ✔ -
96 ࣮ݧ1-1ɿ݁Ռ − ੜྫ2 75 ൣғ vi ৭ ܗ ਖ਼ղจ
• ʮਫ৭ͷཱํମͱਫ৭ͷԁப͕Ϳ͔ͭΔʯ “Cyan cube and cyan cylinder collide.” • ʮਫ৭ͷཱํମ͕ਫ৭ͷԁபʹ͔͡ΕΔʯ “Cyan cube is repulsed by cyan cylinder. ” • ʮਫ৭ͷԁப͕ਫ৭ͷཱํମʹ͔͡ΕΔʯ “Cyan cylinder is repulsed by cyan cube. ” PredNet ϕʔε ਫ৭ͷཱํମ͕੨৭ͷٿʹͿ͔ͭΔ “Cyan cube is repulsed by blue sphere. ” ✘ ✘ PredRNN ϕʔε ਫ৭ͷཱํମ͕੨৭ͷٿʹͿ͔ͭΔ “Cyan cube is repulsed by blue sphere. ” ✘ ✘ PredRNN v2ϕʔε ਫ৭ͷཱํମ͕ਫ৭ͷٿʹͿ͔ͭΔ “Cyan cube is repulsed by cyan sphere. ” ✔ ✘ PreCNet ϕʔε ਫ৭ͷཱํମ͕ਫ৭ͷԁபʹͿ͔ͭΔ “Cyan cube is repulsed by cyan cylinder. ” ✔ ✔ -
96 ࣮ݧ1-1ɿ݁Ռ − ධՁࢦඪʹΑΔਫ਼ൺֱ 76 ϕʔεϞσϧ είΞ BLEU@2↑ BLEU@3↑ BLEU@4↑
METEOR↑ CIDEr↑ PredNet ϕʔε ӳ 80.3 63.0 56.3 68.8 72.9 79.7 74.5 68.8 70.2 72.4 PredRNN ϕʔε ӳ 84.3 66.8 59.1 72.6 74.6 82.5 76.1 73.4 73.5 75.1 PredRNN v2 ϕʔε ӳ 86.2 72.4 62.7 75.9 78.3 85.9 78.9 75.7 77.6 78.2 PreCNet ϕʔε ӳ 90.6 77.1 67.9 78.1 80.3 88.3 80.6 79.2 80.4 81.2 -
96 • GPTΛ༻͍ͯɺ2ͭͷମ͕িಥ͢Δͱ༧ଌͨ͠໘ΛݴޠͰੜ ݕূઃఆ • 100িಥ • ը૾Λ༩͑ͨͱ͖ɺিಥೝࣝ→ݴޠੜ·ͰμΠϨΫτͰߦ͏ ૬ҧ •
ఏҊϞσϧɿিಥ༧ଌͱݴޠੜͷϞσϧ͕ผ • LLMɿશͯಉ͡ਪաఔ ࣮ݧ1-2ɿGPTʹΑΔମͷিಥ༧ଌͷݴޠੜ 77 -
96 ࣮ݧ1-2ɿGPTʹΑΔମͷিಥ༧ଌͷݴޠੜ 78 Model Prompt ੜྫ Accuracy ମ͕͍͋ͬͯΔ͔ simple detail
GPT-4o 㾎 ਫ৭ͷٿମͱ৭ͷԁப͕িಥ͢Δɻ 1.94 㾎 ਫ৭ͷٿମ͕৭ͷԁபʹͿ͔ͭΔ 1.95 GPT-4o-FT 㾎 ਫ৭ͷٿ͕ࠨ͔Β৭ͷٿʹিಥ͢Δɻ 2.10 㾎 ਫ৭ͷٿͱ৭ͷٿ͕Ϳ͔ͭΔɻ 2.06 GPT-4o mini 㾎 ৭ͷٿମ͕৭ͷԁபʹ͔ͬͯస͕Γিಥ͢Δɻ 1.63 㾎 ৭ͷٿମ͕৭ͷԁபʹ͔ͬͯస͕Γিಥ͢Δɻ 1.69 GPT-4 㾎 − 0 㾎 − 0 มԽ༧ଌϞσϧ ৭ͷٿ͕৭ͷԁபʹিಥ͢Δɻ 2.44 • ৭ͷԁபͱ৭ͷٿ͕Ϳ͔ͭΔ • ৭ͷԁப͕৭ͷٿʹ͔͡ΕΔ • ৭ͷٿ͕৭ͷԁபʹ͔͡ΕΔ ਖ਼ղจ -
96 ࣮ݧ1-2ɿGPTʹΑΔମͷিಥ༧ଌͷݴޠੜ 79 Model Prompt BLEU@2↑ BLEU@3↑ BLEU@4↑ METEOR CIDEr
simple detail GPT-4o 㾎 39.3 32.6 26.4 30.2 36.7 㾎 44.5 36.9 28.5 30.1 38.6 GPT-4o-FT 㾎 27.2 20.4 14.3 29.5 30.8 㾎 38.7 29.2 27.1 31.4 31.3 GPT-4o mini 㾎 36.1 28.7 21.3 30.1 33.3 㾎 36.2 28.7 22.7 31.1 32.6 GPT-4 㾎 − − − − − 㾎 − − − − − มԽ༧ଌϞσϧ 88.3 80.6 79.2 80.4 81.2 -
96 • ཧతಛੑͷ༧ଌຒΊࠐΈදݱ͔ΒɺݴޠੜϞσϧΛհͯ͠িಥঢ়گΛੜ ࣮ݧ1-1ɿมԽ༧ଌϞσϧ • ॏͳΓ߹ͬͨମͷਫ਼ͷӨڹ • ମೝࣝͷࣦഊ͕ݴޠੜӨڹ͕ݟΒΕΔ • ج൫ͱͳΔ༧ଌϞσϧͷҧ͍
• ༧ଌϞσϧͷਫ਼͕ݴޠੜͷਫ਼ʹӨڹ • ςϯϓϨʔτΛֶश͍ͤͯ͞ΔͨΊɺදݱͷ෯ͤ·͍ ࣮ݧ1-2ɿGPT • จষΛੜ͢Δ ͱ͍͏؍ͰͲͷϞσϧཧղͰ͖Δจষ͕ੜͰ͖ͨ • ͔͠͠ɺମͷछྨ͕ؒҧ͍ͬͯΔ͜ͱͰɺධՁࢦඪͷείΞ͕͘ͳͬͨ ࣮ݧ1ɿ·ͱΊɾߟ 80 -
96 ࣮ݧ2-1 • มԽ༧ଌϞσϧ4छྨͰ༧ଌͨ͠িಥ༰ͷจষʹରͯ͠ɺڥͷཧৗࣝ Λ༩͑ͨͱ͖ʹΑΓৄࡉͳઆ໌จͷੜ ࣮ݧ2-2 • LLMΛ༻͍ͨিಥͷ༧ଌ͔Βৄࡉͳઆ໌จͷੜ • GPT-4o
• GPT-4o-FT • GPT-4o mini • GPT-4 ࣮ݧ2ɿڥͷཧৗࣝΛՃ͑ͨͱ͖ͷݴޠੜ 81 -
96 ࣮ݧ2-1ɿڥͷཧৗࣝΛՃ͑ͨͱ͖ͷݴޠੜ 82 ڥɾମͷঢ়ଶ 1. চ͕πϧπϧ͍ͯ͠Δ ମA͕ॏ͘ɺମB͍ܰ 2. চ͕βϥβϥ͍ͯ͠Δ ମA͕ॏ͘ɺମB͍ܰ
… 45. ମAܰ͘ɺମBॏ͍ ཧৗࣝͷϦετҰཡ ৗࣝ1〜45 ͔Βબ ৗࣝɿ19 • চ͕πϧπϧ͍ͯ͠Δ • ମAͷॏ͕͞େ͖͘ɺ ମBͷॏ͞খ͍͞ • ମA͘ɺମB͍ T5 Ϋϥυιʔγϯά ৭ͷٿ͍Α͘৭ͷԁபʹিಥͯ͠ɺ ৭ͷԁபԕ͘ʹ͖ඈ͞ΕΔɻ -
96 • 2ͭͷମͷিಥ༧ଌ໘ʹ͍ͭͯɺڥମʹؔ͢ΔཧৗࣝΛ༩͑ͨͱ͖ɺ িಥঢ়گΛΑΓৄࡉʹݴޠͰੜ͢Δ ࣮ݧ2-1ɿڥͷཧৗࣝ 83 ମͷঢ়ଶʢ࣭ྔʣ ମͷঢ়ଶʢεϐʔυʣ 1. ମAͱମBͷ࣭ྔ͕͍͠
2. ମAͷ࣭ྔ͕େ͖͍ ମBͷ࣭ྔ͕খ͍͞ 3. ମAͷ࣭ྔ͕খ͍͞ ମBͷ࣭ྔ͕େ͖͍ 4. ͳ͠ 1. ମAͱମBͷεϐʔυ͕͍͠ 2. ମAͷεϐʔυ͕͍ ମBͷεϐʔυ͕͍ 3. ମAͷεϐʔυ͕͍ ମBͷεϐʔυ͕͍ 4. ͳ͠ 4ύλʔϯ 4ύλʔϯ ྆ํͳ͠Λআ֎ → 15ύλʔϯ ڥ 1. চ͕πϧπϧ͍ͯ͠Δ 2. চ͕βϥβϥ͍ͯ͠Δ 3. ͳ͠ -
96 • T5 (Text-To-Text Transfer Transformer) • TransformerϕʔεͷϞσϧ • ༁ɺ࣭ԠɺྨɺཁͳͲ༷ʑͳλεΫͰ׆༻
• ೖྗ͞ΕͨςΩετʹରͯ͠ɺશͯͷλεΫͰςΩετͰग़ྗ • ༻ͨ͠ࣄલֶशϞσϧT5 • sonoisa/t5-base-japanese • megagonlabs/t5-base-japanese-web • nlp-waseda/comet-t5-base-Japanese • ֶश • ೖྗɿিಥʹؔ͢Δจষɺ݅ • ग़ྗɿΫϥυιʔγϯάͰऩूͨ͠5จ͔ΒϥϯμϜͰબΜͩ1จ ࣮ݧ2-1ɿT5Ϟσϧ 84 -
96 ࣮ݧ2-1ɿT5ͷֶशઃఆ 85 ઃఆ Data (6:1:1) 2,000 Learning rate 5×10%4
batch size 32 Epoch 100 optimization AdamW [Loshchilov+, 17] loss function cross entropy -
96 ࣮ݧ2-1ɿT5Ͱͷ݁Ռ 86 Model Epoch BLEU↑ ROUGE-2↑ ROUGE-L↑ sonoisa/ t5-base-japanese
81 95.2 64.2 74.6 megagonlabs/ t5-base-japanese-web 93 81.6 56.6 67.7 nlp-waseda/ comet-t5-base-japanese 98 80.9 56.2 67.4 -
96 ࣮ݧ2-1ɿ݁Ռ − ੜྫ1 87 มԽ༧ଌϞσϧʹ͓͚Δੜจɿ৭ͷԁப͕৭ͷٿʹ͔͡ΕΔ ମAɿ৭ͷԁப ମBɿ৭ͷٿ চ ࣭ྔ
͞ ཧৗࣝΛ;·͑ͨੜจ ਓखʹΑΔਖ਼ղྫ πϧπϧ A = B A = B ৭ͷԁபͱ৭ͷٿিಥͯ͠ɺ྆ํͱ ରͷํ͖ඈ͞ΕΔɻ ৭ͷԁப͕ಉ͡ͷ৭ͷٿʹͿ͔ͭͬ ͯɺ৭ͷٿԕ͘ʹͶฦ͞ΕΔɻ πϧπϧ − A < B ৭ͷٿ͍Α͘৭ͷԁபʹিಥͯ͠ɺ ৭ͷԁபԕ͘ʹ͖ඈ͞ΕΔɻ ৭ͷԁபͱ৭ͷٿ͕Ϳ͔ͭΓɺ৭ͷԁ ப͕গ͔͠Εɺ৭ͷٿ͕গ͔͠ΕΔɻ βϥβϥ A < B A > B ৭ͷԁப͔͕৭ͷٿʹ͍Α͘িಥͯ͠ɺ ৭ͷٿ͕ԕ͘ʹ͖ඈ͞ΕΔɻ ৭ͷԁப͍Α͘৭ͷٿʹͿ͔ͭͬͯɺ ৭ͷԁப΄Μͷগ͠Ͷฦ͞ΕΔɻ − A > B A < B ৭ͷٿ͍Α͘৭ͷԁபʹিಥͯ͠ɺ ৭ͷٿ͖ඈ͞ΕΔɻ ৭ͷԁப৭ͷٿ͕Ϳ͔ͭΓɺ৭ͷٿ ͕͔ΕΔɻ -
96 ࣮ݧ2-1ɿ݁Ռ − ੜྫ2 88 มԽ༧ଌϞσϧʹ͓͚Δੜจɿਫ৭ͷཱํମ͕ਫ৭ͷԁபʹ͔͡ΕΔ ମAɿਫ৭ͷཱํମ ମBɿਫ৭ͷԁப চ ࣭ྔ
͞ ཧৗࣝΛ;·͑ͨੜจ ਓखʹΑΔਖ਼ղྫ − A = B − ਫ৭ͷཱํମͱਫ৭ͷԁபিಥͯ͠ɺ྆ํ ͱରͷํ͖ඈ͞ΕΔɻ ਫ৭ͷԁபਫ৭ͷཱํମʹিಥͯ͠ɺਫ৭ ͷཱํମ͖ඈ͞ΕΔɻ πϧπϧ A > B A < B ਫ৭ͷཱํମ͕͍Α͘ਫ৭ͷԁபʹিಥ͠ ͯɺਫ৭ͷཱํମ͕ԕ͘ʹ͖ඈ͞ΕΔɻ ਫ৭ͷཱํମਫ৭ͷԁபʹܹ͘͠িಥ͞Ε ͯɺਫ৭ͷཱํମ͚͔ͩ͢͜͠͡ΕΔɻ βϥβϥ − A = B ਫ৭ͷཱํମͱਫ৭ͷԁபিಥͯ͠ɺ྆ํ ͱର ͷํ͖ඈ͞ΕΔɻ ਫ৭ͷԁபͱਫ৭ͷཱํମ͕িಥͯ͠ਫ৭ͷ ԁபͱਫ৭ͷཱํମ͕ಉ͡ఔͶฦΔɻ − A = B A > B ਫ৭ͷཱํମ͍Α͘ਫ৭ͷԁபʹিಥ͠ ͯɺਫ৭ͷԁபԕ͘ʹ͖ඈ͞ΕΔɻ ਫ৭ͷཱํମΏͬ͘Γͱਫ৭ͷԁபૣ͍ ͞Ͱରͷํʹඈ͞ΕΔɻ -
96 ࣮ݧ2-1ɿ݁Ռ − ධՁࢦඪʹΑΔਫ਼ൺֱ 89 ϕʔεͱͳΔ Ϟσϧ BLEU@4 BERTScore BLEURT
ROUGE ؚҙ શจ ؚҙ ׂ G-EVAL- 4o PredNet ϕʔε 37.3 68.2 30.3 34.4 43.2 64.5 80.3 PredRNN ϕʔε 43.6 74.7 36.1 41.8 48.7 69.7 83.1 PredRNN v2 ϕʔε 46.5 79.5 45.6 49.6 56.1 75.3 88.5 PreCNet ϕʔε 55.8 82.2 49.7 56.4 67.9 80.2 92.4 -
96 ࣮ݧ2-2ɿGPTΛ༻͍ͨཧৗࣝΛؚΉিಥঢ়گͷઆ໌ 90 Model ੜྫ ਖ਼ղྫ • ৭ͷٿͱ৭ͷԁபରํʹେ͖͘ඈ͞ΕΔɻ • ͓ޓ͍ʹͿ͔ͭΓɺ৭ͷԁபগ͠ζϨͯ৭ͷۄগ͠స͕͍ͬͯ͘ɻ
GPT-4o ਫ৭ͷٿମ͕͘৭ͷԁபʹিಥ͠ɺਫ৭ͷٿମβϥβϥͨ͠চ্Ͱେ͖͘ ͔͡Εɺ৭ͷԁபΘ͔ͣʹಈ͘ɻ GPT-4o-FT ਫ৭ͷٿ͕৭ͷٿʹͿ͔ͭΓɺਫ৭ͷٿ͕৭ͷٿʹ͔͡ΕΔɻ GPT-4o mini ৭ͷٿମ͕βϥβϥͨ͠চΛΓͳ͕Β৭ͷԁபʹ͔ͬͯస͕Γɺিಥͷॠ ؒʹٿମͷٸܹʹݮগ͠ɺԁபͦͷॏ͞ʹΑΓΘ͔ͣʹಈ͖͕͘ͳΔ͕ɺ ٿମͷ͕͞༏ҐͰ͋ΔͨΊɺٿମԁபΛԡ͠ग़͢Α͏ʹస͕Γଓ͚ɺԁபલ ํʹগ͠స͕Δ͔ɺΘ͔ͣʹճస͠ͳ͕ΒਐΉɻ GPT-4 − มԽ༧ଌϞσϧ ৭ͷٿ͕৭ͷԁபʹ͍Α͘িಥͯ͠ɺ ৭ͷٿ͕ԕ͘ʹ͖ඈ͞ΕΔɻ • চ͕βϥβϥ • ৭ͷٿͷํ͕৭ͷԁபΑΓ͍ܰ • ৭ͷํ͕৭ͷԁபΑΓ͍ ཧ ৗࣝ -
96 ࣮ݧ2-2ɿGPTΛ༻͍ͨཧৗࣝΛؚΉিಥঢ়گͷઆ໌ 91 Model BLEU@4↑ BERTScore↑ BLEURT↑ ROUGE↑ 含意 全⽂↑
含意 分割↑ G-EVAL↑ GPT-4o 31.3 54.5 32.3 32.3 48.5 55.6 65.4 GPT-4o-FT 25.6 59.3 27.8 28.7 52.3 61.4 66.7 GPT-4o mini 16.7 49.2 19.6 13.9 48.9 63.8 63.3 GPT-4 − − − − − − − มԽ༧ଌϞσϧ 55.8 82.2 49.7 56.4 67.9 80.2 92.4 -
96 • 2ͭͷମͷিಥ༧ଌ໘ʹ͍ͭͯɺڥମʹؔ͢ΔཧৗࣝΛ༩͑ͨͱ͖ɺিಥঢ়گΛ ΑΓৄࡉʹݴޠͰੜ ࣮ݧ2-1ɿݴޠੜྫ • ܾ·ͬͨܕʢͱͯɺ͍Α͘ɺԕ͘ʹɺΘ͔ͣʹʣͳͲΛֶश͍ͯ͠ΔͨΊɺ දݱͷ෯ͤ·͍ • ਓखʹΑΔจষఔදݱͷछྨ͕ଟ͍
ධՁࢦඪʹΑΔਫ਼ൺֱ • ୯ޠҰகͰͷධՁɿ • ༰ͷؚҙͰͷධՁɿߴ ࣮ݧ2-2ɿGPTΛ༻͍࣮ͨݧ • จষΛੜ͢Δ͜ͱ ಘҙ • ධՁࢦඪͷείΞɿ65.5 • ମͷೝࣝ༧ଌ͕͏·͘Ͱ͖͍ͯͳ͍͜ͱ͕είΞͷࠩʢ20ʣͱͯ͠දΕͨ ࣮ݧ2ɿ·ͱΊɾߟ 92 -
96 ఏҊݚڀͷ·ͱΊ 1. ҙͷ࣌ؒ෯Ͱͷ༧ଌ 1. PredNetͱTD-AEΛΈ߹Θͤɺώτͷ֊ߏΛͬͨҙͷ࣌ؒ෯Ͱ༧ଌՄೳͳϞσϧ ը૾ͷମͷಈ͖ͷཧղ 1. ը૾ͷڥʹ͓͚Δମͷಈ͖Λଊ͑ɺମͷཧಛੑͷมԽ͔Βɺମಉ͕࢜িಥ͢Δ
λΠϛϯάΛଊ͑ΒΕΔΈͷఏҊ ը૾ͷମͷিಥͷλΠϛϯάͷ༧ଌ 1. ը૾ͷڥͷମಉ͕࢜িಥ͢ΔλΠϛϯάΛɺମͷಈ͖ࢹ֮ใͷมԽ͔Β ༧ଌͰ͖ΔϞσϧͷߏங িಥঢ়گʹؔ͢Δݴޠੜ 1. ڥͷମͷ༧ଌͨ͠িಥঢ়گΛݴޠͰੜ͠ɺ࣮ੈքͱݴޠΛ݁ͼ͚ͭͨ 2. ڥͷ݅ΛՃͨ͠ͱ͖ɺڥͷಛੑΛ౿·͑ͨিಥΛઆ໌Ͱ͖ΔݴޠϞσϧʹΑͬͯɺিಥ ঢ়گΛΑΓৄࡉʹઆ໌Ͱ͖ΔΑ͏ʹͳͬͨ 94 -
96 ߩݙ • ৽ͨͳώτͷ࡞ۀϞσϧ • ώτͷ֊ߏΛදݱͨ͠༧ଌϞσϧͱɺҙͷ࣌ؒ෯Ͱ༧ଌ͕Ͱ͖Δਂੜֶश ϞσϧΛΈ߹Θͤͨ • গ͠ઌͷ༧ଌʢ1ඵʣ͕Ͱ͖ΔΑ͏ʹͳͬͨ •
ώτͷΑ͏ʹࢹ͔֮Βଊ͑ͨใΛͱʹɺମͷಈ͖Λ༧ଌͰ͖ΔϞσϧ • ը૾ಛྔཧγϛϡϨʔλʔͱ͍ܾͬͨΊΒΕͨͰͳ͘ɺ ༩͑ΒΕͨը૾ʢώτͷࢹ֮ใʹ૬ʣʹ͋Δମͷಈ͖Λଊ͑ɺকདྷͷিಥΛ ༧ଌͰ͖ΔΑ͏ʹͳͬͨ • ڥใΛߟྀͨ͠ɺিಥঢ়گΛΑΓৄࡉʹઆ໌Ͱ͖ΔΈ • ༧ଌͨ͠িಥ༰ΛଞऀʹݴޠΛհͯ͠ڞ༗Ͱ͖ΔΑ͏ʹͳͬͨ 95 -
96 ࠓޙͷ՝ • ظ༧ଌͷՄೳੑ • ݚڀ1ɿ1ඵఔ • ώτͷ༧ଌɿ࣍ͷɺܭըͳͲ • ࣮ੈքʹ͍ۙσʔληοτͷར༻
• CLEVRERɿ୯७ͳσʔλ • ंࡌΧϝϥͰͷσʔλɺώτ͕෦ͷதΛಈ͍͍ͯΔσʔλ • େنݴޠϞσϧࣗମͷཧಛੑͷཧղ • LLMΛγεςϜʹΈࠐΜ࣮ͩੈքཧղͷදݱ 96