Share
Embed
Inductive Bias and Graph Networks
Inductive biasとGraph Networkに基づくDeepLearningのトレンドについて取りまとめを行いました。
少々記載が難しいですが、出典の論文などを読むにあたって並行で参照すると良いのではないかと思います。
(細かい記載に誤りがある可能性がありますので、なるべく出典を参照しつつご確認ください。)
なお、完成版は下記で取り扱いました。
https://lib-arts.booth.pm/items/3162254
LiberalArts
June 05, 2021
Transcript
19 slides
-
7 痥 2 畍 Graph Network 睗 2 皹ךעյ"Relational inductive
biases, deep learning, and graph networks"⩧מ Graph Network כ Inductive bias מחַי 牞霼ױն2-1 硼ך Inductive biasյ2-2 硼ך Graph Network מח ַיאב牞霼ג䔿מյ2-3 硼ךאסױכ״脝㷋鉿ַױն 2.1 Inductive bias הכ 2.1.1 Inductive bias ך嚊銲 Inductive bias(䊟等溷فؕؓت) עրDeepLearning סؾشع٠٭ؠ圸ꅎ 脝ֻמֵגזי陭㴻⯼䳀ցכ槏闋כ虘ַ־כ䘼ַױնגכֻ ף氺⦐מ㸐י CNN 氠ַ갾עյ ր氺⦐עꁿ⤒סمؠجٜכ潸◦✑氠 יַסךյنٜؔذ⭦槏鉿ֹ׆כך杅䖇ꓪ䬂⮂ך׀ցכַֹ⯼䳀 氠ַיֽյ׆ Inductive bias כײם׆כֿך׀ױնCNN מꮹ յRNN ךע笠⮬ظ٭ذֿ꽄沁מ潸◦✑氠יַׂ坎㲳銨יֽյ ׆ Inductive bias כ脝ֻ׆כֿך׀ױն ׆סֹמյInductive bias עⷃם㳔肪כעױגꇙזג镄掾־㳔肪 מֵגזיסقنؚ٭ُ٤ت⻔┪雧ײ⺅篁ײכ脝ֻ׆כֿך׀ױ 7 -
8 痥 2 畍 Graph Network 2.1 Inductive bias הכ
ն׆׆ך峜䟨גַסֿ bias כ臝ׂכꇃ㳔肪 (overfitting) 䞯鱍㎇ꉌ ׀כַֹⷦ骭⺇ׄױֿյ րInductive bias ㎇ꉌցכַֹ עրInductive bias 嵛氠ցכַֹ镄掾־饗韢׆כֿ㝂ַ׆כ ךն 嚣锡䲖׳מֵגזיעյ րלסֹם⯼䳀ַֽיٓةٖ٭ٜٝيٜך לסֹם⭦槏圸ꅎ脝ֻ־ց⺅䪒ֹסֿ Inductive bias דכ脝ֻ סֿ虘ַ־כ䘼ַױն 2.1.2 Graph Network ך锷俑ך鎸鯹 [2018] Inductive bias מחַיס饗韢ע♧⯼יַױֿյꁿ䌑ס DeepLearning ס煝疴מֽׄ Inductive bias מחַיס脝㷋מֵגזי䫅 ֻיֽׂכ虘ַסֿ Graph Network[2018] ס韢倀ךն ㎫ 2.1: Graph Network 韢倀 ٬Relational inductive biases, deep learning, and graph networks https://arxiv.org/abs/1806.01261 ׆ס韢倀ךע Graph Neural Network מꫀꅙ MPNN(Message Pass- 8 -
9 痥 2 畍 Graph Network 2.1 Inductive bias הכ
ing Neural Network) յ Transformer מꫀꅙ NLNN(Non-local Neural Network) םל簡⻉溷מ⺅䪒ֹنٝ٭ّ٠٭ؠס Graph Network 㸬 ⪜מֵגזיյInductive bias מחַי⻎免מ饗韢ֿ鉿יַױն ㎫ 2.2: Inductive bias ס阾鼥 (Graph Network 韢倀) ┪阾ֿ Graph Network 韢倀מֽׄ Inductive bias ס阾鼥ךֿյ糽ך 虝♕ׄגכ׆ֿ嚣锡䲖׳מֵגזיעꓨ锡ךն րInductive bias 氠ַ׆כך㳔肪ٜؓإٛثّע⨲⩰꽄⛺尴״׆כֿך׀ցכי ֽյ2-1-1 硼ך牞霼ג⫐㵼כ⻉蔹ױն 9 -
1 0 痥 2 畍 Graph Network 2.1 Inductive bias
הכ ㎫ 2.3: Table.1(Graph Network 韢倀) ㎫ 2.3 ֿ Graph Network 韢倀ס Table.1 ךֿյDeepLearning סٓ ةٖ٭ٜ (׆׆ךע Component כ銨槁יַ) ׇכס Inductive bias מחַי饗韢ֿ鉿יַױն׆׆ך銨מַֽי Entity ע阛砯ס㸐骭յ Relation ע Entity סꫀꅙ䙎յInductive bias ע⭦槏圸ꅎמֽׄ⯼䳀מח ַי銨槁יַױն沑ײꁎײ (Convolutional) ע㹾䨾溷յٛٝ٤ع (Recurrent) עꅙ籽溷 (Sequential)յGraph Network ע⚈䟨 (Arbitrary) כ 銨阾יַסע䫅ֻיֽׂכ虘ַכ䘼ַױն ׆ס雄❿מֵגזיעյ沑ײꁎײٛٝ٤عמ♳㴻 Inductive bias ע㝕׀ׂյGraph Network ע⚈䟨ךյFully connected מ♳㴻 Inductive bias ע㸯ַ׆כֿ鞅ײ⺅ױն׆׆ךם圸ꅎ溷ם䱱筺 鉿ֹמֵגזיע Graph Network 氠ַסֿ┞薭溷דכ䘼ַױֿյ ┞䍲 Fully connected ╚䖥מنؚ٭تי Inductive bias 㸯ׂג ┪ך䱱寛כַֹס脝ֻ偙ס┞חדכ䘼ַױն ׆מꫀꅙי⮂י׀גסֿ MLP-Mixer gMLP דכ脝ֻי虘ַכ 䘼ַױֿյ׆ס MLP-Mixer gMLP Graph Network ס阾岺ך銨 כ脝ֻױסךյאב❈ַ⮔ׄכַֹע⪢✄ Graph Network ס銨阾⩧מ俠槏ג┪ך Inductive bias 脝ֻכַֹסֿꈌ ⮗ךעםַ־כַֹסֿ瞉脢ס锶闋ךն כַֹ׆כךյ籽ׂ 2-2 硼ךע׆׆ױךס需מꫀꅙי Graph Net- work[2018] ס韢倀ס銨阾ס牞霼鉿ַױն 10 -
1 1 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 2.2 Graph Network ה DeepLearning ך侭椚 2.2.1 Graph Network ך邌鎸 (GN block) 2-2-1 硼ךע Graph Network 韢倀מֽׄ Graph Network ס銨阾מח ַי牞霼鉿ַױնGraph Network ס韢倀מַֽיյْؕ٤ס⭦槏ע GN block ךٓةٖ٭ٜⵊױն ㎫ 2.4: GN block(Graph Network 韢倀) ㎫ 2.4 ֿ Graph Network ס韢倀מֽׄ GN block ס俙䑑ס銨阾ךն 剳偆ꫀ俙 (update function) ס φ כ겏笴ꫀ俙 (aggregation function) ס ρ מ זי GN block ע圸䧯ױն剳偆ꫀ俙 (update function) ס φ כ겏笴 ꫀ俙 (aggregation function) ס ρ עאב 3 חֵױֿյ剳偆ꫀ俙ע ؙشةյؿ٭غյءٚنס㺲䙎ס 3 חךյ겏笴ꫀ俙עؙشة̕ؿ٭غյؿ٭ غ̕ءٚنյؙشة̕ءٚنס 3 חֿ銨阾יַױն ׆ס銨阾דׄך槏闋סע㝕㜟םסךյ┞薭溷ם MPNN(Message Passing Neural Network) ❛מյ⪽✄溷מ銨阾牞霼ױն 11 -
1 2 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 ㎫ 2.5: GN block כ MPNN(Graph Network 韢倀) ㎫ 2.5 ע Graph Network 韢倀ס Figure.4 ס┞ꌃךֿյGN block ס䓺 䑑ךס MPNN(Message Passing Neural Network) ס銨阾ךնMPNN ך ע剳偆ꫀ俙עؙشةյؿ٭غյءٚنס㺲䙎ס 3 ח氠ַյ겏笴ꫀ俙ע ؙشة̕ؿ٭غյؿ٭غ̕ءٚنס 2 חֿאב氠ַיַױն׆ס ⭦槏ע MPNN(Graph Neural Network) ס 1 חס㺽ס阛砯מ㸐䗎יַױ ն׆׆ךյu עءٚنס⪢✄ס㺲䙎ךֵյNode Classification ذتؠס ֹמذتؠ陭㴻מזיע┮锡ס㖪⻉ֵױն ׆ס GN block ס銨阾ע㸴չ⫩ꩽםⷦ骭⺇ׄױֿյ RNNյ Transformer םל坎չםؾشع٠٭ؠ圸ꅎ⺅䪒ֹמֵגזי俠槏ַכַֹ׆כ ך㸬⪜גסךעםַ־כ䘼ױֹ㸴虘ַ銨阾ֵאֹם宜 סךֿյGraph Network ס韢倀ע䑛氠ַֹםסך韢倀 ס銨阾מ⻉יֽׂ偙ֿ虘אֹך ն 2-2-2 硼ךע GN block 氠ַי Transformer ס牞霼鉿ַױն 12 -
1 3 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 2.2.2 Transformer ה Graph Network(GN block) 2-2-2 硼ךע 2-2-1 硼ך牞霼ג GN block ס銨阾מ㕈טַי Transformer ס銨阾מחַי牞霼ױնTransformer ע NLNN(Non-local Neural Net- work) ס┞甦כיַסךյNLNN מꫀ阾鼥牞霼ױն ㎫ 2.6: GN block כ NLNN(Graph Network 韢倀) 韢倀ס Figure.4 ס┞ꌃ獏ג㎫ 2.6 ךע NLNN מחַיס阾鼥ֿ鉿 יַױնTransformer ⩧מ脝ֻםյφe כ ρe→v ֿ Dot product attentionյφv ֿ Feed Forward Network(FFN) מ㸐䗎כ脝ֻי虘ַ־ כ䘼ַױն 13 -
1 4 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 ㎫ 2.7: GN block כ NLNN Ώ (Graph Network 韢倀) ㎫ 2.8: GN block כ NLNN ΐ (Graph Network 韢倀) ㎫ 2.9: GN block כ Transformer(Graph Network 韢倀) Transformer ס⭦槏ס霄箖ע㎫ 2.7㎫ 2.9 מ阾鼥יַױն俙䑑מ 14 -
1 5 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 חַיע坎չם煝疴┞חס銨阾ךױכ״ג׆כך韢倀ס阾鼥蔦✄ֿ㸴չ ־מַׂכ䘼ג״յ㎫ 2.7 ⩧מ Transformer ס銨阾⫙㴻聋 ױն φe α (VQ, VK ) = softmax QKT dk φe β (VV al ) = VV al ρe→v(E, VQ, VK, VV al ) = φe α (VQ, VK )φe β (VV al ) = softmax QKT dk VV al = Vres φv = FFN(Vres ) ┪阾ס俙䑑ע㎫ 2.7 כ Transformer ס韢倀脝מ⫙圸䧯鉿ַױ גնVQ յVK յVV al עאב Transformer 韢倀ס Q כ K כ V 銨י ַױնױגյVres ע Dot product attention ס⭦槏䔿ס鉿⮬յFFN ע Transformer מֽׄ Feed Forward Network(MLP כ⻎聋) 銨יַױ ն겏笴ꫀ俙 ρ ס阾鼥祔ⷃמג״מ鉿⮬銨阾氠ַי⪢יסؿ٭غכ ؙشة┞䍲מ⺅䪒ַױגֿյGraph Network ס韢倀ךע┞ח┞ח⺅ 䪒זיַֹך阾鼥ֿ㸴չ沌םױն 2-2-1 硼ךע MPNN מꫀיյ2-2-2 硼ךע Transformer NLNN ס ┞❛ס镄掾־牞霼鉿ַױגն籽ׂ 2-2-3 硼ךע׆׆ױך牞霼ג GN block(Graph Network) ס銨阾ס寯氠䙎ס牞霼מֵגזיյאס♑ס DeepLearning מחַי GN block ס銨阾⩧מ牞霼鉿ַױն 2.2.3 Graph Network כ圫ղז DeepLearning 邌ׇ ׆׆ױךך MPNN NLNN 牞霼ױגֿյ2-2-3 硼ךע Graph Network 氠ַי׆׆ױך⺅䪒זג♧㜽ס DeepLearning סؓ٭؞طؠ زٔ牞霼ױն 15 -
1 6 痥 2 畍 Graph Network 2.2 Graph Network
ה DeepLearning ך侭椚 ㎫ 2.10: GN block כ RNN(Graph Network 韢倀) ㎫ 2.10 ע Graph Network 韢倀ס Figure.4 ס┞ꌃך׆מ RNN 銨כיַֹךնRNN ךעꅙ籽溷 (Sequential) מؙشة־ ؿ٭غמ⚻ꇖֿ鱍׆ג״յ׆ךע銨槁ך׀םַסךעםַ־כ脝ֻ סךֿյGraph Network ס韢倀ךע Figure.6 ך鏿俙ס GN block ס ꅙ篙מחַי⺅䪒יֽյ׆הכס篁ײ⻉ך RNN 銨槁 כ脝ֻיַֹךն 16 -
1 7 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ㎫ 2.11: GN block סꅙ篙 (Graph Network 韢倀) ㎫ 2.11 מ韢倀ס Figure.6 獏ױגն2-2-1 硼ך⺅䪒זג MPNN כ 2-2-2 硼ך⺅䪒זג NLNN(Transformer etc) ךע㎫ 2.11 ס (a) 氠 ַיֽյ׆׆ךע (c) 氠ַגכ脝ֻכ䘼ַױն ׆ ס ֹ מ GN block כ RNN א ס ꅙ 篙 氠 ַ ד ׄ ך 㝂 ׂ ס DeepLearning סؓ٭؞طؠزٔ銨槁ך׀כ脝ֻי虘ַ־כ䘼ַױն 2.3 Inductive bias ה Graph Network 2-1 硼ך Inductive bias מחַיյ2-2 硼ך Graph Network מחַי אב⺅䪒זגסךյ2-3 硼ךעאמ㕈טַיⴭ䭇溷ם镄掾־ DeepLearning סؓ٭؞طؠزٔמחַיױכ״脝㷋鉿ַױն 2.3.1 Inductive bias וך״ֲח⟎㹀ַׅ 2-3-1 硼ךע Inductive bias מ㕈טַיؼٖ٭ٜٚؾشع٠٭ؠס圸ꅎ לסֹמ♳㴻־מחַי脝㷋鉿ַױն2-1-2 硼ס㎫ 2.2 ך牞霼 17 -
1 8 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ג Inductive bias ךֿյؼٖ٭ٜٚؾشع٠٭ؠסؓ٭؞طؠزٔדׄ ךםׂيؕث笠ס䩘岺מֽׄ◜⯼⮔䉘 (prior distribution) յ㳔肪מֵ גזיס婞⯵ⵊ꽃סꃯⲎםל㝂㼜מ廌כיַױն ׆סֹמ Inductive bias מע坎չם锶偙ֵֿױֿյ㝂㸴ꈌ䓜ם锶 偙י虘ַסךֵףյ׆ױךֵ瓦䍲篙卸媘י׀ג煝疴מֽׄ Inductive bias ע㸴ם־䟨⽱ֵֿכ脝ֻי虘ַכ䘼ַױնאסג ״յط؞تع韢倀םלס劔⻏םסעאב潸䗎ס Inductive bias ؆דٓظٛ٤ءֿםיַכ脝ֻי虘ַ־כ䘼ַױն׆ס需עؼٖ٭ ٜٚؾشع٠٭ؠמכלױյب٤وٜם㎇䊟⮔卥յⴢꏕه٭تطؔ٤ ءםלס Tree-based סؓوٞ٭زםלאס♑ס塌唩㳔肪סٜؓإٛثّך ⻎坎ךն ׆⺇ׄיյInductive bias מחַי脝ֻ갾עյ׆ױךס煝疴מ ֽׄٓظٛ٤ءס⯼䳀מאב✇ֿ翝־י׀גס־כַֹ镄掾ך虝չ כ䪻䳢׆כֿꓨ锡דכ䘼ַױնגכֻף Tree-based סٜؓإٛثّ עր✇־ס匛⚂מ㕈טׂ⮔㼜繪ꂉ׆כցֿ⯼䳀מ翝־גٜؓإٛث ّךն׆ Inductive bias כ脝ֻ׆כֿך׀־כ䘼ַױն ׆סֹמյ րאבס䩘岺מַֽיאבלסֹם⯼䳀ֿ翝־ יַ־ցמחַי䊬מ脝䢩םֿ劳ױַ Inductive bias ס♳㴻מחַ י嗱阧כַֹסֿ䖩锡םסךעםַ־כ䘼ַױն 㝕卽ס㕈勓溷ם脝ֻ偙מחַי牞霼ך׀גסךյ2-3-1 硼ע׆׆ױךכյ 籽ׂ 2-3-2 硼ךע DeepLearning מꫀ➳ٓةٖ٭ٜס Inductive bias מ חַי牞霼ױն 2.3.2 CNNծRNNծGNNծTransformerծMLP 2-3-2 硼ךע CNNյRNNյGNNյTransformerյMLP םלמ䞯㴻 Inductive bias מחַי牞霼鉿ַױն CNN מ䞯㴻 Inductive bias עյ ր氺⦐מַֽיעꁿ⤒סمؠجٜס ⡑כ潸ꫀ䭥חցכַֹסֿ╭ם♳㴻ךնCNN ך氠ַיַ沑ײꁎ 18 -
1 9 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ײ⭦槏עنٜؔذ⭦槏כ⻎坎ךյنٜؔذظ٭ذ־蔦ⳛ㳔肪כַֹ סֿ CNN ס㳔肪ס╭ם嵣ךնױגյCNN ך氠ַיַ Pooling ⭦槏ע氺⦐ס㐁緐מ杅䖇ꓪס䬂⮂䟨⽱יֽյ׆闋⦐䍲┫ׅ י氺⦐ס䟨⽱⻉ַע㜟םַכַֹ⯼䳀 (Inductive bias) מ㕈טַיַ כ脝ֻ׆כֿך׀ױն RNN(׆׆ךע LSTMյGRU ײױ) מ䞯㴻 Inductive bias עյ ր笠⮬ظ٭ذע׆ױךס⮂槁篙卸מ㕈טַי姌ֿ气䧯ցכ脝ֻ י虘ַ־כ䘼ַױնRNN 笠ס䩘岺ע♑ס䩘岺מ嬟יאל篙卸ֿ⮂ יַםַכַֹֹמ䘼ױֿյRNN 笠ס䩘岺עꩽַ笠⮬ס⺅䪒 ֵַֿױֹױׂ鉿־םַכַֹסֵֿכיַױնאַֹֹ䟨⽱ך RNN 笠 Transformer סֹם self-attention ي٭تס䩘岺ס偙ֿ笠 ⮬ظ٭ذס⺅䪒ַמֵגזיע虘ַ Inductive bias ♳㴻ך׀גכ 锶׆כך׀ס־ױ؆ն GNN ע⻎免מ┰ֻגءٚنס圸ꅎמ㕈טַי阛砯鉿ֹכַֹסֿ Inductive bias כי氠ַיַכ脝ֻי虘ַכ䘼ַױնGNN 栄聋ך脝ֻג갾ע 2-2-1 硼ך⺅䪒זג MPNN(Message Passing Neural Network) ْؕ٭ةכ虘ַכ䘼ַױֿյ䌮聋ך脝ֻכאא♑ ס Neural Network ע⪢יءٚنמ㕈טַיַכ锶׆כך׀סךյ DeepLearning ⪢✄ֿ GNN דכ脝ֻ׆כך׀כ䘼ַױն2-2 硼ך ⺅䪒זג Graph Network ס煝疴ךע׆סꁊס簡┞溷ם⺅䪒ַֿי ֽյ2021 䌑免ך需꾴ס MLP ي٭تס䩘岺 Graph Network ס┞甦כ锶 ׆כך׀ױն Transformer ע self-attention מ㕈טׂ䩘岺כֿהךֿյ GNN 溷ם ءٚنאבסؿ٭غ阋靣⭦槏ס㖪⻉עⷃ靣ס꿔⛣䍲 (Dot product attention ס Dot product ע⫐畤ךֵյCos 꿔⛣䍲כ㕈勓溷מע⻎聋ך) מ㕈טַי圸碎 MPNN כ⻎坎ס⭦槏鉿ֹכ脝ֻ׆כך׀ױնא סג״յ րGNN 脝ֻמֵגזיסءٚنؿ٭غס꿔⛣䍲מ㕈טַי气 䧯ך׀ցכ脝ֻסֿ Transformer ס Inductive bias כ锶םי虘ַ־ כ䘼ַױն 19 -
2 0 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ױגյ 2021 䌑免掾ךׂ需꾴מ┪ֿסֿ MLP ي٭تס䩘岺ךն րself- attention vs MLPցס镄掾ך锶ֿהךֿյאא Transformer 蔦 ✄յattention ⭦槏♧㜽סכ׆ע MLP ךֵסךյ㵅עאלꇙַֿ ֵ־כ阋ֻףאֹךעםַסךעםַ־כַֹסֿ瞉脢ס锶闋ךն2-2 硼ס Graph Network ס煝疴ךעؿ٭غⷃ⛺ס MLP ⭦槏מؙشة闋 גؿ٭غꪨ⭦槏㸬⪜גס Graph Network כ㴻聋יַגסךյ MLP-Mixer gMLP םלס䩘岺 Transformer כ嬟鼛ג갾מյלה Graph Network כ锶׆כ־אל㝕׀םꇙַעםַסךעםַ ־כ脝ֻי虘ַסךעכ䘼ַױն ׆׆ױך坎չם DeepLearning ס圸䧯מחַי牞霼鉿ַױגֿյ㕈勓 溷מע Graph Network(㴻聋ֿױד㴻ױזיַםַⷦ骭סג״յGNN כ ־לֹ־ס偂阋עꉌׄױ) מ㕈טַיյ Inductive bias 㸬⪜י圸䧯 尴״כ脝ֻסֿ虘ַסךעכ䘼ַױն籽ׂ 2-3-3 硼ךע MLP-Mixerյ 2-3-4 硼ךע gMLP מחַיאב⺅䪒ַױն 2.3.3 MLP-Mixer(MLP ) 2-3-3 硼ךע MLP-Mixer מחַי牞霼鉿ַױն ㎫ 2.12: MLP-Mixer ס⭦槏嚣锡 (MLP-Mixer 韢倀) ٬MLP-Mixer: An all-MLP Architecture for Vision https://arxiv.org/abs/2105.01601 ㎫ 2.12 ע MLP-Mixer ס韢倀ס Figure.1 ־ס䫕穀ךֿյ㎫מֽׄ 20 -
2 1 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network MLP1 עقشزꪨ (ViT םלס氺⦐⭦槏⯼䳀ס阾鼥סג״ Patch כ יַױֿյGNN דכؿ٭غյNLP דכ token מם׆כע䫅ֻיֽׂ כ虘ַכ䘼ַױ) ס MLP מ悍砯յMLP2 עאבסقشز⫐ס MLP 悍砯מ㸐䗎כ槏闋כ虘ַ־כ䘼ַױն MLP-Mixer ס⭦槏 GNN ס镄掾־锶םյMLP2 עؿ٭غ⫐ ךס MLP ךֵյ׆ע Transformer ס FFN(Feed Forward Network) ס⭦槏מ┞蔹ױն┞偙ך MLP1 ע Dot product attention מ㕈טׂ Transformer כע沌םյⷃמؿ٭غꪨ (قشزꪨ) ךס MLP מ悍砯 כםױնMLP1յMLP2 ס悍砯מֵגזיעقْٚ٭ذ氠ַיֽյ 杅מ MLP1 ךע GNN ס걋䱸鉿⮬מ㸐䗎قْٚ٭ذס㳔肪ֿ鉿י ַ׆כע䪻䳢יֽׂכ虘ַ־כ䘼ַױն זׂ脝ֻםյ րTransfomer ךעⷃ靣ס꿔⛣䍲מ㕈טַיءٚن 圸碎יַגֿյMLP-Mixer ךעءٚنס圸ꅎ㳔肪ցכַֹ锶偙 ⺪茣דכ䘼ַױն 2.3.4 gMLP(MLP Ύ) 2-3-4 硼ךע gMLP מחַי牞霼鉿ַױն 21 -
2 2 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ㎫ 2.13: gMLP ס⭦槏嚣锡Ώ (gMLP 韢倀) ٬Pay Attention to MLPs https://arxiv.org/abs/2105.08050 ㎫ 2.13 ע gMLP ס韢倀ס Figure.1 ךն㸴׆ס㎫דׄדכ־מׂ ַךֿյ㎫ 2.14 םלמ俙䑑ס阾鼥ֵֿյChannel Proj ע Transformer ס FFN ס⭦槏כ⻎坎ךֵכ韢倀מ阾鼥יֽյאסג״׆׆ךע ؿ٭غ⫐ךס MLP ס阛砯ֿ鉿יַ׆כֿ־ױն 22 -
2 3 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ㎫ 2.14: gMLP ס⭦槏嚣锡ΐ (gMLP 韢倀) ┞偙ך Spatial Gating Unit ֿ gMLP 栃蔦ס⭦槏ךֿյ┫阾ס㎫ 2.15 ך俙䑑םלס阾鼥ֵֿױն 23 -
2 4 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ㎫ 2.15: Spatial Gating Unit ס⭦槏 (gMLP 韢倀) MLP-Mixer מ嬟鼛י㸴չ⭦槏סْؕ٭ةֿח׀מַׂג״յ׆ה MLP-Mixer ╚䖥ך䪻䳢יֽׂ偙ֿ虘אֹםⷦ骭ךնױגյ 2-3-3 硼ך⺅䪒זג MLP-Mixer כ 2-3-4 硼ך⺅䪒זג gMLP עלה Graph Network ס阾岺ך⺅䪒ֻ׆כ䫅ֻיֽׂכ虘ַ־כ䘼ַ ױն 2.3.5 ➙䖓ך㾜劄חאְגך✮䟝 睗 2 皹ךע׆׆ױך坎չם氠靣מחַי牞霼ױגֿյ րGraph Network ס韢倀םלמ㕈טׂ✇־ס簡┞溷ם銨阾⩧מ Inductive bias לס ֹמ陭阛־ցמ煝疴סنؚ٭تֿ瓌סךעםַ־յכַֹסֿ瞉 脢ס锶闋ךն銨阾מꫀיע✇־ס簡┞銨阾ֿ尴ױכ虘ַךֿյ ֹ㸴免ꪨֿ־־־ױ؆ն 24 -
2 5 痥 2 畍 Graph Network 2.3 Inductive bias
ה Graph Network ױגյ րTransformer vs MLPցמחַיע♀䔿坎չם饗韢ֿ⮂יׂכ 䘼ַױֿյTransformer 蔦✄ MLP כ锶םסךאל㝕׀ם䟨⽱ס םַ嬟鼛דכ䘼ַױնꫀꅙי潲䠊溷םْؕ٭ةֿח־״כ虘ַכ䘼ַ ױסך MPNN ⩧מ GNN סْؕ٭ة䲖ײյGraph Network ס銨阾 ך嵞气ס煝疴״י俠槏כַֹסֿꓨ锡ךעםַ־כ䘼ַױն ױגյInductive bias ס脝㷋ֿ儨♀㙟ֻיַסךյ韢倀攍갾מ ע Inductive bias ס⽟ꁊס饗韢⨲⩰溷מ牞霼כַֹס㝕◜ךעם ַ־כ䘼ַױն 25