Share
Embed
Transformer and Graph Neural Network
下記で取り扱ったTransformerの簡易的な解説を公開します。
・Transformer と画像処理
https://lib-arts.booth.pm/items/2741653
TransformerはLLMのベースに用いられるなど、応用の幅が広いので抑えておくと良いと思います。
LiberalArts
February 22, 2021
Transcript
14 slides
-
ୈ 2 ষ Transformer ͷجຊཧղ ୈ 2 ষͰɺ ຊॻͷٕज़తͳཧղͷϕʔεͱͯ͠ Transformer
ͷج ຊͷཧղʹ͍ͭͯऔΓѻ͍·͢ɻҎԼɺ 2.1 અͰ RNN ͱ Attention ͷ֓ཁʹ͍ͭͯऔΓѻ͍ɺ2.3 અͱ 2.4 અͰ Dot Product Attention Λத৺ͱ͢Δ Transformer Ϟδϡʔϧͷղऍ͕ߦ͑ΔΑ͏ʹ 2.2 અ ͰͦͷϕʔεͱͳΔߟ͑ํͱͯ͠ Message Passing ͱάϥϑχϡʔ ϥϧωοτϫʔΫ (Graph Neural Network) ʹ͍ͭͯ͝հ͠·͢ɻ 2.4 અ·ͰͰ Transformer ʹ͍ͭͯѲ্ͨ͠Ͱɺ2.5 અͰ 3.2 અ ͰऔΓѻ͏ Vision Transformer(ViT) Ͱ༻͍ΒΕΔ BERT ͷωο τϫʔΫߏʹ͍ͭͯ֬ೝ͠·͢ɻ 2.1 RNN ͱ Attention 2.1 અͰ RNN ͱ Attention ʹ͍ͭͯऔΓѻ͍·͢ɻ https://lib-arts.booth.pm/items/1834866 RNN ͱ Attention ʹ্͍ͭͯهͷୈ 1 ষʙୈ 2 ষͰৄ͘͠औΓѻ͍· ͨ͠ͷͰɺຊॻͰ؆୯ͳ֓ཁͱͦͷղऍʹ͍ͭͯ·ͱΊ͍ͨͱࢥ͍·͢ɻ 15 -
ୈ 2 ষ Transformer ͷجຊཧղ ˛ਤ 2.1 RNN ϕʔεͷܥྻมϞσϦϯά (Seq2seq
จ Figure 1) https://arxiv.org/abs/1409.3215 ˛ਤ 2.2 ܥྻมϞσϦϯάͷ Attention ͷಋೖ (Attention จΑΓ) https://arxiv.org/abs/1409.0473 ਤ 2.1 ͱਤ 2.2 ͦΕͧΕ RNN ܥΛ༻͍ͨ Seq2seq ͱɺAttention Λಋ ೖͨ͠ݚڀͰ͢ɻͲͪΒػց༁ (Machine Translation) ʹ͍ͭͯऔΓ 16 -
2.1 RNN ͱ Attention Μͩಉ࣌ظͷݚڀͰ͢ɻRNN ࠶ؼతʹχϡʔϥϧωοτϫʔΫΛܭࢉ͠ ͍ͯ͘ͷʹରͯ͠ɺAttention ӅΕͷॏΈ͚Λܭࢉ͠·͢ɻ ͯ͞ɺͪΖΜ͜ΕΒͷܭࢉͷྲྀΕͷѲॏཁͳͷͰ͕͢ɺΑΓॏཁ ͳͷͦΕͧΕͷॲཧͷߏ͕ͲͷΑ͏ͳҙຯΛ͔࣋ͭͰ͢ɻRNN
ܥܥ ྻͷϞσϦϯάΛऔΓѻ͏ʹ͋ͨͬͯͷγϯϓϧͳߟ͑ํͰ͋ΔҰํͰɺ ͕͋Δͱ͢Εܥྻ͕͘ͳΔʹͭΕͯใͷऔΓѻ͍͕͘͠ͳΔ ͱ͍͏Ͱ͢ɻLSTM GRU ͳͲɺ͍ܥྻΛऔΓѻ͏ͨΊʹ༷ʑͳ ͕ͳ͞Ε͖ͯ·͕ͨ͠ɺ݁ہͷͱ͜ΖϞδϡʔϧશମͰൺྻతʹ ใ͕ࣦΘΕΔ͜ͱʹมΘΓͳ͘ɺ͍ܥྻͷऔΓѻ͍ʹ͕͋Γ·ͨ͠ɻ ͨͱ͑ RNN ͷʹ͋ͨͬͯใ͕͔͠Βͳ͍ͱԾఆ͢Δͱɺ 0.510 = 0.000976 · · · ͷΑ͏ʹܥྻͷ͕͞ 10 Ͱ͋Δ͚ͩͰ࠷ॳͷ୯ޠͷ ใ 0.1 ˋҎԼ͔͠Βͳ͘ͳΓ·͢ɻ ͜Εʹର͠ɺAttention ॏΈ͚ͱͯ͠ܭࢉ͢ΔͨΊɺΕͨใ ͦΕ΄Ͳແཧͳ͘औΓѻ͑·͢*1 ɻAttention ॲཧͰܭࢉॲཧͷաఔ͕ ൺྻతͰͳ͍͚ͩͰɺΕͨใͷऔΓѻ͍͕֨ஈʹ༰қʹͳΓ·͢ɻͪ ͳΈʹൺྻతͳऔΓѻ͍ͱ͍͏ҙຯͰ CNN ͳͲͷͷಉ༷Ͱ͢ ͕ɺ1.2 અͰ͝հͨ͠ ResNet ͷܗࣜΛऔΓೖΕΔ͜ͱͰͪ͜Β͋Δఔ ճආ͕Մೳͩͱࢥ͍·͢ɻ ·ͨɺ Attention Λܭࢉ͢Δ্Ͱཧղ͓͔ͯ͠ͳ͍͚ͯ͘ͳ͍ͷ͕ɺ ॏΈ ͷܭࢉͰ͢ɻ͜ͷॏΈͷܭࢉʹؔͯ͠৭ʑͱํ๏͕͋Γ·͕͢ɺೖྗܥྻ ΛॏΈͷܭࢉʹͦͷ··ར༻Ͱ͖Δ self-attention ͕༗ྗͰɺTransformer self-attention ͷߏʹجͮ͘ͱߟ͑Δ͜ͱ͕Ͱ͖·͢ɻTransformer Ͱ༻͍ΒΕ͍ͯΔ self-attention ͷߏͱͯ͠ɺ2.3 અͷ Dot Product Attention 2.4 અͷ Transformer ϞδϡʔϧͰऔΓѻ͍·͢ɻ *1 LSTM GRU ಉ༷ʹ͍ܥྻ͕औΓѻ͑ΔΑ͏ͳվྑͰ͋Δͱཧղͯ͠ྑ͍ͱࢥ ΘΕ·͕͢ɺ໌ࣔతʹͷܗࣜͰද͞ΕΔ Attention ͷԋࢉͷύϑΥʔϚϯεʹٴ ͳ͍ͱ͍͏ͷ͕ҰൠతͳධՁͱղऍͰ͖Δͱࢥ͍·͢ɻ 17 -
ୈ 2 ষ Transformer ͷجຊཧղ 2.2 Message Passing ͱ GNN
2.2 અͰ Message Passing ͱ Graph Neural Network ʹ͍ͭͯऔΓѻ ͍·͢ɻ https://www.amazon.co.jp/dp/B08JGM3JNP ৄ্͘͠هͷୈ 3 ষʙୈ 4 ষͰऔΓѻ͍·͕ͨ͠ɺTransformer ͷ Dot Product Attention(self-attention ͷҰछ) ϝοηʔδͷΈ (Message Passing Paradigm) Λར༻ͨ͠ Graph Neural Network ͱͯ͠ཧ ղ͢Δ͜ͱ͕Ͱ͖·͢ɻ Dot Product Attention ͷॲཧ 2.3 અͰऔΓѻ͍·͕͢ɺDot Product Attention Λ୯ʹॲཧͷྲྀΕͱͯ͠Ѳ͢Δ͚ͩͩͱͳ͔ͳ͔Πϝʔδ͕༙ ͖ͮΒ͍Ͱ͢ɻͦ͜Ͱ 2.2 અͰ 2.3 અͰҙຯ߹͍Λཧղ͢Δʹ͋ͨͬͯ ͷલஈ֊ͱͯ͠ Message Passing Paradigm ʹج͍ͮͯάϥϑχϡʔϥϧ ωοτϫʔΫ (Graph Neural Network) ͷ֓ཁΛ֬ೝ͠·͢*2 ɻ ˛ਤ 2.3 άϥϑͷྫ (Wikipedia άϥϑཧΑΓ) https://ja.wikipedia.org/wiki/άϥϑཧ *2 ʮMessage Passing Paradigm άϥϑχϡʔϥϧωοτϫʔΫͷݚڀͷ 1 ͭͰ͋Δʯ ͱ͍͏ͷ͕ҰൠతͳධՁͰ͋ΔҰํͰɺ ʮάϥϑχϡʔϥϧωοτϫʔΫ Message Passing Paradigm Ͱ֓ͶදͤΔʯͱղऍ͢Δ͜ͱՄೳͩͱࢥ͍·͢ɻ 18 -
2.2 Message Passing ͱ GNN ·ͣɺάϥϑʹ͍ͭͯਤ 2.3 ͷΑ͏ʹ (ϊʔυ) ͱઢ
(Τοδ) Ͱؔ ੑΛදͨ͠ͷͱཧղ͓ͯ͘͠ͱྑ͍Ͱ͢ɻ۩ମతʹӺͷ࿏ઢਤͳͲ͕Θ ͔Γ͍͢Ͱ͢ɻͯ͞ɺӺͷ࿏ઢਤ͕͋Δࡍʹࠞࡶ۩߹ͳͲΛ༧ଌ͍ͨ͠ࡍ ͳͲͷϞσϦϯάʹศརͳͷ͕ Message Passing ͷߟ͑ํͰ͢ɻ ˛ਤ 2.4 Message Passing ͷࣜ (MPNN จΑΓ) https://arxiv.org/abs/1704.01212 ࣜʹ͍ͭͯਤ 2.4 ͷΑ͏ʹද͢͜ͱ͕Ͱ͖·͕͢ɺͬ͘͟Γ௫ΉͳΒ ਤ 2.4 ͷ (1) ͕ࣜిंʹΔɺ(2) ͕ࣜిं͔Β߱ΓΔΠϝʔδͰ௫Ήͷ͕ ྑ͍͔ͱࢥ͍·͢*3 ɻ ˛ਤ 2.5 Message Passing ͷΠϝʔδ (ۙͷӅΕͷͷܭࢉ) *3 ͋͘·ͰΠϝʔδͳͷͰ͋·Γݫີʹߟ͑ա͗ͳ͍Ͱྑ͍ͱࢥ͍·͢ɻ 19 -
ୈ 2 ষ Transformer ͷجຊཧղ ͜ͷ Message Passing Λ͋ΔӺ v1
ʹ͍ͭͯணͨ͠ͷ͕ਤ 2.5 ͱߟ͑Δ ͜ͱ͕Ͱ͖ΔͷͰ͕͢ɺ͜͜ʹύϥϝʔλͷֶशΛಋೖͨ͠ͷ͕άϥϑΈ ࠐΈͰ͢ɻ ࣜ 2.1: άϥϑΈࠐΈ (ύϥϝʔλॲཧͱΈࠐΈ) h (l+1) i = σ ⎛ ⎝b(l) + j∈N(i) 1 cij h (l) j W(l) ⎞ ⎠ άϥϑΈࠐΈࣜ 2.1 ͷΑ͏ʹද͢͜ͱ͕Ͱ͖ɺW(l) ͕ύϥϝʔλΛද ͍ͯ͠Δͱߟ͑Δͱྑ͍Ͱ͢ɻ͜͜ͰύϥϝʔλΛಋೖ͢ΔҙٛΛӺͷࠞࡶ ۩߹ͷ༧ଌͷྫΛݩʹߟ͑ΔͱɺਓͷྲྀΕͷಋઢͷ֬อͳͲʹΑͬͯ͋ΔӺ Ͱ߱Γͨ٬ͷਓ͕ͦͷ··ࠞࡶ۩߹ʹө͞ΕΔ༁Ͱͳ͍͜ͱʹ͋Γ ·͢*4 ɻ͜ΕΛऔΓѻ͏ʹ͋ͨͬͯύϥϝʔλΛಋೖ͢Δ͜ͱͰɺͦΕͧΕ ͷӺͰͬͨ٬ͷͱ࿏ઢਤΛೖྗͱͯ͠༩͑ͯɺग़ྗΛͦΕͧΕͷӺͷ ࠞࡶͱͯ͠ڭࢣ͋ΓֶशΛ͢ΕɺӺͷࠞࡶͷ༧ଌ͕ՄೳʹͳΓ·͢ɻ ্هͷΑ͏ʹάϥϑΈࠐΈΛ༻͍ͯߏஙͨ͠χϡʔϥϧωοτϫʔΫΛ άϥϑχϡʔϥϧωοτϫʔΫ (GNN; Graph Neural Network) ͱݺͿͷ Ͱ͕͢ɺ͜ͷߟ͑ํΛ༻͍Δ͜ͱͰྡؔΛࢀߟʹͨ͠χϡʔϥϧωο τϫʔΫͷֶश͕ՄೳͰ͢ɻ2.2 અͷྫͰӺΛϊʔυɺ࿏ઢΛΤοδͱݟ ཱͯͯάϥϑχϡʔϥϧωοτϫʔΫΛߟ͑·͕ͨ͠ɺ୯ޠΛϊʔυɺͦΕ ͧΕͷ୯ޠͷྨࣅΛΤοδͱݟཱͯͯάϥϑΈࠐΈ (Attention ܭࢉ + ϑΟʔυϑΥϫʔυͷύϥϝʔλܭࢉ) Λߦͬͨͷ͕ Transformer Ͱ͢ɻ https://www.amazon.co.jp/dp/B08B4SBQL7 ্هͷୈ 5 ষͱୈ 7 ষͰऔΓѻͬͨ BoW Λݩʹͨ͠จॻྨωοτ ϫʔΫੳͷߟ͑ํΛจॻ୯ҐͰͳ͘୯ޠ୯Ґʹద༻ͨ͠ͱߟ͑ͯಉ༷ *4 ͨͱ͑ରԠ͕ૣ͍Ӻһ͕ଟ͍Ӻͱͦ͏Ͱͳ͍ӺͰࠞࡶʹ͕ࠩग़ΔͩΖ͏ͱߟ͑Β ΕΔͱࢥ͍·͢ɻ 20 -
2.3 Dot Product Attention ʹਤ 2.6 ͷΑ͏ͳάϥϑΛ࡞͢Δ͜ͱ͕ՄೳͰ͢*5 ɻ ˛ਤ 2.6
ωοτϫʔΫੳͱάϥϑ ͜ͷΑ͏ʹ୯ޠจষͳͲʹج͍ͮͯάϥϑΛܭࢉ͢Δ͜ͱ͕Ͱ͖ΔͷͰ ͕͢ɺ͜ͷ͜ͱ 2.3 અͷ Dot Product Attention 2.4 અͷ Transformer ϞδϡʔϧΛཧղ্͍ͯ͘͠Ͱඇৗʹʹཱͪ·͢ɻ 2.3 Dot Product Attention 2.3 અͰ Dot Product Attention ͱ Multi-Head Attention ʹ͍ͭͯ֬ ೝ͠·͢ɻ *5 BoW จॻͷ D ͱ୯ޠͷ W ͷߦྻͰ͕͢ɺD ʹରͯ͠ྨࣅΛܭࢉ͢Δͷ W ʹ ରͯ͠ྨࣅΛܭࢉ͢ΔͷͲͪΒ Cos ྨࣅͳͲͰܭࢉͰ͖ΔͷͰɺख๏ͱͯ͠ ಉ༷ʹߟ͓͑ͯ͘͜ͱ͕ՄೳͰ͢ɻ 21 -
ୈ 2 ষ Transformer ͷجຊཧղ ˛ਤ 2.7 Dot Product Attention
ͱ Multi-Head Attention ͷ֓ཁਤ (Trans- former จ Figure 2) https://arxiv.org/abs/1706.03762 ਤ 2.7 ͕ Dot Product Attention ͱ Multi-Head Attention ͷ֓ཁਤͰ ͢ɻࣜ߹Θͤͯ֬ೝ͠·͢ɻ ˛ਤ 2.8 Dot Product Attention(Transformer จΑΓ) 22 -
2.3 Dot Product Attention ˛ਤ 2.9 Multi-Head Attention(Transformer จΑΓ) ਤ
2.8 ͱਤ 2.9 Dot Product Attention ͱ Multi-Head Attention ʹͦ ΕͧΕରԠ͍ͯ͠·͢ɻ ͜͜·Ͱ֬ೝͨ͠จͷਤͱ͚ࣜͩͰগ͠Θ͔ΓͮΒ͍ͷͰҎԼॲཧ ֓ཁΛ֬ೝ͠·͢ɻ·ͣɺԿ͔͠ΒͷλεΫΛલఏͱͯ͠ߟ͑Δํ͕Θ͔Γ ͍͢ͷͰɺseq2seq ͳͲͱಉ༷ͷػց༁ (Machine Translation) λεΫ Λલఏʹߟ͑·͢ɻ ػց༁λεΫͰຊޠΛೖྗ͢Δʹ͋ͨͬͯʮࢲ ࠓ ౦ژ ʹ ߦ͘ʯͷΑ͏ͳܗଶૉղੳͷ݁ՌͷܥྻΛೖྗ͠ɺͦΕͧΕͷ୯ޠʹ Word2vec ͷΑ͏ͳॲཧΛࢪͯ͠୯ޠΛύϥϝʔλͰදͨ͠දݱͰ͋Δࢄ දݱ (Distributed representation) Λಘ·͢*6 ɻ Attention ͷॲཧͰͦΕͧΕͷ୯ޠͷੵΛܭࢉͯ֬͠Խ͠ (Q ͱ K ͷߦྻͷੵʹରͯ͠ softmax ؔΛөͤ͞Δ)ɺͦΕʹج͍ͮͯॏΈ͚ ͷܭࢉ (Attention ॲཧ) Λߦ͍·͢ɻཁ͍ۙ͠ࢄදݱͰද͞ΕΔ୯ ޠ͕ͳΔ͘ޓ͍ʹ૬ޓ࡞༻͞ΕΔΑ͏ͳॲཧΛ࣮ݱ͍ͯ͠Δͱղऍ͢Ε ྑ͍Ͱ͢ɻ·ͨɺ్தͷܭࢉͰੵ (Dot Product) Λܭࢉ͢Δ͜ͱ͔Β Dot Product Attention ͱ໊͚ΒΕ͍ͯΔ͜ͱ͓͑ͯ͘ͱྑ͍ͱࢥ͍ ·͢ɻ ͜͜·Ͱͷ༰͕ Dot Product Attention ͷॲཧͰ͕͢ɺ ͜ͷ Dot Prod- uct Attention ΛݩʹΞϯαϯϒϧతʹܭࢉΛߦͬͨͷ͕ Multi-Head At- *6 Transformer Ͱجຊతʹ୯ޠΛ 512 ࣍ݩͷύϥϝʔλͰද͠·͢ɻ͜ͷࢄදݱΛ ಘΔॲཧͷ͜ͱΛ Embedding ͱ͍͏͜ͱ͋Γ·͢ɻ 23 -
ୈ 2 ষ Transformer ͷجຊཧղ tention Ͱ͢ɻMulti-Head Λ༻͍Δ͜ͱͰΞϯαϯϒϧతͳؤ݈ੑɺฒ ྻॲཧͷͳͲ͕࣮ݱͰ͖Δͱߟ͑ͯྑ͍͔ͱࢥ͍·͢*7 ɻ
2.4 Transformer Ϟδϡʔϧͷղऍ 2.4 અͰ Transformer Ϟδϡʔϧͷղऍʹ͍ͭͯߦ͍·͢ɻ ˛ਤ 2.10 Transformer Ϟδϡʔϧͷશମ૾ (Transformer จ Figure 1) *7 Multi-Head Attention ·Ͱৄ͘͠औΓѻ͏ͱ͍͠ͷͱɺجຊతʹ Dot Product Attention ͚ͩͰ Transformer ͷߴ͍ύϑΥʔϚϯεेཧղͰ͖ΔͷͰຊॻͰ লུ͠·͢ɻ 24 -
2.4 Transformer Ϟδϡʔϧͷղऍ https://arxiv.org/abs/1706.03762 ·ͣɺਤ 2.10 ͕ Transfomer Ϟδϡʔϧͷ֓ཁͰ͢ɻࠨͷΤϯίʔμʔ ͱӈͷσίʔμʔΛ
2.3 અͷ༁ͷྫͰߟ͑ΔͱຊޠΛ෦දݱʹม͑Δ ͷ͕ΤϯίʔμʔɺӳޠΛੜ͢ΔͷΛσίʔμͱͦΕͧΕߟ͑Δͱྑ͍ Ͱ͢ɻ 2.3 અͰऔΓѻͬͨ Dot Product Attention Multi-Head Attention ਤ 2.10 ʹ͓͍ͯΦϨϯδͰදݱ͞Ε͍ͯ·͢ɻ͜Ε 2.2 અͰऔΓѻͬͨ άϥϑΈࠐΈʹ͓͚Δ Message Passing ʹରԠ͍ͯ͠Δͱߟ͑Δ͜ͱ͕ Ͱ͖·͢ɻ·ͨɺਤʹ͓͍ͯਫ৭Ͱࣔ͞Εͨ Feed Forward ֤୯ޠ͝ͱʹ ύϥϝʔλֶशΛߦ͓ͬͯΓɺ͜Ε 2.2 અͰऔΓѻͬͨάϥϑΈࠐΈʹ ͓͍ͯϊʔυ͝ͱʹӅΕͷϕΫτϧͱύϥϝʔλߦྻͷੵΛܭࢉ͢Δ͜ͱ ʹ૬͍ͯ͠Δͱߟ͑Δ͜ͱ͕Ͱ͖·͢ɻ ˛ਤ 2.11 Dot Product Attention ͱ GNN 25 -
ୈ 2 ষ Transformer ͷجຊཧղ https://www.hello-statisticians.com/ml/deeplearning/trans former1.html Transformer ͱάϥϑχϡʔϥϧωοτϫʔΫͱͷରԠਤ 2.11
ͷΑ͏ ʹղऍ͢Δ͜ͱ͕ՄೳͰ͢ɻTransformer GNN ʹ͓͚ΔάϥϑΛͦΕͧ Εͷ୯ޠͷࢄදݱӅΕͷྨࣅʹج͍ͮͯࣗಈੜͨ͠ GNN Ͱ͋Δ ͱ֓ͶղऍͰ͖·͢ɻ ˛ਤ 2.12 Reformer จ https://arxiv.org/abs/2001.04451 ·ͨɺ͜͜·Ͱͷ Transformer Ϟδϡʔϧͷղऍ Reformer ΛབྷΊΔ͜ ͱͰ͞Βʹߟ͕ՄೳʹͳΔͷͰ؆୯ʹ͝հ͠·͢ɻ 26 -
2.4 Transformer Ϟδϡʔϧͷղऍ ˛ਤ 2.13 LSH Attention(Reformer จ Figure 2)
Reformer Ͱਤ 2.13 Ͱද͞ΕΔ LSH(Locality Sensitive Hashing) At- tention ͰɺAttention ॲཧΛಛఆͷ୯ޠ͚ͩʹߜͬͯߦ͏͜ͱͰܭࢉޮ Λ্ͤ͞ɺ1,000 ୯ޠલޙͷऔΓѻ͍͕த৺ͷ Transformer ੜͷݚ ڀʹର͠ɺͦͷेഒҎ্ͷ୯ޠΛऔΓѻ͑ΔΑ͏ʹͨ͠ݚڀͰ͢ɻ͜ͷ Reformer Ͱ༻͍ΒΕ͍ͯΔߟ͑ํͦͷཧల։άϥϑͷऔΓѻ͍Λ͔ ͳΓߟྀͨ͠༰Ͱ͋ΓɺάϥϑχϡʔϥϧωοτϫʔΫͱ߹ΘͤͯѲ͠ ͓ͯ͘ͱཧղ͕ਂ·Γ·͢ɻ ͏গ͠ߟ͢ΔͳΒɺTransformer શͯͷ୯ޠಉ࢜ʹ͍ͭͯ Atten- tion ॲཧΛߦ͏ͱ͍͏ιϑτͳߏɺReformer ಛఆͷؔ࿈ੑͷߴ͍୯ޠ ಉ࢜ʹ͍͔ͭͯ͠ܭࢉΛߦΘͳ͍ϋʔυͳߏͱߟ͑Δ͜ͱ͕Ͱ͖Δ͔ͱࢥ ͍·͢ɻάϥϑχϡʔϥϧωοτϫʔΫͱͯ͠ Transformer Λཧղ͢Δ͜ ͱͰɺ͜ͷΑ͏ʹҰݟෳࡶͰͦ͠͏ʹݟ͑Δ Transformer ੜͷॲཧΛ ୯ޠΛϊʔυͱΈͳͨ͠άϥϑχϡʔϥϧωοτϫʔΫͱͯ͠ཧղ͢Δ͜ͱ ͕Ͱ͖ɺײతͳཧղͱߟ͕ՄೳʹͳΓ·͢ɻ 27 -
ୈ 2 ষ Transformer ͷجຊཧղ 2.5 BERT ͷωοτϫʔΫͷߏ 2.5 અͰ
Transformer Λݩʹͨ͠ݴޠॲཧͷࣄલֶशϞσϦϯάͷ BERT ʹ͍ͭͯ͝հ͠·͢ɻಛʹ 3.2 અͰ͝հ͢Δ Vision Trans- former(ViT) BERT ͷωοτϫʔΫߏΛͦͷ··༻͍͍ͯΔͨΊɺ BERT ͷѲʹ͍ͭͯઌʹߦ͓ͬͯ͘ํ͕·͍͠Ͱ͢ɻBERT Transformer Λϕʔεʹͨ͠ϞσϦϯάͰ͋Γɺࣄલֶशͱͯ͠ Masked Language Modeling Next Sentence Prediction Λ༻͍͍ͯ·͢ɻ ˛ਤ 2.14 BERT ͷ BASE ͱ LARGE(BERT จΑΓ) https://arxiv.org/abs/1810.04805 BERT ͷ BASE ͱ LARGE ্هͷαΠζͰ͋ΓɺେମͷنײΛ͑ ͓ͯ͘ͱྑ͍ͱࢥ͍·͢ɻBERT ͷ BASE ʹ͓͍ͯɺͷ͕ 12ɺӅΕ (୯ޠͷࢄදݱͷ࣍ݩ) ͕ 768ɺ૯ύϥϝʔλ͕ 1.1 ԯͱ͍͏ࣈಛ ʹ͓͑ͯ͘ͱྑ͍Ͱ͢ɻ BERT ʹ͍ͭͯ Transformer ͕ཧղͰ͖͍ͯΕͬ͟ͱཧղ͍ͯ͠Δ ͰेͳͷͰຊॻͰৄ͘͠औΓѻ͍·ͤΜɻΑΓৄ֬͘͠ೝ͍ͨ͠ํԼ هͰৄ͘͠औΓѻ͍·ͨ͠ͷͰԼهͳͲΛ֬͝ೝ͍ͩ͘͞ɻ https://lib-arts.booth.pm/items/1834866 28