【動画あり】Transformer論文解説

ࠓճͷ಺༰ ࠷ॳͷϖʔδ ಺༰ղઆ ࿦จಡΉͱ͖ʹԿΛߟ͍͑ͯΔ͔ʁ ͦͷޙͷల։ ͜ͷ࿦จ୯ମͷཧղʹͱͲ·Βͣ ࿦จͷಡΈํɾ෼໺ͰͷҐஔ෇͚Λ஌Δ 5SBOTGPSNFSఏҊ࿦จΛಡΉ 7BTXBOJ
"TIJTI FUBM"UUFOUJPOJTBMMZPVOFFE"EWBODFTJOOFVSBMJOGPSNBUJPOQSPDFTTJOHTZTUFNT

ͳͥ͜ͷ࿦จ͕ॏཁͳͷ͔ʁ

5SBOTGPSNFSͷԠ༻ൣғ ※: https://blog.google/products/search/search-language-understanding-bert/ FUD 5SBOTGPSNFS ςΩετ຋༁Λओ؟ͱͯ͠ఏҊ #&35 (15 ςΩετ෼ྨFUD
ςΩετੜ੒ ෦෼ΞʔΩςΫνϟͷ࠾༻ ը૾΁ͷద༻ 7J5 %JGGVTJPO 5SBOTGPSNFS ը૾෼ྨFUD ը૾ੜ੒ $IBU(15 -MBNB 4UBCMF%JGGVTJPO 4PSB (PPHMFݕࡧ˞ $-*1 ۃΊͯ޿ൣғʹج൫ٕज़ͱͯ͠׆༂

ਓ޻஌ೳͷ࿦จΛಡΉ੾Γޱ

ਓ޻஌ೳͷ࿦จΛಡΉ੾Γޱ ର߅അ͸Կ͔ ओఏҊ͸Կ͔ ධՁͷ8IBU3FTVMU

ਓ޻஌ೳͷ࿦จΛಡΉ੾Γޱ ର߅അ͸Կ͔ ओఏҊ͸Կ͔ ධՁͷ8IBU3FTVMU *OUSPEVDUJPO #BDLHSPVOE .PEFM"SDIJUFDUVSF
8IZ4FMG"UUFOUJPO 5SBJOJOH 3FTVMUT

࿦จͷߏ଄ ΞϒετϥΫτ ΠϯτϩμΫγϣϯ ؔ࿈ݚڀ ఏҊख๏ ࣮ݧઃఆɾ݁Ռɾٞ࿦ ݁࿦

࿦จͷߏ଄ ΞϒετϥΫτ ΠϯτϩμΫγϣϯ ؔ࿈ݚڀ ఏҊख๏ ࣮ݧઃఆɾ݁Ռɾٞ࿦ ݁࿦ ⁞֓ཁΛ௫Ή

࿦จͷߏ଄ ΞϒετϥΫτ ΠϯτϩμΫγϣϯ ؔ࿈ݚڀ ఏҊख๏ ࣮ݧઃఆɾ݁Ռɾٞ࿦ ݁࿦ ⁞֓ཁΛ௫Ή  ओுΛ௫Ή

⁠ॏΈ෇͚ͯ͠ಡΉ

ਓ޻஌ೳͷ࿦จΛಡΉ੾Γޱ ର߅അ͸Կ͔ ओఏҊ͸Կ͔ ධՁͷ8IBU3FTVMU *OUSPEVDUJPO #BDLHSPVOE .PEFM"SDIJUFDUVSF
8IZ4FMG"UUFOUJPO 5SBJOJOH 3FTVMUT

ܥྻϞσϦϯά

ܥྻϞσϦϯά ॱংͷ͋Δཁૉͷ࿈ͳΓͱΈͳͤΔ΋ͷΛܥྻʢTFRVFODFʣͱݺͼɺ ͜ΕΛର৅ͱ͢ΔϞσϦϯάΛܥྻϞσϦϯάͱݺͿ

ܥྻϞσϦϯά ॱংͷ͋Δཁૉͷ࿈ͳΓͱΈͳͤΔ΋ͷΛܥྻʢTFRVFODFʣͱݺͼɺ ͜ΕΛର৅ͱ͢ΔϞσϦϯάΛܥྻϞσϦϯάͱݺͿ ྫ ຋༁ ஫ҙػߏ͑͋͞Ε͹े෼ "UUFOUJPOJTBMMZPVOFFE ྫ ςΩετੜ੒
Ͳ͏ͧΑΖ͓͘͠ئ͍͠·͢ɻ Կ͔࣭͝໰΍ϦΫΤετ͕͋Ε͹ڭ͍͑ͯͩ͘͞ɻ ΑΖ͘͠པΉ ྫ ߏจղੳ 4 /1ΑΖ͘͠ 71པΉ ΑΖ͘͠པΉ

Ͳ͏ͧΑΖ͓͘͠ئ͍͠·͢ɻ Կ͔࣭͝໰΍ϦΫΤετ͕͋Ε͹ڭ͍͑ͯͩ͘͞ɻ ΑΖ͘͠པΉ ྫ ߏจղੳ 4 /1ΑΖ͘͠ 71པΉ ΑΖ͘͠པΉ TPVSDF

Ͳ͏ͧΑΖ͓͘͠ئ͍͠·͢ɻ Կ͔࣭͝໰΍ϦΫΤετ͕͋Ε͹ڭ͍͑ͯͩ͘͞ɻ ΑΖ͘͠པΉ ྫ ߏจղੳ 4 /1ΑΖ͘͠ 71པΉ ΑΖ͘͠པΉ UBSHFU

ॏཁ՝୊ɿ௕ڑ཭ґଘͷཧղ ൴͕ॻ͍ͨͦͷຊΛɺࢲ͸Ұ౓΋ಡΜͩ͜ͱ͕͋Γ·ͤΜɻ ໨తޠ ओޠ Ґஔతʹ཭Εͨܥྻཁૉؒͷґଘؔ܎

ॏཁ՝୊ɿ௕ڑ཭ґଘͷཧղ ൴͕ॻ͍ͨͦͷຊΛɺࢲ͸Ұ౓΋ಡΜͩ͜ͱ͕͋Γ·ͤΜɻ ໨తޠ ओޠ Ґஔతʹ཭Εͨܥྻཁૉؒͷґଘؔ܎ ௕ڑ཭ґଘΛ೺ѲͰ͖ͳ͍ͱେ൒ͷλεΫ͸ղ͚ͳ͍

ର߅അ͸Կ͔ *OUSPEVDUJPO#BDLHSPVOE

ର߅അ͸Կ͔ ࠶ؼܕχϡʔϥϧωοτϫʔΫ ৞ΈࠐΈχϡʔϥϧωοτϫʔΫ -45.<)PDISFJUFS > (36<$IVOH > FUD #ZUF/FU<,BMDICSFOOFS
> $POW44<(FISJOH > FUD ܥྻͷཁૉΛॱʹೖྗ͍ͯ͘͠ ฒྻܭࢉ͕Ͱ͖ͳ͍ ཁૉؒڑ཭ʹԠͨ͡ܭࢉྔ૿Ճ͕ݦஶ ௕ڑ཭ґଘͷֶश͕ࠔ೉

ର߅അ͸Կ͔ ࠶ؼܕχϡʔϥϧωοτϫʔΫ ৞ΈࠐΈχϡʔϥϧωοτϫʔΫ -45.<)PDISFJUFS > (36<$IVOH > FUD #ZUF/FU<,BMDICSFOOFS
> $POW44<(FISJOH > FUD ܥྻͷཁૉΛॱʹೖྗ͍ͯ͘͠ ฒྻܭࢉ͕Ͱ͖ͳ͍ ཁૉؒڑ཭ʹԠͨ͡ܭࢉྔ૿Ճ͕ݦஶ ௕ڑ཭ґଘͷֶश͕ࠔ೉ ฒྻԽ͕Մೳ͔ͭ௕ڑ཭ґଘΛֶशͰ͖ΔϞσϧͱͯ͠ 5SBOTGPSNFSΛఏҊʢ4FD ʣ

طଘݚڀ͔ΒҾ͖ܧ͙΋ͷ Τϯίʔμɾσίʔμػߏ FODPEFSEFDPEFSNFDIBOJTN ࣗݾ஫ҙػߏ TFMGBUUFOUJPONFDIBOJTN Τϯίʔμ σίʔμ ೖྗ ग़ྗ
ಛ௃நग़ɾܥྻੜ੒ͷೋஈߏ͑ ࢲ ͸ ٢ా ࢲ ͸ ٢ా ࣗܥྻؒͰͷॏΈ෇͚

طଘݚڀ͔ΒҾ͖ܧ͙΋ͷ Τϯίʔμɾσίʔμػߏ FODPEFSEFDPEFSNFDIBOJTN ࣗݾ஫ҙػߏ TFMGBUUFOUJPONFDIBOJTN Τϯίʔμ σίʔμ ೖྗ ग़ྗ
ಛ௃நग़ɾܥྻੜ੒ͷೋஈߏ͑ ࢲ ͸ ٢ా ࢲ ͸ ٢ా ࣗܥྻؒͰͷॏΈ෇͚ Τϯίʔμɾσίʔμͷ༗༻ੑΛੜ͔ͭͭ͠ ࣗݾ஫ҙػߏͰ׬݁͢ΔॳΊͯͷϞσϧʢ4FDʣ

·ͱΊ ର߅അ͸Կ͔ ओఏҊ͸Կ͔ ධՁͷ8IBU3FTVMU ࠶ؼܕωοτϫʔΫ ৞ΈࠐΈωοτϫʔΫ ࣗݾ஫ҙͰ׬݁ͨ͠ ΤϯίʔμɾσίʔμΛఏҊ w
ฒྻԽ͕༰қ w ௕ڑ཭ґଘΛଊ͑Δ .PEFM"SDIJUFDUVSF 8IZ4FMG"UUFOUJPO 5SBJOJOH 3FTVMUT

͜͜·ͰಡΉͱओ؟͕Θ͔Δ ฒྻԽՄೳͰɺ͔ͭ௕ڑ཭ͷґଘؔ܎Λ ଊ͑ΒΕΔϝΧχζϜͱ͸ʁ

ओఏҊ͸Կ͔ .PEFM"SDIJUFDUVSF8IZ4FMG"UUFOUJPO

ఏҊϞσϧͷΞʔΩςΫνϟ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ 'FFE'PSXBSE   /FUXPSL Ճࢉਖ਼نԽ ϚεΫ෇͖
  Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ Ґஔූ߸Խ Ґஔූ߸Խ 🌟 🌟 🌟 /ʷ ʷ/ ˞ਤ͸7BTXBOJ ͷ'JHVSFΛϕʔεͱͯ͠࡞੒

Ґஔූ߸Խ ఏҊϞσϧͷΞʔΩςΫνϟ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ 'FFE'PSXBSE   /FUXPSL Ճࢉਖ਼نԽ
ϚεΫ෇͖   Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ Τϯίʔμ Ґஔූ߸Խ

ϚεΫ෇͖   Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ σίʔμ Ґஔූ߸Խ

ϚεΫ෇͖   Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ σίʔμ Ґஔූ߸Խ ࣗݾճؼ BVUPSFHSFTTJPO ࣌ࠁ ͷग़ྗ͕ ͷೖྗʹͳΔػߏ t t + 1

  Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ ˞ਤ͸7BTXBOJ ͷ'JHVSFΛϕʔεͱͯ͠࡞੒ Ґஔූ߸Խ Ґஔූ߸Խ

  Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ ˞ਤ͸7BTXBOJ ͷ'JHVSFΛϕʔεͱͯ͠࡞੒ ͔͜͜Βઌͷॲཧ͸ೖྗςΩετͷ ޠॱΛೝࣝͰ͖ͳ͍ Ґஔූ߸Խ Ґஔූ߸Խ

ຒΊࠐΈɾҐஔූ߸Խ ς Ω ε τ τ Ϋ ϯ Խ
ࢲ ͸ ٢ా ʜ ʜ ʜ ຒΊࠐΈ e1 e2 e3 ࣍ݩͷϕΫτϧ dmodel

ຒΊࠐΈɾҐஔූ߸Խ ς Ω ε τ τ Ϋ ϯ Խ ࢲ
͸ ٢ా ʜ ʜ ʜ ຒΊࠐΈ e1 e2 e3 ʜ ʜ ʜ Ґஔූ߸Խ p1 p2 p3 ppos [2i] = sin ( pos 10000 2i dmodel ) ppos [2i + 1] = cos ( pos 10000 2i dmodel )

ຒΊࠐΈɾҐஔූ߸Խ ς Ω ε τ τ Ϋ ϯ Խ
ࢲ ͸ ٢ా ʜ ʜ ʜ ຒΊࠐΈ e1 e2 e3 ʜ ʜ ʜ Ґஔූ߸Խ p1 p2 p3 ppos [2i] = sin ( pos 10000 2i dmodel ) ppos [2i + 1] = cos ( pos 10000 2i dmodel ) x1 x2 x3 = de1 + p1 = de2 + p2 = de3 + p3

  Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ϛϧνϔου ࣗݾ஫ҙ Ճࢉਖ਼نԽ Ճࢉਖ਼نԽ ೖྗςΩετ ग़ྗςΩετ 'FFE'PSXBSE   /FUXPSL ઢܗม׵ ιϑτϚοΫεؔ਺ ֬཰෼෍ 🌟 🌟 🌟 /ʷ ʷ/ ˞ਤ͸7BTXBOJ ͷ'JHVSFΛϕʔεͱͯ͠࡞੒ ࣗݾ஫ҙ TFMGBUUFOUJPO w ϕΫτϧྻΛจ຺Λߟྀ͠ͳ͕Βม׵ w 5SBOTGPSNFSͷ࠷΋ॏཁͳ෦඼ Ґஔූ߸Խ Ґஔූ߸Խ

ࣗݾ஫ҙ x1 x2 x3 Q = [ q1 ]
[ q2 ] [ q3 ] K = [ k1 ] [ k2 ] [ k3 ] V = [ v1 ] [ v2 ] [ v3 ] qi = Wq xi , Wq ∈ ℝdk ×dmodel ki = Wk xi , Wk ∈ ℝdk ×dmodel vi = Wv xi , Wv ∈ ℝdv ×dmodel

ࣗݾ஫ҙ x1 x2 x3 Q = [ q1 ]
[ q2 ] [ q3 ] K = [ k1 ] [ k2 ] [ k3 ] V = [ v1 ] [ v2 ] [ v3 ] h1 h2 h3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K QK⊤ = [ q1 ] [ q2 ] [ q3 ] [ k1 ] [ k2 ] [ k3 ] = q1 ⋅ k1 q1 ⋅ k2 q1 ⋅ k3 q2 ⋅ k1 q2 ⋅ k2 q2 ⋅ k3 q3 ⋅ k1 q3 ⋅ k2 q3 ⋅ k3

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K QK⊤ = [ q1 ] [ q2 ] [ q3 ] [ k1 ] [ k2 ] [ k3 ] = q1 ⋅ k1 q1 ⋅ k2 q1 ⋅ k3 q2 ⋅ k1 q2 ⋅ k2 q2 ⋅ k3 q3 ⋅ k1 q3 ⋅ k2 q3 ⋅ k3 ʮ٢ాʯ͔Βݟͨʮࢲʯͷॏཁ౓

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K QK⊤ = [ q1 ] [ q2 ] [ q3 ] [ k1 ] [ k2 ] [ k3 ] = q1 ⋅ k1 q1 ⋅ k2 q1 ⋅ k3 q2 ⋅ k1 q2 ⋅ k2 q2 ⋅ k3 q3 ⋅ k1 q3 ⋅ k2 q3 ⋅ k3 ʮ٢ాʯ͔Βݟͨʮ͸ʯͷॏཁ౓

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K QK⊤ = [ q1 ] [ q2 ] [ q3 ] [ k1 ] [ k2 ] [ k3 ] = q1 ⋅ k1 q1 ⋅ k2 q1 ⋅ k3 q2 ⋅ k1 q2 ⋅ k2 q2 ⋅ k3 q3 ⋅ k1 q3 ⋅ k2 q3 ⋅ k3 ʮ٢ాʯ͔Βݟͨʮ٢ాʯͷॏཁ౓

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ͜Ε͔Βܭࢉ͍ͨ͠΋ͷ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K softmax ( QK⊤ dk ) = a11 a12 a13 a21 a22 a23 a31 a32 a33 εέʔϦϯά ิ ͜ͷߦྻΛ஫ҙߦྻ BUUFOUJPONBUSJY ͱݺͿ ֤੒෼ ͸஫ҙॏΈ BUUFOUJPOXFJHIU ͱݺͿ aij

ࣗݾ஫ҙ h1 h2 h3 = a21 v1 + a22
v2 + a23 v3 ࣗݾ஫ҙͷ࠷ऴग़ྗ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K softmax ( QK⊤ dk ) = a11 a12 a13 a21 a22 a23 a31 a32 a33 εέʔϦϯά = a11 v1 + a12 v2 + a13 v3 = a31 v1 + a32 v2 + a33 v3

ࣗݾ஫ҙ h1 h2 h3 = a21 v1 + a22
v2 + a23 v3 ࣗݾ஫ҙͷ࠷ऴग़ྗ पลจ຺ͷࠞͥ߹Θͤ۩߹Λ˓ͷ஋͕ܾΊΔ ͜ΕΛ Λ༻͍ͯٻΊΔ Q, K softmax ( QK⊤ dk ) = a11 a12 a13 a21 a22 a23 a31 a32 a33 εέʔϦϯά = a11 v1 + a12 v2 + a13 v3 = a31 v1 + a32 v2 + a33 v3 ࣗݾ஫ҙ ʹपลจ຺Λߟྀ͢Δػߏ

ϚεΫ෇͖ࣗݾ஫ҙ softmax ( QK⊤ dk ) = a11 a12
a13 a21 a22 a23 a31 a32 a33

a13 a21 a22 a23 a31 a32 a33 [ 1 0 0 1 1 0 1 1 1 ] ϚεΫߦྻ

a13 a21 a22 a23 a31 a32 a33 [ 1 0 0 1 1 0 1 1 1 ] ϚεΫߦྻ a11 a12 a13 a21 a22 a23 a31 a32 a33 ⊙ [ 1 0 0 1 1 0 1 1 1 ] = a11 0 0 a21 a22 0 a31 a32 a33

ϚεΫ෇͖ࣗݾ஫ҙ h1 h2 h3 = a21 v1 + a22
v2 + 0v3 ࣗݾ஫ҙͷ࠷ऴग़ྗ softmax ( QK⊤ dk ) = a11 a12 a13 a21 a22 a23 a31 a32 a33 = a11 v1 + 0v2 + 0v3 = a31 v1 + a32 v2 + a33 v3 [ 1 0 0 1 1 0 1 1 1 ] ϚεΫߦྻ a11 a12 a13 a21 a22 a23 a31 a32 a33 ⊙ [ 1 0 0 1 1 0 1 1 1 ] = a11 0 0 a21 a22 0 a31 a32 a33

ࣗݾ஫ҙʹ͍ͭͯ੔ཧ

ࣗݾ஫ҙ h1 h2 h3 = ◯v1 + ◯v2 +
◯v3 = ◯v1 + ◯v2 + ◯v3 = ◯v1 + ◯v2 + ◯v3 ܭࢉ͍ͨ͠΋ͷ QK⊤ = [ q1 ] [ q2 ] [ q3 ] [ k1 ] [ k2 ] [ k3 ] = q1 ⋅ k1 q1 ⋅ k2 q1 ⋅ k3 q2 ⋅ k1 q2 ⋅ k2 q2 ⋅ k3 q3 ⋅ k1 q3 ⋅ k2 q3 ⋅ k3 softmax ( QK⊤ dk ) = a11 a12 a13 a21 a22 a23 a31 a32 a33

Ϛϧνϔουࣗݾ஫ҙ 'JHΑΓൈਮ Λ ෼ׂͯ͠ฒྻॲཧ ग़ྗΛܨ͛ͯͻͱͭʹ͢Δ Q, K, V h
࿦จͰ͸ Ͱ࣮ݧ h = 1,4,8,16,32

1PTJUJPOXJTF'FFE'PSXBSE/FUXPSLT ࣗ ݾ ஫ ҙ ࢲ ͸ ٢ా ʜ
ʜ ʜ จ຺ԽຒΊࠐΈ h1 h2 h3 ReLU(h1 W1 + b1 )W2 + b2 ReLU(h2 W1 + b1 )W2 + b2 ReLU(h3 W1 + b1 )W2 + b2 ϕΫτϧͦΕͧΕʹରͯ͠'FFEGPSXBSE/FUXPSLΛద༻

1PTJUJPOXJTF'FFE'PSXBSE/FUXPSLT ࣗ ݾ ஫ ҙ ࢲ ͸ ٢ా ʜ
ʜ ʜ จ຺ԽຒΊࠐΈ h1 h2 h3 ReLU(h1 W1 + b1 )W2 + b2 ReLU(h2 W1 + b1 )W2 + b2 ReLU(h3 W1 + b1 )W2 + b2 ϕΫτϧͦΕͧΕʹରͯ͠'FFEGPSXBSE/FUXPSLΛద༻ 5SBOTGPSNFSʹ͓͚Δ''/ͷ໾ׂʹ͍ͭͯ͸ͦͷޙ͞·͟·ͳٞ࿦͕͋Δ w (FWB .PS FUBM5SBOTGPSNFSGFFEGPSXBSEMBZFSTBSFLFZWBMVFNFNPSJFTBS9JWQSFQSJOUBS9JW w ;IBOH ;IFOHZBO FUBM.PF fi DBUJPO5SBOTGPSNFSGFFEGPSXBSEMBZFSTBSFNJYUVSFTPGFYQFSUTBS9JWQSFQSJOUBS9JW w (FWB .PS FUBM5SBOTGPSNFSGFFEGPSXBSEMBZFSTCVJMEQSFEJDUJPOTCZQSPNPUJOHDPODFQUTJOUIFWPDBCVMBSZTQBDFBS9JWQSFQSJOUBS9JW w FUD

ฒྻԽ͕༰қ w ௕ڑ཭ґଘΛଊ͑Δ 5SBJOJOH 3FTVMUT 5SBOTGPSNFSͷػߏ Ґஔූ߸Խ

ධՁͷ8IBU3FTVMU 5SBJOJOH3FTVMUT

8IBU λεΫ ػց຋༁ σʔληοτ 8.5 w χϡʔε຋༁ͷֶशɾධՁσʔληοτ w FOEFNJMMJPOTFOUFODFQBJST
w FOGSNJMMJPOTFOUFODFQBJST ධՁࢦඪ w #-&6ʢ຋༁ੑೳʣ w '-01Tʢܭࢉྔʣ

3FTVMU ଞϞσϧʹඖఢ͢ΔੑೳΛΑΓগͳֶ͍शίετͰ࣮ݱ

ฒྻԽ͕༰қ w ௕ڑ཭ґଘΛଊ͑Δ 5SBOTGPSNFSͷػߏ Ґஔූ߸Խ 8IBUػց຋༁ 3FTVMU405" ଞϞσϧʹඖఢ͢ΔੑೳΛ ΑΓগͳֶ͍शίετͰ࣮ݱ

ͦͷޙͷల։

೿ੜϞσϧͷ஀ੜ Τϯίʔμͱσίʔμ͸໾ׂ͕ҟͳΔ Τϯίʔμ σίʔμ ೖྗܥྻͷ$POUFYUVBMJ[BUJPO ࣗݾճؼతͳܥྻੜ੒

೿ੜϞσϧͷ஀ੜ Τϯίʔμͱσίʔμ͸໾ׂ͕ҟͳΔ Τϯίʔμ σίʔμ ೖྗܥྻͷ$POUFYUVBMJ[BUJPO ࣗݾճؼతͳܥྻੜ੒ ಛ௃நग़ثͱͯ͠ͷ׆༻ #&357J5FUD ੜ੒Ϟσϧͱͯ͠ͷ׆༻
(15-MBNBFUD ͦΕͧΕΛϕʔεͱͨ͠৽ͨͳϞσϧ͕஀ੜ

ֶशύϥμΠϜͷมભ ݱࡏ εΫϥονֶश ϑΝΠϯνϡʔχϯά *ODPOUFYUMFBSOJOH ಛఆͷλεΫʹಛԽͨ͠ϞσϧΛ ϥϯμϜͳ஋ΛͱΔύϥϝλ͔Βֶश
ࣄલֶशࡁΈϞσϧΛ ݸผλεΫ޲͚ʹඍௐ੔ Ϟσϧͦͷ΋ͷΛௐ੔ͤͣ ࢦࣔʹै༷ͬͯʑͳλεΫΛ͜ͳ͢ w ࠶ؼܕωοτϫʔΫ w ৞ΈࠐΈωοτϫʔΫ w 5SBOTGPSNFS w #&35 w (15 w 3FT/FU w (15 w -MBNB w 1B-.

ֶशύϥμΠϜͷมભ ݱࡏ εΫϥονֶश ϑΝΠϯνϡʔχϯά *ODPOUFYUMFBSOJOH ಛఆͷλεΫʹಛԽͨ͠ϞσϧΛ ϥϯμϜͳ஋ΛͱΔύϥϝλ͔Βֶश
ࣄલֶशࡁΈϞσϧΛ ݸผλεΫ޲͚ʹඍௐ੔ Ϟσϧͦͷ΋ͷΛௐ੔ͤͣ ࢦࣔʹै༷ͬͯʑͳλεΫΛ͜ͳ͢ w ࠶ؼܕωοτϫʔΫ w ৞ΈࠐΈωοτϫʔΫ w 5SBOTGPSNFS w #&35 w (15 w 3FT/FU w (15 w -MBNB w 1B-. #&35ʹΑΔϑΝΠϯνϡʔχϯά(15ʹΑΔ*ODPOUFYUMFBSOJOH ͕ಛʹΤϙοΫϝΠΩϯά

ࣗݾ஫ҙͰදݱ͞Ε͍ͯΔ஌ࣝͱ͸ʁ Ϟσϧ͕ͲͷΑ͏ͳ஌ࣝΛ͍࣋ͬͯΔ͔Λௐࠪ͢ΔݚڀΛ ϓϩʔϏϯά QSPCJOH ͱݺͿ

ࣗݾ஫ҙͰදݱ͞Ε͍ͯΔ஌ࣝͱ͸ʁ Ϟσϧ͕ͲͷΑ͏ͳ஌ࣝΛ͍࣋ͬͯΔ͔Λௐࠪ͢ΔݚڀΛ ϓϩʔϏϯά QSPCJOH ͱݺͿ #&35ͷ࡞ΔຒΊࠐΈ͔Β܎Γड͚ߏ଄͕͓͓ΉͶநग़Ͱ͖Δʢࠇ͕ਖ਼ղɺ੨͕#&35͔Βநग़ͨ͠܎Γड͚ߏ଄ʣ<)FXJUU 'JH> ໌ࣔతʹֶश͍ͯ͠ͳ͍஌ࣝͷ֫ಘՄೳੑ<$MBSL
'JH>

ςΩετΛ௒͑ͨ׆༂ %PTPWJUTLJZ 'JH 3BEGPSE 'JH
7JTJPO5SBOTGPSNFS $-*1 ࣗݾ஫ҙͷՄೳੑΛ୳Δޙଓݚڀ͕ଟൃ

ฒྻԽ͕༰қ w ௕ڑ཭ґଘΛଊ͑Δ 5SBOTGPSNFSͷػߏ Ґஔූ߸Խ 8IBUػց຋༁ 3FTVMU405" ଞϞσϧʹඖఢ͢ΔੑೳΛ ΑΓগͳֶ͍शίετͰ࣮ݱ

⁠ॏΈ෇͚ͯ͠ಡΉ

【動画あり】Transformer論文解説

【動画あり】Transformer論文解説

More Decks by 数理の弾丸

Other Decks in Technology

Featured

Transcript