Share
Embed
機械翻訳とエンコーダデコーダモデル
次のサイトの2021年12月29日版の資料を参考にさせていただいております.
https://web.stanford.edu/~jurafsky/slp3/10.pdf
Kyao
May 24, 2022
Transcript
34 slides
-
Speech and Language Processing. Draft of Dec 29, 2021. Copyright
© 2021 Daniel Jurafsky & James H. Martin. All Rights Reserved. $IBQUFS .BDIJOF5SBOTMBUJPOBOE &ODPEFS%FDPEFS.PEFMT #/BPLJ,PKJNB -
-BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ &ODPEFS%FDPEFS.PEFM &ODPEFS%FDPEFSXJUI3//T "UUFOUJPO #FBN4FBSDI
&ODPEFS%FDPEFSXJUI5SBOTGPSNFST 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT .5&WBMVBUJPO #JBTBOE&UIJDBM*TTVFT 5BCMF -
͋ΒΏΔࣗવݴޠʹ͓͚Δ౷ܭతීวੑʢྨࣅɾ૬ҧʣ -BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ 8PSE0SEFS5ZQPMPHZ 4 7 0ʹ͓͚Δޠॱ ӳޠ υΠπޠ 4 7
0 ຊޠ ώϯσΟʔޠ 4 0 7 ΞΠϧϥϯυޠ 7 4 0 4ओޠ 7ಈࢺ 0తޠ ྫʣӳޠɿ ຊޠɿ )FTQFBLT+BQBOFTF ൴ຊޠΛ͢ ໊ࢺͱܗ༰ࢺͷલޙؔ ӳޠ ژޠ "EK /PVO εϖΠϯޠ ݱϔϒϥΠޠ /PVO "EK "EKܗ༰ࢺ /PVO໊ࢺ ྫʣӳޠɿ εϖΠϯޠɿ HSFFOXJUDI CVSKBWFSEF -BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ -
-BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ ݴޠͷಛͷج४ ·ͱ·Γʹ͓͚Δܗଶૉ ݽཱޠޠኮ͕จதͰมܗͤͣɼಠཱ͍ͯ͠Δ ๊߹ޠจΛߏ͢Δ֤ཁૉ͕ີ͠ɼશମ͕·ΔͰҰ୯ޠʹݟ͑Δ ܗଶૉͷڥքͷ໌֬͞ ணޠޠװʹ͕͖ࣙͭɼཱࣗޠؒΛଐޠͰͭͳ͙ ۶ંޠޠኮ͕จதͰมܗ͠ɼจதͷΛද͢ .PSQIPMPHJDBM5ZQPMPHZ 3FGFSFOUJBMEFOTJUZ
ݴޠؒͷ໊ࢺͳͲʹΑΔࢀরͷলུ߹͍ ˠলུ͞Εͨ୯ޠΛਪ͢Δඞཁ͕͋Δ QSPESPQMBOHVBHF ໊ࢺ͕লུ͞ΕΔݴޠ -
5IF&ODPEFS%FDPEFS.PEFM จ຺ʹԠͨ͡ग़ྗྻΛੜ͢ΔϞσϧʢTFRTFRʣ 5IF&ODPEFS%FDPEFS.PEFM ߏ͢Δཁૉ &ODPEFSೖྗ Λड͚औΓɼͦΕʹରԠ͢Δจ຺දݱ Λੜ ͷؔͰ͋ΔίϯςΩετϕΫτϧ
Λ%FDPEFSʹ͑Δ %FDPEFS Λड͚औΓɼӅΕঢ়ଶ Λੜ͠ɼରԠ͢Δग़ྗ xn 1 hn 1 hn 1 C C hm 1 ym 1 γʔέϯε ϕΫτϧ ೖྗ୯ޠ ग़ྗ xn 1 , hn 1 , ym 1 , hm 1 C n m -
&ODPEFS%FDPEFSXJUI3//T γϯϓϧͳϞσϧ &ODPEFS%FDPEFSXJUI3//T ht = g(ht−1 , xt ) C
= hn yt = f(ht ) จ಄ϚʔΧ ࣗݾճؼ 5BTLӳޠˠεϖΠϯޠ ऑ ೖྗͷจ຺͕࠷ऴతʹ۩ݱԽ͞Εͨ ͷσίʔμͷӨڹྗ͕࣍ୈʹബ·Δ hn ( = C) ׆ੑԽؔ TPGUNBYؔ g f จ ಄ Ϛ ʔ Χ -
&ODPEFS%FDPEFSXJUI3//T &ODPEFS%FDPEFSXJUI3//T C = he n hd 0 = C
hd t = g( ̂ yt−1 , hd t−1 , C) zt = f(hd t ) yt = softmax(zt ) ̂ yt = argmaxw∈V P(w|x, y1 , ⋯, yt−1 ) ࣜԽ ͜ͷ ׆ੑԽؔͰͳ͍ g TPGUNBYؔޙͰϕΫτϧ ֤εςοϓͷ࠷͕֬ߴ͍ग़ྗ ̂ yt &ODPEFSͷӅΕঢ়ଶ he %FDPEFSͷӅΕঢ়ଶ hd ɾɾɾɾɾɾɾɾɾ ɾɾɾɾɾɾɾɾɾ ɾɾɾ ɾɾɾɾ ɾɾɾ ͜ͷ ׆ੑԽؔͰ͋Δ f ɾɾɾɾɾɾɾɾ -
"UUFOUJPO ɿγϯϓϧͳ&ODPEFS%FDPEFSϞσϧͷϘτϧωοΫ ݪจͷใ͕ ʹ͔͠ͳ͘ɼશͯͷใΛදݱ͠ͳ͚ΕͳΒͳ͍ɽ ೖྗ͕จͷ߹ɼ Ͱಉ͡Α͏ʹදݱ͢Δ͜ͱ͕ࠔɽ C C "UUFOUJPO "UUFOUJPO.FDIBOJTNʢղܾํ๏ʣ
&ODPEFSͷશӅΕঢ়ଶ͔ΒใΛσίʔμͰѻ͑Δܗͱͯ͠Ճ ॏΛ༻͍ͯͭͷݻఆϕΫτϧ Λಈతʹੜ͢Δ֓೦ C ֤σίʔυεςοϓ Ͱੜ͞ΕΔ શͯͷ ͷؔ i Ci he hd i = g( ̂ yi−1 , hd i−1 , Ci ) -
"UUFOUJPO Λੜ࣌ʹ͓͍ͯ ͱશͯͷ ͱͷؔ࿈ੑΛٻΊΔ hd i hd i−1 he j
"UUFOUJPO EPUQSPEVDUTDPSF શͯͷ ʹ͓͍ͯੵʹΑΓͭͷ ͷྨࣅΛٻΊΔ j h score(hd i−1 , he j ) αij = softmax(score(hd i−1 , he j )∀j ∈ e) = exp(score(hd i−1 , he j )) ∑ k exp(score(hd i−1 , he k )) score(hd i−1 , he j ) = hd i−1 ⋅ he j ͜ΕΒΛTPGUNBYؔͰਖ਼نԽͨ͠ͷΛ ͱ͢Δͱ αij -
"UUFOUJPO w ͱ ͷՃॏΛ ͱ͢Δ αij he j Ci "UUFOUJPO
Ci = ∑ j αij he j w FOEUPFOEͰֶशՄೳͳύϥϝʔλ Λ༻͍Δʢओྲྀʣ Ws score(hd i−1 , he j ) = hd i−1 Ws he j -
#FBN4FBSDI #FBN4FBSDI ֤εςοϓͰ࠷దͳ ݸͷτʔΫϯΛબ͢Δ k %FDPEFSʹద༻͢Δͱ ιϑτϚοΫεͷग़ྗ͔Β࠷దͳԾઆΛબ͢Δɽ ֤ԾઃΛҟͳΔ%FDPEFSʹೖྗ͢Δɽ
ҟͳΔ%FDPEFS͝ͱʹTPGUNBYͷग़ྗΛٻΊΔɽ ݸͷ֤τʔΫϯྻͰείΞϦϯάͯ͠ɼ࠷దͳ ݸબ͢Δɽ ͦͷதʹจϚʔΧ͕͋Δ߹ɼԾઆ͔Β֎ͯ͠ऴྃ͠ ΛͭݮΒ͢ɽ Ͱ͋ΕʹΔɽ k × V k k k ≠ 0 ɿCFBNXJEUI ϏʔϜ෯ ൪·ͰͷτʔΫϯྻɿIZQPUIFTJT Ծઆ k k Ұൠతʹେ͖ͳ.5γεςϜʹʙݸͰ༻͍ΒΕ͍ͯΔɽ k -
#FBN4FBSDI #FBN4FBSDI είΞϦϯάର֬ʢରؔʣΛ༻͍Δ score(y) = log P(y|x) = log(P(y1 |x)P(y2
|y1 , x)⋯P(yt |y1 , ⋯, yt−1 , x)) = t ∑ i=1 log P(yi |y1 , ⋯, yi−1 , x) ʢDIBJOSVMFʣ ར લͷϊʔυͷείΞΛਵ࣌Ճࢉ͍ͯ͘͜͠ͱͰܭࢉޮ͕ྑ͍ ܭࢉྔO(k × |V| × T) ϏʔϜ෯ શޠኮ ग़ྗ୯ޠ k |V| T ˠείΞΛߴ͍ͷ͔Βॱʹ ݸબ͢Δ k Ξϯμϑϩʔରࡦ log -
#FBN4FBSDI #FBN4FBSDI ͨ͠ԾઆʹΑΔ୯ޠྻͷ͕͞ҟͳΔͱ ͍จࣈྻͷ֬ʻ͍จࣈྻͷ֬ ͱͳΓɼ͍จࣈྻ͕બ͞Εͯ͠·͏ɽ score(y) = − log
P(y|x) = − 1 T t ∑ i=1 log P(yi |y1 , ⋯, yi−1 , x) ղܾྫ ෛͷର֬Λ༻͍ͯ࠷খʹͯ͠ɼਖ਼نԽͷͨΊʹ୯ޠͰׂΔɽ -
&ODPEFS%FDPEFSXJUI5SBOTGPSNFST &ODPEFS%FDPEFSXJUI5SBOTGPSNFST Q = WQHdec[i−1] K = WKHenc V =
WVHenc DSPTTBUUFOUJPO CrossAttn(Q, K, V) = softmax ( QKT dk ) V ʢ4PVSDF5BSHFUܕ"UUFOUJPOʣ 2VFSZ ,FZ 7BMVF &ODPEFSͷӅΕঢ়ଶ %FDPEFSͷӅΕঢ়ଶ Q K V Henc Hdec -
4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ɾXPSEQJFDFࣙॻͷߏஙํ๏ʹ͍ͭͯ 6OJDPEFจࣈͷαϒηοτͳͲͰXPSEQJFDFࣙॻʢ#1ࣙॻʣͷ୯ޠΛॳظԽ 7ݸͷ୯ޠ͕Ͱ͖Δ·Ͱ܁Γฦ͢ ݱࡏͷ୯ޠू߹Λ༻ֶ͍ͯशίʔύεͰOHSBNΛֶश͢Δ
ݱࡏͷࣙॻ͔Βͭͷ୯ޠΛ࿈݁ͯ͠Ͱ͖Δ৽͍͠୯ޠͷՄೳੑͷηοτ Λߟ͑ͯɼֶशίʔύεͷݴޠϞσϧ֬Λ࠷্ͤ͞Δ৽͍͠୯ޠΛ ͭબ͢Δʢ#1ʣ ɾ͔Βݸͷޠኮ͕༻͞ΕΔ 5PLFOJ[BUJPO -
4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ίετؔ TPVSDFจॻͷεύϯɼ UBSHFUจॻͷεύϯ ֤จॻͰϥϯμϜʹબΕͨ4ݸͷεύϯ x y x1 , …,
xS , y1 , …, yS c(x, y) = (1 − cos(x, y))nSents(x) nSent(y) ∑S s=1 (1 − cos(x, ys )) + ∑S s=1 (1 − cos(xs , y)) εύϯʹؚ·ΕΔจͷ nSents() = ྨࣅΛ ਖ਼نԽ͢Δ ίʔύεͷΫϦʔϯΞοϓͰखॻ͖ϧʔϧͰਫ਼ͷ͍จͷϖΞΛআͯ͠ਫ਼Λ্͛Δ 4FOUFODF"MJHONFOU ৽͘͠ίʔύεͷ࡞͢Δ߹ w ιʔεจͱλʔήοτจͷͦΕͧΕεύϯΛऔΓɼ༁Ͱ͋ΔՄೳੑΛଌఆ͢Δ ίετؔ w จॻؒͰͷྑ͍είΞͷ"MJHONFOUʢϖΞɾྻʣΛٻΊΔΞϧΰϦζϜ ˠ࠷খฤूڑΞϧΰϦζϜͷ֦ு -
4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ɿಛఆͷݴޠͰର༁ίʔύε͕ݶΒΕ͍ͯͯɼֶशσʔλ͕ෆ͍ͯ͠Δ TPVSDF .5 .5 .5 UBSHFU CJUFYUͷ TPVSDFςΩετ CJUFYUͷ
UBSHFUςΩετ UBSHFUͷ ίʔύε TPVSDFͷ ςΩετ USBJO JOGFSFODF USBJO CJUFYU ੜͨ͠ςΩετ CJUFYU ίʔύε #BDLUSBOTMBUJPO -
.5&WBMVBUJPO .5&WBMVBUJPO ධՁج४ BEFRVBDZʢଥੑʣɿ.5༁จʹର͢Δݪจͷҙຯͷਖ਼֬͞ flVFODZʢྲྀெੑʣɿ.5༁จ͕5BSHFUݴޠͰͷྲྀெ͞ 6TJOH)VNBO3BUFSTUP&WBMVBUF.5 ਓ͕ؒPSຬͳͲͷईͰධՁʢίετͱ͕͔͔࣌ؒΔʣ flVFODZ
όΠϦϯΨϧʹݪจͱ.5༁จΛൺֱ ϞϊϦϯΨϧʹࢀߟ༁จͱ.5༁จΛൺֱ ˠධՁείΞ͔ΒҟৗΛআͯ͠ਖ਼نԽ͢Δ BEFRVBDZ .5༁จͷΘ͔Γ͢͞ɼ໌ྎ͞ɼಡΈ͢͞ͳͲ -
.5&WBMVBUJPO .5༁จͱࢀߟ༁จʹ͓͚Δग़ݱ͢Δ จࣈOHSBNͷ' "VUPNBUJD&WBMVBUJPO $IBSBDUFS0WFSMBQʢจࣈͷॏͳΓʣ DIS'ʢDIBSBDUFS'TDPSFʣ ύϥϝʔλ OHSBNͷ͞ k DIS1.5༁จͷ
d ͷจࣈOHSBN͕ࢀߟ༁จͰग़ݱ͢Δׂ߹ͷฏۉ DIS3ࢀߟ༁จͷ d ͷจࣈOHSBN͕ࢀߟ༁จͰग़ݱ͢Δׂ߹ͷฏۉ 1 k 1 k chrFβ = (1 + β2) chrP ⋅ chrR β2 ⋅ chrP + chrR .5 ༁จ ࢀߟ༁จ 1SFDJTJPO DIS1 3FDBMM DIS3 Ұൠʹ β = 2 -
.5&WBMVBUJPO DIS'ͳͲࢀߟ༁จͱ.5༁จͷڞ௨͢Δ୯ޠΛਖ਼֬ʹଌఆ͢ΔͨΊݫ͗͢͠Δ ˠ ࢀর ࢀߟ༁จ ͱ ީิ .5༁จ Λಉٛޠͱͯ͠ͷҰகΛߟྀ͢Δ
ͱ ͕ಉٛޠʢྨࣅੑʣͰ͋ΔධՁ x ˜ x r x ˜ x &NCFEEJOH#BTFE.FUIPET -
#JBTBOE&UIJDBM*TTVF #JBTBOE&UIJDBM*TTVF w จԽͷݻఆ֓೦ʹैͬͯδΣϯμʔͰ୯ޠׂ͕ΓͯΒΕΔ w ໊ࢺ͕ࢦ͢୯ޠʹΑͬͯมΘΔ ˠਫ਼͕Լ͕Δ w ઐ༻ޠͷΑ͏ͳ.5γεςϜ͕Βͳ͍୯ޠΛΔଌఆج४͕ະղܾ w
Ϧιʔε͕গͳ͍ݴޠ͋ΔͨΊϦιʔεͰͷ༁ՄೳͳΞϧΰϦζϜ͕ඞཁ ˠ.5ΞϧΰϦζϜͷ։ᚙͷࢀՃΛଅͨ͢ΊͷࢀՃܕϓϩηεσβΠϯͷఏҊ -
4VNNBSZ 4VNNBSZ ػց༁ࣗવݴޠॲཧʹ͓͍ͯ࠷͘ΘΕ͍ͯΔΞϓϦέʔγϣϯͷ̍ͭͰɼ .5ͷͨΊʹ։ൃ͞Εͨ&ODPEFS%FDPEFSϞσϧࠓͰॏཁͳπʔϧͰ͋Δɽ w ݴޠ͕࣋ͭߏޠኮͷࠩҟ͕༁Λ͍ͯ͘͠͠Δ w ݴޠֶͰྨܕతͳ࣍ݩͰݴޠΛྨ͍ͯ͠Δ w 3//T5SBOTGPSNFSTͷ&ODPEFS%FDPEFSωοτϫʔΫ
TFRTFR &ODPEFS͕ೖྗγʔέϯεΛड͚औΓɼͦͷίϯςΩετදݱΛੜ͠ɼ %FDPEFSʹ͞ΕɼλεΫʹಛԽͨ͠ग़ྗγʔέϯεΛੜ͢Δ w 3//ͷ"UUFOUJPO.FDIBOJTN5SBOTGPSNFSTͷDSPTTBUUFOUJPO %FDPEFS͕&ODPEFSͷશӅΕঢ়ଶ͔ΒใΛݟΔ͜ͱ͕Մೳ -
4VNNBSZ 4VNNBSZ w %FDPEFS֤εςοϓͰͷτʔΫϯબʹ HSFFEZEFDPEJOHͱCFBNTFBSDI͕༻͍ΒΕΔ w ର༁ίʔύε QBSBMMFMDPSQVT ͱ͍͏ݴޠͷςΩετ CJUFYU
Λֶश͢Δ w #BDLUSBOTMBUJPOλʔήοτݴޠͷίʔύεΛ༻͍ͯٯ༁͢Δख๏ w .5ͷධՁ༁ͷਓؒʹΑΔଥੑͱྲྀெੑͷଌఆ͕࠷࣮֬ͰجຊͰ͋Δ ͔͠͠ɼίετͱ͕͔͔࣌ؒΔͨΊɼDIS'ͷΑ͏ͳࣗಈධՁଌఆɼ &NCFEEJOHͷྨࣅʹجͮ͘ଌఆ͕͋Δ