Upgrade to Pro — share decks privately, control downloads, hide ads and more …

機械翻訳とエンコーダデコーダモデル

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
Avatar for Kyao Kyao
May 24, 2022
46

 機械翻訳とエンコーダデコーダモデル

次のサイトの2021年12月29日版の資料を参考にさせていただいております.
https://web.stanford.edu/~jurafsky/slp3/10.pdf

Avatar for Kyao

Kyao

May 24, 2022

Transcript

  1. Speech and Language Processing. Draft of Dec 29, 2021. Copyright

    © 2021 Daniel Jurafsky & James H. Martin. All Rights Reserved. $IBQUFS .BDIJOF5SBOTMBUJPOBOE &ODPEFS%FDPEFS.PEFMT #/BPLJ,PKJNB
  2.  -BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ  &ODPEFS%FDPEFS.PEFM  &ODPEFS%FDPEFSXJUI3//T  "UUFOUJPO  #FBN4FBSDI

     &ODPEFS%FDPEFSXJUI5SBOTGPSNFST  4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT  .5&WBMVBUJPO  #JBTBOE&UIJDBM*TTVFT 5BCMF
  3. ͋ΒΏΔࣗવݴޠʹ͓͚Δ౷ܭతීวੑʢྨࣅɾ૬ҧʣ -BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ 8PSE0SEFS5ZQPMPHZ 4 7 0ʹ͓͚Δޠॱ ӳޠ υΠπޠ 4 7

    0 ೔ຊޠ ώϯσΟʔޠ 4 0 7 ΞΠϧϥϯυޠ 7 4 0 4ओޠ 7ಈࢺ 0໨తޠ ྫʣӳޠɿ ೔ຊޠɿ )FTQFBLT+BQBOFTF ൴͸೔ຊޠΛ࿩͢ ໊ࢺͱܗ༰ࢺͷલޙؔ܎ ӳޠ ๺ژޠ "EK /PVO εϖΠϯޠ ݱ୅ϔϒϥΠޠ /PVO "EK "EKܗ༰ࢺ /PVO໊ࢺ ྫʣӳޠɿ εϖΠϯޠɿ HSFFOXJUDI CVSKBWFSEF -BOHVBHF%JWFSHFODFTBOE5ZQPMPHZ
  4. 5IF&ODPEFS%FDPEFS.PEFM จ຺ʹԠͨ͡ग़ྗྻΛੜ੒͢ΔϞσϧʢTFRTFRʣ 5IF&ODPEFS%FDPEFS.PEFM ߏ੒͢Δཁૉ  &ODPEFS͸ೖྗ Λड͚औΓɼͦΕʹରԠ͢Δจ຺දݱ Λੜ੒  ͷؔ਺Ͱ͋ΔίϯςΩετϕΫτϧ

    Λ%FDPEFSʹ఻͑Δ  %FDPEFS͸ Λड͚औΓɼӅΕঢ়ଶ Λੜ੒͠ɼରԠ͢Δग़ྗ xn 1 hn 1 hn 1 C C hm 1 ym 1 γʔέϯε ϕΫτϧ ೖྗ୯ޠ਺ ग़ྗ਺ xn 1 , hn 1 , ym 1 , hm 1 C n m
  5. &ODPEFS%FDPEFSXJUI3//T γϯϓϧͳϞσϧ &ODPEFS%FDPEFSXJUI3//T ht = g(ht−1 , xt ) C

    = hn yt = f(ht ) จ಄ϚʔΧ ࣗݾճؼ 5BTLӳޠˠεϖΠϯޠ ऑ఺ ೖྗͷจ຺͕࠷ऴతʹ۩ݱԽ͞Εͨ ͷσίʔμ΁ͷӨڹྗ͕࣍ୈʹബ·Δ hn ( = C) ׆ੑԽؔ਺ TPGUNBYؔ਺ g f จ ಄ Ϛ ʔ Χ
  6. &ODPEFS%FDPEFSXJUI3//T &ODPEFS%FDPEFSXJUI3//T C = he n hd 0 = C

    hd t = g( ̂ yt−1 , hd t−1 , C) zt = f(hd t ) yt = softmax(zt ) ̂ yt = argmaxw∈V P(w|x, y1 , ⋯, yt−1 ) ਺ࣜԽ ͜ͷ ͸׆ੑԽؔ਺Ͱ͸ͳ͍ g TPGUNBYؔ਺ޙͰϕΫτϧ ֤εςοϓͷ࠷΋֬཰͕ߴ͍ग़ྗ ̂ yt &ODPEFSͷӅΕঢ়ଶ he %FDPEFSͷӅΕঢ়ଶ hd ɾɾɾɾɾɾɾɾɾ ɾɾɾɾɾɾɾɾɾ ɾɾɾ ɾɾɾɾ ɾɾɾ ͜ͷ ͸׆ੑԽؔ਺Ͱ͋Δ f ɾɾɾɾɾɾɾɾ
  7. "UUFOUJPO ໰୊఺ɿγϯϓϧͳ&ODPEFS%FDPEFSϞσϧͷϘτϧωοΫ ݪจͷ৘ใ͕ ʹ͔͠ͳ͘ɼશͯͷ৘ใΛදݱ͠ͳ͚Ε͹ͳΒͳ͍ɽ ೖྗ͕௕จͷ৔߹ɼ Ͱಉ͡Α͏ʹදݱ͢Δ͜ͱ͕ࠔ೉ɽ C C "UUFOUJPO "UUFOUJPO.FDIBOJTNʢղܾํ๏ʣ

    &ODPEFSͷશӅΕঢ়ଶ͔Β৘ใΛσίʔμͰѻ͑Δܗͱͯ͠Ճ ॏ࿨Λ༻͍ͯͭͷݻఆϕΫτϧ Λಈతʹੜ੒͢Δ֓೦ C ֤σίʔυεςοϓ Ͱੜ੒͞ΕΔ ͸શͯͷ ͷؔ਺ i Ci he hd i = g( ̂ yi−1 , hd i−1 , Ci )
  8. "UUFOUJPO Λੜ੒࣌ʹ͓͍ͯ ͱશͯͷ ͱͷؔ࿈ੑΛٻΊΔ hd i hd i−1 he j

    "UUFOUJPO EPUQSPEVDUTDPSF શͯͷ ʹ͓͍ͯ಺ੵʹΑΓͭͷ ͷྨࣅ౓ΛٻΊΔ j h score(hd i−1 , he j ) αij = softmax(score(hd i−1 , he j )∀j ∈ e) = exp(score(hd i−1 , he j )) ∑ k exp(score(hd i−1 , he k )) score(hd i−1 , he j ) = hd i−1 ⋅ he j ͜ΕΒΛTPGUNBYؔ਺Ͱਖ਼نԽͨ͠΋ͷΛ ͱ͢Δͱ αij
  9. "UUFOUJPO w ͱ ͷՃॏ࿨Λ ͱ͢Δ αij he j Ci "UUFOUJPO

    Ci = ∑ j αij he j w FOEUPFOEͰֶशՄೳͳύϥϝʔλ Λ༻͍Δʢओྲྀʣ Ws score(hd i−1 , he j ) = hd i−1 Ws he j
  10. #FBN4FBSDI #FBN4FBSDI ֤εςοϓͰ࠷దͳ ݸͷτʔΫϯΛબ୒͢Δ k %FDPEFSʹద༻͢Δͱ  ιϑτϚοΫεͷग़ྗ͔Β࠷దͳԾઆΛબ୒͢Δɽ  ֤ԾઃΛҟͳΔ%FDPEFSʹೖྗ͢Δɽ

     ҟͳΔ%FDPEFS͝ͱʹTPGUNBYͷग़ྗΛٻΊΔɽ  ݸͷ֤τʔΫϯྻͰείΞϦϯάͯ͠ɼ࠷దͳ ݸબ୒͢Δɽ
 ͦͷதʹจ຤ϚʔΧ͕͋Δ৔߹ɼԾઆ͔Β֎ͯ͠ऴྃ͠ ΛͭݮΒ͢ɽ  Ͱ͋Ε͹ʹ໭Δɽ k × V k k k ≠ 0 ɿCFBNXJEUI ϏʔϜ෯  ൪໨·ͰͷτʔΫϯྻɿIZQPUIFTJT Ծઆ k k ͸Ұൠతʹେ͖ͳ.5γεςϜʹ͸ʙݸͰ༻͍ΒΕ͍ͯΔɽ k
  11. #FBN4FBSDI #FBN4FBSDI είΞϦϯά͸ର਺֬཰ʢର਺໬౓ؔ਺ʣΛ༻͍Δ score(y) = log P(y|x) = log(P(y1 |x)P(y2

    |y1 , x)⋯P(yt |y1 , ⋯, yt−1 , x)) = t ∑ i=1 log P(yi |y1 , ⋯, yi−1 , x) ʢDIBJOSVMFʣ ར఺ ௚લͷϊʔυͷείΞΛਵ࣌Ճࢉ͍ͯ͘͜͠ͱͰܭࢉޮ཰͕ྑ͍ ܭࢉྔ͸O(k × |V| × T) ϏʔϜ෯  શޠኮ਺  ग़ྗ୯ޠ਺ k |V| T ˠείΞΛߴ͍΋ͷ͔Βॱʹ ݸબ୒͢Δ k ͸Ξϯμϑϩʔରࡦ log
  12. #FBN4FBSDI #FBN4FBSDI ໰୊఺ ׬੒ͨ͠ԾઆʹΑΔ୯ޠྻͷ௕͕͞ҟͳΔͱ ௕͍จࣈྻͷ֬཰ʻ୹͍จࣈྻͷ֬཰ ͱͳΓɼ୹͍จࣈྻ͕બ୒͞Εͯ͠·͏ɽ score(y) = − log

    P(y|x) = − 1 T t ∑ i=1 log P(yi |y1 , ⋯, yi−1 , x) ղܾྫ ෛͷର਺֬཰Λ༻͍ͯ࠷খ໰୊ʹͯ͠ɼਖ਼نԽͷͨΊʹ୯ޠ਺ͰׂΔɽ
  13. &ODPEFS%FDPEFSXJUI5SBOTGPSNFST &ODPEFS%FDPEFSXJUI5SBOTGPSNFST Q = WQHdec[i−1] K = WKHenc V =

    WVHenc DSPTTBUUFOUJPO CrossAttn(Q, K, V) = softmax ( QKT dk ) V ʢ4PVSDF5BSHFUܕ"UUFOUJPOʣ 2VFSZ ,FZ 7BMVF &ODPEFSͷӅΕঢ়ଶ %FDPEFSͷӅΕঢ়ଶ Q K V Henc Hdec
  14. 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ɾXPSEQJFDFࣙॻͷߏஙํ๏ʹ͍ͭͯ  6OJDPEFจࣈͷαϒηοτͳͲͰXPSEQJFDFࣙॻʢ#1ࣙॻʣͷ୯ޠΛॳظԽ  7ݸͷ୯ޠ͕Ͱ͖Δ·Ͱ܁Γฦ͢  ݱࡏͷ୯ޠू߹Λ༻ֶ͍ͯशίʔύεͰOHSBNΛֶश͢Δ 

    ݱࡏͷࣙॻ͔Βͭͷ୯ޠΛ࿈݁ͯ͠Ͱ͖Δ৽͍͠୯ޠͷՄೳੑͷηοτ Λߟ͑ͯɼֶशίʔύεͷݴޠϞσϧ֬཰Λ࠷΋޲্ͤ͞Δ৽͍͠୯ޠΛ ͭબ୒͢Δʢ#1ʣ ɾ͔Βݸͷޠኮ͕࢖༻͞ΕΔ 5PLFOJ[BUJPO
  15. 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ίετؔ਺ TPVSDFจॻͷεύϯɼ UBSHFUจॻͷεύϯ ֤จॻͰϥϯμϜʹબ͹Εͨ4ݸͷεύϯ x y x1 , …,

    xS , y1 , …, yS c(x, y) = (1 − cos(x, y))nSents(x) nSent(y) ∑S s=1 (1 − cos(x, ys )) + ∑S s=1 (1 − cos(xs , y)) εύϯʹؚ·ΕΔจͷ਺ nSents() = ෼฼͸ྨࣅ౓Λ ਖ਼نԽ͢Δ ίʔύεͷΫϦʔϯΞοϓͰखॻ͖ϧʔϧͰਫ਼౓ͷ௿͍จͷϖΞΛ࡟আͯ͠ਫ਼౓Λ্͛Δ 4FOUFODF"MJHONFOU ৽͘͠ίʔύεͷ࡞੒͢Δ৔߹ w ιʔεจͱλʔήοτจͷͦΕͧΕεύϯΛऔΓɼ຋༁Ͱ͋ΔՄೳੑΛଌఆ͢Δ ίετؔ਺ w จॻؒͰͷྑ͍είΞͷ"MJHONFOUʢϖΞɾ੔ྻʣΛٻΊΔΞϧΰϦζϜ
 ˠ࠷খฤूڑ཭ΞϧΰϦζϜͷ֦ு
  16. 4PNFQSBDUJDBMEFUBJMTPOCVJMEJOH.5TZTUFNT ໰୊఺ɿಛఆͷݴޠͰ͸ର༁ίʔύε͕ݶΒΕ͍ͯͯɼֶशσʔλ͕ෆ଍͍ͯ͠Δ TPVSDF .5 .5 .5 UBSHFU CJUFYUͷ TPVSDFςΩετ CJUFYUͷ

    UBSHFUςΩετ UBSHFUͷ ίʔύε TPVSDFͷ ςΩετ USBJO JOGFSFODF USBJO CJUFYU 
 ੜ੒ͨ͠ςΩετ CJUFYU  ίʔύε #BDLUSBOTMBUJPO
  17. .5&WBMVBUJPO .5&WBMVBUJPO ධՁج४  BEFRVBDZʢଥ౰ੑʣɿ.5༁จʹର͢Δݪจͷҙຯͷਖ਼֬͞  flVFODZʢྲྀெੑʣɿ.5༁จ͕5BSHFUݴޠͰͷྲྀெ͞ 6TJOH)VNBO3BUFSTUP&WBMVBUF.5 ਓ͕ؒ఺PS఺ຬ఺ͳͲͷई౓ͰධՁʢίετͱ͕͔͔࣌ؒΔʣ flVFODZ

    όΠϦϯΨϧʹ͸ݪจͱ.5༁จΛൺֱ ϞϊϦϯΨϧʹ͸ࢀߟ༁จͱ.5༁จΛൺֱ ˠධՁείΞ͔Βҟৗ஋Λ࡟আͯ͠ਖ਼نԽ͢Δ BEFRVBDZ .5༁จͷΘ͔Γ΍͢͞ɼ໌ྎ͞ɼಡΈ΍͢͞ͳͲ
  18. .5&WBMVBUJPO .5༁จͱࢀߟ༁จʹ͓͚Δग़ݱ͢Δ จࣈOHSBNͷ'஋ "VUPNBUJD&WBMVBUJPO $IBSBDUFS0WFSMBQʢจࣈͷॏͳΓʣ DIS'ʢDIBSBDUFS'TDPSFʣ ύϥϝʔλ OHSBNͷ௕͞ k DIS1.5༁จͷ

    d ͷจࣈOHSBN͕ࢀߟ༁จͰग़ݱ͢Δׂ߹ͷฏۉ஋ DIS3ࢀߟ༁จͷ d ͷจࣈOHSBN͕ࢀߟ༁จͰग़ݱ͢Δׂ߹ͷฏۉ஋ 1 k 1 k chrFβ = (1 + β2) chrP ⋅ chrR β2 ⋅ chrP + chrR .5 ༁จ ࢀߟ༁จ 1SFDJTJPO DIS1 3FDBMM DIS3 Ұൠʹ β = 2
  19. 4VNNBSZ 4VNNBSZ ػց຋༁͸ࣗવݴޠॲཧʹ͓͍ͯ࠷΋޿͘࢖ΘΕ͍ͯΔΞϓϦέʔγϣϯͷ̍ͭͰɼ .5ͷͨΊʹ։ൃ͞Εͨ&ODPEFS%FDPEFSϞσϧ͸ࠓͰ΋ॏཁͳπʔϧͰ͋Δɽ w ݴޠ͕࣋ͭߏ଄΍ޠኮͷࠩҟ͕຋༁Λ೉͍ͯ͘͠͠Δ w ݴޠֶͰ͸ྨܕతͳ࣍ݩͰݴޠΛ෼ྨ͍ͯ͠Δ w 3//T΍5SBOTGPSNFSTͷ&ODPEFS%FDPEFSωοτϫʔΫ

    TFRTFR ͸
 &ODPEFS͕ೖྗγʔέϯεΛड͚औΓɼͦͷίϯςΩετදݱΛੜ੒͠ɼ
 %FDPEFSʹ౉͞ΕɼλεΫʹಛԽͨ͠ग़ྗγʔέϯεΛੜ੒͢Δ w 3//ͷ"UUFOUJPO.FDIBOJTN΍5SBOTGPSNFSTͷDSPTTBUUFOUJPO͸
 %FDPEFS͕&ODPEFSͷશӅΕঢ়ଶ͔Β৘ใΛݟΔ͜ͱ͕Մೳ
  20. 4VNNBSZ 4VNNBSZ w %FDPEFS͸֤εςοϓͰͷτʔΫϯબ୒ʹ͸
 HSFFEZEFDPEJOHͱCFBNTFBSDI͕༻͍ΒΕΔ w ର༁ίʔύε QBSBMMFMDPSQVT ͱ͍͏ݴޠͷςΩετ CJUFYU

    Λֶश͢Δ w #BDLUSBOTMBUJPO͸λʔήοτݴޠͷίʔύεΛ༻͍ͯٯ຋༁͢Δख๏ w .5ͷධՁ͸຋༁ͷਓؒʹΑΔଥ౰ੑͱྲྀெੑͷଌఆ͕࠷΋࣮֬ͰجຊͰ͋Δ
 ͔͠͠ɼίετͱ͕͔͔࣌ؒΔͨΊɼDIS'ͷΑ͏ͳࣗಈධՁଌఆɼ &NCFEEJOHͷྨࣅ౓ʹجͮ͘ଌఆ͕͋Δ