EMNLP2015読み会：Effective Approaches to Attention-based Neural Machine Translation

Transcript

1. Effective Approaches to Attention-based Neural Machine Translation Authors: Minh-Thang LuongɹHieu PhamɹChristopher D. Manning ಡΉਓ: ಙӬ୓೭ ਤ΋શͯ͜ͷ࿦จ͔ΒҾ༻ 
   &./-1ಡΈձ
2. ࣗݾ঺հɿಙӬ୓೭ • Twitter ID: @tkng • εϚʔτχϡʔεגࣜձࣾͰNLP΍ͬͯ·͢
3. ࠓ೔ͷ࿦จ͸ʁ • Effective Approaches to Attention-based Neural Machine Translation • ڈ೥͙Β͍͔ΒྲྀߦΓ࢝Ίͨseq2seqܥͷख ๏ͷ֦ு
4. Seq2seq modelͱ͸ʁ • Encoder/Decoder modelͱ΋ݴ͏ • ຋༁ݩͷจΛݻఆ௕ͷϕΫτϧʹΤϯίʔυ ͯ͠ɺ͔ͦ͜Β຋༁ޙͷจΛσίʔυ͢Δ • Մม௕ͷσʔλ͸औΓѻ͍͕೉͍͠ͷͰɺ׶ ͑ͯݻఆ௕ʹͯ͠͠·͏ͱ͍͏ൃ૝
5. Ͳ͏΍ͬͯݻఆ௕ʹΤϯίʔυ ͢Δͷʁ • recurrent neural networkΛ࢖͏ • http://colah.github.io/posts/2015-08-Understanding-LSTMs/ • http://kaishengtai.github.io/static/slides/treelstm-acl2015.pdf • LSTM = recurrent neural networkͷҰछ
6. Seq2seqϞσϧͰͷ຋༁
7. Seq2seq·ͰͷಓͷΓ (1) • Recurrent Continuous Translation Models (EMNLP2013) • Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (EMNLP2014)
8. Seq2seq·ͰͷಓͷΓ (2) • Sequence to Sequence Learning with Neural Networks (NIPS2014) • ൺֱతγϯϓϧͳStacked LSTM͕ྑ͍ੑೳΛ ࣔ͢͜ͱ͕࣮ݧͰࣔ͞Εͨ • ϏʔϜαʔνɺٯॱͰͷೖྗɺΞϯαϯϒϧ ͷ3छྨͷ޻෉͕ೖ͍ͬͯΔ
9. Seq2seqϞσϧͷऑ఺ • ௕จʹऑ͍ • ݻ༗໊ࢺ͕ೖΕସΘΔ
10. AttentionʹΑΔվળ [Bahdanau+ 2015] • DecodeͷࡍͷContextʹEncodeͷࡍͷ֤࣌ࠁ ʹ͓͚ΔӅΕঢ়ଶͷॏΈ෇͖࿨Λ༻͍Δ • ॏΈࣗମ΋RNNͰܭࢉ͢Δ
11. ࠓճͷ࿦จͷߩݙ • ৽͍͠attention (local attention) ΛఏҊͨ͠ • ຋༁ݩจʹ͓͍ͯɺҐஔɹ͔ΒલޙD୯ޠ෼ ͷӅΕঢ়ଶͷॏΈ෇͖࿨ΛऔΔ • ॏΈͷܭࢉ͸global attentionͷ৔߹ͱಉ༷ • ɹ͸1ͭͣͭਐΊ͍ͯ͘৔߹ʢlocal-mʣ ͱɺ͜Εࣗମ΋RNNʹ͢Δ৔߹ʢlocal- pʣͷ2ͭΛ࣮ݧ͍ͯ͠Δ pt pt
12. local attention
13. local attention΁ͷҹ৅ • ޠॱ͕ࣅ͍ͯΔݴޠؒͰͷ຋༁ͳΒɺ໌Β͔ ʹ͜ͷํ͕ྑͦ͞͏ • ೔ӳΈ͍ͨʹޠॱ͕େ͖͘ҧ͏৔߹͸ɺ஫໨ Ґஔɹͷਪఆࣗମ͕೉͍͠λεΫʹͳͬͪΌ ͍ͦ͏… pt
14. ࣮ݧ݁ՌɿWMT'14
15. ࣮ݧ݁ՌɿWMT'14 • Α͘ݟΔͱɺlocal attentionͰͷੑೳ޲্͸ +0.9ϙΠϯτ • ଞͷςΫχοΫͰՔ͍ͰΔϙΠϯτ͕ଟ͍
16. ࣮ݧ݁ՌɿWMT'15
17. ͍͔ͭ͘຋༁αϯϓϧ
18. ·ͱΊ • Seq2seqϞσϧͷ֦ுͱͯ͠ɺlocal attention ΛఏҊͨ͠ • ఏҊख๏͸͍͔ͭ͘ͷ࣮ݧʹ͓͍ͯɺState of the artͷੑೳΛୡ੒ͨ͠
19. ײ૝ • Local attentionΛඍ෼ • ྨࣅ͢Δख๏ͱ۩ମతʹͲ͏ҧ͏͔͕໌շʹ ॻ͔Ε͓ͯΓɺಡΈ΍͔ͬͨ͢ • AttentionΛཧղͰ͖ͯΑ͔ͬͨʢখฒײʣ