SNLP2019: Improving Neural Conversational Models with Entropy-Based Data Filtering

Transcript

1. Improving Neural Conversational Models with Entropy-Based Data Filtering Richard Csaky, Patrik Purgai, Gabor Recski ACL2019 ˞ਤද͸࿦จΑΓҾ༻ ಡΉਓ: ੺ؒ ྯಸ (౦๺େ/ཧݚAIP) ୈ11ճ࠷ઌ୺NLPษڧձ 2019-09-28
2. ·ͱΊ • ໨త: ଟ༷ੑͷ͋ΔԠ౴Λੜ੒͢Δ • ઓུ: Ϟσϧʹ generic ͳԠ౴Λֶशͤ͞ͳ͍ • ఏҊ: ֶशϖΞσʔλϑΟϧλϦϯά • ൃ࿩ͷ൚༻ੑΛ entropy Ͱදݱ • Entropy ͷߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ • ධՁ: • ର࿩Ԡ౴ੜ੒λεΫ • ෳ਺ࣗಈධՁई౓ͰείΞ޲্ͷ܏޲ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ 
3. ର࿩Ԡ౴ੜ੒ͰΑ͋͘Δ࿩ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  ೖྗ: What color is the sky? ग़ྗ: I don’t know. ೖྗ: What is your name? ग़ྗ: I don’t know. ೖྗ: I’m losing my key. ग़ྗ: I don’t know. ೖྗ: Hey! What's up!? ग़ྗ: I don’t know what you are talking about.
4. ͳͥ generic ͳԠ౴͕ੜ੒͞ΕΔʁ • ର࿩͸ many-to-many ͳੑ࣭Λ࣋ͭ • χϡʔϥϧϞσϧ͸ฏۉΛֶश͢Δ [Wu+’18] “ؒҧΘͳ͍” “҆શͳ” Ԡ౴΁ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  Wu+’18; Why do neural response generation models prefer universal replies? arXiv:1808.09187 What color is the sky? It is blue. The setting sun is amazingly red. I don’t know. what do you mean? I like orange better. You look hot.
5. Many-to-many ͷղܾΞϓϩʔν Ϟσϧ • Ϟσϧ֦ு [Serban+’17; Zhao+’18; Gao+’19; Wang+’18] • જࡏม਺ΛαϯϓϦϯάɾσίʔυΛվળ • ଛࣦؔ਺ [Li+’16] • ڧԽֶशɾఢରతֶश [Li+’16; Serban+’17; Lipton+’18] σʔλ • ෇Ճ৘ใͷར༻ [Li+’16; Liu+’17; Xing+’18; Baheti+’18] • ର࿩ཤྺɾΧςΰϦʢe.g., ϖϧιφɾײ৘ɾτϐοΫɾ஌ࣝʣ • ֶशσʔλϑΟϧλϦϯά 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ 
6. Entropy • Many ౓߹͍Λ entropy ! ͱΈͳ͢ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  "# $# $% $& " "% $
7. Entropy ͷ؍࡯ (DailyDialog) • Entropy ্Ґ20ൃ࿩ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  !" !# !$ % ӳޠֶशऀ༻೔ৗձ࿩
8. Entropy ͷ؍࡯ (DailyDialog) • Entropy ෼෍ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  87% 5% 8% ' = 0 ' = 1 ' > 1
9. Entropy ͷ؍࡯ (DailyDialog) • Entropy ͱൃ࿩ग़ݱස౓ɾ୯ޠ௕ͷؔ܎ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ 
10. ఏҊ: Entropy-based training data filtering Entropy ͕ߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ 1. SOURCE: 2. TARGET: 3. BOTH: SOURCE or TARGET 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  > 1 > 1
11. ఏҊ: Entropy-based training data filtering • SOURCE 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  > 1 (DailyDialog)
12. ࣮ݧ: ର࿩Ԡ౴ੜ੒ • ֶशσʔλ: DailyDialog (90k utterances in 13k dialogs) • Ϟσϧ: Transformer [Vaswani+’17] • ධՁ: ࣗಈධՁई౓ (17छ) • Word and utterance entropy [Serban+’17] • KL divergence • Embedding metrics (average, extrema, greedy) • Coherence • Distinct • BLEU 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  Entropy-based training data filtering  (proposed) vs
    (base) Ϋϥελߟྀ
13. ࣮ݧ݁Ռ: Filtering ༗ > ແ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  • TARGET ϑΟϧλϦϯά͕࠷ྑͷ݁Ռ Random Gold (Table 2) non-genericness diversity
14. ·ͱΊ • ໨త: ଟ༷ੑͷ͋ΔԠ౴Λੜ੒͢Δ • ઓུ: Ϟσϧʹ generic ͳԠ౴Λֶशͤ͞ͳ͍ • ఏҊ: ֶशϖΞσʔλϑΟϧλϦϯά • ൃ࿩ͷ൚༻ੑΛ entropy Ͱදݱ • Entropy ͷߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ • ධՁ: • ର࿩Ԡ౴ੜ੒λεΫ • ෳ਺ࣗಈධՁई౓ͰείΞ޲্ͷ܏޲ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ 
15. +α: Best performance ͸Ͳ͜ʁ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  • valid loss ࠷௿ (10-20k) < training loss ࠷௿ (80-100k) | , | '- '. distinct 567
16. +α: ଞछίʔύεͰ΋ޮՌ͋Γ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ  • Filtering • ࣗಈධՁ݁Ռ ʢөըࣈນʣ ʢ೔ৗձ࿩ʣ Cornellʢөըࣈນʣ: Twitter:
17. +α: Utterance cluster Λߟྀ • ྨࣅൃ࿩ΛΫϥελԽ • AVG-EMBEDDING (AE) • SENT2VEC (SC) • ݁Ռɿfiltering ແ < AE,SC < ID (ඇΫϥελԽ) • tab. 1 • tab. 2 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ