Tweet
Tweet

Slide 1

Slide 1 text

Improving Neural Conversational Models with Entropy-Based Data Filtering Richard Csaky, Patrik Purgai, Gabor Recski ACL2019 ˞ਤද͸࿦จΑΓҾ༻ ಡΉਓ: ੺ؒ ྯಸ (౦๺େ/ཧݚAIP) ୈ11ճ࠷ઌ୺NLPษڧձ 2019-09-28

Slide 2

Slide 2 text

·ͱΊ • ໨త: ଟ༷ੑͷ͋ΔԠ౴Λੜ੒͢Δ • ઓུ: Ϟσϧʹ generic ͳԠ౴Λֶशͤ͞ͳ͍ • ఏҊ: ֶशϖΞσʔλϑΟϧλϦϯά • ൃ࿩ͷ൚༻ੑΛ entropy Ͱදݱ • Entropy ͷߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ • ධՁ: • ର࿩Ԡ౴ੜ੒λεΫ • ෳ਺ࣗಈධՁई౓ͰείΞ޲্ͷ܏޲ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ

Slide 3

Slide 3 text

ର࿩Ԡ౴ੜ੒ͰΑ͋͘Δ࿩ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ ೖྗ: What color is the sky? ग़ྗ: I don’t know. ೖྗ: What is your name? ग़ྗ: I don’t know. ೖྗ: I’m losing my key. ग़ྗ: I don’t know. ೖྗ: Hey! What's up!? ग़ྗ: I don’t know what you are talking about.

Slide 4

Slide 4 text

ͳͥ generic ͳԠ౴͕ੜ੒͞ΕΔʁ • ର࿩͸ many-to-many ͳੑ࣭Λ࣋ͭ • χϡʔϥϧϞσϧ͸ฏۉΛֶश͢Δ [Wu+’18] “ؒҧΘͳ͍” “҆શͳ” Ԡ౴΁ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ Wu+’18; Why do neural response generation models prefer universal replies? arXiv:1808.09187 What color is the sky? It is blue. The setting sun is amazingly red. I don’t know. what do you mean? I like orange better. You look hot.

Slide 5

Slide 5 text

Many-to-many ͷղܾΞϓϩʔν Ϟσϧ • Ϟσϧ֦ு [Serban+’17; Zhao+’18; Gao+’19; Wang+’18] • જࡏม਺ΛαϯϓϦϯάɾσίʔυΛվળ • ଛࣦؔ਺ [Li+’16] • ڧԽֶशɾఢରతֶश [Li+’16; Serban+’17; Lipton+’18] σʔλ • ෇Ճ৘ใͷར༻ [Li+’16; Liu+’17; Xing+’18; Baheti+’18] • ର࿩ཤྺɾΧςΰϦʢe.g., ϖϧιφɾײ৘ɾτϐοΫɾ஌ࣝʣ • ֶशσʔλϑΟϧλϦϯά 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ

Slide 6

Slide 6 text

Entropy • Many ౓߹͍Λ entropy ! ͱΈͳ͢ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ "# $# $% $& " "% $

Slide 7

Slide 7 text

Entropy ͷ؍࡯ (DailyDialog) • Entropy ্Ґ20ൃ࿩ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ !" !# !$ % ӳޠֶशऀ༻೔ৗձ࿩

Slide 8

Slide 8 text

Entropy ͷ؍࡯ (DailyDialog) • Entropy ෼෍ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ 87% 5% 8% ' = 0 ' = 1 ' > 1

Slide 9

Slide 9 text

Entropy ͷ؍࡯ (DailyDialog) • Entropy ͱൃ࿩ग़ݱස౓ɾ୯ޠ௕ͷؔ܎ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ

Slide 10

Slide 10 text

ఏҊ: Entropy-based training data filtering Entropy ͕ߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ 1. SOURCE: 2. TARGET: 3. BOTH: SOURCE or TARGET 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ > 1 > 1

Slide 11

Slide 11 text

ఏҊ: Entropy-based training data filtering • SOURCE 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ > 1 (DailyDialog)

Slide 12

Slide 12 text

࣮ݧ: ର࿩Ԡ౴ੜ੒ • ֶशσʔλ: DailyDialog (90k utterances in 13k dialogs) • Ϟσϧ: Transformer [Vaswani+’17] • ධՁ: ࣗಈධՁई౓ (17छ) • Word and utterance entropy [Serban+’17] • KL divergence • Embedding metrics (average, extrema, greedy) • Coherence • Distinct • BLEU 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ Entropy-based training data filtering (proposed) vs (base) Ϋϥελߟྀ

Slide 13

Slide 13 text

࣮ݧ݁Ռ: Filtering ༗ > ແ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ • TARGET ϑΟϧλϦϯά͕࠷ྑͷ݁Ռ Random Gold (Table 2) non-genericness diversity

Slide 14

Slide 14 text

·ͱΊ • ໨త: ଟ༷ੑͷ͋ΔԠ౴Λੜ੒͢Δ • ઓུ: Ϟσϧʹ generic ͳԠ౴Λֶशͤ͞ͳ͍ • ఏҊ: ֶशϖΞσʔλϑΟϧλϦϯά • ൃ࿩ͷ൚༻ੑΛ entropy Ͱදݱ • Entropy ͷߴ͍ൃ࿩Λֶशσʔλ͔Βআڈ • ධՁ: • ର࿩Ԡ౴ੜ੒λεΫ • ෳ਺ࣗಈධՁई౓ͰείΞ޲্ͷ܏޲ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ

Slide 15

Slide 15 text

+α: Best performance ͸Ͳ͜ʁ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ • valid loss ࠷௿ (10-20k) < training loss ࠷௿ (80-100k) | , | '- '. distinct 567

Slide 16

Slide 16 text

+α: ଞछίʔύεͰ΋ޮՌ͋Γ 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ • Filtering • ࣗಈධՁ݁Ռ ʢөըࣈນʣ ʢ೔ৗձ࿩ʣ Cornellʢөըࣈນʣ: Twitter:

Slide 17

Slide 17 text

+α: Utterance cluster Λߟྀ • ྨࣅൃ࿩ΛΫϥελԽ • AVG-EMBEDDING (AE) • SENT2VEC (SC) • ݁Ռɿfiltering ແ < AE,SC < ID (ඇΫϥελԽ) • tab. 1 • tab. 2 2019-09-28 ୈ11ճ࠷ઌ୺NLPษڧձ