Share
Embed
Transcript
20 slides
-
• Stylized response generation • ച ղীࢲ queryী ೠ ߸ਸ
ਗೞח झఋੌ۽ ࢤࢿ • polite, professional, friendly, … • ޙઁ 1. Parallel ؘఠ ࠗ • Text style transfer Ӕࠄ ޙઁ -> ӝઓ unsupervised approach۽ ೧Ѿೡ ࣻ হਸө? 2. Non-parallel ച ؘఠ ࠗ • ച ؘఠ non-parallel झఋੌ ؘఠ݅ ઓ (झ, ࣗࢸ, ࠶۽Ӓ ١) ➡ ӝઓ ߑधਵ۽ח less style-specificೞѢա less context-relevantೣ 3 Motivation -
• S2S + LM (Niu and Bansal, 2018) • ച
ؘఠ۽ seq2seq ݽ؛ ण • झఋੌ ؘఠ۽ LM ण • ف ݽ؛ ഛܫ оਵ۽ ష ஏ ➡ ъઁ۽ biasܳ ӝ ٸޙী relevance ↓ • Multi-task learning (Luan et al., 2017) • ച ؘఠ۽ seq2seq ݽ؛ ण • झఋੌ ؘఠ۽ autoencoder ण • ف ؘఠܳ э latent space ࢚ী mapping दఇ ➡ ৈ ܻ࠙ػ ۞झఠܳ ഋࢿೞӝ ٸޙী style intensity ↓ 4 Previous Work -
• ച-झఋੌ р shared latent space ഋࢿ • ਵ۽ ਬࢎೠ
stylized sentenceٜ ࠺तೠ Ҕী ݽب۾ • SPACEFUSION (Gao et al., 2019)ਸ non-parallel ؘఠ۽ ഛ • Automatic, human evaluation ݽف baselineী ࠺೧ ࣻ 5 Contributions -
• Joint optimizationਸ ా೧ ߸ relevance৬ diversityܳ ֫ • relevance৬
diversity shared latent spaceܳ ഋࢿ • ف ࢿਸ ઑೡ ࣻ ب۾ э ҕр ࢚ী align • Lossী ҙ۲ regularization termਸ ୶оೞח ߑध 7 Recap: SPACEFUSION -
• ܲ latent spaceՙܻ оөਕѱ ݅٘ח term • Cross-space distance
• • • • Same-space distance • • • dconv = ∑ i∈batch dE (zS2S(xi ), zAE(yi )) n l dstyle = 1 2 dcross NN ({zS2S(xi )}, {zAE(si )}) + 1 2 dcross NN ({zAE(si )}, {zS2S(xi )}) dcross NN ({ai }, {bi }) = ∑ i∈batch dE (ai , bNN of ai ) n l dspread-out = min[dsame NN ({zAE(yi )}), dsame NN ({zAE(si )}), dsame NN ({zS2S(xi )})] dsame NN ({ai }) = ∑ i∈batch dE (ai , aNN of ai ) n l ℒfuse,{conv,style} = d{conv,style} − dspread-out 9 STYLEFUSION: Fusion Objective -
• ҕр ࢚ীࢲ ߭ఠ زী ٮܲ ߸ചо ݒՍۣب۾ ೞח
term • Smoothing between prediction and target • • • Smoothing between non-stylized and random stylized sentence • • ✓ ℒsmooth,conv = − 1 |y| log p(y|zconv) zconv = (1 − u)zAE(y) + uzS2S(x) + ϵ ℒsmooth,style = − (1 − u) 1 |y| log p(y|zstyle) − u 1 |s| log p(s|zstyle) zstyle = (1 − u)zAE(y) + uzAE(s) + ϵ u ∼ U(0,1), ϵ ∼ N(0,σ2I) 10 STYLEFUSION: Smoothness Objective -
• • ੌ߈ਵ۽ • Overfitting ߑܳ ਤ೧ ۽ pretrain •
Data augmentation • ۽ <unk>۽ ݃झఊ • ℒ = − 1 |y| log p(y|zS2S) + ℒconv + ℒstyle Dstyle ≪ Dconv Dconv si ∈ Dstyle P(mask) ∝ (freq)−1 11 STYLEFUSION: Training Objective -
• • ߸ীࢲ ࢠ݂ • ରਗী ೱ ߉ ঋب۾
normalize: • • ܻ णػ style classifier ഛܫҗ о • neural: 2-layer GRU • ngram: logistic regression using multi-hot features (n=1,2,3,4) z = zS2S(x) + r zS2S(x) z ρ = |r|/(σ l) score(hi ) = (1 − λ)P(hi |zS2S(x)) + λPstyle(hi ) 12 STYLEFUSION: Inference -
• ؘఠࣇ • Reddit: ച, 10M context-response ह • arXiv,
Holmes: п 1M, 38K ޙ • : pretrained style classifier۽ Redditীࢲ ఠ݂ • Baselines • MTask (Luan et al., 2017) • S2S+LM (Niu and Bansal, 2018) • Retrieval, Rand, Human Dtest 14 Experimental Setup -
• Rand: style intensity ↑, but relevance ↓ • S2S+LM:
BLEU ↓ • MTask: style intensity ↓, diversity ↓ 18 Results: Automatic Evaluation -
• MTask: style intensity ↓ • S2S+LM: relevance ↓ 19
Results: Human Evaluation & Visualization zS2S(x) zAE(s) zAE(y)