Delete, Retrieve, Generate: Approach for Text Style Transfer

Transcript

1. Delete,Retrieve,Generate Approach for Text Style Transfer ੿׮਍ (ML Engineer, Pingpong)
2. ݾର ݾର 1. ࣁ޷ա ѐਃ 2. Delete,Retrieve,Generate: 
 A Simple Approach to Sentiment and Style Transfer (NAACL 2018) 3. Transforming Delete,Retrieve,Generate Approach 
 for Controlled Text Style Transfer (EMNLP 2019)
3. ࣁ޷ա ѐਃ " ࣁ޷ա ѐਃ
4. • Style Transfer • ࠺੹ ࠙ঠীࢲח ੜ ঌ۰૓ ӝࣿ۽
 ੉޷૑ ղ੄ ௑బஎ (য়࠳ં౟)ח ਬ૑ೞݶࢲ, झఋੌ݅ਸ ߸ജೞৈ ੉޷૑ܳ ੤ҳࢿೞח ӝࣿ ࣁ޷ա ѐਃ What is Style Transfer?
5. • Text Style Transfer • ఫझ౟੄ ղਊ਷ ਬ૑ೞݶࢲ, ఫझ౟੄ झఋੌਸ ߸ജೞৈ ఫझ౟ܳ ੤ҳࢿೞח ӝࣿ • Ӓ۞ա, ఫझ౟ীࢲ ݈ೞח ղਊҗ झఋੌ੉ۆ…? • গݒೞҊ ੿੄ೞӝ য۰਍ ѐ֛ • ٮۄࢲ, ೟ࣿ੸ੋ োҳٜ਷ ఫझ౟੄ хࢿ ߂ दઁ ١ ౠ੿ attributeী ୡ੼ਸ ݏ୶ח ҃਋о ݆਺ ࣁ޷ա ѐਃ What is Text Style Transfer? The chicken was delicious The chicken was terrible positive negative
6. • Delete,Retrieve,Generate (DRG)ۄח ߑߨۿਸ ੉ਊೠ Text Style Transferী ҙೠ ֤ޙਸ 2ѐ ࣗѐ • ఫझ౟੄ Style Transferܳ ా೧ Controllableೠ ޙ੢੄ ࢤࢿਸ ԫ೧ࠅ ࣻ ੓਺ • ઁয оמೠ (Controllable) ޙ੢੄ ࢤࢿ਷ ਋ܻীѱ ੓যࢲ рҗೡ ࣻ হח ࠗ࠙ 1. ೝಯ੉о ޖ঺ਸ ݈ೞח૑ب ઺ਃೞ૑݅ যڌѱ ݈ೞח૑ী ٮۄ ؀ച੄ ݅઒بח ௼ѱ ׳ۄ૗ • झ௼݀౴੄ ੗زച, ಕܰࣗա੄ ాੌ ١ 2. ೝಯ੉੄ ݈ਸ য়۽૑ ݽ؛ী݅ ੄ઓೞח Ѫ੉ ইפۄ ӝദ੗੄ ੄بী ٮۄ ੸੺൤ ઁয оמ • ੉ߣ ఢ਷ negativeೠ ׹߸݈Ҋ positiveೠ ׹߸ਸ ೞ੗! ۄ؍о • ੉ߣ ఢ਷ ಣࢲޙ੉ ইפۄ ૕ޙਸ ؍ઉࠁ੗! ۄ؍о ࣁ޷ա ѐਃ ࣁ޷ա ѐਃ
7. 1. Delete,Retrieve,Generate: 
 A Simple Approach to Sentiment and Style Transfer (NAACL 2018) Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer
8. • ಕয ؘ੉ఠࣇ੉ ইצ non-parallel ؘ੉ఠࣇਸ ੉ਊೞৈ झఋੌਸ ߸ജೞח ߑߨۿਸ ઁউ • ੌ߈੸ਵ۽ زੌೠ ஶబஎী झఋੌ݅ ׮ܲ ఫझ౟ ಕযहਸ ҳೞӝח য۰਑ • ੉ ֤ޙ ੉੹ীח ఫझ౟۽ࠗఠ content latent representations৬ style latent representationsܳ implicit ೞѱ ܻ࠙ೞח ݽ؛ਸ ೟णೞח ߑधী ؀ࣁ৓਺. Ӓ۞ա… 1. ӝࠄ੸ਵ۽ ࢤࢿػ ޙ੢੄ fluencyо ڄয૑Ѣա झఋੌ ߸ജ੉ ੜ উؽ 2. content ࠁઓਯ ߂ style ߸ജਯ੄ ౟ۨ੉٘ য়೐ܳ ઑ੺ೞӝо ൨ٝ • ୭Ӕ Text Style Transfer ҙ۲ ؀ࠗ࠙੄ ֤ޙٜীࢲ ߬੉झۄੋਵ۽ ॳ੉ח ݽ؛ Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer ֤ޙ ѐਃ
9. • ߑߨۿ 1. Delete: ޙ੢ਵ۽ࠗఠ ਗې झఋੌী ೧׼ೞח 
 attribute words݅ਸ ઁѢ 2. Retrieve: target style corpus۽ࠗఠ 
 ਗ ޙ੢җ ਬࢎೠ ஶబஎܳ ׸਷ ޙ੢ਸ Ѩ࢝ೞৈ, 
 Ӓ ޙ੢ ղ੄ attribute wordsܳ ୶୹ 3. Generate: Delete җ੿ীࢲ झఋੌ ࣘࢿ੉ ઁѢػ ޙ੢җ 
 Retrieve җ੿ীࢲ Ѩ࢝ػ झఋੌ ࣘࢿ ױযٜਸ ੉ਊೞৈ 
 target झఋੌ۽ ߸ജػ ޙ੢ਸ ࢤࢿ Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Method ӛ੿ ࠗ੿ Delicious So good Friendly Awesome Loves … Ugly Very rude Sucks Not good … Great food but horrible staff and very rude workers Great food but staff and workers 1.Delete 2.Retrieve Awesome, Friendly 3.Generate Great food, awesome staff and friendly workers
10. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Delete • ௏ಌझ ղীࢲ झఋੌী ؀ೠ ध߹۱੉ о੢ ֫਷ (झఋੌ ӓࢿ੉ о੢ ௾) word-ngramsܳ ઁѢ • п n-gramsо пп੄ ௏ಌझী ١੢ೞח ࠼بܳ ష؀۽ ੼ࣻܳ ҅࢑ • ౠ੿ thresholdܳ ֈ਷ n-gramী ؀೧ࢲ attribute words۽ р઱
11. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Retrieve • ੸੺ೠ target style attribute wordsܳ ୶୹ೞӝ ਤ೧
 source sentence৬ ਬࢎೠ ޙ੢ਸ target style corpusীࢲ Ѩ࢝ • This food is (delicious) -> This food is ugly (X), This food is bland (O) • ਤ धীࢲ ف ޙ੢ ࢎ੉੄ Ѣܻܳ ੤ח ೣࣻ dח ޙ੢ р Ѣܻܳ ੤ח ইޖ metric੉ա ࢚ҙ হ਺ 1. TF-IDF weighted word overlap 2. Euclidean distance using sentence embedding 3. Edit distance ١١…
12. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Generate • ୨ 4о૑ ߑߨਸ ઁউ 1. RetrieveOnly: Retrieve Ѿҗ Ѩ࢝ػ target sentenceܳ Ӓ؀۽ ੉ਊ 2. TemplateBased: Deleteػ ޙ੢җ Retrieveীࢲ Ѩ࢝ػ target attribute markersܳ ױࣽ concat 3. DeleteOnly: Deleteػ ޙ੢җ target style ࣘࢿਸ ੑ۱ೞݶ 
 target style sentenceо ࢤࢿغب۾ ೞח LSTMӝ߈ encoder-decoder ݽ؛ 4. DeleteAndRetrieve: Deleteػ ޙ੢җ Retrieveীࢲ Ѩ࢝ػ target attribute markersܳ ੑ۱ೞݶ
 target style sentenceо ࢤࢿغب۾ ೞח LSTMӝ߈ encoder-decoder ݽ؛
13. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Experiments • ؘ੉ఠࣇ • Yelp, Amazon ܻ࠭੄ positive, negative ؘ੉ఠࣇ • ੉޷૑ ஭࣌੄ romantic, humorous ؘ੉ఠࣇ • ಣоח ௼ۄ਋٘ ࣗयਸ ੉ਊೞৈ 5ױ҅ ಣо • ղਊ ࠁઓਯ • ࣘࢿ (झఋੌ) ߸҃ਯ • ޙ੢੄ ੗োझ۞਑
14. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Experimental Results • 4 ഑਷ 5੼ਸ ߉਷ ҃਋ Success۽ р઱ • RetrieveOnly, TemplateBased ١੄ ߬੉झۄੋ ݽ؛ٜ੉ ӝઓ ݽ؛ٜࠁ׮ જ਷ ࢿמ • DeleteAndRetrieve ݽ؛੄ ࢿמ੉ о੢ ਋ࣻೞա Human Ѿҗীח ݆੉ ޅ ޷ஜ
15. 2. Transforming Delete,Retrieve,Generate Approach 
 for Controlled Text Style Transfer (EMNLP 2019) Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer
16. • Pre-trained language modelҗ Transformer੄ ౠࢿਸ ੉ਊ೧ࢲ ӝઓ ֤ޙ੄ ױ੼ਸ ࠁ৮ೞ੗ 1. Delete җ੿ীࢲ ղਊী ೧׼ೞח ೨ब ױযܳ ઁѢೣ 2. Source style ױযܳ ઁѢೞ૑ ޅೣ 3. LSTMӝ߈ encoder-decoder ݽ؛੉ Ӓ׮૑ ۽ߡझ౟ೞ૑ ޅ೧ ੗োझ۞਍ ޙ੢ਸ ղߜ૑ ޅೣ 4. ӟ ޙ੢ী ஂডೣ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer ֤ޙ ѐਃ
17. • Delete, Retrieve, Generateۄח ௾ ߑߨۿ ੗୓ח زੌ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Method
18. • Input reduction (Feng et al., 2018) • style classificationਸ ೡ ٸ о੢ ӝৈبо ௾ ױযٜ੉ attribute wordsੌ Ѫ • BERT-based transformerܳ Delete Transformer۽ ࢎਊ • [CLS] ష௾ী ؀ೠ attention scoreо ࢚ਤੋ ష௾ٜਸ attribute words۽ р઱ೞৈ ઁѢ • BERTח ӝࠄ੸ਵ۽ (Multi-head, Multi-layer)੉Ҋ п க, ೻٘݃׮ ׮ܲ ৉ೡਸ ыਵ޲۽ 
 attribute words ઁѢী ੸೤ೠ attention headܳ ଺ח җ੿੉ ೙ਃೣ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Delete
19. • attribute words ઁѢী ੸೤ೠ attention headܳ ଺ח җ੿ • ޙ੢ ղীࢲ ੐੄۽ ѐ੄ ష௾ਸ ઁѢೠ ޙ੢ਸ ۽ ಴അ • п (h, l)ী ؀೧ (4)৬ э੉ scoreܳ ݒӣ • (5)৬ э੉ validation datasetী ؀೧ ੉ झ௏য੄ ୨೤੉ ઁੌ ੘਷ (h, l)ਸ 
 о੢ ੸೤ೠ attention head۽ Ѿ੿ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Delete γ|x| x′h,l
20. • ӝઓ ࣻߨҗ زੌೞѱ, source sentence৬ ਬࢎೠ ஶబஎܳ ׸Ҋ ੓ח target sentenceܳ Ѩ࢝ • ف ޙ੢ ࢎ੉੄ Ѣܻܳ ੤ח ೣࣻ d 1. TF-IDF weighted 2. Averaged-GloVe over all tokens of a sentence 3. Universal Sentence Encoder (Cer et al., 2018) • 3о૑ metric ઺ 1.੉ о੢ retrieval Ѿҗо જও׮Ҋ ೣ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Retrieve
21. • GPTܳ ӝ߈ਵ۽ ೞח Generative Style Transformer (GST)ܳ ઁউ • Blind Generative Style Transformer (B-GST), Guided Generative Style Transformer(G-GST) • B-GST • Deleteػ ޙ੢җ target styleਸ ੑ۱ • ୹۱ ޙ੢ਸ ࢤࢿೞחؘী ੓যࢲ ੗ਬبо ੓Ҋ
 ਗ ޙ੢җ ਬࢎೠ ޙ੢ਸ target corpus۽ࠗఠ retrieveೞӝ য۰਍ ҃਋ী ਬਊೣ • G-GST • Deleteػ ޙ੢җ target attribute wordsܳ ੑ۱ • ਗ ޙ੢җ ਬࢎೠ ޙ੢ਸ retrieveೠ ҃਋ ݽ؛ীѱ target attributes੄ ੿ࠁܳ ઁҕ೧઱ח Ѻ • ୹۱ ޙ੢ী ؀ೠ Fine-grained control੉ оמ Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Generate
22. • झಕ࣍ ష௾ٜਸ ୶о۽ ੉ਊೞৈ ޙ੢ਸ ੑ۱ • B-GST : <ATTR> <CONT_START> Contents <OUTPUT_START> • G-GST : <ATTR_START> Attribute Words <CONT_START> Contents <OUTPUT_START> Transforming Delete,Retrieve,Generate Approach for Controlled Text Style Transfer Generate
23. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Experiments • ؘ੉ఠࣇ • Yelp, Amazon ܻ࠭੄ positive, negative ؘ੉ఠࣇ • ੉޷૑ ஭࣌੄ romantic, humorous ؘ੉ఠࣇ • Political ؘ੉ఠࣇ - ಕ੉झ࠘ ҕച׼ ૑૑੗, ޹઱׼ ૑૑੗੄ ؆Ӗ • Gender ؘ੉ఠࣇ - Yelp ਺ध੼ ܻ࠭੄ ࢿ߹ • ಣоח ௼ۄ਋٘ ࣗयਸ ੉ਊೞৈ ߬੉झۄੋҗ ઁউ ݽ؛੄ ਋ਤܳ ࠺Ү • ղਊ ߸҃ਯ • ࣘࢿ ߸҃ਯ • ޙ੢੄ ੗োझ۞਑
24. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Experimental Results • ӝઓ੄ Delete&Retrieve ݽ؛җ B-GST ݽ؛ ઺ যו ଃ੉ ؊ ੸੺ೠ૑ܳ ௼ۄ਋٘ ࣗयਵ۽ ಣо • D&R: Delete & Retrieve, BT:Back-Translation • ੿۝੸ੋ ಣо Ѿҗب ֤ޙী ঱әغয੓૑݅ ৈӝࢲח ࢤۚ
25. Delete,Retrieve,Generate: A Simple Approach to Sentiment and Style Transfer Experimental Results
26. хࢎ೤פ׮✌ ୶о ૕ޙ ژח ҾӘೠ ੼੉ ੓׮ݶ ঱ઁٚ ইې োۅ୊۽ োۅ ઱ࣁਃ! ੿׮਍ (ݠन۞׬ ূ૑פয, ೝಯ) Email.dawoon@scatterlab.co.kr