SNLP2021: Style is NOT a single variable: Case Studies for Cross-Stylistic Language Understanding

Transcript

1. Style is NOT a single variable: Case Studies for Cross-Stylistic Language Understanding Dongyeop Kang and Eduard Hovy ACL-IJCNLP2021 ಡΈख: ੺ؒ ྯಸʢ౦๺େʣ @ୈ13ճ࠷ઌ୺NLPษڧձ
2. ຊݚڀͷ࢟੎ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 2 • Systemic Functional Linguistics (SFL) Theory [Halliday 2003] ʹجͮ͘ 選択体系機能⾔語学: ݴޠ͕ࣾձͷதͰͲͷΑ͏ʹ࢖ΘΕΔ͔ ίϛϡχέʔγϣϯʹ͓͚Δ໾ׂɾཱͪҐஔ Textual Interpersonal Ideational Dongyeop Kang; Linguistically Informed Language Generation: A Multi-faceted Approach (CMU-LTI-20-003)
3. • Systemic Functional Linguistics (SFL) Theory [Halliday 2003] ʹجͮ͘ ຊݚڀͷ࢟੎ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 3 選択体系機能⾔語学: ݴޠ͕ࣾձͷதͰͲͷΑ͏ʹ࢖ΘΕΔ͔ Dongyeop Kang; Linguistically Informed Language Generation: A Multi-faceted Approach (CMU-LTI-20-003) ίϛϡχέʔγϣϯʹ͓͚Δ໾ׂɾཱͪҐஔ ֤ػೳΛߏ੒͢Δཁૉྫ
4. ελΠϧͷཁૉΛͲ͏੔ཧ͢Δ͔ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 4 • ैདྷͷࣾձݴޠֶͷҰྫ • Hovy (1987): pragmatics constraints e.g., strong or neutral stance about the topic of a text or a different level of formality • Biber (1991): participant roles and characteristics and the relationships among participants • ຊݚڀͷ৽͍͠෼ྨ • Social participation [Biber 1991] ͱ Content coupledness ʹجͮ͘4ͭͷΧςΰϦʹ෼ྨ スタイルが会話の中で 話し⼿と聞き⼿の どちらに関係しているか スタイルが原⽂の内容に どれだけ影響を受けているか
5. Cross-style Language Understanding 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 5 • Cross-style Language Understanding • ςΩετͷޠኮతಛ௃ͱײ৘نൣͱͷ૬ؔ [Warriner et al. 2013] • formality, frustration, and politeness ؒͷ૬ؔؔ܎ [Chhaya et al. 2018] • demographic information (e.g., gender, age) ͱ emotion ؒͷ૬ؔؔ܎ [Preotiuc-Pietro and Ungar 2018] • ϝλϑΝʔͱײ৘ͷ૬ޓ࡞༻ [Dankers et al. 2019; Mohammad et al. 2016]
6. ֓ཁ • ͜Ε͔Β͸包括的・横断的ͳ Cross-style language understanding ΁ • ʮελΠϧ͸Φʔέετϥʯ • ґଘؔ܎ɺڞଘՄೳੑΛ޿͘ௐࠪ • ߩݙ • ^ ΛՄೳʹ͢ΔϕϯνϚʔΫσʔλ X SLUE Λ੔උ • Total 1.6M sentences on 15 stylesʢطଘσʔλ 14 + ৽ن 1ʣ • Cross-style diagnostic set (500 sentences with 15 styles) • XSLUE Λ༻͍ͨ cross-style ͳ case-study Λ঺հ • ϥϕϧ෼ྨɺ૬ؔ෼ੳɺ৚݅෇͖ੜ੒ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 6
7. • ᶃ Individual style dataset: طଘσʔλΛେن໛ʹू໿ɾܗࣜΛ౷Ұ • ᶄ Cross-style diagnostic set: ελΠϧԣஅతʹݕূ͍ͨ͠ɺશελΠϧϥϕϧΛਓखͰ෇༩ X SLUE: A Benchmark for Cross-Style Language Understanding and Evaluation 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 7 SarcasmDetection [Gosh and Veale 2016] PASTEL [Kang et al. 2019] Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male Text: I just imagined you dancing like this Style: Sarcasm:NonSarcastic Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, …
8. X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 8 包括的 • ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ੒ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k ௒ NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞੒ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male
9. X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 9 包括的 • ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ੒ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k ௒ NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞੒ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male
10. X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 10 包括的 • ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ੒ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k ௒ NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞੒ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male
11. X SLUE ― Individual style dataset 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 11 包括的 • ελΠϧશ15छɺ4ΧςΰϦΛຬͨ͢ • طଘ 21 + ৽ن 1 σʔληοτ͔Βߏ੒ ⼤規模 • ߹ܭ 1.6M sentences • ֤ελΠϧ 1k ௒ NLGͷ܇࿅σʔλͱͯ͠ར༻Մೳ ✔ ✔ widely-used in *ACL and publicly available ৽ͨʹ࡞੒ Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male
12. X SLUE ― Cross-style diagnostic set 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 12 診断セット作成⽅法： 1. ೋ௨Γͷํ๏Ͱ਍அ༻αϯϓϧ 500 sentences Λ֫ಘ • Individual style dataset ʹ༻͍Δ֤σʔλͷςετηοτ͔Βແ࡞ҝʹ 250 sentences • ࣄલֶशࡁ෼ྨثʹΑΔελΠϧ༧ଌͰείΞʹେ͖ͳ͹Β͖͕ͭੜͨ͡πΠʔτ͔Βແ࡞ҝʹ 250 tweets 2. ֤αϯϓϧʹର͠ɺશελΠϧ (e.g., Formality, Gender) ʹ͍ͭͯϥϕϧ (e.g., Informal, Male) ΛਓखͰ෇༩ • 1αϯϓϧ͋ͨΓ5ਓ͕Ξϊςʔγϣϯɺ֤ελΠϧͷ࠷ऴతͳϥϕϧ͸5ਓͷ majority vote Ͱܾఆ Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, … ✔ 複数スタイル間の依存関係や相互作⽤が検証可能に
13. Three Case studies on XSLUE 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 13 • Case #1: スタイルラベル分類タスク • Style is NOT a single variable! • ݸผʹϞσϧԽ͢ΔΑΓ΋·ͱΊͯϞσϧԽ͢Δ΄͏্͕ख͍͘͘͸ͣ • Case #2: スタイル属性間の依存関係（相関）分析 • ελΠϧଐੑؒͷ݁ͼ͖ͭͷେখΛௐࠪ • Case #3: スタイル属性の組み合わせと⽣成 • ελΠϧଐੑʹ͸ڞଘՄೳͳ૊Έ߹Θͤͱɺͦ͏Ͱͳ͍૊Έ߹Θ͕ͤ͋Δ
14. Case #1: ελΠϧϥϕϧ෼ྨλεΫ ― ઃఆ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 14 モデル • Single models: • Majority classifier • biLSTM [Hochreiter and Schmidhuber 1997] • BERT [Devlin et al. 2019] • RoBERTa [Liu et al. 2019] • T5 [Raffel et al., 2019] タスクと評価尺度 • (1) individual-set, (2) cross-set ্ͰͷελΠϧϥϕϧ෼ྨ • F1 score (multi-label ͷ৔߹͸ϚΫϩฏۉ஋Λ༻͍Δ) • Cross model: (based on pre-trained T5) ⼊⼒: Style + Text → 出⼒: Label スタイル毎にモデル化 vs 複数スタイルをまとめてモデル化
15. スタイル毎にモデル化 Case #1: ελΠϧϥϕϧ෼ྨλεΫ ― ݁Ռ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 15 Text: The natural lightning made the apartment look quite nice for the upcoming tour. Style: Gender:Male Text: :) They always had pizza around, too! Style: Formality:Informal, Politeness:Polite, Humor:NonHumorous, Sarcasm:NonSarcasm, …, Gender:NonBinary, Age:18-24, Country:USA, PoliticalView:RightWing, … Cross ͕ single Λେ෯ʹվળ Cross Ͱੑೳ௿Լ Metaphor ͷ৔߹ target metaphor verb ͕ Text ͷจ಄ʹདྷΔͨΊ ଞͱएׯλεΫઃఆ͕ҟͳΔ: Style + Text (=Verb:Text) → Label → ྆ऀΛҰॹʹֶश͢Δͱੑೳ͕௿Լ͢ΔՄೳੑ 全体としては Cross で性能が改善する傾向 ෳ਺ελΠϧΛಉ࣌ʹϞσϧԽ͢Δ͜ͱͷ༗ޮੑΛࣔࠦ ✔ T5 < Cross (T5-based)
16. Case #2: ελΠϧؒͷґଘؔ܎෼ੳ ― ઃఆ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 16 χϡʔεهࣄ౳ʹൺ΂ͯελΠϧ͕ଟ༷ 分析⽅法 1. ৽ͨʹऩूͨ͠πΠʔτ1Mʹର͠ɺ ελΠϧ෼ྨث (cross) Λ༻͍ͯશ53छͷελΠϧϥϕϧ (=ଐੑ) (e.g., Informal, Male) ͷ֬཰Λ༧ଌ 2. ෼ྨث͕ࢉग़ͨ͠༧ଌ஋ΑΓɺϐΞιϯͷ૬ؔ܎਺ͱϢʔΫϦουڑ཭Λ༻͍ͯଐੑؒͷ૬ؔߦྻΛಘΔ ਖ਼ͷ૬ؔ ෛͷ૬ؔ スタイル属性間の結びつきの⼤⼩を調査
17. 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 17 Case #2: ελΠϧؒͷґଘؔ܎෼ੳ ― ݁Ռ ✔得られた属性間の依存関係は⼈間の主観と⼀致 (Target Style, Correlated Style) ϖΞʹ͍ͭͯɺ͜ΕΒͷଐੑ͕૬ޓʹґଘ ͍ͯ͠Δ͜ͱͷଥ౰ੑΛධՁऀ2໊͕5఺ຬ఺ͰධՁɻ2ਓͷฏۉ఺ΛHɻ ✔Ward法よる可視化で解釈可能なクラスタを確認 ΞδΞܥຽ଒ (SouthAsian, EastAsian), த೥૚ (35-44, 45-54, 55-74)ɺ ੵۃੑ (happiness, dominance, positive, polite)ɺѱ͍ײ৘ (anger, disgust, sadness, fear) ͳͲɻ スタイル属性間の結びつきの⼤⼩を調査
18. Case #3: ελΠϧͷ૊Έ߹Θͤͱੜ੒ ― ઃఆɾ݁Ռ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 18 条件付き⽂⽣成モデル • PPLM [Dathathri et al. 2019] ! "|$! , … , $" ∝ !(") * !($! |") ⋯ !($" |") タスクと評価尺度 • ৚݅෇͖จੜ੒ ೖྗ: Prompt “Every natural text is”, Target style {Polite, Impolite}º{Positive, Negative} → ग़ྗ: Stylized text • ਓखධՁ Stylistic appropriateness score (1-5), ධՁऀ3໊ͷฏۉ 結果 スタイル属性には共存可能な組み合わせと、そうでない組み合わせがある ݴޠϞσϧ (GPT-2) ελΠϧ෼ྨث (cross) ✔⽣成例 (!=1 ͷ৔߹) ਓखධՁ݁Ռ ૬ؔείΞ (Case #2) ✔属性の組み合わせによりスコアが低下、相関とも関連か PosºImpolite, NegºPolite ͕௿͍ɻ͜ͷ৔߹ʹ͍ͭͯ͸ଐੑؒͷ૬ؔͱ܏޲͕Ұகɻ
19. ·ͱΊ 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 19 • Cross-style language understanding ݚڀͷͨΊͷϕϯνϚʔΫσʔλ XSLUE Λ࡞੒ • Social participation ͱ Content coupledness ʹج͖ͮελΠϧଐੑΛ4ΧςΰϦʹ෼ྨ • จϨϕϧͷελΠϧ෼ྨσʔληοτΛू໿͠ܗࣜΛ౷Ұ 15छͷελΠϧΛؚΉ 1M ௒ͷσʔληοτ: Individual style dataset • ಉҰςΩετʹશ15छͷελΠϧϥϕϧΛਓखͰ෇༩ͨ͠ධՁηοτ: Cross-style diagnostic set • XSLUE Λ༻͍ͨ cross-style ͳ case-study Λ঺հ • ϥϕϧ෼ྨɿෳ਺ελΠϧΛಉ࣌ʹϞσϧԽ͢Δ͜ͱͷ༗ޮੑΛࣔࠦ • ૬ؔ෼ੳɿελΠϧଐੑؒͷ݁ͼ͖ͭͷେখΛௐࠪɺਓؒͷओ؍ͱҰ؏͢Δ͜ͱΛ֬ೝ • ৚݅෇͖ੜ੒ɿελΠϧଐੑʹ͸૊Έ߹ΘͤՄೳͳ΋ͷͱɺͦ͏Ͱͳ͍΋ͷ͕͋Δ 感想 • ར༻Մೳͳσʔλ͕͜͜5೥͘Β͍Ͱ૿͑ͨҹ৅ɺྑ͍λΠϛϯάͰͷ੔ཧ • ஶऀʮελΠϧͷछྨ΋ͬͱ૿΍͍ͨ͠ɺΑΓแׅతͳϕϯνϚʔΫʹ͍ͨ͠ʯ • Ξϊςʔγϣϯ࡞ۀͷઃఆͷઆ໌͕ͱͯ΋ஸೡ (ΞϊςʔγϣϯλεΫ࣮ࢪը໘ɺใुͷܾఆํ๏)
20. Appendix 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 20
21. XSLUE ―Cross-style diagnostic set 2021-09-16 ୈ13ճ࠷ઌ୺NLPษڧձ 21 ਍அηοτ࡞੒ํ๏ɿ 1. ೋ௨Γͷํ๏Ͱ਍அ༻αϯϓϧ 500 sentences Λ֫ಘ • Individual style dataset ʹ༻͍Δ֤σʔλͷςετηοτ͔Βແ࡞ҝʹ 250 sentences • ࣄલֶशࡁ෼ྨثʹΑΔελΠϧ༧ଌͰείΞʹେ͖ͳ͹Β͖͕ͭੜͨ͡πΠʔτ͔Βແ࡞ҝʹ 250 tweets 2. ֤αϯϓϧʹର͠ɺશελΠϧ (e.g., Formality, Gender) ʹ͍ͭͯϥϕϧ (e.g., Informal, Male) ΛਓखͰ෇༩ • 1αϯϓϧ͋ͨΓ5ਓ͕Ξϊςʔγϣϯɺ֤ελΠϧͷ࠷ऴతͳϥϕϧ͸5ਓͷ majority vote Ͱܾఆ Style IAA ೉͍͠ ओ؍తͳελΠϧ