【輪講資料】From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers

研究室内の輪講用に作った資料です

Transcript

1. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers 5݄2೔ ෢ాɾ࡫໺ݚڀࣨɹM2 ໼໺ઍߛ Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavas 2020 EMNLP
2. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers ֓ཁ • ଟݴޠϞσϧͷݴޠؒసҠೳྗͷݶքʹ͍ͭͯௐࠪ 1. ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζ͸ݴޠؒసҠೳྗʹͲͷΑ͏ͳӨڹΛ༩͑Δ͔ʁ 2. ԼҐλεΫͱݴޠؒసҠೳྗʹؔ܎͸͋Δ͔ʁ 3. ݴޠؒసҠೳྗͷ༧ଌ͸Մೳ͔ʁ 4. few-shotʹίετΛࢧ෷͏΂͖͔ʁ • ιʔεݴޠ͔Βԕ͍ݴޠ΍௿ϦιʔεͳݴޠͰ͸zero-shotਪ࿦͸೉͍͠ • ௿ϨϕϧλεΫͷసૹੑೳʹ͸ߏ଄తͳݴޠྨࣅੑ͕େ͖͘Өڹ • ߴϨϕϧͷLUλεΫͷసૹੑೳʹ͸ࣄલֶशͷίʔύεαΠζ͕Өڹ • zero-shot͕೉͍͠৔߹΄Ͳfew-shotͰੑೳ͕վળ • few-shotͷඅ༻ରޮՌ͸ߴ͍ 2
3. બΜͩཧ༝ • ݴޠؒͷసҠೳྗʹ͍͍ͭͯΖ͍ΖͳλεΫͰධՁ͍ͯ͠Δ • ߏจతͳಛ௃΍Իӆతͳಛ௃ͳͲɺݴޠͷৄࡉͳಛ௃Λར༻ͨ͠ݴޠؒྨࣅ౓ ͰసҠೳྗͷ෼ੳΛߦͳ͍ͬͯͯ໘ന͍ 3
4. Cross-lingual Transfer ✦ ڭࢣ͋Γֶश (Supervised Learning) • े෼ͳαΠζͷσʔληοτΛڭࢣ৴߸ͱֶͯ͠श
 ✖︎ σʔληοτ͕५୔ͳݴޠ͸ҰѲΓ • λεΫ͝ͱʹҟͳΔΞϊςʔτ͕ඞཁ… ➡ߴϦιʔεݴޠͷϥϕϧ෇͖σʔλΛར༻ͯ͠௿ϦιʔεݴޠͰ ͷਪ࿦Λߦ͍͍ͨʂ ✦ ݴޠؒసҠֶश (Cross-lingual Transfer Learning) • ݴޠԣஅతʹλεΫΛղ͘ྗΛ֫ಘ͢Δ 4
5. Cross-lingual Transfer ✦ Cross-Lingual Word Embeddings (CLWE) • ݴޠԣஅతͳ୯ޠຒΊࠐΈ ✦ Massively Multilingual Transformer networks 
 (MMTs) • ଟݴޠίʔύεͰࣄલֶश͞ΕͨݴޠϞσϧ • mBERTɺXLM-R… • ݴޠؒసҠೳྗΛ࣋ͭ • FineTuning࣌ʹ༩͍͑ͯͳ͍ݴޠͰ΋ੑೳվળ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ 5 ▪ ӳޠ ▪ ΠλϦΞޠ CLWE࿦จΑΓҾ༻ ݴޠԣஅతͳ୯ޠຒΊࠐΈۭؒ
6. Massively Multilingual Transformers ✦ multilingual BERT (mBERT) • MLMͱNSPͰࣄલֶश • Mask Language ModelingɿϥϯμϜʹϚεΫͨ͠จதͷτʔΫϯΛ༧ଌ • Next Sentence Predictionɿ̎ͭͷจ͕ྡΓ߹͍ͬͯΔ͔༧ଌ • 104ݴޠͷWikipediaΛֶशʹར༻ ✦ XLM RoBERTa (XLM-R) • MLMͷΈͰࣄલֶश • 100ݴޠͷCommonCrawlΛֶशʹར༻ • ΢ΣϒςΩετ • mBERTΑΓ΋େ͖͍ 6 XLM-R࿦จΑΓҾ༻
7. Massively Multilingual Transformers ✓ ݴޠؒసҠೳྗΛ࣋ͭ ✦ Zero-shot • FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ࿦ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ (΋)ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ 7 ଟݴޠࣄલֶशʹΑͬͯ
 ֤ݴޠͷදݱۭ͕ؒྨࣅ͠ɺ
 ݴޠؒసૹ͕ՄೳͳͷͰ͸ʁ[1] <>
   &NFSHJOH
   $SPTTMJOHVBM
   4USVDUVSF
   JO
   1SFUSBJOFE
   -BOHVBHF
   .PEFMT
8. MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ ✓ ݴޠؒసҠੑೳʹେ͖ͳӨڹΛ༩͑Δ৚݅ͱཁҼΛ෼ੳʢQ1~4ʣ 1. ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζ͸ݴޠؒసҠೳྗʹͲͷΑ͏ͳӨ ڹΛ༩͑Δ͔ʁ 2. ԼҐλεΫͱݴޠؒసҠೳྗʹؔ܎͸͋Δ͔ʁ 3. ݴޠؒసҠೳྗͷ༧ଌ͸Մೳ͔ʁ 4. few-shotʹίετΛࢧ෷͏΂͖͔ʁ 8 Q Q Q Q
9. Q1:ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζ͸ݴޠؒసҠೳྗʹͲͷΑ͏ͳ ɹɹӨڹΛ༩͑Δ͔ʁ • ଟݴޠ୯ޠຒΊࠐΈ (CLWE)͸ྨࣅͨ͠ݴޠؒ΍े෼ͳίʔύε͕ଘࡏ ͢ΔݴޠͰͷΈੑೳΛൃش • CLWEͰ͸֤ݴޠͷ୯ޠຒΊࠐΈۭ͕ؒࣅ͍ͯΔ͜ͱ͕લఏͱͯ͠ඞཁ • ྫ) ӳޠͱࣅ͍ͯΔݴޠ • υΠπޠɺεΧϯδφϏΞޠɺϑϥϯεޠɺεϖΠϯޠ • ಉ͡ޠ଒ (ΠϯυɾϤʔϩούޠ଒)ʹଐ͢Δ ➡MMTsͷݴޠؒసҠೳྗ͸Ͳ͏͔ʁ ✓ ෳ਺ݴޠؒͰͷݴޠؒసҠੑೳΛௐࠪ 9 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ
10. Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ܎͸͋Δ͔ʁ • POS-tagging (඼ࢺλά෇͚) • Dependency Parsing (ґଘߏ଄ղੳ) • Named Entity Recognition (ݻ༗දݱநग़)
 • Natural Language Inference (ࣗવݴޠਪ࿦) • Question Answering (࣭໰Ԡ౴) 10 ߴϨϕϧ (ݴޠཧղ)λεΫ ௿ϨϕϧλεΫ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ ✓ ෳ਺ͷλεΫͰθϩγϣοτݴޠؒసҠೳྗΛଌΔ
11. Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ܎͸͋Δ͔ʁ ✦ POS-tagging (඼ࢺλά෇͚) • ςΩετதͷ୯ޠʹର͠඼ࢺΛϥϕϦϯά͢Δ ✦ Dependency Parsing (ґଘߏ଄ղੳ) • จதͷґଘؔ܎ʹج͍ͮͯߏจղੳΛߦ͏ ✦ Named Entity Recognition (ݻ༗දݱநग़) • ςΩετ͔Βݻ༗දݱ (ਓ໊ɺ૊৫໊ɺ஍໊ɺ࣌ؒ……)Λநग़͢Δ ✦ Natural Language Inference (ࣗવݴޠਪ࿦) • ̎ͭͷςΩετͷؒʹ੒Γཱͭਪ࿦తؔ܎ (ؚҙ/தཱ/ໃ६)Λ༧ଌ͢Δ ✦ Question Answering (࣭໰Ԡ౴) • จॻ͔Β࣭໰ͷճ౴ͱͳΔςΩετΛ୳ࡧ͢Δ 11 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ
12. Q3:ݴޠؒసҠೳྗͷ༧ଌ͸Մೳ͔ʁ • ݴޠͷྨࣅੑɺࣄલֶश࣌ͷίʔύεαΠζͳͲ͔ΒಛఆλεΫɺ
 ݴޠͰͷੑೳΛ༧ଌͰ͖Δ͔ʁ
 ※ ݴޠͷྨࣅੑ • lang2vecͰݴޠͷಛ௃ΛϕΫτϧԽ͠ɺͦͷίαΠϯྨࣅ౓Λར༻ • ݴޠͷಛ௃͕ऩू͞Εͨσʔλϕʔε͔Βಛ௃ྔΛநग़ 12 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ
13. Q4: few-shotʹίετΛࢧ෷͏΂͖͔ʁ • few-shotͷඅ༻ରޮՌ͸Ͳͷఔ౓ͳͷ͔ʁ ✓ zero-shotͱfew-shotͷੑೳΛൺֱ ✓ few-shotͷΞϊςʔγϣϯʹ͔͔Δඅ༻Λྫࣔ 13 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ ✦ Zero-shot • FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ࿦ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ (΋)ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ + ੑೳ + ίετ
14. ຊ࿦จͰѻΘΕΔओͳݴޠʹ͍ͭͯ ޠ଒ɿಉ͡ىݯΛ࣋ͭݴޠͷάϧʔϓ • ڞ௨ͷޠኮɺจ๏ɺԻӆମܥͳͲΛ࣋ͭ͜ͱ͕͋Γɺࣅͨಛ௃Λڞ༗͢Δݴޠ܈ • ΠϯυɾϤʔϩούޠ଒ EN (ӳޠ), RU (ϩγΞޠ), HI (ώϯσΟʔޠ), IT (ΠλϦΞޠ), SV (ε΢Σʔσϯޠ), ES (εϖΠϯޠ), 
 EL (ΪϦγϟޠ),DE (υΠπޠ), FR (ϑϥϯεޠ), BG (ϒϧΨϦΞޠ), UR (΢ϧυΡʔޠ), … • ΞϑϩɾΞδΞޠ଒ AR (ΞϥϏΞޠ), HE (ϔϒϥΠޠ), … • ଞͷޠ଒ TR (τϧίޠ): Φάζޠ܈, ZH (தࠃޠ): γφɾνϕοτޠ଒, EU (όεΫޠ): όεΫޠ଒, FI (ϑΟϯϥϯ υޠ): ΢ϥϧޠ଒, JA (೔ຊޠ): ೔ຊޠ଒, KO (ؖࠃޠ): ؖࠃޠ଒, VI (ϕτφϜޠ): ΦʔετϩΞδΞޠ ଒, TH (λΠޠ): λΠɾΧμΠޠ଒, SW (εϫώϦޠ): χδΣʔϧɾίϯΰޠ଒, … 14
15. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ෳ਺ͷλεΫΛෳ਺ͷݴޠʹର͠θϩγϣοτͰਪ࿦ • Q1ͱQ2ʹճ౴͢Δ • Q1: ੑೳͱݴޠྨࣅੑɺࣄલֶशσʔλαΠζͷؔ܎ʹ͍ͭͯ • Q2: ੑೳͱλεΫͷؔ܎ʹ͍ͭͯ • λεΫ: ݴޠ (ISO 639-1ܗࣜ) • POS(඼ࢺλά෇͚)ɿZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • DEP (ґଘߏ଄ղੳ): ZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • NER (ݻ༗දݱநग़): ZH, TR, RU, AR, HI, EU, FI, HE, IT, JA, KO, SV • NLI (ࣗવݴޠਪ࿦): ZH, TR, RU, AR, HI,VI, TH, ES, EL, DE, ER, BG, SW, UR • QA (࣭໰Ԡ౴): ZH, TR, RU, AR, HI,VI, TH, ES, EL, DE 15
16. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ෳ਺ͷλεΫΛෳ਺ͷݴޠʹର͠θϩγϣοτͰਪ࿦ • λεΫ: σʔληοτ • POS(඼ࢺλά෇͚)ɿUniversal Dependency • DEP (ґଘߏ଄ղੳ): Universal Dependency • NER (ݻ༗දݱநग़): NER WikiANN dataset [2] • NLI (ࣗવݴޠਪ࿦): XNLI [3] • QA (࣭໰Ԡ౴): XQuAD [4] 16 <>
    .BTTJWFMZ
    .VMUJMJOHVBM
    5SBOTGFS
    GPS
    /&3
    <>
    9/-*
    &WBMVBUJOH
    $SPTTMJOHVBM
    4FOUFODF
    3FQSFTFOUBUJPOT
    <>
    0O
    UIF
    $SPTTMJOHVBM
    5SBOTGFSBCJMJUZ
    PG
    .POPMJOHVBM
    3FQSFTFOUBUJPOT
17. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ෳ਺ͷλεΫΛෳ਺ͷݴޠʹର͠θϩγϣοτͰਪ࿦ • λεΫ: ධՁࢦඪ • POS(඼ࢺλά෇͚)ɿAccuracy • DEP (ґଘߏ଄ղੳ): Unlabeled Attachment Scores (܎Γઌ͕ਖ਼͚͠Ε͹ྑ͍) • NER (ݻ༗දݱநग़): Accuracy • NLI (ࣗવݴޠਪ࿦): Accuracy • QA (࣭໰Ԡ౴): Exact Match 17
18. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ෳ਺ͷλεΫΛෳ਺ͷݴޠʹର͠θϩγϣοτͰਪ࿦ • શͯͷλεΫͰӳޠͷֶशσʔλͷΈར༻ • ௿ϨϕϧλεΫ (POS, DEP, NER)Ͱ͸λεΫݻ༗ʹ௥Ճͨ͠ϨΠϠʔͷΈֶश͠ɺ
 Ϟσϧͷύϥϝʔλ͸ౚ݁ 18
19. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ӳޠͰͷ݁Ռͱͷࠩ෼Λදࣔ • ࣄલֶशϞσϧ • B: mBERT base-cased • X: XLM-R base • 19
20. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • ௿ϨϕϧλεΫͷθϩγϣοτసҠೳྗ͸ݴޠͷۙ͞ʹӨڹΛड͚Δ 20 ⬛︎ ΠϯυɾϤʔϩούޠ଒
21. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • ߴϨϕϧλεΫͷθϩγϣοτసҠೳྗ͸ݴޠͷۙ͞Ҏ֎ͷӨڹ΋ड͚Δ • XNLIʹ͓͍ͯZH͸HIͱURΑΓ΋ߴ͍ੑೳ • (XLM-RͰͷ)XQuADʹ͓͍ͯTH͸RUͱESͱಉఔ౓ͷੑೳ 21 ⬛︎ ΠϯυɾϤʔϩούޠ଒
22. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ ✓ ݴޠྨࣅੑͷਪఆʹ͸lang2vec͔ΒҎԼͷಛ௃ྔΛར༻ • Syntax: ߏจ৘ใΛදݱ ओޠͱಈࢺͷલޙؔ܎ͳͲ • Phonology:Իӆతಛ௃Λදݱ ࢠԻͱ฼Իͷൺ཰ͳͲ • Inventory: Իӆ࿦ʹ͓͚Δࣗવྨ (ಛఆͷൃԻΛߦ͏Ի)ͷ༗ແ • Language families: ޠ଒Λදݱ • Geography: ஍ٿ্Ͱͷڑ཭Λදݱ ➡͜ΕΒͷίαΠϯྨࣅ౓Λ࢖͏͜ͱͰݴޠྨࣅੑΛਪఆ
 ✓ ࣄલֶश࣌ͷίʔύεαΠζΛඪ४Խͯ͠ಛ௃ྔSIZEͱͯ͠ར༻ 22
23. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • θϩγϣοτసҠੑೳͱ֤ಛ௃ྔͷ૬ؔ܎਺ (P: ϐΞιϯ, S: εϐΞϚϯ ) 23
24. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • ߏจ৘ใΛද͢SYN͸௿ϨϕϧλεΫͰ͋ΔDEPɺPOSͱڧ͍૬ؔ • Ұ෦Λআ͘ߴϨϕϧλεΫͱ΋૬͕ؔ͋Δ 24
25. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • PHON͸XNLIΛআ͘શͯͷλεΫͰXLM-Rͱߴ͍૬ؔ 25
26. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • SIZE͸ߴϨϕϧλεΫʹͷΈߴ͍૬ؔΛࣔ͢ • ߴϨϕϧͷݴޠཧղλεΫ͸๛෋ͳදݱͷཧղΛඞཁͱ͠ɺ௿Ϩ ϕϧλεΫΑΓ΋ଟ༷ͳσʔλ͕ඞཁ 26
27. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • Q3:సҠೳྗͷ༧ଌ͸Մೳ͔ʁ ➡ͦΕͧΕͷಛ௃Λઢܗʹ૊Έ߹ΘͤͯಛఆͷλεΫʹ͓͚Δ৽͠ ͍ݴޠͰͷసૹੑೳΛ༧ଌͰ͖Δ͔ʁ • ϝλճؼ෼ੳΛར༻ • ໨తม਺Λઆ໌ม਺ʹΑͬͯͲͷ͙Β͍આ໌Ͱ͖Δͷ͔ʁ 27
28. θϩγϣοτసҠೳྗͷ࣮ݧͱ෼ੳ • ߴϨϕϧλεΫͷੑೳΛઆ໌͢Δ
 ͨΊʹ͸SIZE͕ඞཁ 28
29. Few-shotసҠೳྗͷ࣮ݧͱ෼ੳ • Q4: few-shotʹίετΛ͔͚Δ΂͖͔ʁ ➡few-shotʹΑΔݴޠؒసҠੑೳͷมԽΛௐ΂Δ • ӳޠʹՃ͑ͯkݸͷλʔήοτݴޠͷσʔλΛֶशʹར༻ 29 ✦ Zero-shot • FineTuning࣌ʹ༩͍͑ͯͳ͍Ϋϥε (ݴޠ)Ͱਪ࿦ • ྫ) ӳޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ ✦ Few-shot • FineTuning࣌ʹগྔͷڭࢣσʔλ (λʔήοτݴޠ)Λ (΋)ར༻ • ྫ) ӳޠͱগྔͷΠλϦΞޠͰλεΫAʹରֶͯ͠श
 -> ΠλϦΞޠͰλεΫAʹରͯ͠ਪ࿦ + ੑೳ + ίετ
30. Few-shotసҠೳྗͷ࣮ݧͱ෼ੳ • ௿ϨϕϧλεΫͰେ෯ͳੑೳ޲্ • ಛʹDEP ,POSʹ͓͍ͯগ਺ͷkͰ΋େ෯ͳੑೳ޲্ • ߴϨϕϧλεΫͰ͸ଟগͷ޲্ 30
31. Few-shotసҠೳྗͷ࣮ݧͱ෼ੳ • λεΫ͝ͱɺݴޠ͝ͱɺγϣοτ਺͝ͱͷੑೳΛਤࣔ 31
32. Few-shotసҠೳྗͷ࣮ݧͱ෼ੳ • ιʔεݴޠͱݴޠྨࣅੑ͕௿͍ݴޠͷํ͕few-shotͷԸܙΛड͚΍͍͢ • zero-shotͰͷੑೳ͕ߴ͍ݴޠ͸few-shotͷԸܙΛड͚ʹ͍͘ (SV) • zero-shotͰͷੑೳ͕௿͍ݴޠ͸few-shotͰੑೳ͕େ͖͘޲্ (JA, AR) 32 ◀︎ few-shotࣄྫ਺ͱ
 ੑೳͷਪҠ mBERTΛ༻͍ͨ
 ґଘߏ଄ղੳλεΫ
 Ͱͷ݁Ռ
33. Few-shotసҠೳྗͷ࣮ݧͱ෼ੳ Direct Target Language Few-Shot Fine-Tuning ӳޠͰͷλεΫ΁ͷֶशΛߦΘͣɺfew-shotͰͷΈֶश • ߴϨϕϧλεΫʹ͓͍ͯӳޠେن໛σʔλͰͷֶशΛল͘ͱ௿͍ੑೳ • ιʔεݴޠͰͷλεΫֶशͱλʔήοτݴޠͰͷֶशɺ྆ํ͕ඞཁ • ௿ϨϕϧλεΫʹ͓͍ͯ͸ӳޠͰͷֶशΛলུՄೳ • ΑΓ୯७ͳ௿ϨϕϧλεΫʹରͯ͠͸গ਺ͷσʔλͰ΋ֶशՄೳ 33
34. Q1:ݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζ͸ͲͷΑ͏ͳӨڹΛ༩͑Δ͔ʁ 34 ✓ సҠֶशͷੑೳ͸ݴޠྨࣅੑɺࣄલֶश࣌ͷίʔύεαΠζʹґଘ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ • ଟݴޠ୯ޠຒΊࠐΈ (CLWE)͸ྨࣅͨ͠ݴޠؒ΍े෼ͳίʔύε͕ଘࡏ͢ΔݴޠͰͷΈੑೳΛൃش • CLWEͰ͸֤ݴޠͷ୯ޠຒΊࠐΈۭ͕ؒࣅ͍ͯΔ͜ͱ͕લఏͱͯ͠ඞཁ • ྫ) ӳޠͱࣅ͍ͯΔݴޠ • υΠπޠɺεΧϯδφϏΞޠɺϑϥϯεޠɺεϖΠϯޠ • ಉ͡ޠ଒ (ΠϯυɾϤʔϩούޠ଒)ʹଐ͢Δ ➡ MMTsͷݴޠؒసૹೳྗ͸Ͳ͏͔ʁ ✓ ෳ਺ݴޠؒͰͷݴޠؒసૹੑೳΛௐࠪ
35. Q2:ԼҐλεΫͱݴޠؒసҠೳྗʹؔ܎͸͋Δ͔ʁ • POS-tagging (඼ࢺλά෇͚) • Dependency Parsing (ґଘߏ଄ղੳ) • Named Entity Recognition (ݻ༗දݱநग़)
 • Natural Language Inference (ࣗવݴޠਪ࿦) • Question Answering (࣭໰Ԡ౴) 35 ߴϨϕϧ (ݴޠཧղ)λεΫ ௿ϨϕϧλεΫ ✓ zero-shot, few-shotͷ྆ઃఆͰݴޠؒసҠੑೳͷ܏޲͕λεΫͷϨϕϧʹґଘ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ
36. Q3:ݴޠؒసҠೳྗͷ༧ଌ͸Մೳ͔ʁ ✓ ಛఆͷλεΫͷసҠ݁Ռ͕ଘࡏ͢Ε͹ɺ؆୯ͳճؼΛ༻͍Δ͜ͱͰ৽͍͠ ݴޠͰͷసҠೳྗ͕͓͓·͔ʹ༧ଌՄೳ ✓ λεΫ͝ͱʹґଘ͢Δಛ௃ྔʹ܏޲͕ଘࡏ • ௿ϨϕϧλεΫɿݴޠྨࣅੑʹґଘ • ߴϨϕϧλεΫɿݴޠྨࣅੑͱࣄલֶश࣌ͷίʔύεαΠζʹґଘ 36 • ݴޠͷྨࣅੑɺࣄલֶश࣌ͷίʔύεαΠζͳͲ͔ΒಛఆλεΫɺ
 ݴޠͰͷੑೳΛ༧ଌͰ͖Δ͔ʁ
 ※ ݴޠͷྨࣅੑ • lang2vecͰݴޠͷಛ௃ΛϕΫτϧԽ͠ɺͦͷίαΠϯྨࣅ౓Λར༻ • ݴޠͷಛ௃͕ऩू͞Εͨσʔλϕʔε͔Βಛ௃ྔΛநग़ MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ
37. Q4:few-shotʹίετΛࢧ෷͏΂͖͔ʁ • few-shotͷඅ༻ରޮՌ͸Ͳͷఔ౓ͳͷ͔ʁ 37 MMTsͷθϩγϣοτݴޠؒసҠೳྗͷ෼ੳ ✓ ಛʹzero-shot͕͏·͍͔͘ͳ͍λεΫ/ݴޠʹରͯ͠few-shot͸͔ͳΓޮՌత ✓ ௿Ϧιʔεݴޠʹରͯ͠΋͋Δఔ౓ͷίετͰΞϊςʔγϣϯՄೳ ◀︎ ΞϊςʔγϣϯίετͱੑೳͷมԽͷฏۉ
38. Conclusion • ιʔεݴޠ͔Βԕ͍ݴޠ΍௿ϦιʔεͳݴޠͰ͸zero-shotਪ࿦͸೉͍͠ • ௿ϨϕϧλεΫͷసҠੑೳʹ͸ߏ଄తͳݴޠྨࣅੑ͕େ͖͘Өڹ • ߴϨϕϧͷLUλεΫͷసҠੑೳʹ͸ࣄલֶशͷίʔύεαΠζ͕Өڹ • zero-shot͕೉͍͠৔߹΄Ͳfew-shotͰੑೳ͕վળ • few-shotͷඅ༻ରޮՌ͸ߴ͍ 38
39. ΑΓେ͖ͳଟݴޠϞσϧͰͷθϩγϣοτݴޠؒసҠੑೳʹ͍ͭͯ ✓ ͜ͷ࿦จ͸2020೥ʹൃද͞Ε͓ͯΓɺݱ୅Ͱ͸ঢ়گ͕ҟͳΔՄೳੑ͕͋Δ • ࿦จ಺Ͱ࣮ݧʹ༻͍ΒΕ͍ͯΔϞσϧ • mBERT (base): 110M • XLM-R (base): 270M • ݱࡏൃද͞Ε͍ͯΔେن໛ଟݴޠϞσϧ • mT5-XXL[5]: 13B • Bloomz[6]: 176B 39 <>
    N5
    "
    .BTTJWFMZ
    .VMUJMJOHVBM
    1SFUSBJOFE
    5FYUUP5FYU
    5SBOTGPSNFS
    <>
    $SPTTMJOHVBM
    (FOFSBMJ[BUJPO
    UISPVHI
    .VMUJUBTL
    'JOFUVOJOH
40. ΑΓେ͖ͳଟݴޠϞσϧͰͷθϩγϣοτݴޠؒసҠੑೳʹ͍ͭͯ • XNLIλεΫʹ͓͍ͯmT5ͷzero-shotͰͷੑೳ͸mBERTɺXLM-RͰͷ
 few-shotͰͷੑೳΑΓྑ͍ • ಉ༷ͷ݁ՌΛXQuADʹ͓͍ͯ΋֬ೝ 40 L L N#&35   9-.3   MBSHF mT5࿦จΑΓҾ༻ ˛ ຊ࿦จͰࣔ͞Ε͍ͯͨXNLIͷੑೳ k = few-shot਺
41. ΑΓେ͖ͳଟݴޠϞσϧͰͷθϩγϣοτݴޠؒసҠੑೳʹ͍ͭͯ • ຊ࿦จதͰ͸೉͍͠ͱ͞Ε͍ͯͨԕ͍ݴޠؒͰͷసҠੑೳ͕େ෯ʹ޲্ 41 ⬛︎ ΠϯυɾϤʔϩούޠ଒Ҏ֎ MBSHF mT5࿦จΑΓҾ༻
42. ݱ୅ʹ͓͚ΔConclusion • ιʔεݴޠ͔Βԕ͍ݴޠ΍௿ϦιʔεͳݴޠͰ͸zero-shotਪ࿦͸೉͍͠ • ͕ɺେ͖ͳϞσϧͰ͸ԕ͍ݴޠʹରͯ͠΋zero-shotਪ࿦͕Մೳ • ௿ϨϕϧλεΫͷసҠੑೳʹ͸ߏ଄తͳݴޠྨࣅੑ͕େ͖͘Өڹ • ߴϨϕϧͷLUλεΫͷసҠੑೳʹ͸ࣄલֶशͷίʔύεαΠζ͕Өڹ • zero-shot͕೉͍͠৔߹΄Ͳfew-shotͰੑೳ͕վળ • zero-shotੑೳ͕޲্͓ͯ͠Γɺfew-shotʹΑΔӨڹ͸վΊͯݕ౼͍ͨ͠ • few-shotͷඅ༻ରޮՌ͸ߴ͍ 42