Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
【輪講資料】Zero-shot Cross-lingual Semantic Parsing
Search
Yano
May 03, 2023
0
120
【輪講資料】Zero-shot Cross-lingual Semantic Parsing
研究室内の輪講で使った資料です。
Yano
May 03, 2023
Tweet
Share
More Decks by Yano
See All by Yano
[WIP] How Do Large Language Models Acquire Factual Knowledge During Pretraining?
yano0
0
8
NLP2025参加報告
yano0
0
530
【輪講資料】ReAct: Synergizing Reasoning and Acting in Language Models / Tree of Thoughts: Deliberate Problem Solving with Large Language Models
yano0
0
170
【輪講資料】SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
yano0
2
320
【輪講資料】From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
yano0
0
84
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
4k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Agile that works and the tools we love
rasmusluckow
329
21k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
890
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Making the Leap to Tech Lead
cromwellryan
134
9.5k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.4k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
It's Worth the Effort
3n
187
28k
Side Projects
sachag
455
43k
Transcript
Zero-shot Cross-lingual Semantic Parsing Tom Sherborne, Mirella Lapata ACL 2022
֓ཁ 2 ✓ θϩγϣοτଟݴޠҙຯղੳϞσϧɺZX-parseͷఏҊ • ݴޠؒજࡏදݱͷΞϥΠϝϯτʹண͠సҠֶशͷޡࠩΛ࠷খԽ • ෳͷଛࣦؔΛಋೖ͢ΔϚϧνλεΫֶशʹΑΓҟͳΔݴޠͷ જࡏදݱ͕ྨࣅ͢ΔΑ͏ʹ •
ରݴޠͷϖΞσʔλΛඞཁͱͤͣɺӳޠϖΞͱରݴޠͷࣗવ จͷΈར༻ ✓ θϩγϣοτҙຯղੳλεΫʹ͓͍ͯෳͷݴޠͰߴ͍ੑೳ
બΜͩཧ༝ • ݴޠԣஅతͳݚڀʹڵຯ͕͋ΔͨΊ • ಛʹݴޠԣஅతͳજࡏۭؒΛ࡞ΔͰ໘നͦ͏ • ࣗͷݚڀΛؚΉ͞·͟·ͳసҠֶशλεΫͰར༻Ͱ͖ͦ͏ 3
ҙຯղੳ (Semantic Parsing) • ࣗવݴޠͷൃΛཧܗࣜ(Logical Form)ʹม • ͞·͟·ͳλεΫͰॏཁͳΠϯϑϥ • ࣭ԠɺରγεςϜɺػցͷࢦࣔ…
4 -JTU fl JHIUTGSPN4BO'SBODJTDPUP1JUUTCVSHI 4&-&$5%*45*/$5 fl JHIU@ fl JHIU@JE'30.ʜ
• ෳͷݴޠ͔ΒͳΔࣗવจΛಉ͡ཧܗࣜʹม[1] ଟݴޠҙຯղੳ (Cross-lingual Semantic Parsing) 5 <>.VMUJMJOHVBM4FNBOUJD1BSTJOH1BSTJOH.VMUJQMF-BOHVBHFTJOUP4FNBOUJD3FQSFTFOUBUJPOT 4&-&$5%*45*/$5 fl
JHIU@ fl JHIU@JE'30.ʜ -JTU fl JHIUTGSPN4BO 'SBODJTDPUP1JUUTCVSHI αϯϑϥϯγείൃϐοπ όʔάߦ͖ͷϑϥΠτΛ ϦετΞοϓ͍ͯͩ͘͠͞ɻ ʹ
ઌߦݚڀ • ӳޠͷࣗવจ-ཧܗࣜͷฒྻσʔλΛ༁͠ɺ֤ରݴޠͷࣗવ จ-ཧܗࣜϖΞσʔλΛར༻ • ػց༁Λ༻͍Δ߹[2] • ػց༁ᘳͰͳ͍ • ಛʹϦιʔεݴޠʹ͓͍ͯ
ߴ࣭ͳػց༁͍͠ • ਓख༁Λ༻͍Δ߹[3] • ߴίετ 6 <>#PPUTUSBQQJOHB$SPTTMJOHVBM4FNBOUJD1BSTFS <>/FVSBM"SDIJUFDUVSFTGPS.VMUJMJOHVBM4FNBOUJD1BSTJOH ཧܗࣜ ӳޠ ରݴޠ ༁
• ରݴޠͷฒྻσʔλΛΘͳ͍ θϩγϣοτͱ͢Δ ➡ ӳޠͷࣗવจ-ཧܗࣜϖΞσʔλͱ ରݴޠͷࣗવจͷΈར༻ ✓ Ϟσϧߏ • ӳޠ͔Βಘͨજࡏදݱ͔Β
ཧܗࣜΛੜ͢Δσίʔμ • ݴޠԣஅతͳʢӳޠͱྨࣅͨ͠ʣ જࡏදݱΛ֫ಘ͢ΔΤϯίʔμ ఏҊख๏ 7 αϯϑϥϯγείൃϐοπ όʔάߦ͖ͷϑϥΠτΛ ϦετΞοϓ͍ͯͩ͘͠͞ɻ 4&-&$54&-&$54&-&$5 8)&3& '30.4&-&$5 Τϯίʔμ σίʔμ ❌ 4&-&$5%*45*/$5 fl JHIU@ fl JHIU@JE'30.ʜ -JTU fl JHIUTGSPN4BO 'SBODJTDPUP1JUUTCVSHI ⭕ Τϯίʔμ σίʔμ ˛ΞϥΠϝϯτΛߦΘͳ͍߹ɺݴޠ͝ͱʹ ҟͳΔજࡏදݱͱͳΓग़ྗ͕ҟͳΔ જࡏදݱ
ϚϧνλεΫֶश • ̍ͭͷΤϯίʔμʹର͠ෳͷతؔΛ༻͍ɺಉ࣌ʹ࠷దԽ[4] • ଟݴޠλεΫʹ͓͍ͯɺʮιʔεݴޠ (ӳޠͳͲ) ͰͷඪλεΫʴ ιʔεݴޠͱରݴޠͷΞϥΠϝϯτʯΛಉ࣌ʹ࠷దԽ͢Δݚڀ ͕ଘࡏ •
ԻݴޠཧղɺςΩετ؆ུԽɺґଘੑߏจղੳɺػց༁ 8 <>.VMUJUBTL4FRVFODFUP4FRVFODF-FBSOJOH
ఏҊϞσϧɿZX-Parse • ϚϧνλεΫSeq2seqϞσϧ ✓ తͷλεΫʹՃ͑ิॿతͳλεΫΛಋೖ • DLF ɿཧܗࣜͷੜ • DNL
ɿࣗવݴޠͷੜ • LPɿݴޠ༧ଌ 9 Τϯίʔμ z DLF LP ࣗવจ DNL ֶश(EN) ਪ જࡏදݱ Transformer x 6 (mBARTͷΤϯίʔμ) Transformer x 6 ֶश(ଞݴޠ) ཧܗࣜͷੜ ӳޠͰͷΈֶश
DLF ɿ Generating Logical Forms 10 Τϯίʔμ z DLF LP
List fl ights from San Francisco to Pittsburgh? DNL SELECT DISTINCT fl ight_1. fl ight_id FROM … • ӳޠͷࣗવจ͔Βಉ͡ҙຯͷཧܗࣜΛੜ͢Δ ➡ཧܗࣜੜೳྗΛʹ͚ͭΔ ֶश࣌ ਪ࣌ જࡏදݱ
• ରݴޠͷࣗવจʹϊΠζΛՃ͑ɺ࠶ߏங͘͠ӳޠ༁ ➡֤ݴޠͷࣗવจʹΤϯίʔμʔΛదԠͤ͞Δ • ݴޠݻ༗ͷಛੑΛʹ͚Δ DNL ɿ Generating Natural Language
11 Τϯίʔμ z DLF LP αϯϑϥϯγείൃϐοπ όʔάߦ͖ͷϑϥΠτΛ ϦετΞοϓ͍ͯͩ͘͠͞ɻ DNL Н ϊΠζ Н αϯϑϥϯγείൃϐοπ όʔάߦ͖ͷϑϥΠτΛ ϦετΞοϓ͍ͯͩ͘͠͞ɻ List fl ights from San Francisco to Pittsburgh? ֶश࣌ ਪ࣌ જࡏදݱ
LPɿ Language Prediction 12 Τϯίʔμ z DLF LP List fl
ights from San Francisco to Pittsburgh? DNL English ֶश࣌ ਪ࣌ • ೖྗจͷݴޠΛྨثͰ༧ଌ͢Δ ➡ΤϯίʔμʹݴޠΛ۠ผͤ͞Δ જࡏදݱ
LPɿ Language Prediction 13 Τϯίʔμ z DLF LP List fl
ights from San Francisco to Pittsburgh? DNL English ֶश࣌ ਪ࣌ • ೖྗจͷݴޠΛྨثͰ༧ଌ͢Δ ➡ΤϯίʔμʹݴޠΛ۠ผͤ͞Δ ✦ ٯ࣌ʹޯΛసͤ͞Δ ➡ ݴޠΛ۠ผͤ͞ͳ͍ • ݴޠʹͱΒΘΕͳ͍දݱ જࡏදݱ
• ಉ࣌ʹ̏ͭͷతؔΛ࠷దԽ • ཧܗࣜͷੜʴݴޠ͝ͱͷಛΛֶशʴݴޠΛ۠ผ͠ͳ͍ • ରݴޠͷϖΞίʔύεΛඞཁͱ͠ͳ͍θϩγϣοτҙຯղੳ ZX-Parse 14 Τϯίʔμ z
DLF LP DNL ٯ࣌ ཧܗࣜͷੜ ࣗવݴޠͷੜ ʢ࠶ߏ/༁ʣ ݴޠ༧ଌ − ∂LLP ∂θ ∂LNL ∂θ ∂LLF ∂θ જࡏදݱ
σʔληοτ ✦ ҙຯղੳσʔληοτ • ATIS • ཱྀߦυϝΠϯͷଟݴޠࣗવจͱཧܗࣜͷϖΞίʔύε • ӳޠ, ϑϥϯεޠ,
ϙϧτΨϧޠ, εϖΠϯޠ, υΠπޠ, தࠃޠ, ώϯσΟʔޠ, τϧίޠ • Overnight • ̔υϝΠϯͷӳޠࣗવจͱཧܗࣜͷϖΞίʔύε • ਪ࣌ͷΈ༁Ͱ࡞͞ΕͨதࠃޠͱυΠπޠσʔλΛར༻ ✦ ࣗવݴޠσʔληοτ • MKQA • ࣭จͷର༁ίʔύε • ӳޠ, ϑϥϯεޠ, ϙϧτΨϧޠ, εϖΠϯޠ, υΠπޠ, தࠃޠ • ParaCrawl • Webର༁ίʔύε 15
σʔληοτ ✦ ҙຯղੳσʔληοτ • ATIS • ཱྀߦυϝΠϯͷଟݴޠࣗવจͱཧܗࣜͷϖΞίʔύε • ӳޠ, ϑϥϯεޠ,
ϙϧτΨϧޠ, εϖΠϯޠ, υΠπޠ, தࠃޠ, ώϯσΟʔޠ, τϧίޠ • Overnight • ̔υϝΠϯͷӳޠࣗવจͱཧܗࣜͷϖΞίʔύε • ਪ࣌ͷΈ༁Ͱ࡞͞ΕͨதࠃޠͱυΠπޠσʔλΛར༻ ✦ ࣗવݴޠσʔληοτ • MKQA • ࣭จͷର༁ίʔύε • ӳޠ, ϑϥϯεޠ, ϙϧτΨϧޠ, εϖΠϯޠ, υΠπޠ, தࠃޠ • ParaCrawl • Webର༁ίʔύε 16
༁ϕʔεϥΠϯ ✦ “࠷খݶͷྗ”ϕʔεϥΠϯ • ػց༁Λ༻͍ͯ࡞ͨ͠ϕʔεϥΠϯ • Translate-Test • ςετηοτΛӳޠʹ༁ •
Translate-Train • ֶशηοτΛରݴޠʹ༁ ✦ “࠷େݶͷྗ”ϕʔεϥΠϯ • Monolingual Training • ֶशηοτΛਓखͰରݴޠʹ༁ 17
࣮ݧ݁Ռɿ༁ϕʔεϥΠϯͱͷൺֱ • ӳޠҎ֎ͷݴޠͷθϩγϣοτҙຯղੳλεΫͰSOTA 18 • Monolingual Training: ֶशηοτΛਓखͰରݴޠʹ༁ • Translate-Train:
ֶशηοτΛରݴޠʹ༁ • Translate-Test: ςετηοτΛӳޠʹ༁ • ATIS: ࣭จͱཧܗࣜͷϖΞσʔλ • Overnight: ෳυϝΠϯͷจͱཧܗࣜͷϖΞσʔλ ※ώϯσΟʔޠ(HI)ɺτϧίޠ(TR)ର༁ίʔύε͕ ଘࡏ͠ͳ͍ͨΊิॿతͰͷֶशʹؚ·Ε͍ͯͳ͍
࣮ݧ݁Ռɿ࠷খݶͷྗϕʔεϥΠϯͱͷൺֱ • ิॿతͰͷֶशΛߦͳͬͨશͯͷݴޠ(HI,TRҎ֎)ʹ͓͍ͯੑೳ্ • OvernightATIS΄Ͳੑೳ্͍ͯ͠ͳ͍ • υϝΠϯ͕ଟ༷ͳͨΊɺ՝͕ෳ߹తʹͳ͍ͬͯΔ • ӳޠʹ͍ۙݴޠͰTranslate-TrainTranslate-TestΛԼճΔ͕ɺԕ͍ݴޠ Ͱੑೳ͕Լ͢Δ
• ػց༁ͷํͷӨڹʁ 19
࣮ݧ݁ՌɿαϒλεΫͷӨڹ 20 • LFͷΈͷ߹ɺTranslate-TestΑΓੑೳ͕͍ • NLͱLPͲͪΒ͔Ճ͢Δ࣌ɺNLΛՃͨ͠ํ͕ੑೳ͕ߴ͍ • ֤ݴޠͷࣗવݴޠจʹదԠ͢Δํ͕େࣄ
࣮ݧ݁ՌɿαϒλεΫͷӨڹ 21 • LFͷΈͷ߹ɺTranslate-TestΑΓੑೳ͕͍ • NLͱLPͲͪΒ͔Ճ͢Δ࣌ɺNLΛՃͨ͠ํ͕ੑೳ͕ߴ͍ • ֤ݴޠͷࣗવจʹదԠ͢Δ͜ͱ͕େࣄ
࣮ݧ݁ՌɿαϒλεΫͷӨڹ • αϒλεΫͰֶशʹؚ·Εͳ͍ݴޠ(HI,TR)Ͱੑೳ্ • ֶश͍ͯ͠ͳͯ͘ݴޠؒͷજࡏදݱ͕վળ͞Ε͍ͯΔ 22
࣮ݧ݁ՌɿαϒλεΫͷӨڹ • αϒλεΫͰֶशʹؚ·Εͳ͍ݴޠ(HI,TR)Ͱੑೳ্ • ֶश͍ͯ͠ͳͯ͘ݴޠؒͷજࡏදݱ͕վળ͞Ε͍ͯΔ 23 ˝ຒΊࠐΈͷՄࢹԽ
࣮ݧ݁ՌɿαϒλεΫͷӨڹ • ୯ݴޠͷΈར༻͢Δ߹(τ = 0.0)ΑΓ༁ߦ͏߹(τ = 0.5) ͷੑೳ͕ߴ͍ • దͳจΛར༻͢Δ߹(ParaCrawl)ΑΓਪ࣌ͷೖྗͱಉ࣭͡
จΛֶशʹར༻͢Δ߹(MKQA)ͷੑೳ͕ߴ͍ 24
·ͱΊ ✓ θϩγϣοτଟݴޠҙຯղੳϞσϧɺZX-parseͷఏҊ • જࡏදݱͷΞϥΠϝϯτʹண͠సҠֶशͷޡࠩΛ࠷খԽ • ෳͷଛࣦؔΛಋೖ͢ΔϚϧνλεΫֶशʹΑΓҟͳΔݴޠͷ જࡏදݱ͕ྨࣅ͢ΔΑ͏ʹ • ରݴޠͷϖΞσʔλΛඞཁͱͤͣɺӳޠϖΞͱରݴޠͷࣗવ
จͷΈར༻ ✓ θϩγϣοτҙຯղੳλεΫʹ͓͍ͯෳͷݴޠͰߴ͍ੑೳ 25