Share
Embed
alacarte-snlp2018
Hitoshi Manabe
August 04, 2018
Transcript
21 slides
-
— ୈ10ճ࠷ઌNLPษڧձ — A La Carte Embedding: Cheap but Effective
Induction of Semantic Feature Vectors Hitoshi Manabe Mikhail Khodak, Nikunj Saunshi, Yingyu Liang, Tengyu Ma, Brandon Stewart, and Sanjeev Arora. In ACL 2018. ※ εϥΠυURL : https://speakerdeck.com/manaysh/alacarte-snlp2018-1 -
֓ཁ • Γ͍ͨ͜ͱ : ςΩετʹؔ͢Δ༷ʑͳૉੑΛຒΊࠐΈ͍ͨ • සɾະޠ, n-gram, synsetͳͲͳͲ •
ఏҊ : จ຺Λ׆༻͠ɺૉੑͷࢄදݱͷߏங • ϞσϧࣗମͷֶशࣄલֶशࡁΈ୯ޠࢄදݱͱ େنίʔύεͷΈͰجຊతʹ࣮ߦՄೳ • ຒΊࠐΈ͍ͨૉੑͷจ຺Λ༩͑ɺࢄදݱΛon the flyͰ֫ಘ • ݁Ռ : ୯ޠࢄදݱͷFew-shotֶश, จॻྨλεΫͳͲͷԠ༻ • ࣄྫͷจ຺Λ༩͑Δ͚ͩͰྑ͍ײ͡ͷදݱ͕ಘΒΕΔ • ͦͷଞԠ༻λεΫͰͷ༗ޮੑΛ֬ೝ -
ઃఆ ࣄલʹඞཁͳͷ • ޠኮ ΛͭେنͳςΩετίʔύε • ֤ରͷ୯ޠ ʹର͢Δग़ݱจ຺ ͷू߹ͱͯ͠දݱ •
(ͦΕΛͬͯ)ࣄલֶशࡁΈͷ୯ޠࢄදݱ • Կ͔͠Βͷରͷૉੑ(৽ޠɾsynsetͳͲ) ͱͦͷग़ݱจ຺ Λ༩͑ͯɺରૉੑͷࢄදݱ Λon the flyͰ֫ಘ CV vw f Cf V w Cw vf vBarackObama Cf -
ઃఆ ࣄલʹඞཁͳͷ • ޠኮ ΛͭେنͳςΩετίʔύε • ֤ରͷ୯ޠ ʹର͢Δग़ݱจ຺ ͷू߹ͱͯ͠දݱ •
(ͦΕΛͬͯ)ࣄલֶशࡁΈͷ୯ޠࢄදݱ • Կ͔͠Βͷରͷૉੑ(৽ޠɾsynsetͳͲ) ͱͦͷग़ݱจ຺ Λ༩͑ͯɺରૉੑͷࢄදݱ Λon the flyͰ֫ಘ CV vw f Cf V w Cw vf beef up the army we must beef up our organization … v(beef,up) Cf -
ઌߦݚڀ Additive [Lazaridou+. 2017] • featureͷจ຺ͱؚͯ͠·ΕΔ୯ޠࢄදݱͷฏۉͰදݱ • ͨͩͷͰྑ͍ͷ͔ʁ • ୯ޠࢄදݱʹڞ௨ͷํΛ࣋ͭɻͦͷͨΊଟ͘ͷදݱ
Λ͢͜ͱͰ༗༻ͳใ͕ڞ௨ͷใͰຒΕΔ • ετοϓϫʔυcommon wordϊΠζʹͳΔ • ղܾࡦͱͯ͠ த৺Խ+ओআ[Mu+. 2018] ग़ݱසͷٯ ͰॏΈ͚[Arora+. 2017] ͨͩ͠ɺेʹνϡʔχϯά͕ඞཁ(…?) vadditive f = 1 |Cf | X c2Cf 1 |c| X w2c vw -
ఏҊख๏ ఏҊख๏: a la carte • ݩͷ୯ޠࢄදݱ͕Additive + ઢܗมͰۙࣅͰ͖ΔΑ͏ֶश
(ͨͩ͠ ֶश͞ΕΔࣸ૾ύϥϝʔλ) • ಋग़͞ΕΔ ͋Γ͖ͨΓͳํ (common wordͷํ) Λshrinkͤ͞Δಇ͖͕͋Δ[Arora+. 2018] • ดܗࣜͰ࠷దԽͰ͖ΔͷͰಛʹ༨ܭͳνϡʔχϯά͍Βͳ͍ ͨͩ͠ɺίʔύεͷසͰॏΈ͚͢ΔͳͲ͍ͯ͠Δ vw ⇡ Avadditive w = A ✓ 1 |Cw | X c2Cw X w02c vw0 ◆ A A -
ఏҊख๏ • ΞϧΰϦζϜ ݩͷදݱͱઌड़ͷεςοϓͰ֫ಘ͞Εͨදݱ͕ۙ͘ͳΔΑ͏ʹ ઢܗมͷύϥϝʔλΛֶश (࣮ࡍʹग़ݱසͰॏΈ͚) A = argmin X
w2V ↵(cw)||vw Auw ||2 2 -
࣮ݧ • ୯ޠࢄදݱͷFew-shot Learning • ୯ޠؒྨࣅλεΫ • ͦͷଞͷૉੑΛຒΊࠐΜͰ (Synset, n-gram)
Ԡ༻λεΫʹద༻ • synsetͷࢄදݱΛͬͨޠٛᐆດੑղফλεΫ • n-gramͷࢄදݱΛͬͨจॻྨλεΫ -
࣮ݧઃఆ : ୯ޠؒྨࣅ • ୯ޠؒྨࣅλεΫ (CRW dataset) • ୯ޠϖΞͷྨࣅࢉग़ (ϖΞͷยํ͕rare
word) • Westbury Wikipedia Corpus (WWC) Ͱword2vecٴͼઢܗม ύϥϝʔλͷֶश (ධՁରͱͳΔrare wordΛؚΜͩจ༧Ί আࡁΈ) • ֤Rare wordʹରͯ͠ඦఔͷจ຺Λޙ͔Β༩͑ͯࢄදݱΛ ࡞ ※ Rare wordWWCͰͷग़ݱස͕512 ʙ 10,000ճͷ୯ޠΛࢦ͢ -
࣮ݧ݁Ռ : ୯ޠؒྨࣅ • ਓखͱͷ૬ؔ vs จ຺ ※ Rare wordͷग़ݱจ຺Λશ෦ίʔύεʹؚΜͰී௨ͷword2vecͰֶशͤͨ͞
߹, 0.45ͷ૬ؔ • all-but-the-top: த৺Խ + ओͷআڈ • SIF weighted: ग़ݱසʹԠͨ͡ॏΈ͚ฏۉ -
࣮ݧઃఆ : ޠٛᐆດੑղফ • ޠٛᐆດੑղফλεΫ (SemCor dataset) • ೖྗ :
ରͷ୯ޠͱจ຺, ग़ྗ : ର୯ޠͷsense • ୯ޠࢄදݱwikipediaͰֶश͞Εͨgloveɺઢܗมύϥ ϝʔλϒϥϯίʔύεͰֶश • ֤synsetͷࢄදݱsynsetͷఆٛจ(glosses) + ίʔύεͰ synset͕ग़ݱ͢Δจͷશ୯ޠͷࢄදݱͷͱઢܗมͰ֫ಘ • ςετจͷ༧ଌରͷ୯ޠʹରͯ͠ɺपล୯ޠ͔Βࢄදݱ Λ֫ಘ͠ɺsynsetͷࢄදݱͱͷྨࣅ͔ΒҰ൪͍ۙsenseΛ༧ ଌ -
࣮ݧ݁Ռ : ޠٛᐆດੑղফ • ޠٛᐆດੑղফλεΫ (SemCor dataset) • glosses :
senseͷఆٛจ͚ͩΛ༩͑Δ • MFS : ༩͑ΒΕͨ୯ޠʹ͍ͭͯɺίʔύεͰҰ൪ස͕ߴ͍senseΛग़ྗ͢Δ ͚ͩ(͚ͩͲڧ͍) • [Ragonato+. 2017] : RNNϕʔεͷख๏ • ྨࣅϕʔεͷγϯϓϧͳख๏͚ͩͲϕʔεϥΠϯΑΓྑ͍ -
࣮ݧઃఆ : จॻྨ • จॻྨλεΫ • n-gramΛຒΊࠐΉ • ୯ޠࢄදݱAmazon Product
CorpusͰGloveΛֶशޙ ಉίʔύεͰn-gramͷࢄදݱΛߏங • จ຺ͷࢄදݱͷߏங࣌ͷwindowαΠζ10 • ҎԼࣜͰn-gramͷࢄදݱ͔ΒจॻͷࢄදݱΛ֫ಘޙ, ϩδεςΟοΫճؼͰ༧ଌ ※ n͕େ͖͍ͱจ຺͕গͳ࣭͘తʹඍົͳͷͰnͰׂΔ vT D = L X t=1 vT wt . . . 1 n L n+1 X t=1 vT (wt,...,wL+n 1) -
࣮ݧ݁Ռ : จॻྨ • จॻྨλεΫ • • DLܥͱcomparableͳ݁Ռ •
ۃੑྨ(SST, IMDB)݁ߏྑ͍(= ϩʔΧϧͳޠॱͰेͰ?) -
·ͱΊ • ະޠͳͲ༷ʑͳςΩετૉੑΛ୯ޠࢄදݱͱग़ݱจ຺͔Β on the flyͰ৽ͨʹࢄදݱΛߏஙՄೳʹ͢Δख๏ • จ຺ͷ୯ޠࢄදݱͷ + ઢܗมͰૉੑΛදݱ
• ग़ݱจ຺ϕʔεͰͷຒΊࠐΈ • จ຺ϕʔεͷΘΓʹൺֱతܭࢉػʹ༏͍͠ • αϒϫʔυܥख๏ͱͷڞଘ • ஶऀ࣮ : https://github.com/NLPrinceton/ALaCarte -
ࢀߟจݙ • Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma,
and Andrej Risteski. 2018. Linear algebraic structure of word senses, with applications to polysemy. TACL • Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to- beat baseline for sentence embeddings. In Proc. ICLR. • Angeliki Lazaridou, Marco Marelli, and Marco Baroni. 2017. Multimodal word meaning induction from minimal exposure to natural text. Cognitive Science. • Jiaqi Mu and Pramod Viswanath. 2018. All-but-thetop: Simple and effective post-processing for word representations. In Proc. ICLR. • Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017. Neural sequence learning models for word sense disambiguation. In Proc. EMNLP.