Relation such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for LexicalEntailment

Stephen Roller, Katrin Erk.Relation such as Hypernymy:Identifying and Exploiting Hearst Patterns inDistributional Vectors for LexicalEntailment. arXiv:1605.05433. 2016
輪読で発表した時の資料です

Transcript

1. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 म࢜ྠಡ (2016/5/25) Relation such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment ࢁࠜ ৎ྄ ஌ೳ਺ཧݚڀࣨ M1 2016/5/25 1 / 20
2. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 1 ֓ཁ 2 طଘݚڀ 3 طଘݚڀ〣ൺֱ࣮ݧ 4 ఏҊख๏ 5 〳〝〶 2 / 20
3. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࿦จ֓ཁ ֓ཁ Stephen Roller, Katrin Erk. Relation such as Hypernymy: Identifying and Exploiting Hearst Patterns in Distributional Vectors for Lexical Entailment. arXiv:1605.05433. 2016 • ؚҙؔ܎ʢlexical entailmentʣぇਪఆ『぀のとぜ • ෼ࢄදݱぇ࢖〘〔ぎゆ゜がば • ෳ਺〣ぶがのなひぷ〜 state-of-the-art • ղੳ「〛〴぀〝ɺゑぶ゚〤 Hearst Patterns ぇݟ〙々぀ 〽⿸〠ֶश「〛⿶぀〈〝⿿෼⿾〘〔 ˞ ਤ〤『〮〛࿦จத⿾〾Ҿ༻ 1 / 20
4. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 طଘݚڀ〝〒〣໰୊఺ Concat (Baroni, 2012) Diff, Asym (Fu, 2014; Weeds, 2014; Roller, 2014) Ksim (Levy, 2015) طଘݚڀ طଘݚڀ〜〤ؚҙؔ܎ぇ〝〾⿺぀〈〝〠ࣦഊ「〛⿶぀ (Weeds, 2014; Roller, 2014) • ଟ。〣ޠぇؚҙ『぀ޠぇ౴⿺〛⿶぀〕々ʢ”lexical memorization”ʣ • e.g.) animal 〤ଟ。〣ޠぇؚҙ『぀ ⇒ 〝〿⿴⿺』Կ〠〜〷 animal 〝౴⿺〛⿼。 • ˓ cat→animal, ʷ sofa→animal 2 / 20
5. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 طଘݚڀ〝〒〣໰୊఺ Concat (Baroni, 2012) Diff, Asym (Fu, 2014; Weeds, 2014; Roller, 2014) Ksim (Levy, 2015) طଘݚڀʛConcatゑぶ゚ (Baroni, 2012) ্Ґޠ〝ԼҐޠ〣෼ࢄදݱよぜぷ゚ぇ〒ぁ〓ぁ H,w • ݁߹よぜぷ゚ ⟨H, w⟩ ぇೖྗ • SVM 〜ؚҙؔ܎〣らぎぇ෼ྨ • lexical memorization ⿿ى　぀〷〣〣ґવ〝「〛ڧ⿶ よがと゘ぐアख๏ 3 / 20
6. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 طଘݚڀ〝〒〣໰୊఺ Concat (Baroni, 2012) Diff, Asym (Fu, 2014; Weeds, 2014; Roller, 2014) Ksim (Levy, 2015) طଘݚڀʛDiff, Asymゑぶ゚ Diffゑぶ゚(Fu, 2014; Weeds, 2014) • よぜぷ゚〣ࠩ H − w ぇೖྗ • Concat ゑぶ゚〣〰⿸⿿ੑೳ⿿ྑ⿶ (Weeds, 2014) • ٯぇ〝〟⿺぀ਓ〷⿶぀ • ݁࿦ɼ〞〖〾⿿⿶⿶⿾い⿾〾〟⿶ Asymゑぶ゚(Roller, 2014) • H − w 〝 H − w 〣 2 ৐ぽ゚わぇೖྗ • Concat ゑぶ゚〽〿〷ੑೳ⿿ྑ⿶ (Roller, 2014) 4 / 20
7. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 طଘݚڀ〝〒〣໰୊఺ Concat (Baroni, 2012) Diff, Asym (Fu, 2014; Weeds, 2014; Roller, 2014) Ksim (Levy, 2015) طଘݚڀʛ Ksimゑぶ゚ (Levy, 2015) • Concat 〝 Diff ゑぶ゚〤ޠኮؔ܎ぇֶश〜　〛⿶〟⿶ • ୯ޠ⿿〞ぁ〕々্Ґޠ〝「〛ग़ݱ「〹『⿶⿾ぇ༧ଌ「 〛⿶぀〕々 (lexical memorization) • ೖྗ〠 H 〝 w 〣ぢつぐアڑ཭ぇಋೖ • sofa→animal 〝⿶⿸༧ଌ〤๷〆〒⿸ • ෼ࢄදݱ〠〽぀ぎゆ゜がば〤〈〣のとぜ〠޲⿶〛⿶〟 ⿶〝⿶⿸݁࿦ 5 / 20
8. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ طଘݚڀ〣ൺֱ࣮ݧ • طଘݚڀख๏〣⿸〖〞ぁ⿿ྑ⿶⿾〣݁࿦⿿ग़〛⿶〟⿶ • ⿶あ⿶あ〟ぶがの〜ൺֱ࣮ݧ「〛〴぀ • શ෦〜 4 〙〣ぶがのなひぷ • 2 〙〤্ҐɾԼҐؔ܎〣〴ぇؚ〵 • ࢒〿〣 2 〙〤〽〿Ұൠత〟ؚҙؔ܎ぇؚ〵 6 / 20
9. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ぶがのなひぷ LEDS (Baroni, 2012) • 1,385 ૊ (from WordNet) 〣্ҐɾԼҐޠらぎʢਖ਼ྫʣ • 1,385 ૊〣ෛྫʢਖ਼ྫぇてをひや゚「〛ແ࡞ҝ〠つアゆ ゙アそʣ BLESS (Baroni and Lenci, 2011) • 17 छ〣ޠኮؔ܎⿿゘よ゚෇々《ぁ〔 200 ૊〣୯ޠらぎ 〝ෛྫ (random words) • ࣮ݧ〜〤 hypernymy Ҏ֎〤ෛྫ〝「〛ѻ⿸ 7 / 20
10. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ぶがのなひぷ Medical (Levy, 2015) • ҩྍؔ܎〣จষ⿾〾நग़「〔ޠኮؔ܎⿿゘よ゚෇々《 ぁ〔୯ޠらぎ 12,600 らぎʢ⿸〖ؚҙؔ܎〤 945 らぎʣ • ඞ』「〷ଞ〣ぶがの〣ؚҙؔ܎〠Ұக「〟⿶〷〣⿿⿴぀ • e.g.) doctor→hospital 〟〞 • ଟ。〣ޠኮؔ܎⿿⿴぀〣〜೉「⿶ TM14 • SemEval2012 Shared Task 〣ぶがのなひぷ • ଟछ〣ޠኮؔ܎⿿゘よ゚෇《ぁ〛⿶぀ • Turney and Mohammad (2015) 〠〽〘〛ؚҙؔ܎〝〒⿸ 〜〟⿶〷〣〣୯ޠらぎ〠෼々〾ぁ〔〷〣ぇ࢖༻ • 2,188 らぎ〣⿸〖 1,084 らぎ⿿ਖ਼ྫ 8 / 20
11. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ෼ࢄදݱよぜぷ゚ • Gigaword,Wikipedia,BNC,ukWaC ぇぢがむと〝「〛࢖༻ • Word2Vec 〣〽⿸〟ぺゔが゘゚ぼひぷܥ〣෼ࢄදݱ〜〤 〟。ɼڞىじげアぷよがと〣 BOW • PPMIɾSVD 〜 300 ࣍ݩ〠མ〝『 • ୯Ґよぜぷ゚〠ぽがろ゘ぐど 9 / 20
12. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ධՁํ๏ • F ஋ • 20-fold cross validation 〣ฏۉ஋〜ൺֱ • ֤ゑぶ゚〣ൺֱ〝ɼsensitive 〟む゘ゐがの〠〙⿶〛࣮ݧ 10 / 20
13. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ࣮ݧ݁Ռʛsensitive〟む゘ゐがの ෼ࢄදݱ〣 window size • શ〛〣ぶがの〜 window size 〤খ《⿶〰⿸⿿ྑ⿶ 〞〣෼ྨثぇ࢖⿸⿾ • શ〛〣ぶがの〜゜でと ふくひぜճؼ⿿ྑ⿶ 11 / 20
14. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ ゑぶ゚〣ൺֱ • よがと゘ぐア〝「〛ぢつ ぐアڑ཭〣ゑぶ゚ぇ௥Ճ • Ksim ⿿ 3 〙〣ぶがのなひぷ〜উ〘〛⿶぀〝〤⿶⿺ɼ〞 〣ゑぶ゚⿿ྑ⿶⿾〤ぶがのґଘ • Concat ⿿ LEDS Ҏ֎〣ぶがのなひぷ〜 2 ൪໨〠ྑ⿶݁ Ռぇग़「〛⿶぀ ⇒ ߟ࡯ 12 / 20
15. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ Concat〠〙⿶〛〣ߟ࡯ Concat 〣෼ྨ〠⿼々぀෼཭௒ฏ໘ぇ ˆ p = ⟨ ˆ H, ˆ w ⟩ 〝『぀〝ɼ Linear (⟨H, W⟩) = ˆ pT ⟨H, W⟩ = ⟨ ˆ H, ˆ w ⟩ T ⟨H, W⟩ = ˆ HT H + ˆ wT w • ্Ґޠ H 〝ԼҐޠ w 〠ಠཱ〣ॏ〴⿿⿾⿾〘〛⿶⿶぀〔 〶ɼ୯ޠؒ〣”ؔ܎”ぇݟ぀〈〝⿿〜　〛⿶〟⿶ • lexical memorization ⿿ى　〛⿶぀ • LEDS 〜 F ஋⿿௿⿾〘〔〣〤ɼෛྫ⿿׬શ〠゘アはわ〟 ୯ޠらぎ〕⿾〾 • ಉ」୯ޠ⿿ਖ਼ྫ〝「〛〷ෛྫ〝「〛〷ग़ݱ『぀ ⇒lexical memorization ⿿ى　〛⿶぀〣〠〟】〈え〟〠ੑೳ ⿿ྑ⿶〣⿾ʁ 13 / 20
16. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ࣮ݧઃఆ ࣮ݧ݁Ռ ߟ࡯ Concat〠〙⿶〛〣ߟ࡯ (ଓ　) • Concat ゑぶ゚〤 Hearst Patterns ぇݟ〙々ग़「〛⿶぀ • ෼཭௒ฏ໘ ˆ p 〝จ຺〣よぜぷ゚〤ಉۭؒ〠ଘࡏ『぀〣 〜ൺֱ〜　぀ • จ຺ぇߟྀ〜　〛⿶぀〣〜⿴぀ఔ౓ੑೳ⿿ྑ⿶ (?) 14 / 20
17. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ֓ཁ ぎ゚っ゙どわ ࣮ݧ ఏҊख๏ • Concat 〣 feature detector 〣ੑ࣭〠஫໨ • ˆ p 〝 x = ⟨H, w⟩ 〤্Ґޠ⿿ग़ݱ「〹『⿶จ຺ (Hearst Patterns) 〣ಛ௃ぇؚえ〜⿶぀ • ؆୯〟ྲྀぁ Concat 〜 ˆ p (Hearst Patterns) ぇֶश ⇒ ڭࢣ〣෼ࢄදݱ⿾〾 ˆ p ੒෼ぇҾ。 (vector rejection) ⇒ ৽〔〟 Hearst Patterns ⿿ಘ〾ぁ぀ 15 / 20
18. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ֓ཁ ぎ゚っ゙どわ ࣮ݧ ぎ゚っ゙どわ 1 Concat 〜෼཭௒ฏ໘ ˆ p ぇֶश 2 ڭࢣ〣෼ࢄදݱ x 〣த〣 ˆ p ੒෼よぜぷ゚ projˆ p (x) = ( xT ˆ p ∥ˆ p∥ ) ˆ p ぇܭࢉ (vector projection) 3 ڭࢣ〣෼ࢄදݱ⿾〾 projˆ p ぇҾ⿶〔よぜぷ゚ rejˆ p (x) = x − projˆ p (x) ぇܭࢉ (vector rejection) 4 rejˆ p (x) ぇਖ਼نԽ「ɼ৽〔〟ڭࢣ〝『぀ 5 ৽〔〟ڭࢣ〜৽〔〟௒ฏ໘ぇܭࢉ『぀ 〈ぁぇ܁〿ฦ『〈〝〜⿶。〙〷〣 Hearst pattern detectors (ˆ p1 , . . . , ˆ pn :෼཭௒ฏ໘) ぇݟ〙々぀〈〝⿿〜　぀ 16 / 20
19. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ֓ཁ ぎ゚っ゙どわ ࣮ݧ ૉੑよぜぷ゚Fi iteration i ʢˆ pi 〣ֶश〣ஈ֊ʣ〜ɼૉੑよぜぷ゚ Fi 〤ɼ 1 Hi 〝 wi 〣ྨࣅ౓: HT i wi 2 ্Ґޠ〝「〛 Hi ⿿ग़ݱ「〹『⿶จ຺〣よぜぷ゚: HT i ˆ pi 3 ԼҐޠ〝「〛 wi ⿿ग़ݱ「〹『⿶จ຺〣よぜぷ゚: wT i ˆ pi 4 2 〝 3 〣จ຺よぜぷ゚〣ࠩ: HT i ˆ pi − wT i ˆ pi ぇؚ〵〽⿸〠ఆٛ『぀ɽ ∴ Fi = ⟨ HT i wi , HT i ˆ pi , wT i ˆ pi , HT i ˆ pi − wT i ˆ pi ⟩ 17 / 20
20. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ֓ཁ ぎ゚っ゙どわ ࣮ݧ ࣮ݧ݁Ռ • Similarity ൈ。〝 Concat 〝〰〱ಉ」 ⇒Similarity 〤ॏཁ • Detectors ൈ。〝 LEDS Ҏ֎〜େ　。 F ஋⿿Լ⿿぀ • LEDS 〜〤 Hearst Patterns ぇֶश〜　〟⿶ • ਖ਼ྫ〝ෛྫ〠ಉ」୯ޠ⿿ଟ。ग़ݱ『぀⿾〾 • Inclusion ൈ。〝 F ஋⿿Լ⿿぀ • ”ԼҐޠ〣จ຺〠〤্Ґޠ〷౰〛〤〳぀” (Distributional Inclusion Hypothesis) 〣ཪ෇々 18 / 20
21. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 ֓ཁ ぎ゚っ゙どわ ࣮ݧ ࣮ݧ݁Ռ (ଓ　) • iteration 1 〜〤”such as” • iteration 2 〜〤”including”〣จ຺ぇݟ〙々〛⿶぀ • iteration 3, 4 〜〤〽。い⿾〾〟⿶จ຺ぇݟ〙々〛⿶぀ ⿿ɼF ஋〤ྑ。〟぀ • iteration 5 Ҏ߱〤 F ஋⿿Լ⿿〘〛⿶。 19 / 20
22. ֓ཁ طଘݚڀ طଘݚڀ〣ൺֱ࣮ݧ ఏҊख๏ 〳〝〶 〳〝〶 • ؚҙؔ܎෼ྨ〣のとぜ〜෼ࢄදݱぇ࢖〘〔⿶。〙⿾〣 ゑぶ゚ぇෳ਺〣ぶがのなひぷ〜ධՁ • ֤ゑぶ゚〣ར఺〝ܽ఺⿿໌〾⿾〠〟〘〔 • Concat 〜〤 lexical memorization ⿿ى　〛「〳⿸⿿ Hearst Pattern ぇݟ〙々ग़『〈〝⿿〜　぀〈〝⿿෼ ⿾〘〔 • Concat ぇݩ〠 4 〙〣ૉੑぇߟྀ「〔৽〔〟ゑぶ゚ぇ ఏҊ • ఏҊゑぶ゚〤⿶。〙⿾〣ぶがのなひぷ〜 state-of-the-art 〣 F ஋ • ఏҊゑぶ゚〤৽〔〟 Hearst Pattern ぇݟ〙々぀〈〝⿿〜 　぀ 20 / 20