[論紹] QueryingWord Embeddings for Similarity and Relatedness

弊研究室で行なったNAACL読み会の発表資料です。

Transcript

1. Paper introduction QueryingWord Embeddings for Similarity and Relatedness 2018/07/04 M1 Kazuki Ashihara
2. Contents • Introduction • Dataset of Similarity and Relatedness • Word Embedding and Context Embedding • Experiment – Quantitative Evaluation – Qualitative Evaluation • Conclusion 2
3. Introduction • Similarity – Often assumed to be a direction-less measure. – (car → truck) = (truck → car) • Relatedness – Often assumed to be asymmetry – (stork → baby) ≠ (baby → stork) 3
4. Introduction • In the current embedding evaluation, the concept of similarity and relatedness and asymmetric are ignored. – Similarity – Basically do not think about direction
   truck ⇔ car. – People judge
   tiger → leopardis more similar to
   leopard → tiger. – Relatedness –
   storkco-occurs with
   baby. –
   babydoesn`t co-occurs so far with
   stork. • Experiment influence of similarity and relatedness. • Make the dataset 4
5. Dataset of Similarity and Relatedness • SimLex – Pay attention only to the similarity between words. – Annotation by people. – coast - shore : 9.00 – clothes – closet : 1.96 • ProNorm – Pay attention only to the relatedness between words. – People generate the words. – Annotator generates words that are not similar but related. – Generate 1169 related words for 100 words. • WordSim353 – Including similarity pairs and relatedness pairs. – Response based either on taxonomic similarity or associative relatedness. 5
6. Word Embedding and Context Embedding • Skipgram – Output word embedding only. – Context embedding will eventually be discarded. 6 AA measure • Improved to save two vector information* • Consider symmetry and asymmetry between words * Improving distributional similarity with lessons learned from word embeddings (O.Levy et al. TACL 2015)
7. How to Set Other Method • WW – Both word A and word B are word embedding. – Commonly used similarity • CC – Both word A and word B are context embedding. • WC – Word A is word embedding, and word B is context embedding. • CW – Word A is context embedding, and word B is word embedding. 7 Similarity between word A and word B. (A, B)
8. Experiment • Word2Vec using Skipgram – Using dump data of Wikipedia. – Generate word embedding, and context embedding. • Experiment under various conditions – dim (dimension) 100/200/3000 – win (windowsize) 3/6/10 – min (min occurrence) 1/5 – neg (negative sampling) 1/5 – iter (iteretor) 1/5 8
9. Quantitative Evaluation • Calculate scores in each model – Measure accuracy with ProNorm and SimLex. – Use output score and correct data score • hypothesis – SimLex (Similarity) – WW measure is the best. – The direction between two words is not a factor. – ProNorm (Relatedness) – WC measure is the best. – Tell us how likely we would see w2 and similar words in the context of w1. 9
10. Result (Similarity) 10 • The last row of the table includes score between human similarity judgment and a linear regression. • Numbers in this row show an upper bound for Spearman scores of the individual measures.
11. Result (Relatedness) 11
12. Result (Balanced) 12 Consider both symmetric and asymmetric is most important. dim : 300 win : 3
13. Qualitative Evaluation • Production experiment – WW and WC – Using ProNorm (Relatedness) data. – w1 is given and generates the n most similar words. – Count how many people responses were contained 13 dim : 300 win : 10
14. Result 14
15. Output example 15 restaurant People door, family, window, roof, furniture, chimney, kitchen WW mansion, farmhouse, cottage WC barn, residence, room, stable, fireplace, family, kitchen
16. Conclusion • Making the dataset • Evaluation from Similarity and Relatedness • WW was consistently better in scoring similarity between two words. • WC was better in measuring the thematic relatedness. 16