文献紹介：Revisiting Word Embedding for Contrasting Meaning

Transcript

1. 1
   文献紹介 (2016.03.03)
   長岡技術科学大学　　自然言語処理
   　　 Nguyen Van Hai
   Revisiting Word Embedding for Contrasting Meaning
   Zhigang Chen, Wei Lin, Qian Chen,
   Ziaoping Chen, Si Wei, Hui Jiang and Xiaodan Zhu
   Proceedings of the 53th Annual Meeting of the Association for
   Computational Linguistics and the 7th International Joint
   Conference on Natural language processing, p106-115
   

   View Slide

2. 2
   Introduction
   ●
   Contrasting meaning is a basic aspect of semantics
   ●
   This embedding models achieve 92% F-score on
   the GRE dataset, which is “most contrasting word”
   questions (Mohammad et al.,2008)
   ●
   Direction for this basic semantics modeling
   problem.
   

   View Slide

3. 3
   Related work
   ●
   The terms contrasting, opposite, and antonym have
   different definitions in the literature.
   – Contrasting word pairs having some non-zero degree of binary
   incompatibility
   ●
   Word Embedding
   ●
   Modeling Contrasting Meaning
   – PILSA (Yih et al., 2012) get a further improvement on the GRE
   benchmark, where an 81% of F-score
   – Bayesian probabilistic tensor factorization (Zhang et al., 2014)
   

   View Slide

4. View Slide

5. 5
   Top Hidden Layer(s)
   ●
   It is widely recognized that contrasting words
   – e.g., good and bad
   ●
   Intend to appear in similar context or co-occur
   with each other.
   

   View Slide

6. 6
   Stochastic Contrast Embedding (SCE)
   ●
   Hinton and Roweis (2002) proposed a stochastic
   neighbor embedding (SNE) framework.
   ●
   Use the concept of “neighbors” to encode the
   contrast pairs
   ●
   Three respects are different from SNE
   – “neighbors” here are actually contrasting pairs
   – Train SCE using only lexical resources
   – Semantic closeness or contrast are not independent
   

   View Slide

7. 7
   Marginal Contrast Embedding (MCE)
   ●
   This paper used another training criteria, motivated
   by pairwise ranking approach (Cohen et al., 1998)
   ●
   This motivation
   – To explicitly enforce the distances between contrasting
   pairs to be larger than distances between unrelated word
   pairs.
   – Enforce the distances between semantically close pairs
   by another margin
   

   View Slide

8. 8
   Corpus Relatedness Modeling (CRM)
   ●
   Mohammad et al., (2013) proposed a degree of
   contrast hypothesis:
   – “if a pair of words, A and B, are contrasting, then their
   degree of contrast is proportional to their tendency to
   co-occur in a large corpus.”
   ●
   Use the skip-gram model (Mikolov et al., 2013) to
   learn the relatedness embedding
   

   View Slide

9. 9
   Semantic Differential Reconstruction
   (SDR)
   ●
   Using factor analysis, Osgood et al.,(1957) identified
   three dimensions of semantic that account for
   most of the variation in the connotative meaning
   of adjective.
   ●
   Three dimensions
   – Evaluative: good-bad
   – Potency: strong-weak
   – Activity: active-passive
   

   View Slide

10. 10
    Experiment Set-Up
    ●
    Data
    – Used the “most contrasting word” questions collected by
    Mohammad et al., (2008)
    ●
    Lexical Resources
    – WordNet (Miller, 1995) version 3.0
    – Roget's Thesaurus (Kipfer, 2009)
    ●
    Google Billion-Word Corpus
    ●
    Evaluation Metric
    

    View Slide

11. 11
    Experiment Results
    ●
    The result is presented in Table 1 and the
    F-score on test set is 91%
    ●
    The pie graph in Figure 2 shows the percentages of
    target-gold-answer word pairs
    ●
    Figure 3 draws the envelope of histogram of cosine
    distance between all target-choice
    ●
    The performances of different models are show in
    Figure 4
    

    View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide