Upgrade to Pro — share decks privately, control downloads, hide ads and more …

文献紹介:Revisiting Word Embedding for Contrasting Meaning

Van Hai
March 03, 2016

文献紹介:Revisiting Word Embedding for Contrasting Meaning

Van Hai

March 03, 2016

More Decks by Van Hai


  1. 1
    文献紹介 (2016.03.03)
    長岡技術科学大学  自然言語処理
       Nguyen Van Hai
    Revisiting Word Embedding for Contrasting Meaning
    Zhigang Chen, Wei Lin, Qian Chen,
    Ziaoping Chen, Si Wei, Hui Jiang and Xiaodan Zhu
    Proceedings of the 53th Annual Meeting of the Association for
    Computational Linguistics and the 7th International Joint
    Conference on Natural language processing, p106-115

    View Slide

  2. 2

    Contrasting meaning is a basic aspect of semantics

    This embedding models achieve 92% F-score on
    the GRE dataset, which is “most contrasting word”
    questions (Mohammad et al.,2008)

    Direction for this basic semantics modeling

    View Slide

  3. 3
    Related work

    The terms contrasting, opposite, and antonym have
    different definitions in the literature.
    – Contrasting word pairs having some non-zero degree of binary

    Word Embedding

    Modeling Contrasting Meaning
    – PILSA (Yih et al., 2012) get a further improvement on the GRE
    benchmark, where an 81% of F-score
    – Bayesian probabilistic tensor factorization (Zhang et al., 2014)

    View Slide

  4. 4

    View Slide

  5. 5
    Top Hidden Layer(s)

    It is widely recognized that contrasting words
    – e.g., good and bad

    Intend to appear in similar context or co-occur
    with each other.

    View Slide

  6. 6
    Stochastic Contrast Embedding (SCE)

    Hinton and Roweis (2002) proposed a stochastic
    neighbor embedding (SNE) framework.

    Use the concept of “neighbors” to encode the
    contrast pairs

    Three respects are different from SNE
    – “neighbors” here are actually contrasting pairs
    – Train SCE using only lexical resources
    – Semantic closeness or contrast are not independent

    View Slide

  7. 7
    Marginal Contrast Embedding (MCE)

    This paper used another training criteria, motivated
    by pairwise ranking approach (Cohen et al., 1998)

    This motivation
    – To explicitly enforce the distances between contrasting
    pairs to be larger than distances between unrelated word
    – Enforce the distances between semantically close pairs
    by another margin

    View Slide

  8. 8
    Corpus Relatedness Modeling (CRM)

    Mohammad et al., (2013) proposed a degree of
    contrast hypothesis:
    – “if a pair of words, A and B, are contrasting, then their
    degree of contrast is proportional to their tendency to
    co-occur in a large corpus.”

    Use the skip-gram model (Mikolov et al., 2013) to
    learn the relatedness embedding

    View Slide

  9. 9
    Semantic Differential Reconstruction

    Using factor analysis, Osgood et al.,(1957) identified
    three dimensions of semantic that account for
    most of the variation in the connotative meaning
    of adjective.

    Three dimensions
    – Evaluative: good-bad
    – Potency: strong-weak
    – Activity: active-passive

    View Slide

  10. 10
    Experiment Set-Up

    – Used the “most contrasting word” questions collected by
    Mohammad et al., (2008)

    Lexical Resources
    – WordNet (Miller, 1995) version 3.0
    – Roget's Thesaurus (Kipfer, 2009)

    Google Billion-Word Corpus

    Evaluation Metric

    View Slide

  11. 11
    Experiment Results

    The result is presented in Table 1 and the
    F-score on test set is 91%

    The pie graph in Figure 2 shows the percentages of
    target-gold-answer word pairs

    Figure 3 draws the envelope of histogram of cosine
    distance between all target-choice

    The performances of different models are show in
    Figure 4

    View Slide

  12. 12

    View Slide

  13. 13

    View Slide

  14. 14

    View Slide

  15. 15

    View Slide