Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set

Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set

Yuki Muramatsu, Kunihiro Udaka, and Kazuhide Yamamoto. Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set. Proceedings of The Second Workshop on Cognitive Aspects of the Lexicon: Enhancing the Structure and Lookup Mechanisms of Electronic Dictionaries (COGALEX-II), pp.18-27 (2010.8)

C04e17d9b3810e5c0ad22cb8a12589de?s=128

自然言語処理研究室

August 31, 2010
Tweet

Transcript

  1. Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath

    Set Yuki Muramatsu, Kunihiro Udaka and Kazuhide Yamamoto Nagaoka University of Technology (Japan)
  2. Back Ground Textual Entailment Recognition (RTE) T: Google files for

    its long-awaited IPO. H: Google goes public. Entailment Judgment: True. Such the RTE task will contribute to paraphrasing, summarization, question answering and machine translation. 1/21
  3. Our Goal We propose the method of new RTE. Our

    new RTE method was devised based on a past RTE workshop. We will contribute for NLP applications. The different expressions of the same content are merged by RTE. Solving the problem of RTE advances the research of the semantic analysis. 2/21
  4. Relevant Study The PASCAL Recognizing Textual Entailment Challenge (Dagan et

    al, 2005) Proposing how to build evaluation data Introduced several RTE methods. Learning Textual Entailment using SVMs and String Similarity Measures (Prodromos Malakasiotis and Ion Androutsopoulos, 2007) Similarity of words, POS tags and chunk tags approximately 62%. 3/21
  5. Our Method (New RTE system) Training Data Word Overlap Mutual

    Information Subpath Set SVM True False T:xxxxx H:yyyyy SVM features Resource Processing 4/21
  6. Our Method (New RTE system) Training Data Word Overlap Mutual

    Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 5/21 SVM features
  7. Training Data Text Entailment and recognition of inferencing relation based

    on automatic achieved similar expression (Odani et al., 2008) Training data is generally available to the public at the Kyoto University http://www.nlp.kuee.kyoto-u.ac.jp/nl-resource The number of evaluation data open to the public now stands at 2471. 6/21
  8. Training Data 7/21 T: Toyota open a luxury car shop.

    H: Lexus is a luxury car. Entailment Judgment: Talw Talw : When the text is true, the hypothesis is always true. Talw : When the text is true, the hypothesis is almost true. Fmay : When the text is true, the hypothesis may be true. Falw : When the text is true, the hypothesis is false.
  9. Our Method (New RTE system) Training Data Word Overlap Mutual

    Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 8/21 SVM features
  10. Word Overlap Application of the Bleu algorithm for recognizing textual

    entailment. ( Pérez and Alfonseca., 2005) The concordance rate of the text and the hypothesis was then calculated for judging the text and the hypothesis of the inclusion relation. Their accuracy was about 50%. 9/21
  11. Word Overlap ( ) 1 ( , ) exp( log(

    ) / ) 1 {1, / } n i i Bleu A B BP p n BP exp max r c = = = − ∑ A,B : sentence. pi : the matching rate of n-gram. c : length of the sentence A r : length of the sentence B n : n-gram. 10/21
  12. Our Method (New RTE system) Training Data Word Overlap Mutual

    Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 11/21 SVM features
  13. Mutual Information Web Based Probabilistic Textual Entailment (Glickman et al.,

    2005) They assumed that the entailment judgment was ‘true’ when the probability of co-occurrence between the text and the hypothesis was high. Their accuracy was about 58%. 12/21
  14. Mutual Information , 1 ( 1| ) max ( ,

    ) ( ) ( , ) log ( ) ( ) u h v t u v u v P Trh t lep u v p n lep u v p n p n ∈ ∈ = = ∏ ≈ − ⋅ u u : hypothesis word, v : text word. u : the number of words in hypothesis. P(nu ) : probability of word u P(nv ) : probability of word v P(nu,v ) : co-occurrence probability of word u and v 13/21
  15. Our Method (New RTE system) Training Data Word Overlap Mutual

    Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 14/21 SVM features
  16. Subpath Set Textual Entailment Recognition Based on Dependency Analysis and

    WordNet. (Herrera et al., 2005) They assumed that the entailment judgment was ‘true’ when the syntactic similarity of the text and the hypothesis was high. Their accuracy was about 57%. 15/21
  17. Subpath Set 16/21 New methods to retrieve sentences based on

    syntactic similarity. (Ichikawa et al., 2005) Syntactic similarity was calculates partial routes from the root to the leaf of the syntax tree. We used the Japanese version of WordNet (Bond et al., 2009), in which a word with a different surface can be treated as the same expressions.
  18. Subpath Set 17/21 Sim(T1,T2)=15/27 (about 55%) (i) (a,g),(g,j) (a,g,i),(e,g,j) (b,e,g,j)

    (a,b,e,g,j) (a), (b), (d), (e), (g), (i), (a,b), (b,d), (b,e), (e,g), (g,i) (a,b,d),(a,b,e),(b,e,g) (a,b,e,g) Partial route of T1,T2 (c) (a,c) (e,g,i) (b,e,g,i) (a,b,e,g,i) Common route of T1,T2
  19. Evaluation The evaluation method used was a Confidence-Weight Score (Dagan

    et al., 2005) Open test 10-fold cross-validations Training Data True Data Talw :924,Talm : 662 False Data Fmay :262 ,Falw :624 18/21
  20. Results of RTE experiments (Open Test) 19/21 64.10% 61.90% 49.90%

    SVM 61.10% 59.70% 45.00% Subpath Set 67.40% 55.60% 53.40% Mutual Informaition 59.30% 60.20% 39.00% Word Overlap Talw and Talm Talm Talw CWS Talw : Hypothesis is always true. Talm : Hypothesis is almost true.
  21. Discussion (Closed Test of Subpath Set) 20/21 20 25 30

    35 40 45 50 55 60 65 70 0 0.2 0.4 0.6 0.8 1 Subpath Set CWS[%] Talw Talm Talw and Talm
  22. Conclusion We built a Japanese textual entailment recognition system based

    on the past methods of RTE. Matching rate of the words Mutual Information Similarity of the syntax tree The method of using mutual information and the use of three methods of SVM turned out to be effective. 21/21
  23. Thank you for your attention.