Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set

Yuki Muramatsu, Kunihiro Udaka, and Kazuhide Yamamoto. Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set. Proceedings of The Second Workshop on Cognitive Aspects of the Lexicon: Enhancing the Structure and Lookup Mechanisms of Electronic Dictionaries (COGALEX-II), pp.18-27 (2010.8)

Transcript

1. Textual Entailment Recognition using Word Overlap, Mutual Information and Subpath Set Yuki Muramatsu, Kunihiro Udaka and Kazuhide Yamamoto Nagaoka University of Technology (Japan)
2. Back Ground Textual Entailment Recognition (RTE) T: Google files for its long-awaited IPO. H: Google goes public. Entailment Judgment: True. Such the RTE task will contribute to paraphrasing, summarization, question answering and machine translation. 1/21
3. Our Goal We propose the method of new RTE. Our new RTE method was devised based on a past RTE workshop. We will contribute for NLP applications. The different expressions of the same content are merged by RTE. Solving the problem of RTE advances the research of the semantic analysis. 2/21
4. Relevant Study The PASCAL Recognizing Textual Entailment Challenge (Dagan et al, 2005) Proposing how to build evaluation data Introduced several RTE methods. Learning Textual Entailment using SVMs and String Similarity Measures (Prodromos Malakasiotis and Ion Androutsopoulos, 2007) Similarity of words, POS tags and chunk tags approximately 62%. 3/21
5. Our Method (New RTE system) Training Data Word Overlap Mutual Information Subpath Set SVM True False T:xxxxx H:yyyyy SVM features Resource Processing 4/21
6. Our Method (New RTE system) Training Data Word Overlap Mutual Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 5/21 SVM features
7. Training Data Text Entailment and recognition of inferencing relation based on automatic achieved similar expression (Odani et al., 2008) Training data is generally available to the public at the Kyoto University http://www.nlp.kuee.kyoto-u.ac.jp/nl-resource The number of evaluation data open to the public now stands at 2471. 6/21
8. Training Data 7/21 T: Toyota open a luxury car shop. H: Lexus is a luxury car. Entailment Judgment: Talw Talw : When the text is true, the hypothesis is always true. Talw : When the text is true, the hypothesis is almost true. Fmay : When the text is true, the hypothesis may be true. Falw : When the text is true, the hypothesis is false.
9. Our Method (New RTE system) Training Data Word Overlap Mutual Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 8/21 SVM features
10. Word Overlap Application of the Bleu algorithm for recognizing textual entailment. ( Pérez and Alfonseca., 2005) The concordance rate of the text and the hypothesis was then calculated for judging the text and the hypothesis of the inclusion relation. Their accuracy was about 50%. 9/21
11. Word Overlap ( ) 1 ( , ) exp( log( ) / ) 1 {1, / } n i i Bleu A B BP p n BP exp max r c = = = − ∑ A,B : sentence. pi : the matching rate of n-gram. c : length of the sentence A r : length of the sentence B n : n-gram. 10/21
12. Our Method (New RTE system) Training Data Word Overlap Mutual Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 11/21 SVM features
13. Mutual Information Web Based Probabilistic Textual Entailment (Glickman et al., 2005) They assumed that the entailment judgment was ‘true’ when the probability of co-occurrence between the text and the hypothesis was high. Their accuracy was about 58%. 12/21
14. Mutual Information , 1 ( 1| ) max ( , ) ( ) ( , ) log ( ) ( ) u h v t u v u v P Trh t lep u v p n lep u v p n p n ∈ ∈ = = ∏ ≈ − ⋅ u u : hypothesis word, v : text word. u : the number of words in hypothesis. P(nu ) : probability of word u P(nv ) : probability of word v P(nu,v ) : co-occurrence probability of word u and v 13/21
15. Our Method (New RTE system) Training Data Word Overlap Mutual Information Subpath Set SVM True False T:xxxxx H:yyyyy Resource Processing 14/21 SVM features
16. Subpath Set Textual Entailment Recognition Based on Dependency Analysis and WordNet. (Herrera et al., 2005) They assumed that the entailment judgment was ‘true’ when the syntactic similarity of the text and the hypothesis was high. Their accuracy was about 57%. 15/21
17. Subpath Set 16/21 New methods to retrieve sentences based on syntactic similarity. (Ichikawa et al., 2005) Syntactic similarity was calculates partial routes from the root to the leaf of the syntax tree. We used the Japanese version of WordNet (Bond et al., 2009), in which a word with a different surface can be treated as the same expressions.
18. Subpath Set 17/21 Sim(T1,T2)=15/27 (about 55%) (i) (a,g),(g,j) (a,g,i),(e,g,j) (b,e,g,j) (a,b,e,g,j) (a), (b), (d), (e), (g), (i), (a,b), (b,d), (b,e), (e,g), (g,i) (a,b,d),(a,b,e),(b,e,g) (a,b,e,g) Partial route of T1,T2 (c) (a,c) (e,g,i) (b,e,g,i) (a,b,e,g,i) Common route of T1,T2
19. Evaluation The evaluation method used was a Confidence-Weight Score (Dagan et al., 2005) Open test 10-fold cross-validations Training Data True Data Talw :924,Talm : 662 False Data Fmay :262 ,Falw :624 18/21
20. Results of RTE experiments (Open Test) 19/21 64.10% 61.90% 49.90% SVM 61.10% 59.70% 45.00% Subpath Set 67.40% 55.60% 53.40% Mutual Informaition 59.30% 60.20% 39.00% Word Overlap Talw and Talm Talm Talw CWS Talw : Hypothesis is always true. Talm : Hypothesis is almost true.
21. Discussion (Closed Test of Subpath Set) 20/21 20 25 30 35 40 45 50 55 60 65 70 0 0.2 0.4 0.6 0.8 1 Subpath Set CWS[%] Talw Talm Talw and Talm
22. Conclusion We built a Japanese textual entailment recognition system based on the past methods of RTE. Matching rate of the words Mutual Information Similarity of the syntax tree The method of using mutual information and the use of three methods of SVM turned out to be effective. 21/21
23. Thank you for your attention.