Applying Relation Extraction and Graph Matching to Answering Multiple-Choice Questions

Presentation slide at NeLaMKRR 2025. Experimental codes and datasets are available.

Authors
N. Shimoda and A. Yamamoto (homepage)

Affiliation
Graduate School of Informatics, Kyoto University, Japan

Abstract
In this research, we combine Transformer-based relation extraction with matching of knowledge graphs (KGs) and apply them to answering multiple-choice questions (MCQs) while maintaining the traceability of the output process.

KGs are structured representations of factual knowledge consisting of entities and relations. Due to the high construction cost, they had been regarded as static databases with validated links. However, the recent development of Transformer-based relation extraction (RE) methods has enabled us to generate KGs dynamically by giving them natural language texts, and thereby opened the possibility for representing the meaning of the input sentences with the created KGs.

Using this effect, we propose a method that answers MCQs in the “fill-in-the-blank” format, taking care of the point that RE methods generate KGs that represent false information if provided with factually incorrect texts. We measure the truthfulness of each question sentence by (i) converting the sentence into a relational graph using an RE method and (ii) verifying it against factually correct KGs under the closed-world assumption.

The experimental results demonstrate that our method correctly answers up to around 70% of the questions, while providing traceability of the procedure. We also highlight that the question category has a vast influence on the accuracy.

Transcript

1. Graduate School of Informatics, Kyoto University *Naoki Shimoda, Akihiro Yamamoto NeLaMKRR 2025, 12th November Applying Relation Extraction and Graph Matching to Answering Multiple Choice Questions Contents • Abstract • Related Works • Method • Experiments • Conclusion
2. Traceability of Knowledge • LLM reasoning is still unreliable in some areas (ex. education) a. Knowledge used for generating responses are not analyzable. b. LLMs might provide incorrect clues for the utilized knowledge. • Our goal: Construct an inference mechanism whose use of knowledge is traceable. 2 Vincent van Gogh created a famous painting “The Starry Night”. Claim Background Knowledge Starry Night notable work Gogh
3. Options 𝐶 = {𝑐1, 𝑐2, 𝑐3 , 𝑐4 } 𝑐1: “Shohei Ohtani” 𝑐2: “Munetaka Murakami” 𝑐3: “Ichiro Suzuki” 𝑐4: “Mike Trout” Abstract Problem Select an option for a multiple-choice questions with reliability. Background Traceable answer generation is needed. - ex. Education Method Compute the likelihood of a sentence mcq 𝒄𝒊 for each choice 𝒄𝒊 using Knowledge Graphs Results Achieved 53.5% accuracy on curated dataset, while clearly visualizing the background knowledge as graphs 3 “{𝑥} is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB.” Question mcq 𝑥 Proposition mcq 𝑐1 {Shohei Ohtani} is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB.
4. Related Works Multiple-Choice Question Answering (MCQA) • Used for evaluation of language models (LLMs) [1] • Issue: Low interpretability of the knowledge used for generating the output. Fact Verification • A task of deciding whether a given sentence is factually correct or not. • Some works proposes to use KG for interpretability [2] We propose a method using KG to enhance traceability of MCQA. 4 [1] Hendrycks et al., “Measuring Massive Multitask Language Understanding” (2020) [2] Yuan et al., “Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs.” (2024) Shohei Ohtani plays for a Major League Baseball team in US. What‘s the evidence?
5. Method Input: MCQ in the “fill-in-the-blank” format • Question sentence with a blank 𝑥: mcq(𝑥) • Choice of 4 words 𝐶 = 𝑐1 , 𝑐2 , 𝑐3 , 𝑐4 ※ There is only one correct answer Output • The correct option: Ƹ 𝑐 • Explanation: A relational graph that visualizes the background knowledge for each decision 5 Choice 𝐶 = {𝑐1, 𝑐2, 𝑐3 , 𝑐4 } 𝑐1: “Shohei Ohtani” 𝑐2: “Munetaka Murakami” 𝑐3: “Ichiro Suzuki” 𝑐4: “Mike Trout” “{𝑥} is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB.” Question mcq 𝑥 The correct option: ො 𝒄, Explanation
6. Method Strategy 1. Convert the proposition mcq(𝒄𝒊 ) into a relational graph 2. Measure the truthfulness of the graph by comparing it with factually correct KGs. 6 Options 𝐶 = {𝑐1, 𝑐2, 𝑐3 , 𝑐4 } 𝑐1: “Shohei Ohtani” 𝑐2: “Munetaka Murakami” 𝑐3: “Ichiro Suzuki” 𝑐4: “Mike Trout” “{𝑥} is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB.” Question mcq 𝑥 Proposition mcq 𝒄𝟏 {Shohei Ohtani} is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB. Extract Relations Measure likelihood Background Knowledge
7. Method Compute the truthfulness of 𝐦cq(𝒄𝒊 ) Step 1: Create propositional and knowledge graphs Step 2: Estimate the corresponding nodes Step 3: Decide truthfulness of the proposition Relation Extraction [3] ➢ Convert natural language sentence s into a list of semantic triplets. ➢ We use REBEL and UniRel in this research. 7 [3] Zhao et al., “A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers.” (2024) ex. Semantic triplet MLB league Ohtani (subject, relation, object) mcq 𝑐𝑖 Shohei Ohtani is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB.
8. Step 1. Create Propositional & Knowledge Graphs 8 Propositional Graph (PG) • Relational graph to be verified • Converted from the proposition mcq(𝑐𝑖 ) Knowledge Graph (KG) • Represents the background knowledge • Created from related articles of Wikipedia Relation Extraction KG Shohei Ohtani is a Japanese baseball player, playing for the Los Angeles Dodgers in MLB. PG Relation extraction Generated from a Wikipedia article (https://en.wikipedia.org/wiki/Shohei_Ohtani) mcq 𝑐𝑖
9. Step 2. Estimate the Corresponding Nodes Estimate the node correspondence between PG and KG. ➢ Usually, node names fluctuate reflecting the original expression. ➢ ex. Major League Baseball → MLB, Barack Obama → Barack Hussein Obama 9 Propositional Graph Knowledge Graph “Major League Baseball” “MLB” (𝒏𝟏 , 𝒓𝒆𝒍, 𝒏𝟐 ) 𝒏𝟐 𝒓𝒆𝒍 𝒏𝟏
10. Step 2. Estimate the Corresponding Nodes Goals Maximize the sum of node similarity for corresponding nodes Similarity sim 𝑥, 𝑦 between two node labels 𝑥, 𝑦 is needed. Node similarity measure ➢ Sentence Transformer ➢ Convert a sentence with multiple words into an embedding vector. ➢ Define sim 𝑥, 𝑦 as the cosine similarity. 10 0.791 0.094 0.379 0.335 0.512 Ohtani Dodgers MLB baseball 𝑽𝑷 ′ Shohei Ohtani July 5, 1994 Major League Baseball Los Angeles Dodgers pitcher 𝑽𝑲 ′ ※ Figure ignores the edges from nodes other than “Ohtani”.
11. Step 2. Estimate the Corresponding Nodes Method See the problem as a weighted matching problem on a complete- bipartite graph • Set of PG and KG nodes: 𝑽𝑷 ′ , 𝑽𝑲 ′ • Injective node correspondence 𝜑′: 𝑽𝑷 ′ → 𝑽𝑲 ′ • A complete-bipartite graph with edge weight sim label 𝑣 , label 𝑣′ for each edge between node 𝒗 ∈ 𝑽𝑷 ′ and 𝒗′ ∈ 𝑽𝑲 ′ . 11 ො 𝜑′ = arg max 𝜑′ ෍ 𝑣∈𝑽𝑷 ′ sim label 𝑣 , label φ′ 𝑣 0.791 0.094 0.379 0.335 0.512 Ohtani Dodgers MLB baseball 𝑽𝑷 ′ Shohei Ohtani July 5, 1994 Major League Baseball Los Angeles Dodgers pitcher 𝑽𝑲 ′ ※ Figure ignores the edges from nodes other than “Ohtani”.
12. Step 3. Compute the Truthfulness of PG Define the truthfulness as the ratio of “verified” edges in PG. Likelihood of mcq 𝑐𝑖 is the ratio of edges (𝒏𝟏 , 𝒓𝒆𝒍, 𝒏𝟐 ) ∈ PG that also appear in KG Select the correct option as the one maximizes this value. ➢ Additionally, consider the average of node similarity if multiple 𝑐𝑖 maximize this value. ➢ If 𝑐𝑖 could not be narrowed down to one, we randomly select from the candidates. 12 Knowledge Graph Propositional Graph (𝒏𝟏 , 𝒓𝒆𝒍, 𝒏𝟐 ) 𝒏𝟐 𝒓𝒆𝒍 𝒏𝟏
13. Experiments 13 Datasets: KR-200m & KR-200s Created two original datasets with 10 categories using GPT-4o. 1. Art & Music 2. General Knowledge 3. Geography 4. History 5. Literature & Language 6. Mathematics 7. Philosophy & Logic 8. Pop Culture 9. Science 10. Technology & Computing • Each category has 20 questions (N=200). • Two datasets differ in the length of question sentence. ➢ KR-200m (long): 20.1 words ➢ KR-200s (short): 7.5 words
14. Experiments Relation Extraction (RE) Methods ➢ REBEL [4] ➢ Solve RE as a seq2seq generation. ➢ Variant models trained on different RE datasets: REBEL, mREBEL32 , and mREBEL400 ➢ UniRel [5] ➢ Solve RE as a relation classification on pairs of entities. 14 Model Training Dataset Number of Relation Types REBEL REBEL dataset 220 mREBEL400 REDFM 400 mREBEL32 SREDFM 32 UniRel NYT 24 [4] Cabot et al., “REBEL: Relation Extraction By End-to- End Language Generation.” (2021) [5] Tang et al., “UniRel: Unified Representation and Interaction for Joint Relational Triple Extraction.” (2022)
15. Results: Comparison of Accuracy REBEL and mREBEL400 achieved the highest accuracy. Variety of relation types benefits the performance. 15 RE methods KR-200m KR-200s Number of relation types REBEL 53.5 48.0 200 mREBEL400 52.2 49.7 400 mREBEL32 49.0 43.6 32 UniRel 28.6 26.9 24 Most RE methods performed better on KR-200m. Questions with longer sentences are easier to solve.
16. Results: Accuracy per Question Category Accuracy per category (KR-200m with REBEL) 16
17. Case Study of Traceability 17 Propositional Graph for the correct choice: “Starry Night” ✓ All relations were verified from the KG. Vincent van Gogh, a Dutch post- impressionist painter, created several masterpieces, including the famous painting called {x}. Question mcq 𝑥 Choice 𝐶 = {𝑐1 , 𝑐2 , 𝑐3 , 𝑐4 } 1. Starry Night 2. The Persistence of Memory 3. Guernica 4. The Scream A question about “Starry Night”, Art & Music Category of KR-200m
18. Case Study of Traceability 18 PG for the incorrect choice: “The Scream” Only one edge was verified. Vincent van Gogh, a Dutch post- impressionist painter, created several masterpieces, including the famous painting called {x}. Question mcq 𝑥 Choice 𝐶 = {𝑐1 , 𝑐2 , 𝑐3 , 𝑐4 } 1. Starry Night 2. The Persistence of Memory 3. Guernica 4. The Scream A question about “Starry Night”, Art & Music Category of KR-200m
19. Case Study of Traceability Visualization of the Knowledge Graph ✓ Edges used for verification are traceable. 19 Knowledge Graph
20. Conclusion Summary • Proposed a MCQA method using RE and graph matching. • Demonstrated the effectiveness in curated dataset, while keeping the output process traceable. Future Works ➢ Enhance the edge verification mechanism 20 Knowledge Graph Propositional Graph compare
21. References 1. Dan Hendrycks et al., “Measuring Massive Multitask Language Understanding,” paper presented at International Conference on Learning Representations, October 2, 2020, https://openreview.net/forum?id=d7KBjmI3GmQ. 2. Moy Yuan and Andreas Vlachos, “Zero-Shot Fact-Checking with Semantic Triples and Knowledge Graphs,” in Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), ed. Russa Biswas et al. (Association for Computational Linguistics, 2024), https://doi.org/10.18653/v1/2024.kallm-1.11. 3. Zhao, Xiaoyan, Yang Deng, Min Yang, et al. “A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers.” ACM Computing Surveys 56, no. 11 (2024): 1–39. https://doi.org/10.1145/3674501. 4. Pere-Lluís Huguet Cabot and Roberto Navigli, “REBEL: Relation Extraction By End-to-End Language Generation,” in Findings of the Association for Computational Linguistics: EMNLP 2021, ed. Marie-Francine Moens et al. (Association for Computational Linguistics, 2021), https://doi.org/10.18653/v1/2021.findings-emnlp.204. 5. Wei Tang et al., “UniRel: Unified Representation and Interaction for Joint Relational Triple Extraction,” Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2022, 7087–99, https://doi.org/10.18653/v1/2022.emnlp-main.477. 21