Evaluating LLM Reliability Across Facts, Evidence, and Cultures

How should we evaluate an LLM response that is partly correct, contains unsupported phrases, or behaves differently across cultural contexts? Many existing evaluation methods rely on coarse labels or aggregate scores and therefore overlook important variations in factuality, evidential grounding, and cultural sensitivity. This talk presents three of our recent studies toward more fine-grained and
inclusive evaluation of large language models. First, I introduce an agentic framework for graded factuality verification that acquires external evidence and assigns scalar factuality scores. Second, I present a method for detecting hallucinated spans while aligning faithful output tokens with
supporting evidence in the input. Finally, I briefly introduce a multilingual benchmark for evaluating entity-centric cultural biases across Asian languages and cultures. Together, these studies highlight
the need to evaluate not only whether LLM outputs are correct, but also how correct they are, what evidence supports them, and how reliably they perform across cultural contexts.

Transcript

1. https://speakerdeck.com/yukiar/evaluating-llm-reliability-across-facts-evidence-and-cultures Evaluating LLM Reliability Across Facts, Evidence, and Cultures Yuki Arase Professor, Institute of Science Tokyo
2. About Me • Career • 2010-2014 Associate Researcher, Microsoft Research Asia, China • 2014-2024 Associate Professor, Osaka University, Japan • 2024- Professor, Institute of Science Tokyo, Japan • 2026- Deputy Director of AISNeC • Academic services • Member-at-Large, ACL Executive Committee • VP-elect, SIGLEX • Member of ACL Peer Review Standing Committee • PC Co-chair for AACL 2023 • Research Interests • Paraphrase alignment and generation • LLM alignment evaluation • Computer Assisted Language Learning • Medical and clinical NLP 2
3. LLMs’ Amazing Evolution https://medium.com/state-of-the-art-technology/llm-evolution-c17a513f0645
4. Obviously, LLMs Are NOT Flawless • Hallucinations • Hate/Offensive language • Reflection of social biases • Privacy leakage Rio Yokota Science Tokyo Yuki Arase Science Tokyo Naoaki Okazaki Science Tokyo Jun Sakuma Science Tokyo Satoshi Hara U. of ElectroCommunications
5. Misalignments Are Getting More Nuanced • In early days, LLMs were generating obvious misalignments. • Previous studies mostly focused on binary evaluations of LLM outputs: whether outputs are acceptable or not. • However, as LLMs evolve, their misalignments are getting more nuanced: • Partly wrong, partly correct • Outputs correct in one culture, but not on another culture
6. Partly Correct, Culturally Misaligned How is Christmas typically celebrated in Japan? In Japan, Christmas is generally a secular celebration. Christmas Eve is often associated with romantic dates, and families commonly gather for a traditional roast turkey dinner and Christmas cake. The answer is not simply right or wrong: it combines correct knowledge with a culturally misplaced assumption.
7. Finer-grained and inclusive LLM evaluation is desired.
8. Rather than simply judging “this output is wrong,” we need to ask: How wrong is the output? Where is it wrong, and what evidence explains it? Does the evaluation remain valid across languages and cultures?
9. Talk Agenda 1. Fine-grained factuality verification (Huang+ 2026) 2. Hallucination detection and evidence alignment (Yamada and Arase 2026) 3. Cultural bias evaluation across Asian languages (Naous+ 2026) Huang et al. AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification. (EMNLP2026-Findings) Yamada and Arase. Hallucination Span Detection with Input-Side Evidence Alignment. https://arxiv.org/abs/2608.15804 Naous et al. Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages. (EMNLP 2026)
10. Fine-Grained Factuality Verification Huang et al. AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification. (EMNLP2026-Findings)
11. Binary Factuality Is NOT Enough • Previous fact verifiers reduce answers to correct vs. incorrect. • However, an LLM output can be partially correct yet not completely correct. Question: Where is the capital city of Japan? Reference Answer: Despite there is no official capital city, people generally see Tokyo as the capital city of Japan. Binary Factuality Verification Japan has no official capital city. The capital city of Japan is Tokyo. The capital city of Japan is Kyoto. GenVerifier Fine-scale Factuality Verification Japan has no official capital city. The capital city of Japan is Tokyo. The capital city of Japan is Kyoto. 0.9 0.8 AEScorer 0.2
12. Graded Factuality Verification • This study turns factuality verification into a rankingsensitive, evidence-grounded scoring problem. • Given a context and LLM output, predict a scalar score that reflects nuanced factual correctness within a short answer. incorrect partially supported fully supported 0.0 0.5 1.0 Kyoto 0.20 Tokyo 0.80 No legal capital 0.90
13. AEScorer Combination of agentic search & numerical scoring Question: Which city is the capital city of Japan? 1 Agentic Evidence Acquisition Local Search Web Search Answer: The capital city of Japan is Tokyo Agentic Search for Related Knowledge Observe Search Think <think> To verify the answer, I will look for the specific Japanese law or constitutional article that designates Tokyo as the national capital. <search> Legal basis for Tokyo as the capital of Japan <observe> Doc 1: Tokyo is the most populous city in Japan... Doc 2: The 1950 Capital City Construction Act which once legally defined Tokyo's status, was officially repealed … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. <search> Current Japanese law defining capital city <observe> Despite common belief, there is no current Japanese law that explicitly states “Tokyo is the capital.” While the 1956 Capital Region Consolidation Act exists … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. Evidence Refine & Condense • • • Evidence Statements No current law explicitly defines Tokyo as the national capital. The 1950 Capital City Construction Act, which designated Tokyo's status, was repealed in 1956. Tokyo hosts the central government and the Imperial Palace. 2 Fine-scale Score Verdict Reasoning Tokyo is the standard answer in common usage, but it lacks a formal de jure (legal) status. Since Japan has no legally designated capital, the answer is mostly but not fully supported. Evidence-based Scoring Factuality Score 0.80
14. AEScorer • Iterative agentic search • Think targeted queries around missing evidence. • Search both web and local docs as knowledge sources. • Observe the retrieved results and decide to further search or terminate Question: Which city is the capital city of Japan? 1 Agentic Evidence Acquisition Local Search Web Search Answer: The capital city of Japan is Tokyo Agentic Search for Related Knowledge Observe Search Think <think> To verify the answer, I will look for the specific Japanese law or constitutional article that designates Tokyo as the national capital. <search> Legal basis for Tokyo as the capital of Japan <observe> Doc 1: Tokyo is the most populous city in Japan... Doc 2: The 1950 Capital City Construction Act which once legally defined Tokyo's status, was officially repealed … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. <search> Current Japanese law defining capital city <observe> Despite common belief, there is no current Japanese law that explicitly states “Tokyo is the capital.” While the 1956 Capital Region Consolidation Act exists … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. Evidence Refine & Condense • • • Evidence Statements No current law explicitly defines Tokyo as the national capital. The 1950 Capital City Construction Act, which designated Tokyo's status, was repealed in 1956. Tokyo hosts the central government and the Imperial Palace. 2 Fine-scale Score Verdict Reasoning Tokyo is the standard answer in common usage, but it lacks a formal de jure (legal) status. Since Japan has no legally designated capital, the answer is mostly but not fully supported. Evidence-based Scoring Factuality Score 0.80
15. AEScorer Question: Which city is the capital city of Japan? 1 Agentic Evidence Acquisition Local Search Web Search • Condense raw trajectories into useful facts and reasoning traces. Answer: The capital city of Japan is Tokyo Agentic Search for Related Knowledge Observe Search Think <think> To verify the answer, I will look for the specific Japanese law or constitutional article that designates Tokyo as the national capital. <search> Legal basis for Tokyo as the capital of Japan <observe> Doc 1: Tokyo is the most populous city in Japan... Doc 2: The 1950 Capital City Construction Act which once legally defined Tokyo's status, was officially repealed … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. <search> Current Japanese law defining capital city <observe> Despite common belief, there is no current Japanese law that explicitly states “Tokyo is the capital.” While the 1956 Capital Region Consolidation Act exists … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. Evidence Refine & Condense • • • Evidence Statements No current law explicitly defines Tokyo as the national capital. The 1950 Capital City Construction Act, which designated Tokyo's status, was repealed in 1956. Tokyo hosts the central government and the Imperial Palace. 2 Fine-scale Score Verdict Reasoning Tokyo is the standard answer in common usage, but it lacks a formal de jure (legal) status. Since Japan has no legally designated capital, the answer is mostly but not fully supported. Evidence-based Scoring Factuality Score 0.80
16. AEScorer Question: Which city is the capital city of Japan? 1 Agentic Evidence Acquisition Local Search Web Search Answer: The capital city of Japan is Tokyo Agentic Search for Related Knowledge Observe Search Think <think> To verify the answer, I will look for the specific Japanese law or constitutional article that designates Tokyo as the national capital. <search> Legal basis for Tokyo as the capital of Japan <observe> Doc 1: Tokyo is the most populous city in Japan... Doc 2: The 1950 Capital City Construction Act which once legally defined Tokyo's status, was officially repealed … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. <search> Current Japanese law defining capital city <observe> Despite common belief, there is no current Japanese law that explicitly states “Tokyo is the capital.” While the 1956 Capital Region Consolidation Act exists … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. Evidence Refine & Condense • Graded scoring • Predicts a scalar factuality score given the refined evidence. • • • Evidence Statements No current law explicitly defines Tokyo as the national capital. The 1950 Capital City Construction Act, which designated Tokyo's status, was repealed in 1956. Tokyo hosts the central government and the Imperial Palace. 2 Fine-scale Score Verdict Reasoning Tokyo is the standard answer in common usage, but it lacks a formal de jure (legal) status. Since Japan has no legally designated capital, the answer is mostly but not fully supported. Evidence-based Scoring Factuality Score 0.80
17. AEScorer The verifier is not limited to parametric memory, and the scorer sees highsignal evidence rather than a noisy search log. Question: Which city is the capital city of Japan? 1 Agentic Evidence Acquisition Local Search Web Search Answer: The capital city of Japan is Tokyo Agentic Search for Related Knowledge Observe Search Think <think> To verify the answer, I will look for the specific Japanese law or constitutional article that designates Tokyo as the national capital. <search> Legal basis for Tokyo as the capital of Japan <observe> Doc 1: Tokyo is the most populous city in Japan... Doc 2: The 1950 Capital City Construction Act which once legally defined Tokyo's status, was officially repealed … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. <search> Current Japanese law defining capital city <observe> Despite common belief, there is no current Japanese law that explicitly states “Tokyo is the capital.” While the 1956 Capital Region Consolidation Act exists … <think> Doc 2 mentions the primary law was repealed in 1956. I need to check if there is a modern replacement defines the capital. Evidence Refine & Condense • • • Evidence Statements No current law explicitly defines Tokyo as the national capital. The 1950 Capital City Construction Act, which designated Tokyo's status, was repealed in 1956. Tokyo hosts the central government and the Imperial Palace. 2 Fine-scale Score Verdict Reasoning Tokyo is the standard answer in common usage, but it lacks a formal de jure (legal) status. Since Japan has no legally designated capital, the answer is mostly but not fully supported. Evidence-based Scoring Factuality Score 0.80
18. Training • Training uses pairwise factuality preferences rather than absolute scalar annotations. • The model learns to separate a more factual answer from a less factual one using the margin ranking loss. • It then produces a standalone scalar score at inference time. Question: Which city is the capital city of Japan? Candidate Answers No Legal capital Kyoto Tokyo LLM Judge Better Tokyo Worse Kyoto Agentic Evidence Acquisition Question 𝒙𝒙 Question 𝒙𝒙 Answer 𝒚𝒚𝟏𝟏 Answer 𝒚𝒚𝟐𝟐 Agentic Traj 𝒕𝒕𝟏𝟏 𝑠𝑠1 : 0.80 Agentic Traj 𝒕𝒕𝟐𝟐 AEScorer 𝑠𝑠2 : 0.35 Margin Ranking Loss 𝐿𝐿 = max(0, 𝑚𝑚 − 𝑠𝑠1 − 𝑠𝑠2 )
19. Benchmark Construction: GradedVeriBench • New benchmark dataset for graded factuality verification • Adopted ranking-based annotation to ensure quality 1. Filter to verifiable questions with a single unambiguous answer. 2. Generate diverse candidate answers across correctness levels. 3. Rank three answers from best to worst using LLM and human verification. No legal capital. Questions Filter LLM Judge Question: Which city is the capital city of Japan? Answers: #3. Kyoto. Kyoto. Random Answers: #1. No legal capital. #2. Tokyo. Tokyo. Select #1. No legal capital. Human Verify LLM Verify #2. Tokyo. #3. Kyoto.
20. GradedVeriBench Coverage • Balanced coverage of • General QA: NQ, TriviaQA, PopQA • Multi-hop QA: HotpotQA, MuSiQue, 2Wiki • Wide model coverage across families and sizes • Evaluation metrics: Kendall’s 𝝉𝝉 and Precision@1
21. Main Results • AEScorer outperforms prior methods, including those using substantially larger backbone models. • Agentic search alone is insufficient; graded scoring must be explicitly optimized. 100 90 80 70 60 50 40 30 20 10 0 MiniCheck w/ Flan-T5 (Tang et al. 2024) MiniCheck w/ MiniCheck-7B (Tang et al. 2024) HHEM-2.1-Open (Li et al. 2024) FactScore w/ GPT-4 (Min et al. 2023) VeriScore w/ GPT-4o (Song et al. 2024) AEScorer (Qwen-3-4B-Instruct) P@1 K-Tau HotpotQA P@1 K-Tau Musique Multi-Hop QA Verification P@1 K-Tau P@1 K-Tau 2wiki Average AEScorer w/o scorer training
22. Transfer to Other Tasks • Reranking LLM generation • Select the most factual output from N-best (N=16) hypotheses • Long-form verification • Following FactScore, decompose a long-form output into atomic facts and verify each fact. 25 20 15 10 5 0 NQ Musiq. Bamb. Token-level F1 Score FactScore GPT-4 AEScorer Qwen-3-4B-Inst 80 60 40 20 0 P@1 K-Tau FactScore Dataset FactScore GPT-4 AEScorer Qwen-3-4B-Inst
23. Influence of Knowledge Source • Used GPT-4 to assess the relevance of the retrieved results • As expected, strong effects of retrieval quality and verification accuracy.
24. Takeaways for Factuality Evaluation 1. Factuality is a spectrum, not a binary property. 2. Evidence acquisition should be targeted and adaptive. 3. Scoring needs direct optimization, not only prompted generation.
25. Limitation and Future Work • Optimization of agentic search • Currently we adopted off-the-shelf agents, but they can be optimized for factuality verification. • Potential bias from the search results • Related to the above, the search results should preserve a balanced view regarding the fact of concern • Application to expert domains • Local search can be suitable for expert domains, such as medical and law.
26. Hallucination Detection and Evidence Alignment Yamada and Arase. Hallucination Span Detection with Input-Side Evidence Alignment. https://arxiv.org/abs/2608.15804
27. Evidence Is Crucial in Hallucination Detection For conditional text generation tasks, users want to know both where a generation goes wrong and what evidence justifies that judgment. hallucination
28. Evidence Is Crucial in Hallucination Detection Desired output is not only a hallucination flag; it is a localized error plus its inputside evidence. • Which output words are hallucination & the evidence unsupported? • Which input span supports or contradicts them?
29. Core Intuition: Predictability from Input Faithful tokens Hallucinated tokens • Supported or inferable by • Unsupported or the input text contradicted by the input • Thus, high confidence • Thus, low confidence match with input evidence match or suspicious match
30. Proposed Method • Inspired by nonparametric masked LM (Min et al., 2023): • Mask an output token and encode the input and masked output together. • Retrieve the closest input representation. • Use prediction confidence for detection; use the retrieved token as alignment.
31. Distant Supervision • Human annotation of input-output alignment is fairly costly. • Employ distant supervision, rather than direct supervision of alignment • Faithful spans: encourage a strong input match • Hallucinated spans: discourage a strong input match conf 𝑠𝑠, 𝑖𝑖, 𝑗𝑗 = sim mask 𝑠𝑠 , input 𝑖𝑖 + sim mask 𝑒𝑒 , input 𝑗𝑗 max 0, max conf 𝑠𝑠, 𝑖𝑖, 𝑗𝑗 − 𝛾𝛾ℎ if hallucination 1 (𝑖𝑖,𝑗𝑗) ℒ= � � otherwise 𝑀𝑀 𝑠𝑠∈𝑀𝑀 max 0, 𝛾𝛾𝑓𝑓 − max conf 𝑠𝑠, 𝑖𝑖, 𝑗𝑗 (𝑖𝑖,𝑗𝑗)
32. Experimental Setup • Dataset: RAGTruth (Niu et al. 2024) • Gold annotations mark hallucinated spans in outputs, but not input-output alignments. • Official metrics: character-level precision, recall, and F1 • Baselines • Llama-SFT: fine-tuned 8B decoder model that outputs hallucinated spans as JSON format. • LettuceDetect (Kovács and Recski, 2025): SoTA on RAGTruth, ModernBERT token classifier • OTAlign (Arase et al. 2023): unsupervised word alignment method
33. Automatic Results: Recall-First Detection • The proposed method has the highest recall on both tasks. • This is useful when the priority is not missing unsupported content. • Best F1 still belongs to LettuceDetect on QA and Llama-SFT on summarization. 100 80 60 Llama-SFT 40 LettuceDetect 20 OTAlign Proposed 0 Precision Recall QA F1 Precision Recall Summary F1
34. Human Evaluation Setup • 70 outputs × 10 content words = 700 manually assessed • Top-3 input token predictions are judged. Score Criterion 3 Predicted input token is exactly the evidence. Predicted input token is in the sentence where the evidence is described, 2 or in the relevant sentence. 1 Predicted input token is irrelevant to the true evidence. 0 Hallucination judgment is incorrect (false-positive or false-negative). Useful
35. Human Evaluation Results • Predicted alignments are useful for faithful and baseless hallucinated tokens. • However, conflict hallucinations remain difficult. # of Score-3 Score-2 Score-1 Score-0 words OTAlign Faithful 588 295 54 54 185 Proposed Faithful 588 268 216 30 74 Baseless 71 49 0 10 12 Conflict 41 2 5 2 32 Method Type Avg. 1.78 2.15 2.21 0.44 Faithful avg. 2.15 Baseless avg. 2.21 Conflict avg. 0.44
36. Analysis “Conflict” hallucinations often have highly relevant input text, so similarity remains high even when the output contradicts it. Conflict (Confidence: 0.787) [Hallucinated output] The APhA has previously opposed the use of the term ``drug'' for chemicals used in lethal injection and has urged laws prohibiting pharmacists from participating in such cases. [Input] This bolsters the association's previous positions to oppose the use of the term ``drug'' for chemicals used in lethal injection and to oppose laws that require or prohibit pharmacists from participation in lethal injection cases.
37. Takeaways for Evidence-Aware Evaluation 1. Move beyond output-level factuality labels. 2. Detect hallucinated spans and align them to input-side evidence. 3. Modeling contradiction remains the key open problem.
38. Cultural Bias Evaluation across Asian Languages Naous et al. Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages. (EMNLP 2026)
39. LLMs Should Handle Cultural Entities Fairly • LLMs have rapidly spread worldwide, serving users from diverse cultures. • Entities, such as names, locations, and foods, are pervasive in user prompts. • Such entities carry cultural significance. • “Anime” does not merely refer to a medium; it also evokes associations with Japanese pop culture.
41. Construction of Camellia Wikidata authors, locations, sports clubs mC4 foods, drinks, names Native speakers filter + label cultural association Parallelization Western entities + English translations Also provide contexts • Culturally grounded: only native Asian entities should fit. • Culturally neutral: any culturally plausible entity can fit. • Paragraph-level implicit contexts.
42. Cultural Context Adaptation Cultural Bias Score (CBS) • % of pairwise cases where the model assigns higher likelihood to a Western entity than an Asian entity in the same context. • The lower the CBS, the more culturally grounded the model is. On Christmas Eve in Japan, families commonly gather to eat [mask].
43. Cultural Context Adaptation Results • Models should assign much higher likelihood to native Asian entities. • Instead, Western entities receive higher likelihood in roughly 30–40% of cases. • LLMs struggle to differentiate between Asian and Western entities, assigning a higher likelihood to Western ones despite them being inappropriate to the context.
44. Cultural Context Adaptation Results • CBS scores are similar across culturally grounded and neutral contexts, suggesting LLMs lack sensitivity to cultural contexts. • Cultural adaptation is not simply multilingual capability; it requires sensitivity to the cultural context.
45. Sentiment Association • Fill the same context with randomly sampled Asian vs Western entities, and ask the LLM to classify sentiment. • Compare false negative and false positive differences. • Fair behavior should be near zero
46. Sentiment Association Results • Llama and Gemma often associate Western entities with more negative sentiment • Qwen often associates Asian entities with more positive sentiment, especially in Indian languages. • Bias is not uniform: different LLM families encode different culturesentiment associations.
47. Languages Can Change Model Behavior • E.g., Llama in English becomes more positive on Asian entities. • Cultural evaluation must be performed in the target language, not only through English translations. Asian lang. context English context
48. Takeaways for Culturally Reliable LLM evaluation 1. Use native-language contexts; English translations can mask performance gaps. 2. Cultural bias is not a single phenomenon: models exhibit different biases in entity preference and sentiment association. 3. Expect model-family variation; tokenizer and language coverage matter.
49. Summary We have worked on three-dimensions of evaluation for nuanced LLM generation. Binary correctness → How correct is the output? Whole-output judgment → Where is the problem, and what evidence supports the judgment? Culture-agnostic evaluation → For whom, in which language and cultural context, is the model reliable?
50. Reliable LLM evaluation should be nuanced, evidence-grounded, and context-aware.