Judgement Day: Benchmarking “Black Box” LLMs With Open Legal Datasets

Presented at the Linux Foundation Open Source Summit India (June 16, 2026) by Kannan Murugapandian.

This talk introduces Statutory Myopia — a newly documented failure mode in Retrieval-Augmented Generation (RAG) systems, where LLMs over-rely on retrieved text and suppress their own pre-trained knowledge of conflicting or superseding legal rules.

Using an open evaluation harness built on the LegalBench dataset (3 doctrinal task subsets, 239 unique cases, 4,302 reasoning traces), three frontier models — GPT-5, DeepSeek R1, and Gemini 2.5 Pro — were benchmarked across No-RAG and Statute-Only RAG conditions. The results were counterintuitive: incomplete retrieval caused a 9 percentage point drop in accuracy globally (0.87 → 0.78), with a worst-case 21 pp decline in the Hearsay task when the superseding Crawford v. Washington ruling was withheld from the context.

The talk also covers the custom async evaluation harness with SHA-256 cryptographic caching for reproducibility, the blacklist protocol for controlled retrieval experiments, and implications for any hierarchical domain — including medical guidelines, API deprecation, and compliance specifications.

Transcript

1. Judgement Day Benchmarking “Black Box” LLMs With Open Legal Datasets Kannan Murugapandian June 16, 2026 1 / 43
2. Session Roadmap 1 The Challenge 2 Background 3 Evaluation Harness 4 Experimental Design 5 Results 6 Implications 2 / 43
3. Conventional Method User LLM 3 / 43
4. Multi-LLM Council User LLM 1 LLM 2 LLM 3 LLM 4 LLM 5 Judge Response 4 / 43
5. The Challenge
6. The Challenge The Black Box Problem Proprietary models dominate professional domains No access to weights, architecture, or training data Cannot verify claims through code inspection Benchmarking becomes the proxy for transparency 6 / 43
7. The Challenge When Rules Override Other Rules Reasoning requires: Knowing which rule applies Knowing which rule overrides Applying the full authority chain Not just: Retrieving matching documents Simple Q&A Pattern matching The Gap What happens when retrieval surfaces the rule — but misses the override? 7 / 43
8. The Challenge Research Question Does RAG always improve reasoning in hierarchical domains? Common assumption: RAG improves accuracy by grounding in facts Reality: reasoning requires complete hierarchical context Gap: what happens when retrieval is incomplete? 8 / 43
9. The Challenge Key Contribution: Statutory Myopia Definition (Statutory Myopia) The tendency of LLMs to prioritize explicitly retrieved text over implicit pre-trained knowledge of conflicting or superseding rules. Finding: Incomplete retrieval harms performance more than no retrieval Magnitude: 9 percentage point decline (0.87 → 0.78) Worst case: 21 pp decline when a superseding rule is withheld 9 / 43
10. Background
11. Background Retrieval-Augmented Generation (RAG) Standard RAG Pipeline: 1 User submits query 2 System retrieves relevant documents 3 Documents injected into context 4 Model generates answer Assumed benefit: Reduced hallucinations Improved accuracy Verifiable sources Our Challenge This assumption fails when retrieved context is hierarchically incomplete 11 / 43
12. Background The Hierarchy Problem Authority Chain in Legal Domains 1 Constitution — supreme law, overrides everything below 2 Statutes — legislative rules 3 Case Law — judicial interpretations of statutes 4 Regulations — administrative rules Retrieving level 2 without level 3 doesn’t give you less context. It gives you wrong context. 12 / 43
13. Background Statutory Myopia: The Mechanism No-RAG 1 Input query taken 2 Persona prompt constructed 3 Retrieval skipped 4 Query passed directly to model 5 Model draws on full pre-trained knowledge Statute-Only RAG 1 Input query taken 2 Persona prompt constructed 3 Vector DB queried — override withheld 4 Query + partial docs passed to model 5 Attention concentrates on injected text Key Insight Injecting incomplete context suppresses correct pre-trained knowledge 13 / 43
14. Evaluation Harness
15. Evaluation Harness Evaluation Harness Custom Python asynchronous pipeline Standardizes prompts across model APIs Vector retrieval with blacklist protocol Cryptographic caching for reproducibility 4,302 reasoning traces evaluated 15 / 43
16. Evaluation Harness System Architecture Input Query RAG System Prompt Construction Models Evaluator Async execution — slow models don’t block fast ones SHA-256 caching ensures deterministic re-runs Exponential backoff on API failures 16 / 43
17. Evaluation Harness Asynchronous Execution Python asyncio event loop Semaphore caps concurrent requests at 5 Prevents head-of-line blocking Why This Matters DeepSeek R1 (heavy chain-of-thought) doesn’t stall GPT-5 or Gemini 2.5 Pro 17 / 43
18. Evaluation Harness Cryptographic Caching Unique signature per API request Cache hit returns stored response — no API call Fully deterministic, zero-cost re-runs Hreq = SHA256(Mid ∥Psys∥Pusr ∥Sseed ) Mid : model Psys : system prompt Pusr : user prompt Sseed = 42 18 / 43
19. Evaluation Harness Fault Tolerance Exponential backoff on 503 errors Retries until valid response received Production Reality Any benchmark running concurrent API calls needs this — instability is guaranteed at scale 19 / 43
20. Experimental Design
21. Experimental Design Dataset: LegalBench Open, collaboratively constructed benchmark 162 legal reasoning tasks with ground-truth labels Why LegalBench? Each task has a well-defined correct answer — making automated evaluation clean and reproducible We evaluate 3 doctrinal subsets across 239 unique cases 21 / 43
22. Experimental Design Three Tasks — Three Scenarios Task N What it tests Hearsay 94 Statute says yes; constitutional ruling says no. Which wins? Abercrombie 95 Pure definitions — no overriding rule exists Personal Jurisdiction 50 Single well-known test, baked into ev- ery model’s training Three tasks deliberately chosen to span the spectrum: conflict, no conflict, memorised. 22 / 43
23. Experimental Design Model Configuration Model Type Temp / Seed GPT-5 Unified multimodal 0.0 / 42 DeepSeek R1 RL reasoning 0.0 / 42 Gemini 2.5 Pro Extended thinking 0.0 / 42 Three different reasoning paradigms Fixed temperature and seed for reproducibility 23 / 43
24. Experimental Design The Blacklist Protocol Core experimental control Retrieval engine returns statutes — withholds the override Cretrieved = Topk (Q, D \ {d | dsource ∈ BlacklistQ}) Example Hearsay task: FRE 801–803 retrieved, Crawford v. Washington withheld 24 / 43
25. Experimental Design Prompt Engineering Structured JSON output schema — enforced across all trials Only the system prompt (persona) varies between conditions { "reasoning": "Step-by-step analysis...", "verdict": "Exact label from options" } 25 / 43
26. Experimental Design Interpretive Personas Textualist Follow the text strictly — no inferred intent Purposivist Consider the goal the rule was written to achieve Legal Realist Prioritise real-world outcome over formal rules The Question Does interpretive framing change how much the model trusts injected context? 26 / 43
27. Experimental Design Evaluation Protocol Layer 1: Strict string matching against ground-truth labels Layer 2: LLM judge for semantic equivalence (fallback) Result 100% strict-match coverage — Layer 2 never invoked across all 4,302 outputs 27 / 43
28. Results
29. Results Global Performance Impact No-RAG accuracy: 0.87 Statute-Only RAG accuracy: 0.78 9 percentage point decline across 4,302 trials The Finding More information. Worse results. Consistently. 29 / 43
30. Results Model-Specific Vulnerabilities DeepSeek R1: −9 pp (0.85 → 0.76) GPT-5: −8 pp (0.86 → 0.78) Gemini 2.5 Pro: −6 pp (0.81 → 0.75) Implication Statutory Myopia is model-agnostic — it is a property of the RAG setup, not the model 30 / 43
31. Results Persona Sensitivity Textualist — largest drops: up to −12.1 pp (DeepSeek R1) Purposivist — moderate drops: −5.9 to −7.9 pp Legal Realist — most resilient: as low as −3.8 pp (Gemini) Takeaway Instruct a model to privilege explicit text — it trusts incomplete context more aggressively. Your system prompt is not neutral. 31 / 43
32. Results Domain-Specific Results Hearsay: −21 pp (0.77 → 0.56) — override withheld; largest drop Abercrombie: 0 pp change — no override exists; statute was sufficient Personal Jurisdiction: −2 pp — override so well-known retrieval couldn’t suppress it Rule of Thumb RAG risk scales with how much a correct answer depends on an override the retrieval missed 32 / 43
33. Results The Crawford Trap What was retrieved: FRE 803 — excited utterances are admissible What was withheld: Crawford v. Washington — testimonial statements are barred by the Sixth Amendment No-RAG: Model correctly applies Crawford from memory Statute-Only RAG: Model reads FRE 803, rules admissible, stops there 33 / 43
34. Results Manual Audit of Failures 25 Hearsay failure cases audited manually: 60% — Trap cases: Retrieved text was relevant but incomplete without the override 40% — Distractor cases: Retrieved text was not sufficiently relevant Key Insight On-topic retrieval can still mislead — relevance alone does not guarantee correctness 34 / 43
35. Results Mechanism: Contextual Overwriting No-RAG Distributed representation Rules and overrides coexist Full hierarchy applied Statute-Only RAG Attention concentrates on injected text Override absent from context Correct knowledge suppressed Partial context can be worse than no context. 35 / 43
36. Results Why Incomplete Retrieval > No Retrieval No-RAG: model accesses its full pre-trained knowledge Partial RAG: injected text concentrates attention, suppressing correct internal knowledge The model doesn’t know what it’s missing The Danger Incomplete context creates false confidence — the model answers wrongly, but assertively 36 / 43
37. Implications
38. Implications Implications for Any Hierarchical Domain Retrieval design must respect your domain’s structure — not just semantic similarity Graph-structured knowledge bases — link rules to their overrides explicitly Verify completeness before injection — does the retrieved set include all controlling authority? Applies Beyond Law Medical guidelines. API deprecation. Compliance specs. Any domain where one rule overrides another. 38 / 43
39. Implications Limitations Artificial worst-case blacklist Only 3 of 162 tasks No formal statistical testing Temperature fixed at 0.0 Case law encoding inferred, not verified Personas not externally validated 39 / 43
40. Implications Future Directions All 162 LegalBench tasks Pure statutory domains and common-law reasoning Graph-structured knowledge bases with hierarchy verification Formal hypothesis testing with confidence intervals Open to the Community Harness is documented. Dataset is public. Methodology is transferable to any domain. 40 / 43
41. Conclusion
42. Implications Conclusion Statutory Myopia is real and measurable across three frontier models Incomplete RAG degrades performance by 9 pp globally; 21 pp worst case Chain-of-thought does not confer immunity System prompt framing amplifies or dampens the effect “Incomplete retrieval may harm reasoning more than no retrieval at all” 42 / 43
43. Kannan Murugapandian Thank You DOI: 10.5281/zenodo.19522950