A Systematic Comparison of RAG Architectures for Geographic POI Question Answering

本研究は、地理的関心地点（POI）クエリに対する最適なRAGアーキテクチャを明示するため、複数のRAGアーキテクチャ（Structured、Graph、Hybrid、Adaptive、Agentic）を同一条件で比較評価したものです。東京の約3,600件のOSMデータと、5段階の階層的なプロンプトのテストケースを用いてルールベースで評価を行いました。

比較の結果、Structured RAGが全体精度89.1%を記録し、GraphRAGやAdaptive RAGを凌駕しました。座標に基づく決定論的な空間計算が、グラフ構造よりも正確な距離・方向推論を可能にすることを示唆しています。

Transcript

1. A Systematic Comparison of RAG Architectures for Geographic POI Question Answering Using OpenStreetMap Data This study compares five RAG architectures for geographic POI questions. Noboru Otsuka · Geolonia Inc. FOSS4G 2026, Hiroshima · September 1–3, 2026 RAG architectures for geographic POI QA · FOSS4G 2026 1
2. Geographic questions need spatial calculations Geographic questions often need explicit calculations for distance, direction, and proximity. – Example: “Which supermarket is nearest, and how many restaurants are within 500 m to its east?” – Vector retrieval finds similar text; coordinates support distance, direction, and proximity calculations. – Few studies compare several POI RAG designs under the same conditions. Example POI query “Which is the nearest supermarket to Shibuya Station, and how many restaurants lie within 500 m to its east?” nearest → distance to its east → direction within 500 m → proximity how many → density / count RAG architectures for geographic POI QA · FOSS4G 2026 2
3. One main question, two design axes and two evaluation metrics I tested which processing and orchestration strategies work well for different geographic questions. – Main question: which strategy works well for each question type? Roadmap – Design axis 1 — processing: spatial calculation or graph relations – Design axis 2 — orchestration: fixed, rule-based, or LLM-selected – Evaluation — expected-keyword success and overall answer quality RAG architectures for geographic POI QA · FOSS4G 2026 Question Setup Evaluation Results Implications One question guides the setup, evaluation, results, and implications. 3
4. Two phases, two comparison goals The study used two phases to answer different questions. – Phase 1: Shibuya · 90 cases · Structured · Graph · Adaptive – Phase 2: four Tokyo areas · 130 prompts per system · Hybrid · Graph · Adaptive · Agentic – Hybrid carries Structured RAG's spatial modules into the Phase 2 fixed pipeline. Phase 1 Shibuya 90 cases Structured · Graph · Adaptive fixed paths + rule-based routing Phase 2 four Tokyo areas 130 prompts per system Hybrid · Graph · Adaptive · Agentic fixed paths + rule/LLM orchestration RAG architectures for geographic POI QA · FOSS4G 2026 4
5. Five configurations built from shared components Two processing families are combined with fixed, rule-based, and LLM-based orchestration. – Processing families: spatial + vector, and graph relations – Adaptive routes by rules; Agentic selects individual tools with the LLM Structured Hybrid GraphRAG Adaptive Agentic Phase 1 — fixed spatial + vector — Phase 2 — same fixed family — fixed — relational path — rule-based — route selection — LLM-selected — tools and steps — – Five evaluated configurations, not five independent retrieval mechanisms Shared vector-retrieval base Fixed pipelines · rule-based routing · LLM tool orchestration RAG architectures for geographic POI QA · FOSS4G 2026 5
6. A controlled comparison within each phase Conditions were controlled within each phase; the two phases used different evaluation settings. – Data: Shibuya in Phase 1; four Tokyo areas in Phase 2 – Shared within each phase: Qwen2.5-7B, multilingual-e5-base, ChromaDB, prompts, and data – Compare scores within a phase, not directly across Phase 1 and Phase 2 Same conditions within each phase Model Embeddings Qwen2.5-7B multilingual-e5-base Prompts Data shared in each phase shared in each phase Phase 1 — Shibuya RAG architectures for geographic POI QA · FOSS4G 2026 Phase 2 — four Tokyo areas 6
7. Five levels test increasingly complex geographic reasoning Representative prompts show the progression from retrieval to multi-step reasoning. – Levels reflect required capability—not question length Level Capability Representative evaluated prompt – L1–L2: retrieve POIs and calculate spatial relations L1 Basic retrieval Where is Shibuya Station? – L3–L5: satisfy constraints, support decisions, and reason across steps L2 Spatial reasoning Which convenience store is nearest? L3 Constraint satisfaction Which cafés within 500 m have both phone numbers and websites? L4 Decision support Where should a cram school open, considering station and convenience-store access? L5 Advanced reasoning Does ‘many cafés’ still hold at 300 m instead of 500 m? – Scoring: expected keywords + reasoning · evidence · constraints · uncertainty, weighted by level Representative prompts translated from the Phase 1 base set · 55 prompts · 12 subcategories RAG architectures for geographic POI QA · FOSS4G 2026 7
8. Test design The same questions compare configurations within each phase The phases use different prompt sets, areas, and evaluated configurations. – Controlled comparison: the same cases are used for every configuration within a phase – Phase 1: architecture comparison on Shibuya data – Phase 2: multi-area generalization across four Tokyo areas Set Question coverage Evaluated configurations Phase 1 Shibuya · 90 cases Structured · Graph · Adaptive Phase 1 · base 55 L1–L5 prompts Same cases for each configuration Phase 1 · graph 35 relation-focused prompts Competitor · complementary · brand Phase 2 130 prompts per configuration Hybrid · Graph · Adaptive · Agentic Phase 2 · coverage four areas + cross-area prompts 520 total system queries Compare scores within a phase—not directly between Phase 1 and Phase 2 RAG architectures for geographic POI QA · FOSS4G 2026 8
9. Paired tests compare the same questions within each phase Paired answers are tested within each phase; prompt sets and composite weighting differ between phases. – Same prompts produce paired answers from each configuration Wilcoxon — continuous composite scores – Tests are applied within each phase – Bonferroni correction controls multiple comparisons McNemar — binary success outcomes Interpretation — non-significant does not mean equal performance RAG architectures for geographic POI QA · FOSS4G 2026 9
10. Phase 1 Phase 1: Structured RAG had the highest score Among the Phase 1 configurations, Structured RAG had the highest score. – Structured: fixed spatial-processing + vector-retrieval pipeline – Same 90 Shibuya cases: Structured 89.1 · Adaptive 86.1 · GraphRAG 76.7 • Evidence: Structured vs GraphRAG, Wilcoxon signed-rank, Bonferronicorrected p<0.001 – Coordinate-computable questions likely favored explicit spatial calculations RAG architectures for geographic POI QA · FOSS4G 2026 10
11. Phase 1 Different questions favored different systems The best architecture depended on the question type. – GraphRAG led comparison and competitor questions – Structured RAG led representative calculation and attribute questions – Category results are descriptive—not significance-tested – Explicit POI relations may help; this study does not isolate the cause RAG architectures for geographic POI QA · FOSS4G 2026 11
12. Phase 2 Phase 2 had no clear non-agentic winner Among the Phase 2 configurations, Hybrid had the highest numerical score, without a clear nonagentic winner. – Hybrid: Structured spatial modules + vector retrieval in a fixed multi-area pipeline – Phase 2: Hybrid 67.1 · Adaptive 65.2 · Graph 62.3 · Agentic 54.8 • Hybrid, Graph, and Adaptive were not clearly separated (pairwise corrected p = 0.056–0.353) – Agentic used LLM-selected tools and steps; its longer loop did not improve quality • Evidence versus Agentic: Hybrid and Adaptive p<0.001; Graph p=0.016 RAG architectures for geographic POI QA · FOSS4G 2026 12
13. Phase 2 High keyword success did not guarantee high-quality answers Systems often mentioned expected entities but lacked grounded reasoning, evidence, constraints, or uncertainty. – Expected keywords: above 90% for every system – Overall answer quality: 54.8–67.1 – Binary success: no significant pairwise differences after correction – L1–L2: entities, coordinates, and spatial calculation – L3–L5: evidence, constraints, decisions, and uncertainty RAG architectures for geographic POI QA · FOSS4G 2026 13
14. What the level-based results mean for system design Choose processing, orchestration, and evaluation together. – Compute coordinate-derived relations explicitly – Use graph links for explicit POI relations Compute — distance · direction · proximity – Match fixed, rule-based, or LLM control to the model and task Link — stored POI relations when they help retrieval Select + evaluate — fixed · rules · LLM tools · grounded quality RAG architectures for geographic POI QA · FOSS4G 2026 14
15. The results do not cover every setting These results are useful, but they are not universal. – Other regions and languages may behave differently Scope of this study – Larger models may change routing and tool use – Broader geographic tests and human evaluation are still needed Dense urban Tokyo Japanese questions One 7B model Rule-based automatic scoring Phase-specific comparisons · no direct Structured-versus-Hybrid head-to-head test RAG architectures for geographic POI QA · FOSS4G 2026 15
16. Design the evaluation before choosing the RAG configuration Define capability → choose processing and orchestration → test the final answer. – Start from the geographic capability the question requires – Build the processing path and selection policy around that capability – Evaluate grounded answer quality—not only keyword success Define L1 retrieval → L5 reasoning and uncertainty Build Spatial calculation · graph relations · fixed/rules/LLM Evaluate Reasoning · evidence · constraints · uncertainty RAG architectures for geographic POI QA · FOSS4G 2026 16