Slide 51
Slide 51 text
Resource Track
Benchmarking Geospatial Question Answering Engines Using the Dataset
GEOQUESTIONS1089
Sergios-Anestis Kefalidis1, Dharmen Punjani2, Eleni Tsalapati1,Konstantinos Plas1, Mariangela Pollali1, Michail Mitsios1, Myrto Tsokanaridou1, Manolis Koubarakis1, and Pierre Maret2/1 National and
Kapodistrian University of Athens, 2 Universit´e St. Monnet
• 一言でいうと
地理空間QAエンジンの評価用ベンチマークデータセットを作成, SOTAのエンジンを評価．
• Resource Type (Benchmark data)
• 動機
地理空間QAエンジンの有効性(effectiveness), 効率(efficiency)の評価のため, 意味
的に複雑なもの等, 多様なタイプの質問を含む大規模なベンチマークデータセットを作成．
• リソースの設計方針
- 対象: YAGO2, YAGO2geo(YAGO2のうち, 緯度経度情報をもつエンティティのサブセット)
- 従来のデータセット(GEOQUESTIONS201)より複雑な質問を多く含む．自然言語理解や
GeoSPARQLの高度なfeatures(ネスト, no-existsフィルタ, 算術演算)の必要な質問等．
• 新規性
地理空間QAの最大のベンチマーク(1089トリプル), 意味的に複雑な質問を含む
• リソースの再利用性／活用実績
エンジン(GeoQA2,Hamzeiら(2021))を評価(複雑な質問には両者とも対応が不十分) ．
• 品質
クエリのタイプ別/作成方法別サブセットによりエンジンの詳細な分析が可能(報告者コメント)．
• リソースの可用性 https://github.com/AI-team-UoA/GeoQuestions1089
• 次にすべきこと
地理空間QAエンジンの深層学習のトレーニングに使えるようデータセットを拡張
カテ
ゴリ
説明
(３．The GEOQUESTIONS1089 Datasetの本文より引用, 太字は報告者による)
A Asking for a thematic or a spatial attribute of a feature, e.g.,
“Where is Loch Goil located?”
B Asking whether a feature is in a geospatial relation with another
feature or features, e.g., “Is Liverpool east of Ireland?”.
C Asking for features of a given class that are in a geospatial
relation with another feature. E.g., “Which counties border
county Lincolnshire?” or “Which hotels in Belfast are at most
2km from George Best Belfast City Airport?”
D Asking for features of a given class that are in a geospatial
relation with any features of another class, e.g., “Which churches
are near castles?”.
E Asking for features of a given class that are in a geospatial
relation with an unspecified feature of another class, and either
one or both, is/are in another geospatial relation with a feature
specified explicitly. E.g., “Which churches are near a castle in
Scotland?” or “In Greece, which beaches are near villages?”
F As in categories C, D and E above, plus more thematic and/or
geospatial characteristics of the features expected as answers,
e.g., “Which mountains in Scotland have height more than
1000 m?”
G Questions with quantities and aggregates, e.g., “What is the total
area of lakes in Monaghan?” or “How many lakes are there in
Monaghan?”.
H Questions with superlatives or comparatives, e.g., “Which is the
largest island in Greece?” or “Is the largest island in France
larger than Crete?”.
I Questions with quantities, aggregates, and
superlatives/comparatives, e.g.,“Which city in the UK has the
most hospitals?” or “Is the total size of lakes in Greece larger
than lake Loch Lomond in Scotland?”