---
title: "Automatic Generation of a Compositional QA Benchmark for"
description: "This paper addresses that gap with a pipeline that automatically builds a compositional QA benchmark. 18 references, cited by 1 paper here."
authors: ["Tetsuhisa Suizu", "Shohei Higashiyama", "Hiroyuki Shindo", "Hiroki Ouchi", "Sakriani Sakti"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.eacl-srw.61/"
doi: "10.18653/v1/2026.eacl-srw.61"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/suizu-2026-automatic-generation-compositional-qa-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/suizu-2026-automatic-generation-compositional-qa-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / Automatic Generation of a Compositional QA Benchmark for Geospatial Reasoning under Spatial and Entity Constraints

# Automatic Generation of a Compositional QA Benchmark for Geospatial Reasoning under Spatial and Entity Constraints

**Authors:** [Tetsuhisa Suizu](https://md.research.elamur.ai/free/en/authors/tetsuhisa-suizu/), Shohei Higashiyama, [Hiroyuki Shindo](https://md.research.elamur.ai/free/en/authors/hiroyuki-shindo/), [Hiroki Ouchi](https://md.research.elamur.ai/free/en/authors/hiroki-ouchi/) and [Sakriani Sakti](https://md.research.elamur.ai/free/en/authors/sakriani-sakti/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-srw.61/) · [PDF](https://aclanthology.org/2026.eacl-srw.61.pdf) · [DOI 10.18653/v1/2026.eacl-srw.61](https://doi.org/10.18653/v1/2026.eacl-srw.61) · [OpenAlex](https://openalex.org/W7140076393)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Existing geospatial QA benchmarks mostly report overall accuracy and rarely reveal which reasoning skills a question tests. They also tend to focus either on geometry or on factual recall. This paper addresses that gap with a pipeline that automatically builds a compositional QA benchmark. Each question combines explicit spatial constraints (distance, direction, relative position, containment) with entity constraints, such as links between a place and a person, event, or cultural property. Slot-based templates are filled using geographic data from OpenStreetMap and entity facts from Wikidata. This yields questions with controllable complexity and a clear structure, from purely spatial queries to hybrid spatial-entity ones. The authors test GPT-5.2 and Gemini 3 Flash on the generated Japanese-language set. Both models do comparatively well when questions are anchored in rich entity knowledge. Their accuracy falls sharply when exact quantitative spatial judgment is needed, especially for distances and containment. The authors conclude that LLMs lean on associative knowledge rather than genuine spatial computation. They argue this motivates explicit geographic representations or spatial modules in future systems.

## Key points
- The benchmark contains 5,309 automatically generated questions built from 14 slot-based compositional templates. A sample template pattern asks for an entity type linked to a person within a given metric of an origin, such as a Kūkai-related temple within 1 km of Kyoto Station.
- Data sources are OpenStreetMap (ODbL 1.0) for geometry and Wikidata (CC0 1.0) for entity attributes and relations.
- GPT-5.2 and Gemini 3 Flash were evaluated through their APIs with a single prompt format. Weakest results came on distance estimation and containment, while the strongest came on entity-rich questions involving people or cultural properties.
- The current scope is limited to Japanese-language questions and mostly Japanese geo-entities. The pipeline itself is described as language- and region-agnostic.
- The dataset is to be released at github.com/NAIST-geo-and-lang/Compositional_GeoQA_Benchmark. Planned extensions include multi-hop and temporal reasoning, multilingual settings, and map-based interfaces.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nara Institute of Science and Technology](https://md.research.elamur.ai/free/en/universities/nara-institute-of-science-and-technology/) — University, Ikoma, Japan · 46 works · 4,565 citations
- [National Institute of Information and Communications Technology](https://md.research.elamur.ai/free/en/labs/national-institute-of-information-and-communications-technology/) — Research institute, Tokyo, Japan · 23 works · 2,090 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Tetsuhisa Suizu, Shohei Higashiyama, Hiroyuki Shindo, Hiroki Ouchi and Sakriani Sakti. 2026. Automatic Generation of a Compositional QA Benchmark for Geospatial Reasoning under Spatial and Entity Constraints. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{suizu2026automatic,
  title = {{Automatic Generation of a Compositional QA Benchmark for Geospatial Reasoning under Spatial and Entity Constraints}},
  author = {Tetsuhisa Suizu, Shohei Higashiyama, Hiroyuki Shindo, Hiroki Ouchi and Sakriani Sakti},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-srw.61},
  url = {https://doi.org/10.18653/v1/2026.eacl-srw.61}
}
```

- [← previous Automatic Detection of Metaphorical Expressions in Classical Japanese Using WLSP-Enhanced BERT](https://md.research.elamur.ai/free/en/papers/zhu-2026-automatic-detection-metaphorical-expressions-classical/)
- [next → Automatic Machine Translation Detection Using a Surrogate Multilingual Translation Model](https://md.research.elamur.ai/free/en/papers/romero-2026-automatic-machine-translation-detection-surrogate/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. 2022. MapQA: A dataset for question answering on choropleth maps. Preprint, arXiv:2211.08545. — [arXiv:2211.08545](https://arxiv.org/abs/2211.08545) [find](https://scholar.google.com/scholar?q=Shuaichen+Chang%2C+David+Palzer%2C+Jialin+Li%2C+Eric+Fosler%2DLussier%2C+and+Ningchuan+Xiao%2E+2022%2E+MapQA%3A+A+dataset+for+question+answering+on+choropleth+maps%2E+Preprint%2C+arXiv%3A2211%2E08545%2E+%E2%80%94+%5BarXiv%3A2211%2E08545%5D%28ht)
2. Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. 2021. [GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 513–523, Online. Association…
3. Ishita Dasgupta, Danyal Guo, Andreas Stuhlmüller, Samuel J. Gershman, and Noah D. Goodman. 2022. Closure: Assessing systematic generalization of combinatorial structure. In Advances in Neural Information Processing Systems (NeurIPS 2022). [find](https://scholar.google.com/scholar?q=Ishita+Dasgupta%2C+Danyal+Guo%2C+Andreas+Stuhlm%C3%BCller%2C+Samuel+J%2E+Gershman%2C+and+Noah+D%2E+Goodman%2E+2022%2E+Closure%3A+Assessing+systematic+generalization+of+combinatorial+structure%2E+In+Advances+in+Neural+Informat)
4. Max J Egenhofer. 1991. Categorizing binary topological relations between regions, lines, and points in geographic databases. Technical Report, Department of Surveying Engineering, University of Maine. [find](https://scholar.google.com/scholar?q=Max+J+Egenhofer%2E+1991%2E+Categorizing+binary+topological+relations+between+regions%2C+lines%2C+and+points+in+geographic+databases%2E+Technical+Report%2C+Department+of+Surveying+Engineering%2C+University+of+Maine%2E)
5. Jie Feng, Jun Zhang, Tianhui Liu, Xin Zhang, Tianjian Ouyang, Junbo Yan, Yuwei Du, Siqi Guo, and Yong Li. 2025. CityBench: Evaluating the capabilities of large language models for urban tasks. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, KDD ’25, page… [find](https://scholar.google.com/scholar?q=Jie+Feng%2C+Jun+Zhang%2C+Tianhui+Liu%2C+Xin+Zhang%2C+Tianjian+Ouyang%2C+Junbo+Yan%2C+Yuwei+Du%2C+Siqi+Guo%2C+and+Yong+Li%2E+2025%2E+CityBench%3A+Evaluating+the+capabilities+of+large+language+models+for+urban+tasks%2E+In+Proc)
6. Google. 2025. Gemini 3 Flash: frontier intelligence built for speed. [find](https://scholar.google.com/scholar?q=Google%2E+2025%2E+Gemini+3+Flash%3A+frontier+intelligence+built+for+speed%2E)
7. Drew A. Hudson and Christopher D. Manning. 2019. [Gqa: A new dataset for real-world visual reasoning and compositional question answering](https://md.research.elamur.ai/free/en/papers/hudson-2019-gqa-new-dataset-real-world/). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), pages 6700–6709.
8. Wenbin Li, Di Yao, Ruibo Zhao, Wenjie Chen, Zijie Xu, Chengxue Luo, Chang Gong, Quanliang Jing, Haining Tan, and Jingping Bi. 2025a. Stbench: Assessing the ability of large language models in spatiotemporal analysis. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, page 749–752,… [find](https://scholar.google.com/scholar?q=Wenbin+Li%2C+Di+Yao%2C+Ruibo+Zhao%2C+Wenjie+Chen%2C+Zijie+Xu%2C+Chengxue+Luo%2C+Chang+Gong%2C+Quanliang+Jing%2C+Haining+Tan%2C+and+Jingping+Bi%2E+2025a%2E+Stbench%3A+Assessing+the+ability+of+large+language+models+in+spatiote)
9. Zekun Li, Malcolm Grossman, Eric, Qasemi, Mihir Kulkarni, Muhao Chen, and Yao-Yi Chiang. 2025b. Mapqa: Open-domain geospatial question answering on map data. Preprint, arXiv:2503.07871. — [arXiv:2503.07871](https://arxiv.org/abs/2503.07871) [find](https://scholar.google.com/scholar?q=Zekun+Li%2C+Malcolm+Grossman%2C+Eric%2C+Qasemi%2C+Mihir+Kulkarni%2C+Muhao+Chen%2C+and+Yao%2DYi+Chiang%2E+2025b%2E+Mapqa%3A+Open%2Ddomain+geospatial+question+answering+on+map+data%2E+Preprint%2C+arXiv%3A2503%2E07871%2E+%E2%80%94+%5BarXiv%3A2503%2E)
10. David M Mark. 1999. Spatial representation: A cognitive view. Geographical information systems: Principles and applications, 1:81–89. [find](https://scholar.google.com/scholar?q=David+M+Mark%2E+1999%2E+Spatial+representation%3A+A+cognitive+view%2E+Geographical+information+systems%3A+Principles+and+applications%2C+1%3A81%E2%80%9389%2E)
11. OpenAI. 2025. Update to GPT-5 system card: GPT-5.2. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Update+to+GPT%2D5+system+card%3A+GPT%2D5%2E2%2E)
12. James Pustejovsky, Kiyong Lee, Harry Bunt, and Laurent Romary. 2010. ISO-TimeML: An international standard for semantic annotation. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=James+Pustejovsky%2C+Kiyong+Lee%2C+Harry+Bunt%2C+and+Laurent+Romary%2E+2010%2E+ISO%2DTimeML%3A+An+international+standard+for+semantic+annotation%2E+In+Proceedings+of+the+Seventh+International+Conference+on+Language+R)
13. James Pustejovsky and Zachary Yocum. 2013. Capturing motion in ISO-SpaceBank. In Proceedings of the 9th Joint ISO - ACL SIGSEM Workshop on Interoperable Semantic Annotation, pages 25–34, Potsdam, Germany. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=James+Pustejovsky+and+Zachary+Yocum%2E+2013%2E+Capturing+motion+in+ISO%2DSpaceBank%2E+In+Proceedings+of+the+9th+Joint+ISO+%2D+ACL+SIGSEM+Workshop+on+Interoperable+Semantic+Annotation%2C+pages+25%E2%80%9334%2C+Potsdam%2C+Germ)
14. David A. Randell, Zhan Cui, and Anthony G. Cohn. 1992. A spatial logic based on regions and connection. In Proceedings of the Third International Conference on Principles of Knowledge Representation and Reasoning, KR’92, page 165–176, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=David+A%2E+Randell%2C+Zhan+Cui%2C+and+Anthony+G%2E+Cohn%2E+1992%2E+A+spatial+logic+based+on+regions+and+connection%2E+In+Proceedings+of+the+Third+International+Conference+on+Principles+of+Knowledge+Representation+a)
15. Leonard Talmy. 2000. Toward a Cognitive Semantics, Volume 1: Concept Structuring Systems. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Leonard+Talmy%2E+2000%2E+Toward+a+Cognitive+Semantics%2C+Volume+1%3A+Concept+Structuring+Systems%2E+MIT+Press%2C+Cambridge%2C+MA%2E)
16. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on… [find](https://scholar.google.com/scholar?q=Daniel+Keysers%2C+Nathanael+Sch%C3%A4rli%2C+Nathan+Scales%2C+Hylke+Buisman%2C+Daniel+Furrer%2C+Sergii+Kashubin%2C+Nikola+Momchev%2C+Danila+Sinopalnikov%2C+Lukasz+Stafiniak%2C+Tibor+Tihon%2C+Dmitry+Tsarkov%2C+Xiao+Wang%2C+Marc+van)
17. Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning (ICML 2018), pages 2879–2888. PMLR. [find](https://scholar.google.com/scholar?q=Brenden+M%2E+Lake+and+Marco+Baroni%2E+2018%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Proceedings+of+the+35th+International+Conferenc)
18. Stephen C. Levinson. 2003. Space in Language and Cognition: Explorations in Cognitive Diversity. Language Culture and Cognition. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Stephen+C%2E+Levinson%2E+2003%2E+Space+in+Language+and+Cognition%3A+Explorations+in+Cognitive+Diversity%2E+Language+Culture+and+Cognition%2E+Cambridge+University+Press%2E)

*Updated 31 October 2026*