---
title: "KnowledgeBerg — paper by Zhang et al. (2026)"
description: "They describe each question along two axes: knowledge width, the size of the needed universe, and reasoning… 4.35 points. 40 references."
authors: ["Xiao Zhang", "Qianru Meng", "Yongjian Chen", "Yumeng Wang", "Johan Bos"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.548/"
doi: "10.18653/v1/2026.findings-acl.548"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2026-knowledgeberg-evaluating-systematic-knowledge-coverage/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2026-knowledgeberg-evaluating-systematic-knowledge-coverage/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/) / KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

# KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

**Authors:** [Xiao Zhang](https://md.research.elamur.ai/free/en/authors/xiao-zhang/), Qianru Meng, Yongjian Chen, [Yumeng Wang](https://md.research.elamur.ai/free/en/authors/yumeng-wang/) and [Johan Bos](https://md.research.elamur.ai/free/en/authors/johan-bos/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.548/) · [PDF](https://aclanthology.org/2026.findings-acl.548.pdf) · [DOI 10.18653/v1/2026.findings-acl.548](https://doi.org/10.18653/v1/2026.findings-acl.548) · [OpenAlex](https://openalex.org/W7166837687)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Questions that look easy often quietly require a model to know a complete, well-defined set of entities and to run several set operations over it. One example is asking which theme dominates among all 88 IAU constellations. The authors call this the tip-of-the-iceberg problem. They describe each question along two axes: knowledge width, the size of the needed universe, and reasoning depth, the number of chained set operations.

To test this, they built KnowledgeBerg. The universes come from curated authoritative sources such as Gene Ontology, NCBI Taxonomy, textbooks, UN documents and official websites. From these, GPT-4o generated multiple-choice reasoning questions, which human annotators then checked. The authors also propose Iceberg Gap, a metric that compares benchmarks by how far a simple surface hides heavy hidden demands.

Open-source LLMs did poorly at both listing the universes and reasoning over them. Their errors fall into three groups: missing knowledge, not noticing what the question requires, and executing the reasoning wrongly. Inference-time methods help only modestly. This matters for high-stakes uses like clinical diagnosis, where every candidate must be considered.

## Key points
- **Scale:** The benchmark has 4,800 knowledge-grounded reasoning questions (KRQs) built from 1,183 enumeration questions (EQ–EA pairs). It covers 10 domains and 17 languages; the 16 non-English versions were made with Google Translate. Each question has 8–10 options, so random guessing scores 10–12.5%. The median universe has only 12 entities.
- **Baseline results:** Models scored 5.26–36.88 F1 on universe enumeration and 16.00–44.19 accuracy on KRQs.
- **Test-time strategies:**
  - Self-consistency with 16 samples gave the best results for all 11 models tested.
  - Self-verification came second.
  - Hybrid BM25+dense retrieval was the strongest RAG variant.
  - Extra rounds of self-refinement lowered accuracy.
  - The largest gains were 4.35 points from test-time compute and 3.78 points from retrieval.
- **Construction:** Reasoning programs are linear chains of operators such as filtering, counting, union, partitioning and comparison. Candidates were reviewed in three stages, with adjudication between two annotators.
- **Availability:** The dataset is on Hugging Face as `2npc/KnowledgeBerg`. Stated limitations include no validation of translations by native speakers and few very large universes.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 74 works · 5,142 citations
- [Leiden University](https://md.research.elamur.ai/free/en/universities/leiden-university/) — University, Leiden, Netherlands · 37 works · 1,408 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Xiao Zhang, Qianru Meng, Yongjian Chen, Yumeng Wang and Johan Bos. 2026. KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2026knowledgeberg,
  title = {{KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models}},
  author = {Xiao Zhang, Qianru Meng, Yongjian Chen, Yumeng Wang and Johan Bos},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.548},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.548}
}
```

- [← previous Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains](https://md.research.elamur.ai/free/en/papers/yuan-2026-knowledge-verification-exploring-rlvr-llms/)
- [next → KOCOH: Korean Context-Dependent Hate Speech Dataset](https://md.research.elamur.ai/free/en/papers/park-2026-kocoh-korean-context-dependent-hate/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/38/)

## References

40 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Ashburner, Catherine A. Ball, Judith A. Blake, David Botstein, Heather L. Butler, J. Michael Cherry, Allan Peter Davis, Kara Dolinski, Selina S. Dwight, Janan T. Eppig, Midori A. Harris, David P. Hill, Laurie Issel-Tarver, Andrew Kasarskis, Suzanna E. Lewis, John C. Matese, Joel E. Richardso… [find](https://scholar.google.com/scholar?q=Michael+Ashburner%2C+Catherine+A%2E+Ball%2C+Judith+A%2E+Blake%2C+David+Botstein%2C+Heather+L%2E+Butler%2C+J%2E+Michael+Cherry%2C+Allan+Peter+Davis%2C+Kara+Dolinski%2C+Selina+S%2E+Dwight%2C+Janan+T%2E+Eppig%2C+Midori+A%2E+Harris%2C+David)
2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
3. Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022. KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Computati… [find](https://scholar.google.com/scholar?q=Shulin+Cao%2C+Jiaxin+Shi%2C+Liangming+Pan%2C+Lunyiu+Nie%2C+Yutong+Xiang%2C+Lei+Hou%2C+Juanzi+Li%2C+Bin+He%2C+and+Hanwang+Zhang%2E+2022%2E+KQA+pro%3A+A+dataset+with+explicit+compositional+programs+for+complex+question+answe)
4. Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. [Rq-rag: Learning to refine queries for retrieval augmented generation](https://md.research.elamur.ai/free/en/papers/chan-2024-rq-rag-learning-refine-queries/). Preprint, arXiv:2404.00610.
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). Preprint, arXiv:2110.14168.
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv eprints, pages arXiv–2407.
8. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
9. Matteo Gabburo, Nicolaas Paul Jedema, Siddhant Garg, Leonardo F. R. Ribeiro, and Alessandro Moschitti. 2024. Measuring retrieval complexity in question answering systems. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14636–14650, Bangkok, Thailand. Association for Com… [find](https://scholar.google.com/scholar?q=Matteo+Gabburo%2C+Nicolaas+Paul+Jedema%2C+Siddhant+Garg%2C+Leonardo+F%2E+R%2E+Ribeiro%2C+and+Alessandro+Moschitti%2E+2024%2E+Measuring+retrieval+complexity+in+question+answering+systems%2E+In+Findings+of+the+Associatio)
10. David Halliday, Robert Resnick, and Jearl Walker. 2013. Fundamentals of physics. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=David+Halliday%2C+Robert+Resnick%2C+and+Jearl+Walker%2E+2013%2E+Fundamentals+of+physics%2E+John+Wiley+%26+Sons%2E)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). Proceedings of the International Conference on Learning Representations (ICLR).
12. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
13. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
14. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. [Natural…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
15. Huiyuan Lai and Malvina Nissim. 2024. [mCoT: Multilingual instruction tuning for reasoning consistency in language models](https://md.research.elamur.ai/free/en/papers/lai-2024-mcot-multilingual-instruction-tuning-reasoning/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12012– 12026, Bangkok, Thailand. Association for Computa…
16. Huiyuan Lai, Xiao Zhang, and Malvina Nissim. 2025. Multidimensional consistency improves reasoning in language models. Preprint, arXiv:2503.02670. — [arXiv:2503.02670](https://arxiv.org/abs/2503.02670) [find](https://scholar.google.com/scholar?q=Huiyuan+Lai%2C+Xiao+Zhang%2C+and+Malvina+Nissim%2E+2025%2E+Multidimensional+consistency+improves+reasoning+in+language+models%2E+Preprint%2C+arXiv%3A2503%2E02670%2E+%E2%80%94+%5BarXiv%3A2503%2E02670%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E02670)
17. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Proceedings of the 34th Internation…
18. Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024. Understanding and patching compositional reasoning in LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9668–9688, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhaoyi+Li%2C+Gangwei+Jiang%2C+Hong+Xie%2C+Linqi+Song%2C+Defu+Lian%2C+and+Ying+Wei%2E+2024%2E+Understanding+and+patching+compositional+reasoning+in+LLMs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
19. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
20. Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. [Chameleon: plug-and-play compositional reasoning with large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-chameleon-plug-play-compositional-reasoning/). In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red…
21. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+Shashank+Gupta%2C+Bodhisattwa+Prasad+Majumder%2C+Katherine+Her)
22. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (V… [find](https://scholar.google.com/scholar?q=Alex+Mallen%2C+Akari+Asai%2C+Victor+Zhong%2C+Rajarshi+Das%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+2023%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+of+parametric+and+non%2Dparametric+mem)
23. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
24. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. [KILT: a benchmark for knowledge intensive language tasks](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/). In Proceedings of th…
25. Mubashar Raza, Zarmina Jahangir, Muhammad Riaz, and Muhammad Sattar. 2025. Industrial applications of large language models. Scientific Reports, 15. [find](https://scholar.google.com/scholar?q=Mubashar+Raza%2C+Zarmina+Jahangir%2C+Muhammad+Riaz%2C+and+Muhammad+Sattar%2E+2025%2E+Industrial+applications+of+large+language+models%2E+Scientific+Reports%2C+15%2E)
26. Conrad Schoch, Stacy Ciufo, Carol Hotton, Sivakumar Kannan, Rogneda Khovanskaya, Detlef Leipe, Richard McVeigh, Kathleen O’Neill, Barbara Robbertse, Shobha Sharma, Vladimir Soussov, John Sullivan, Lu Sun, Sean Turner, and Ilene Karsch-Mizrachi. 2020. Ncbi taxonomy: A comprehensive update on curation… [find](https://scholar.google.com/scholar?q=Conrad+Schoch%2C+Stacy+Ciufo%2C+Carol+Hotton%2C+Sivakumar+Kannan%2C+Rogneda+Khovanskaya%2C+Detlef+Leipe%2C+Richard+McVeigh%2C+Kathleen+O%E2%80%99Neill%2C+Barbara+Robbertse%2C+Shobha+Sharma%2C+Vladimir+Soussov%2C+John+Sullivan%2C+Lu+)
27. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. [ASQA: Factoid questions meet long-form answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
28. Alon Talmor and Jonathan Berant. 2018. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, L…
29. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
30. Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. 2024. [ProxyQA: An alternative framework for evaluating long-form text generation with large language models](https://md.research.elamur.ai/free/en/papers/tan-2024-proxyqa-alternative-framework-evaluating-long/). In Proceedings of the 62nd Annual Meeting of the Associ…
31. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, and Morgane Rivière. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+and+Morgane+Rivi%C3%A8re%2E+2025%2E+Gemma+3+technical+report%2E+arXiv)
32. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In ICLR 2023.
33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
34. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
35. Xiao Zhang, Huiyuan Lai, Qianru Meng, and Johan Bos. 2025. Ontourl: A benchmark for evaluating large language models on symbolic ontological understanding, reasoning and learning. Preprint, arXiv:2505.11031. — [arXiv:2505.11031](https://arxiv.org/abs/2505.11031) [find](https://scholar.google.com/scholar?q=Xiao+Zhang%2C+Huiyuan+Lai%2C+Qianru+Meng%2C+and+Johan+Bos%2E+2025%2E+Ontourl%3A+A+benchmark+for+evaluating+large+language+models+on+symbolic+ontological+understanding%2C+reasoning+and+learning%2E+Preprint%2C+arXiv%3A2505)
36. Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Confere… [find](https://scholar.google.com/scholar?q=Yuyu+Zhang%2C+Hanjun+Dai%2C+Zornitsa+Kozareva%2C+Alexander+J%2E+Smola%2C+and+Le+Song%2E+2018%2E+Variational+reasoning+for+question+answering+with+knowledge+graph%2E+In+Proceedings+of+the+Thirty%2DSecond+AAAI+Conference)
37. Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. [Take a step back: Evoking reasoning via abstraction in large language models](https://md.research.elamur.ai/free/en/papers/zheng-2023-take-step-back-evoking-reasoning/). In The Twelfth International Conference on Learning Representations.
38. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. [Mmlu-pro: a more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)…
39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Advances in Neural Information Processing Systems.
40. Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. [Large language models are better reasoners with self-verification](https://md.research.elamur.ai/free/en/papers/weng-2023-large-language-models-are-better/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575, Singapore. Association for Computat…

*Updated 2 November 2026*