---
title: "What Breaks Knowledge Graph based RAG? Benchmarking and"
description: "The authors propose a general recipe for building benchmarks that test reasoning when knowledge is… 48 references, cited by 3 papers here."
authors: ["Dongzhuoran Zhou", "Yuqicheng Zhu", "Xiaxia Wang", "Hongkuan Zhou", "Yuan He", "Jiaoyan Chen", "Steffen Staab", "Evgeny Kharlamov"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.114/"
doi: "10.18653/v1/2026.eacl-long.114"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhou-2026-what-breaks-knowledge-graph-based/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhou-2026-what-breaks-knowledge-graph-based/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge

# What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge

**Authors:** Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab and Evgeny Kharlamov · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.114/) · [PDF](https://aclanthology.org/2026.eacl-long.114.pdf) · [DOI 10.18653/v1/2026.eacl-long.114](https://doi.org/10.18653/v1/2026.eacl-long.114) · [OpenAlex](https://openalex.org/W7140148804)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 149,439 papers whose bibliographies we read

## Summary
KG-RAG systems are meant to combine LLM reasoning with structured knowledge graph evidence. Standard benchmarks such as WebQSP, CWQ and GrailQA usually include a triple that states the answer directly, so a correct response may reflect lookup rather than inference. Evaluation is also inconsistent: "accuracy" can mean exact match or loose substring matching, and some codebases compute Hits@1 without any ranking.

The authors propose a general recipe for building benchmarks that test reasoning when knowledge is missing, and they use it to create BRINK. They mine Horn rules with AMIE3, delete the triple that would answer each question directly, and keep the rule body facts so the answer can still be inferred. Questions are generated from these rule groundings, and answer frequencies are balanced through downsampling.

Experiments on data derived from Family, FB15k-237 and Wikidata5m show that most KG-RAG methods fail once direct evidence is removed. Trained methods hold up better. Results also depend heavily on whether entities have textual labels. Together this suggests models rely on memorized knowledge rather than reasoning over graph structure, which motivates better retrieval of alternative paths.

## Key points
- AMIE3 mines rules of length 4 or less, keeping only those that reach a minimum confidence of 0.3 and a minimum head coverage of 0.1. Questions are built only from removed triples that the remaining graph can still support.
- Earlier ways of simulating incompleteness, such as random deletion or cutting shortest paths, can leave questions unanswerable. BRINK guarantees that supporting evidence remains for every question.
- Training-based systems (RoG, GNN-RAG) cope with incompleteness better than non-trained ones. However, they show signs of overfitting to particular relation patterns.
- Freebase IDs (e.g., /m/02mjmr) give F1 results almost identical to random private IDs, while textual labels raise performance considerably. This indicates that LLMs treat IDs as opaque tokens.
- The case studies identify two failure types. One is retrieval failure, where spurious paths lead to hallucinated answers. The other is reasoning failure, where the correct fact is retrieved but distracting context produces a wrong answer such as "Marriage".

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations
- [University of Stuttgart](https://md.research.elamur.ai/free/en/universities/university-of-stuttgart/) — University, Stuttgart, Germany · 91 works · 46,376 citations
- [University of Manchester](https://md.research.elamur.ai/free/en/universities/university-of-manchester/) — University, Manchester, United Kingdom · 87 works · 44,938 citations
- [University of Oslo](https://md.research.elamur.ai/free/en/universities/university-of-oslo/) — University, Oslo, Norway · 51 works · 4,155 citations
- [University of Southampton](https://md.research.elamur.ai/free/en/universities/university-of-southampton/) — University, Southampton, United Kingdom · 10 works · 2,495 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 192 works

## Cite

Cite as: Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab and Evgeny Kharlamov. 2026. What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhou2026what,
  title = {{What Breaks Knowledge Graph based RAG? Benchmarking and Empirical Insights into Reasoning under Incomplete Knowledge}},
  author = {Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Steffen Staab and Evgeny Kharlamov},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.114},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.114}
}
```

- [← previous VIB-Probe: Detecting and Mitigating Hallucinations in Vision-Language Models via Variational Information Bottleneck](https://md.research.elamur.ai/free/en/papers/zhang-2026-vib-probe-detecting-mitigating-hallucinations/)
- [next → ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching](https://md.research.elamur.ai/free/en/papers/zhu-2026-zipvoice-dialog-non-autoregressive-spoken/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinheon Baek, Alham Fikri Aji, Jens Lehmann, and Sung Ju Hwang. 2023. Direct fact retrieval from knowledge graphs without entity linking. arXiv preprint arXiv:2305.12416. [find](https://scholar.google.com/scholar?q=Jinheon+Baek%2C+Alham+Fikri+Aji%2C+Jens+Lehmann%2C+and+Sung+Ju+Hwang%2E+2023%2E+Direct+fact+retrieval+from+knowledge+graphs+without+entity+linking%2E+arXiv+preprint+arXiv%3A2305%2E12416%2E)
2. Russa Biswas, Lucie-Aimée Kaffee, Michael Cochez, Stefania Dumbrava, Theis E Jendal, Matteo Lissandrini, Vanessa Lopez, Eneldo Loza Mencía, Heiko Paulheim, Harald Sack, and 1 others. 2023. Knowledge graph embeddings: open challenges and opportunities. Transactions on Graph Data and Knowledge, 1(1):4… [find](https://scholar.google.com/scholar?q=Russa+Biswas%2C+Lucie%2DAim%C3%A9e+Kaffee%2C+Michael+Cochez%2C+Stefania+Dumbrava%2C+Theis+E+Jendal%2C+Matteo+Lissandrini%2C+Vanessa+Lopez%2C+Eneldo+Loza+Menc%C3%ADa%2C+Heiko+Paulheim%2C+Harald+Sack%2C+and+1+others%2E+2023%2E+Knowledge+g)
3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cas… [find](https://scholar.google.com/scholar?q=Sebastian+Borgeaud%2C+Arthur+Mensch%2C+Jordan+Hoffmann%2C+Trevor+Cai%2C+Eliza+Rutherford%2C+Katie+Millican%2C+George+van+den+Driessche%2C+Jean%2DBaptiste+Lespiau%2C+Bogdan+Damoc%2C+Aidan+Clark%2C+Diego+de+Las+Casas%2C+Aureli)
4. Lihu Chen, Simon Razniewski, and Gerhard Weikum. 2023. Knowledge base completion for long-tail entities. In Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023), pages 99–108. [find](https://scholar.google.com/scholar?q=Lihu+Chen%2C+Simon+Razniewski%2C+and+Gerhard+Weikum%2E+2023%2E+Knowledge+base+completion+for+long%2Dtail+entities%2E+In+Proceedings+of+the+First+Workshop+on+Matching+From+Unstructured+and+Structured+Data+%28MATCHIN)
5. Liyi Chen, Panrong Tong, Zhongming Jin, Ying Sun, Jieping Ye, and Hui Xiong. 2024. Plan-on-graph: Self-correcting adaptive planning of large language model on knowledge graphs. In Proceedings of the 38th Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Liyi+Chen%2C+Panrong+Tong%2C+Zhongming+Jin%2C+Ying+Sun%2C+Jieping+Ye%2C+and+Hui+Xiong%2E+2024%2E+Plan%2Don%2Dgraph%3A+Self%2Dcorrecting+adaptive+planning+of+large+language+model+on+knowledge+graphs%2E+In+Proceedings+of+the+3)
6. Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. 2025. Empowering llms with logical reasoning: A comprehensive survey. arXiv preprint arXiv:2502.15652. [find](https://scholar.google.com/scholar?q=Fengxiang+Cheng%2C+Haoxuan+Li%2C+Fenrong+Liu%2C+Robert+van+Rooij%2C+Kun+Zhang%2C+and+Zhouchen+Lin%2E+2025%2E+Empowering+llms+with+logical+reasoning%3A+A+comprehensive+survey%2E+arXiv+preprint+arXiv%3A2502%2E15652%2E)
7. Xin Luna Dong, Evgeniy Gabrilovich, Geremy Heitz, Wiko Horn, Kevin Murphy, Shaohua Sun, and Wei Zhang. 2014. From data fusion to knowledge fusion. Proceedings of the VLDB Endowment, 7(10). [find](https://scholar.google.com/scholar?q=Xin+Luna+Dong%2C+Evgeniy+Gabrilovich%2C+Geremy+Heitz%2C+Wiko+Horn%2C+Kevin+Murphy%2C+Shaohua+Sun%2C+and+Wei+Zhang%2E+2014%2E+From+data+fusion+to+knowledge+fusion%2E+Proceedings+of+the+VLDB+Endowment%2C+7%2810%29%2E)
8. Jinyuan Fang, Zaiqiao Meng, and Craig Macdonald. 2024. Reano: Optimising retrieval-augmented reader models through knowledge graph generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2094–2112. [find](https://scholar.google.com/scholar?q=Jinyuan+Fang%2C+Zaiqiao+Meng%2C+and+Craig+Macdonald%2E+2024%2E+Reano%3A+Optimising+retrieval%2Daugmented+reader+models+through+knowledge+graph+generation%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of+the+Associat)
9. Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. 2015. Fast rule mining in ontological knowledge bases with amie+. The VLDB Journal, 24(6):707–730. [find](https://scholar.google.com/scholar?q=Luis+Gal%C3%A1rraga%2C+Christina+Teflioudi%2C+Katja+Hose%2C+and+Fabian+M+Suchanek%2E+2015%2E+Fast+rule+mining+in+ontological+knowledge+bases+with+amie%2B%2E+The+VLDB+Journal%2C+24%286%29%3A707%E2%80%93730%2E)
10. Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, pages 3477–3488. ACM. [find](https://scholar.google.com/scholar?q=Yu+Gu%2C+Sue+Kase%2C+Michelle+Vanni%2C+Brian+Sadler%2C+Percy+Liang%2C+Xifeng+Yan%2C+and+Yu+Su%2E+2021%2E+Beyond+iid%3A+three+levels+of+generalization+for+question+answering+on+knowledge+bases%2E+In+Proceedings+of+the+Web)
11. Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, and 1 others. 2024. Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309. [find](https://scholar.google.com/scholar?q=Haoyu+Han%2C+Yu+Wang%2C+Harry+Shomer%2C+Kai+Guo%2C+Jiayuan+Ding%2C+Yongjia+Lei%2C+Mahantesh+Halappanavar%2C+Ryan+A+Rossi%2C+Subhabrata+Mukherjee%2C+Xianfeng+Tang%2C+and+1+others%2E+2024%2E+Retrieval%2Daugmented+generation+with)
12. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. [G-retriever: Retrieval-augmented generation for textual graph understanding and question answering](https://md.research.elamur.ai/free/en/papers/he-2024-g-retriever-retrieval-augmented-generation/). Advances in Neural Information Processing Systems, 37:132876–132907.
13. Yuan He, Bailan He, Zifeng Ding, Alisia Lupidi, Yuqicheng Zhu, Shuo Chen, Caiqi Zhang, Jiaoyan Chen, Yunpu Ma, Volker Tresp, and 1 others. 2025. [find](https://scholar.google.com/scholar?q=Yuan+He%2C+Bailan+He%2C+Zifeng+Ding%2C+Alisia+Lupidi%2C+Yuqicheng+Zhu%2C+Shuo+Chen%2C+Caiqi+Zhang%2C+Jiaoyan+Chen%2C+Yunpu+Ma%2C+Volker+Tresp%2C+and+1+others%2E+2025%2E)
14. Supposedly equivalent facts that aren’t? entity frequency in pre-training induces asymmetry in llms. arXiv preprint arXiv:2503.22362. — [arXiv:2503.22362](https://arxiv.org/abs/2503.22362) [find](https://scholar.google.com/scholar?q=Supposedly+equivalent+facts+that+aren%E2%80%99t%3F+entity+frequency+in+pre%2Dtraining+induces+asymmetry+in+llms%2E+arXiv+preprint+arXiv%3A2503%2E22362%2E+%E2%80%94+%5BarXiv%3A2503%2E22362%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E22362%29)
15. Alfred Horn. 1951. On sentences which are true of direct unions of algebras1. The Journal of Symbolic Logic, 16(1):14–21. [find](https://scholar.google.com/scholar?q=Alfred+Horn%2E+1951%2E+On+sentences+which+are+true+of+direct+unions+of+algebras1%2E+The+Journal+of+Symbolic+Logic%2C+16%281%29%3A14%E2%80%9321%2E)
16. Gautier Izacard and Edouard Grave. 2021. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). In EACL, pages 874– 880. Association for Computational Linguistics.
17. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645. [find](https://scholar.google.com/scholar?q=Jinhao+Jiang%2C+Kun+Zhou%2C+Zican+Dong%2C+Keming+Ye%2C+Wayne+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+2023%2E+Structgpt%3A+A+general+framework+for+large+language+model+to+reason+over+structured+data%2E+arXiv+preprint+arXiv%3A2305%2E)
18. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+Omer+Levy%2C+Dan+Jurafsky%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2020%2E+Generalization+through+memorization%3A+Nearest+neighbor+language+models%2E+In+ICLR%2E+OpenReview%2Enet%2E)
19. Jonathan Lajus, Luis Galárraga, and Fabian Suchanek. 2020. Fast and exact rule mining with amie 3. In The Semantic Web: 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31–June 4, 2020, Proceedings 17, pages 36–52. Springer. [find](https://scholar.google.com/scholar?q=Jonathan+Lajus%2C+Luis+Gal%C3%A1rraga%2C+and+Fabian+Suchanek%2E+2020%2E+Fast+and+exact+rule+mining+with+amie+3%2E+In+The+Semantic+Web%3A+17th+International+Conference%2C+ESWC+2020%2C+Heraklion%2C+Crete%2C+Greece%2C+May+31%E2%80%93June+)
20. Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. A survey on complex knowledge base question answering: Methods, challenges and solutions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 4483–4491. International J… [find](https://scholar.google.com/scholar?q=Yunshi+Lan%2C+Gaole+He%2C+Jinhao+Jiang%2C+Jing+Jiang%2C+Wayne+Xin+Zhao%2C+and+Ji%2DRong+Wen%2E+2021%2E+A+survey+on+complex+knowledge+base+question+answering%3A+Methods%2C+challenges+and+solutions%2E+In+Proceedings+of+the+T)
21. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024. Reasoning on graphs: Faithful and interpretable large language model reasoning. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Linhao+Luo%2C+Yuan%2DFang+Li%2C+Gholamreza+Haffari%2C+and+Shirui+Pan%2E+2024%2E+Reasoning+on+graphs%3A+Faithful+and+interpretable+large+language+model+reasoning%2E+In+ICLR%2E+OpenReview%2Enet%2E)
22. Costas Mavromatis and George Karypis. 2024. Gnn-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139. [find](https://scholar.google.com/scholar?q=Costas+Mavromatis+and+George+Karypis%2E+2024%2E+Gnn%2Drag%3A+Graph+neural+retrieval+for+large+language+model+reasoning%2E+arXiv+preprint+arXiv%3A2405%2E20139%2E)
23. Aisha Mohamed, Shameem Parambath, Zoi Kaoudi, and Ashraf Aboulnaga. 2020. Popularity agnostic evaluation of knowledge graph embeddings. In Conference on Uncertainty in Artificial Intelligence, pages 1059– 1068. PMLR. [find](https://scholar.google.com/scholar?q=Aisha+Mohamed%2C+Shameem+Parambath%2C+Zoi+Kaoudi%2C+and+Ashraf+Aboulnaga%2E+2020%2E+Popularity+agnostic+evaluation+of+knowledge+graph+embeddings%2E+In+Conference+on+Uncertainty+in+Artificial+Intelligence%2C+pages+1)
24. OpenAI. 2024. Chatgpt(3.5)[large language model]. [https://chat.openai.com](https://chat.openai.com). [↗](https://chat.openai.com)
25. Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. 2024. [Logicbench: Towards systematic evaluation of logical reasoning ability of large language models](https://md.research.elamur.ai/free/en/papers/parmar-2024-logicbench-systematic-evaluation-logical-reasoning/). arXiv preprint arXiv:2404.15522.
26. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024. [Graph retrieval-augmented generation: A survey](https://md.research.elamur.ai/free/en/papers/peng-2024-graph-retrieval-augmented-generation-survey/). arXiv preprint arXiv:2408.08921.
27. Nico Potyka, Yuqicheng Zhu, Yunjie He, Evgeny Kharlamov, and Steffen Staab. 2024. Robust knowledge extraction from large language models using social choice theory. In Proceedings of the 23rd International Conference on Autonomous Agents and Multi-agent Systems, pages 1593–1601. [find](https://scholar.google.com/scholar?q=Nico+Potyka%2C+Yuqicheng+Zhu%2C+Yunjie+He%2C+Evgeny+Kharlamov%2C+and+Steffen+Staab%2E+2024%2E+Robust+knowledge+extraction+from+large+language+models+using+social+choice+theory%2E+In+Proceedings+of+the+23rd+Internat)
28. Yunqi Qiu, Yuanzhuo Wang, Xiaolong Jin, and Kun Zhang. 2020. Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In Proceedings of the 13th international conference on web search and data mining, pages 474–482. [find](https://scholar.google.com/scholar?q=Yunqi+Qiu%2C+Yuanzhuo+Wang%2C+Xiaolong+Jin%2C+and+Kun+Zhang%2E+2020%2E+Stepwise+reasoning+for+multi%2Drelation+question+answering+over+knowledge+graph+with+weak+supervision%2E+In+Proceedings+of+the+13th+internation)
29. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. [In-context retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/ram-2023-context-retrieval-augmented-language-models/). Trans. Assoc. Comput. Linguistics, 11:1316–1331.
30. Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in neural information processing systems, 32. [find](https://scholar.google.com/scholar?q=Ali+Sadeghian%2C+Mohammadreza+Armandpour%2C+Patrick+Ding%2C+and+Daisy+Zhe+Wang%2E+2019%2E+Drum%3A+End%2Dto%2Dend+differentiable+rule+mining+on+knowledge+graphs%2E+Advances+in+neural+information+processing+systems%2C+32%2E)
31. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697. [find](https://scholar.google.com/scholar?q=Jiashuo+Sun%2C+Chengjin+Xu%2C+Lumingyuan+Tang%2C+Saizhuo+Wang%2C+Chen+Lin%2C+Yeyun+Gong%2C+Lionel+M+Ni%2C+Heung%2DYeung+Shum%2C+and+Jian+Guo%2E+2023%2E+Think%2Don%2Dgraph%3A+Deep+and+responsible+reasoning+of+large+language+model)
32. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In ICLR. OpenReview.net. [find](https://scholar.google.com/scholar?q=Jiashuo+Sun%2C+Chengjin+Xu%2C+Lumingyuan+Tang%2C+Saizhuo+Wang%2C+Chen+Lin%2C+Yeyun+Gong%2C+Lionel+M%2E+Ni%2C+Heung%2DYeung+Shum%2C+and+Jian+Guo%2E+2024%2E+Think%2Don%2Dgraph%3A+Deep+and+responsible+reasoning+of+large+language+mode)
33. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2020. [Proofwriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/). arXiv preprint arXiv:2012.13048.
34. Alon Talmor and Jonathan Berant. 2018a. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). In NAACL-HLT, pages 641–651. Association for Computational Linguistics.
35. Alon Talmor and Jonathan Berant. 2018b. [The web as a knowledge-base for answering complex questions](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/). arXiv preprint arXiv:1803.06643.
36. Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021. [Diagnosing the first-order logical reasoning ability through logicnli](https://md.research.elamur.ai/free/en/papers/tian-2021-diagnosing-first-order-logical-reasoning/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3738–3747.
37. Kristina Toutanova and Danqi Chen. 2015. [Observed versus latent features for knowledge base and text inference](https://md.research.elamur.ai/free/en/papers/toutanova-2015-observed-versus-latent-features-knowledge/). In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 57–66.
38. Denny Vrandečić and Markus Krötzsch. 2014. [Wikidata: a free collaborative knowledgebase](https://md.research.elamur.ai/free/en/papers/vrandecic-2014-wikidata-free-collaborative-knowledgebase/). Communications of the ACM, 57(10):78–85.
39. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176–194. [find](https://scholar.google.com/scholar?q=Xiaozhi+Wang%2C+Tianyu+Gao%2C+Zhaocheng+Zhu%2C+Zhengyan+Zhang%2C+Zhiyuan+Liu%2C+Juanzi+Li%2C+and+Jian+Tang%2E+2021%2E+Kepler%3A+A+unified+model+for+knowledge+embedding+and+pre%2Dtrained+language+representation%2E+Transacti)
40. Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. 2024. Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering. arXiv preprint arXiv:2404.14741. — [arXiv:2404.14741](https://arxiv.org/abs/2404.14741) [find](https://scholar.google.com/scholar?q=Yao+Xu%2C+Shizhu+He%2C+Jiabei+Chen%2C+Zihao+Wang%2C+Yangqiu+Song%2C+Hanghang+Tong%2C+Guang+Liu%2C+Kang+Liu%2C+and+Jun+Zhao%2E+2024%2E+Generate%2Don%2Dgraph%3A+Treat+llm+as+both+agent+and+kg+in+incomplete+knowledge+graph+questi)
41. Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193. [find](https://scholar.google.com/scholar?q=Liang+Yao%2C+Chengsheng+Mao%2C+and+Yuan+Luo%2E+2019%2E+Kg%2Dbert%3A+Bert+for+knowledge+graph+completion%2E+arXiv+preprint+arXiv%3A1909%2E03193%2E)
42. Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2021. Rng-kbqa: Generation augmented iterative ranking for knowledge base question answering. arXiv preprint arXiv:2109.08678. [find](https://scholar.google.com/scholar?q=Xi+Ye%2C+Semih+Yavuz%2C+Kazuma+Hashimoto%2C+Yingbo+Zhou%2C+and+Caiming+Xiong%2E+2021%2E+Rng%2Dkbqa%3A+Generation+augmented+iterative+ranking+for+knowledge+base+question+answering%2E+arXiv+preprint+arXiv%3A2109%2E08678%2E)
43. Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. [The value of semantic parse labeling for knowledge base question answering](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/). In ACL (2). The Association for Computer Linguistics.
44. Dongzhuoran Zhou, Yuqicheng Zhu, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, and Steffen Staab. 2025a. Evaluating knowledge graph based retrieval augmented generation methods under knowledge incompleteness. arXiv preprint arXiv:2504.05163. — [arXiv:2504.05163](https://arxiv.org/abs/2504.05163) [find](https://scholar.google.com/scholar?q=Dongzhuoran+Zhou%2C+Yuqicheng+Zhu%2C+Yuan+He%2C+Jiaoyan+Chen%2C+Evgeny+Kharlamov%2C+and+Steffen+Staab%2E+2025a%2E+Evaluating+knowledge+graph+based+retrieval+augmented+generation+methods+under+knowledge+incompletene)
45. Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Jiaoyan Chen, Steffen Staab, Yuan He, and Evgeny Kharlamov. 2025b. Gr-agent: Adaptive graph reasoning agent under incomplete knowledge. arXiv preprint arXiv:2512.14766. — [arXiv:2512.14766](https://arxiv.org/abs/2512.14766) [find](https://scholar.google.com/scholar?q=Dongzhuoran+Zhou%2C+Yuqicheng+Zhu%2C+Xiaxia+Wang%2C+Hongkuan+Zhou%2C+Jiaoyan+Chen%2C+Steffen+Staab%2C+Yuan+He%2C+and+Evgeny+Kharlamov%2E+2025b%2E+Gr%2Dagent%3A+Adaptive+graph+reasoning+agent+under+incomplete+knowledge%2E+arX)
46. Yuqicheng Zhu, Daniel Hernández, Yuan He, Zifeng Ding, Bo Xiong, Evgeny Kharlamov, and Steffen Staab. 2025a. Predicate-conditional conformalized answer sets for knowledge graph embeddings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4145–4167, Vienna, Austria. Assoc… [find](https://scholar.google.com/scholar?q=Yuqicheng+Zhu%2C+Daniel+Hern%C3%A1ndez%2C+Yuan+He%2C+Zifeng+Ding%2C+Bo+Xiong%2C+Evgeny+Kharlamov%2C+and+Steffen+Staab%2E+2025a%2E+Predicate%2Dconditional+conformalized+answer+sets+for+knowledge+graph+embeddings%2E+In+Findings)
47. Yuqicheng Zhu, Nico Potyka, Daniel Hernández, Yuan He, Zifeng Ding, Bo Xiong, Dongzhuoran Zhou, Evgeny Kharlamov, and Steffen Staab. 2025b. Ar-grag: Explainable retrieval augmented generation using quantitative bipolar argumentation. In Conference on Neurosymbolic Learning and Reasoning, pages 697–7… [find](https://scholar.google.com/scholar?q=Yuqicheng+Zhu%2C+Nico+Potyka%2C+Daniel+Hern%C3%A1ndez%2C+Yuan+He%2C+Zifeng+Ding%2C+Bo+Xiong%2C+Dongzhuoran+Zhou%2C+Evgeny+Kharlamov%2C+and+Steffen+Staab%2E+2025b%2E+Ar%2Dgrag%3A+Explainable+retrieval+augmented+generation+using+qu)
48. Yuqicheng Zhu, Nico Potyka, Mojtaba Nayyeri, Bo Xiong, Yunjie He, Evgeny Kharlamov, and Steffen Staab. 2024. Predictive multiplicity of knowledge graph embeddings in link prediction. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 334–354, Miami, Florida, USA. Associa… [find](https://scholar.google.com/scholar?q=Yuqicheng+Zhu%2C+Nico+Potyka%2C+Mojtaba+Nayyeri%2C+Bo+Xiong%2C+Yunjie+He%2C+Evgeny+Kharlamov%2C+and+Steffen+Staab%2E+2024%2E+Predictive+multiplicity+of+knowledge+graph+embeddings+in+link+prediction%2E+In+Findings+of+th)

*Updated 31 October 2026*