---
title: "LongTailQA: Benchmarking LLMs and RAG Models on — Xion 2026"
description: "Benchmarks for long-tail question answering are often used to show that LLMs forget rare facts and that RAG helps."
authors: ["William Xion", "Uwe Hadler", "Tim Cofala", "Maximilian Idahl", "Soumyadeep Roy", "Wolfgang Nejdl"]
year: 2026
venue: "Proceedings of the Fifteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2026.lrec-1.405/"
doi: "10.63317/4tdekxzqph7x"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xion-2026-longtailqa-benchmarking-llms-rag-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xion-2026-longtailqa-benchmarking-llms-rag-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / LongTailQA: Benchmarking LLMs and RAG Models on Disambiguated Long-Tail Entities

# LongTailQA: Benchmarking LLMs and RAG Models on Disambiguated Long-Tail Entities

**Authors:** William Xion, Uwe Hadler, Tim Cofala, Maximilian Idahl, Soumyadeep Roy, [Wolfgang Nejdl](https://md.research.elamur.ai/free/en/authors/wolfgang-nejdl/) · **Year:** 2026 · **Venue:** Proceedings of the Fifteenth Language Resources and Evaluation Conference · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.lrec-1.405/) · [PDF](https://aclanthology.org/2026.lrec-1.405.pdf) · [DOI 10.63317/4tdekxzqph7x](https://doi.org/10.63317/4tdekxzqph7x) · [OpenAlex](https://openalex.org/W7160107058)

**Citations:** 0 citations (openalex-snapshot · 2026-10-14)

## Summary

Benchmarks for long-tail question answering are often used to show that LLMs forget rare facts and that RAG helps. The authors find that these benchmarks contain a hidden flaw. In the long-tail parts of PopQA, WITQA and EntityQA, 8–30% of questions name a subject that several different Wikidata entities share. A model can therefore give a correct answer about the wrong namesake and still be marked wrong.

To fix this, the authors took the low-popularity subsets of the three datasets, based on monthly Wikipedia page views. They rewrote ambiguous questions by adding Wikidata entity descriptions and removing any detail that would give away the answer. The result is LongTailQA, a released benchmark.

They then tested vanilla LLMs (Llama 2, Llama 3.1, Ministral, Qwen 2.5, GPT-4o) and RAG systems (Self-RAG, InstructRAG), varying both the retriever and the number of retrieved documents. Removing ambiguity raised accuracy noticeably, and RAG systems gained the most. The benchmark lets researchers separate a model's recall of rare facts from simple name confusion.

## Key points

- **Benchmark size and sources:** LongTailQA has 6,537 disambiguated questions covering 55 relation types. The questions come from the long-tail subsets of three datasets:
  - PopQA: 1,399 questions
  - EntityQA: 2,882 questions
  - WITQA: 2,256 questions
- **Cost of ambiguity:** In a manual check of 100 wrong answers from Self-RAG 13B on PopQA, 35% were correct facts about a different entity with the same name. Disambiguation improved accuracy by up to 24.7%.
- **RAG vs. vanilla LLMs:** InstructRAG-FT reached up to 90.2% accuracy, while vanilla LLMs mostly stayed under about 30%.
- **Use of retrieved evidence:** RAG systems achieved only about 80–85% of the accuracy their retrieval recall would allow. Raising retrieval depth from 5 to 20 documents did not reliably improve results.
- **Upper bound:** When given gold documents, models reached 90.9% accuracy. This suggests better retrieval and filtering of noisy documents could still bring gains. Code and data: github.com/williamx854/LongTailQA-Benchmark.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xion2026longtailqa,
  title = {{LongTailQA: Benchmarking LLMs and RAG Models on Disambiguated Long-Tail Entities}},
  author = {Xion, William and Hadler, Uwe and Cofala, Tim and Idahl, Maximilian and Roy, Soumyadeep and Nejdl, Wolfgang},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4tdekxzqph7x},
  url = {https://doi.org/10.63317/4tdekxzqph7x}
}
```

- [← previous LombardoGraphia: Automatic Classification of Lombard Orthography Variants](https://md.research.elamur.ai/free/en/papers/signoroni-2026-lombardographia-automatic-classification-lombard-orthography/)
- [next → LoRA Fine-Tuning of English–Norwegian NMT for the Oil & Gas Industry](https://md.research.elamur.ai/free/en/papers/yang-2026-lora-fine-tuning-english-norwegian/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mistral AI. 2024. Un ministral, des ministraux. [find](https://scholar.google.com/scholar?q=Mistral+AI%2E+2024%2E+Un+ministral%2C+des+ministraux%2E)
2. Akari Asai, Zeqiu Wu, Yizhong Wang, et al. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. [find](https://scholar.google.com/scholar?q=Akari+Asai%2C+Zeqiu+Wu%2C+Yizhong+Wang%2C+et+al%2E+2024%2E+Self%2Drag%3A+Learning+to+retrieve%2C+generate%2C+and+critique+through+self%2Dreflection%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2C+IC) [our card](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/)
3. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783. [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E+CoRR%2C+abs%2F2407%2E21783%2E)
4. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6465– 6488. Association for Computati… [find](https://scholar.google.com/scholar?q=Tianyu+Gao%2C+Howard+Yen%2C+Jiatong+Yu%2C+and+Danqi+Chen%2E+2023%2E+Enabling+large+language+models+to+generate+text+with+citations%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Methods+in+Natural+Language) [our card](https://md.research.elamur.ai/free/en/papers/gao-2023-enabling-large-language-models-generate/)
5. Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A. Rossi, Subhabrata Mukherjee, Xianfeng Tang, Qi He, Zhigang Hua, Bo Long, Tong Zhao, Neil Shah, Amin Javari, Yinglong Xia, and Jiliang Tang. 2025. Retrieval-augmented generation with graphs (graphrag… [find](https://scholar.google.com/scholar?q=Haoyu+Han%2C+Yu+Wang%2C+Harry+Shomer%2C+Kai+Guo%2C+Jiayuan+Ding%2C+Yongjia+Lei%2C+Mahantesh+Halappanavar%2C+Ryan+A%2E+Rossi%2C+Subhabrata+Mukherjee%2C+Xianfeng+Tang%2C+Qi+He%2C+Zhigang+Hua%2C+Bo+Long%2C+Tong+Zhao%2C+Neil+Shah%2C+Ami)
6. Gautier Izacard, Mathilde Caron, Lucas Hosseini, et al. 2022. Unsupervised dense information retrieval with contrastive learning. [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Mathilde+Caron%2C+Lucas+Hosseini%2C+et+al%2E+2022%2E+Unsupervised+dense+information+retrieval+with+contrastive+learning%2E) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-unsupervised-dense-information-retrieval-contrastive/)
7. Trans. Mach. Learn. Res., 2022. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2022%2E)
8. Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, et al. 2023. Atlas: Few-shot learning with retrieval augmented language models. J. Mach. Learn. Res., 24:251:1–251:43. [find](https://scholar.google.com/scholar?q=Gautier+Izacard%2C+Patrick+S%2E+H%2E+Lewis%2C+Maria+Lomeli%2C+et+al%2E+2023%2E+Atlas%3A+Few%2Dshot+learning+with+retrieval+augmented+language+models%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+24%3A251%3A1%E2%80%93251%3A43%2E)
9. Nikhil Kandpal, Haikang Deng, Adam Roberts, et al. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 15696–15707. PMLR. [find](https://scholar.google.com/scholar?q=Nikhil+Kandpal%2C+Haikang+Deng%2C+Adam+Roberts%2C+et+al%2E+2023%2E+Large+language+models+struggle+to+learn+long%2Dtail+knowledge%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2023%2C+23%2D29+July+2023%2C+Honol) [our card](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/)
10. Sewon Min, Kalpesh Krishna, Xinxi Lyu, et al. 2023. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 12076–12100. A… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Kalpesh+Krishna%2C+Xinxi+Lyu%2C+et+al%2E+2023%2E+Factscore%3A+Fine%2Dgrained+atomic+evaluation+of+factual+precision+in+long+form+text+generation%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+Meth) [our card](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/)
11. Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2… [find](https://scholar.google.com/scholar?q=Christopher+Sciavolino%2C+Zexuan+Zhong%2C+Jinhyuk+Lee%2C+and+Danqi+Chen%2E+2021%2E+Simple+entity%2Dcentric+questions+challenge+dense+retrievers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/)
12. Freda Shi, Xinyun Chen, Kanishka Misra, et al. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 31210–31227. PM… [find](https://scholar.google.com/scholar?q=Freda+Shi%2C+Xinyun+Chen%2C+Kanishka+Misra%2C+et+al%2E+2023%2E+Large+language+models+can+be+easily+distracted+by+irrelevant+context%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2023%2C+23%2D29+July+2023%2C+) [our card](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/)
13. Anmol Singhal, Chirag Jain, Preethu Rose Anish, Arkajyoti Chakraborty, and Smita Ghaisas. 2024. Generating clarification questions for disambiguating contracts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024… [find](https://scholar.google.com/scholar?q=Anmol+Singhal%2C+Chirag+Jain%2C+Preethu+Rose+Anish%2C+Arkajyoti+Chakraborty%2C+and+Smita+Ghaisas%2E+2024%2E+Generating+clarification+questions+for+disambiguating+contracts%2E+In+Proceedings+of+the+2024+Joint+Intern)
14. Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288. wangrui6. GitHub - wangrui6/Zhihu-KOL — github.com. [https://github.com/wangrui6/Zhihu-KOL](https://github.com/wangrui6/Zhihu-KOL). [Accessed 02-12-2024]. [↗](https://github.com/wangrui6/Zhihu-KOL)
15. Yuchen Wei and Milton King. 2024. Sense of the day: Short timeframe temporal-aware word sense disambiguation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 14676–14686… [find](https://scholar.google.com/scholar?q=Yuchen+Wei+and+Milton+King%2E+2024%2E+Sense+of+the+day%3A+Short+timeframe+temporal%2Daware+word+sense+disambiguation%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+Linguistics%2C+La)
16. Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025. Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales. [find](https://scholar.google.com/scholar?q=Zhepei+Wei%2C+Wei%2DLin+Chen%2C+and+Yu+Meng%2E+2025%2E+Instructrag%3A+Instructing+retrieval%2Daugmented+generation+via+self%2Dsynthesized+rationales%2E)
17. An Yang, Baosong Yang, Beichen Zhang, et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+et+al%2E+2024%2E+Qwen2%2E5+technical+report%2E+CoRR%2C+abs%2F2412%2E15115%2E)
18. Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, et al. 2023. Interpretable unified language checking. CoRR, abs/2304.03728. [find](https://scholar.google.com/scholar?q=Tianhua+Zhang%2C+Hongyin+Luo%2C+Yung%2DSung+Chuang%2C+et+al%2E+2023%2E+Interpretable+unified+language+checking%2E+CoRR%2C+abs%2F2304%2E03728%2E)
19. Zihan Zhang, Meng Fang, and Ling Chen. 2024. Retrievalqa: Assessing adaptive retrieval-augmented generation for short-form open-domain question answering. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 6963–697… [find](https://scholar.google.com/scholar?q=Zihan+Zhang%2C+Meng+Fang%2C+and+Ling+Chen%2E+2024%2E+Retrievalqa%3A+Assessing+adaptive+retrieval%2Daugmented+generation+for+short%2Dform+open%2Ddomain+question+answering%2E+In+Findings+of+the+Association+for+Computatio)
20. Language Resource References Xinyu Ma, Xuebo Liu, Derek F. Wong, Jun Rao, Bei Li, Liang Ding, Lidia S. Chao, Dacheng Tao, and Min Zhang. 2024. 3am: An ambiguity-aware multi-modal machine translation dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lang… [find](https://scholar.google.com/scholar?q=Language+Resource+References+Xinyu+Ma%2C+Xuebo+Liu%2C+Derek+F%2E+Wong%2C+Jun+Rao%2C+Bei+Li%2C+Liang+Ding%2C+Lidia+S%2E+Chao%2C+Dacheng+Tao%2C+and+Min+Zhang%2E+2024%2E+3am%3A+An+ambiguity%2Daware+multi%2Dmodal+machine+translation+d)
21. Seiji Maekawa, Hayate Iso, Sairam Gurajada, and Nikita Bhutani. 2024. Retrieval helps or hurts? A deeper dive into the efficacy of retrieval augmentation to language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… [find](https://scholar.google.com/scholar?q=Seiji+Maekawa%2C+Hayate+Iso%2C+Sairam+Gurajada%2C+and+Nikita+Bhutani%2E+2024%2E+Retrieval+helps+or+hurts%3F+A+deeper+dive+into+the+efficacy+of+retrieval+augmentation+to+language+models%2E+In+Proceedings+of+the+2024)
22. Alex Mallen, Akari Asai, Victor Zhong, et al. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada,… [find](https://scholar.google.com/scholar?q=Alex+Mallen%2C+Akari+Asai%2C+Victor+Zhong%2C+et+al%2E+2023%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+of+parametric+and+non%2Dparametric+memories%2E+In+Proceedings+of+the+61st+Annual+Meeting+)
23. Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021. Simple entity-centric questions challenge dense retrievers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2… [find](https://scholar.google.com/scholar?q=Christopher+Sciavolino%2C+Zexuan+Zhong%2C+Jinhyuk+Lee%2C+and+Danqi+Chen%2E+2021%2E+Simple+entity%2Dcentric+questions+challenge+dense+retrievers%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/sciavolino-2021-simple-entity-centric-questions-challenge/)