---
title: "Benchmarking LLM's Capability in Reasoning over Conflicting"
description: "To create the gold annotations, an LLM pulled answers and justifications from each document, and annotators grouped them… 21 references."
authors: ["Yizhen Yuan", "Rui Kong", "Dongze Li", "Yuanchun Li", "Yunxin Liu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.11/"
doi: "10.18653/v1/2026.acl-long.11"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yuan-2026-benchmarking-llm-s-capability-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuan-2026-benchmarking-llm-s-capability-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / Benchmarking LLM's Capability in Reasoning over Conflicting Web References

# Benchmarking LLM's Capability in Reasoning over Conflicting Web References

**Authors:** [Yizhen Yuan](https://md.research.elamur.ai/free/en/authors/yizhen-yuan/), [Rui Kong](https://md.research.elamur.ai/free/en/authors/rui-kong/), Dongze Li, [Yuanchun Li](https://md.research.elamur.ai/free/en/authors/yuanchun-li/) and [Yunxin Liu](https://md.research.elamur.ai/free/en/authors/yunxin-liu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.11/) · [PDF](https://aclanthology.org/2026.acl-long.11.pdf) · [DOI 10.18653/v1/2026.acl-long.11](https://doi.org/10.18653/v1/2026.acl-long.11) · [OpenAlex](https://openalex.org/W7166841539)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Web-search assistants built on RAG put every retrieved page into the model's context. These pages often disagree, so the LLM has to notice and handle conflicts that a search-engine user would normally sort out by hand. Existing contradiction datasets are small, synthetic, or limited to two short opposing sentences. The authors therefore built ConfRAG, which pairs 1,814 open-domain questions with real web paragraphs, averaging 9.58 per question. To create the gold annotations, an LLM pulled answers and justifications from each document, and annotators grouped them into viewpoint clusters with a human-in-the-loop process. Models are then tested on three tasks:
- assigning documents to the correct viewpoint groups;
- recovering every distinct answer;
- reproducing the reasons behind each answer.

Seven proprietary and open models were evaluated, and all of them fell well short of the gold annotations. Even GPT-4.1, the best performer, recovered fewer than half of the answers and few of the rationales. The results show that factual accuracy alone is not enough. Models also need to be aware of contradictions and sensitive to sources. The construction pipeline can be reused to build benchmarks for new domains.

## Key points
- **Dataset:** ConfRAG has 1,814 questions drawn from Natural Questions, ELI5, Yahoo Answers and hand-curated examples. Documents were found by keyword search and converted to clean text with jina.ai. Strong contradictions appear in 57.2% of the questions.
- **Metrics:** Answer clustering is scored with Normalized Mutual Information (NMI). Answer and reason coverage use bipartite maximum matching on keyword-based substring similarity. All scores fall between 0 and 1.
- **Models tested:** GPT-4.1, GPT-4.1-mini, Claude-3-7-Sonnet, Llama-3.1-70B/8B-Instruct and Qwen2.5-72B/7B-Instruct.
- **Results:**
  - GPT-4.1 led with NMI 0.47, answer coverage 0.45 and reason coverage 0.15.
  - The remaining models scored below 0.37 on answer coverage and generally under 0.14 on reason coverage.
  - Claude produced the most format-compliant outputs.
  - The 72B Qwen model clearly outperformed the 7B one.
- **Setup and availability:** Models receive the true number of clusters k as input, so a setting where k is unknown is left for future work. The data is released under CC BY 4.0 on GitHub (XaiverYuan/ConfRAG).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Yizhen Yuan, Rui Kong, Dongze Li, Yuanchun Li and Yunxin Liu. 2026. Benchmarking LLM's Capability in Reasoning over Conflicting Web References. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yuan2026benchmarking,
  title = {{Benchmarking LLM's Capability in Reasoning over Conflicting Web References}},
  author = {Yizhen Yuan, Rui Kong, Dongze Li, Yuanchun Li and Yunxin Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.11},
  url = {https://doi.org/10.18653/v1/2026.acl-long.11}
}
```

- [← previous Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis](https://md.research.elamur.ai/free/en/papers/tian-2026-benchmarking-large-language-models-game/)
- [next → Benchmarking LLMs for Aspect-Based Sentiment Classification in Slovene Historical Periodicals](https://md.research.elamur.ai/free/en/papers/munda-2026-benchmarking-llms-aspect-based-sentiment/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2024. The claude 3 model family: Opus, sonnet, haiku. Technical report by Anthropic. [find](https://scholar.google.com/scholar?q=2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+Technical+report+by+Anthropic%2E)
2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
3. Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. Geo: Generative engine optimization. In KDD 2024 - Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Proceedings of the ACM SIGKDD International Co… [find](https://scholar.google.com/scholar?q=Pranjal+Aggarwal%2C+Vishvak+Murahari%2C+Tanmay+Rajpurohit%2C+Ashwin+Kalyan%2C+Karthik+Narasimhan%2C+and+Ameet+Deshpande%2E+2024%2E+Geo%3A+Generative+engine+optimization%2E+In+KDD+2024+%2D+Proceedings+of+the+30th+ACM+SIGK)
4. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
5. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). Advances in Neural Information Processing Systems, 31.
6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
7. Pablo A Estévez, Michel Tesmer, Claudio A Perez, and Jacek M Zurada. 2009. Normalized mutual information feature selection. IEEE Transactions on neural networks, 20(2):189–201. [find](https://scholar.google.com/scholar?q=Pablo+A+Est%C3%A9vez%2C+Michel+Tesmer%2C+Claudio+A+Perez%2C+and+Jacek+M+Zurada%2E+2009%2E+Normalized+mutual+information+feature+selection%2E+IEEE+Transactions+on+neural+networks%2C+20%282%29%3A189%E2%80%93201%2E)
8. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
9. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. [A survey on rag meeting llms: Towards retrieval-augmented large language models](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/). In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.
10. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997, 2:1.
11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
12. Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, and Prasanna Sattigeri. 2024. Wikicontradict: A benchmark for evaluating llms on real-world knowledge conflicts from wikipedia. Advances in Neural Information Processing Systems, 37:… [find](https://scholar.google.com/scholar?q=Yufang+Hou%2C+Alessandra+Pascale%2C+Javier+Carnerero%2DCano%2C+Tigran+Tchrakian%2C+Radu+Marinescu%2C+Elizabeth+Daly%2C+Inkit+Padhi%2C+and+Prasanna+Sattigeri%2E+2024%2E+Wikicontradict%3A+A+benchmark+for+evaluating+llms+on+r)
13. Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. Won’t get fooled again: Answering questions with false premises. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5626– 5643, Toronto, Can… [find](https://scholar.google.com/scholar?q=Shengding+Hu%2C+Yifan+Luo%2C+Huadong+Wang%2C+Xingyi+Cheng%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Won%E2%80%99t+get+fooled+again%3A+Answering+questions+with+false+premises%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+t)
14. Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, and Deepak Ramachandran. 2023. Boardgameqa: A dataset for natural language reasoning with contradictory information. Advances in Neural Information Processing Systems, 36:39052–39074. [find](https://scholar.google.com/scholar?q=Mehran+Kazemi%2C+Quan+Yuan%2C+Deepti+Bhatia%2C+Najoung+Kim%2C+Xin+Xu%2C+Vaiva+Imbrasaite%2C+and+Deepak+Ramachandran%2E+2023%2E+Boardgameqa%3A+A+dataset+for+natural+language+reasoning+with+contradictory+information%2E+Adv)
15. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Lin…
16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
17. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+Gales%2E+2023%2E+SelfCheckGPT%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empir)
18. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa…
19. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. [RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models](https://md.research.elamur.ai/free/en/papers/niu-2024-ragtruth-hallucination-corpus-developing-trustworthy/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu…
20. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gabriel+Poesia%2C+Alex+Polozov%2C+Vu+Le%2C+Ashish+Tiwari%2C+Gustavo+Soares%2C+Christopher+Meek%2C+and+Sumit+Gulwani%2E+2022%2E+Synchromesh%3A+Reliable+code+generation+from+pre%2Dtrained+language+models%2E+In+International+)
21. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and 1 others. 2024. [jina-embeddings-v3: Multilingual embeddings with task lora](https://md.research.elamur.ai/free/en/papers/sturua-2024-jina-embeddings-v3-multilingual-embeddings/). arXiv preprint arXiv:2409.10173.

*Updated 2 November 2026*