---
title: "Conflicting Needles in a Haystack: How LLMs behave when"
description: "Retrieval-augmented and long-context applications often give LLMs documents that contradict… 22 references, cited by 5 papers here."
authors: ["Murathan Kurfali", "Robert Östling"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1742/"
doi: "10.18653/v1/2025.emnlp-main.1742"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kurfali-2025-conflicting-needles-haystack-how-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kurfali-2025-conflicting-needles-haystack-how-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information

# Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information

**Authors:** [Murathan Kurfali](https://md.research.elamur.ai/free/en/authors/murathan-kurfali/) and [Robert Östling](https://md.research.elamur.ai/free/en/authors/robert-ostling/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1742/) · [PDF](https://aclanthology.org/2025.emnlp-main.1742.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1742](https://doi.org/10.18653/v1/2025.emnlp-main.1742) · [OpenAlex](https://openalex.org/W4416035030)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/kurfali-2025-conflicting-needles-haystack-how-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Retrieval-augmented and long-context applications often give LLMs documents that contradict themselves, yet little is known about how models react. Kurfalı and Östling turn the Needle-in-a-Haystack (NIAH) test into an adversarial probe. Instead of one hidden fact, they plant three incompatible answers to a single question inside Wikipedia-based texts. The answers are invented pseudo-words, so none of them is correct. The test is whether a model notices the conflict and, if it does not, which option it picks. The authors vary how often each needle occurs, its order, the insertion layout, the question domain and whether the surrounding article is topically related. They test eight hypotheses with conditional logistic regression, applied both to normalized answer probabilities and to greedy generations. Open models of several families and sizes, plus two commercial systems, almost never flag the contradiction. Their choices follow systematic shortcuts: above all repetition, plus model-specific preferences for position and surface form. This suggests that RAG pipelines can be steered by noisy or deliberately manipulated content.

## Key points
- **Setup:** The needles are the pseudo-words Dralveth, Fyrundell and Zequarith, which had no Google hits. Each is the answer to one fictional "Xaltruvia" question in one of three domains: Geography, History or Biology. Haystacks are Wikipedia excerpts capped at 5,000 words (about 7,000 tokens).
- **Manipulated factors:** Each needle is repeated 1, 2 or 5 times. Needles can appear in 1st, 2nd or 3rd position. Layouts are placed at the beginning, middle or end and are either sequential or spaced 5% apart, with an even 25/50/75% baseline. The haystack's topical relatedness to the question also varies.
- **No hedging:** In generation mode, refusals or expressions of uncertainty were nearly absent. The two commercial models never refused and gave a single answer 92.6% of the time on average. Probability-based and generation-based choices agreed only 67.9% of the time.
- **Repetition dominates:** A needle repeated five times was far more likely to be chosen. Odds ratios were about 63 for Llama-3.1-8B in the probability analysis and about 181 for Gemma-2-9b in generation. Position effects differed by model: some favoured later slots, while others penalised them.
- **Scope:** Open models up to 70B parameters were tested, with English data only. The work is diagnostic and does not propose fixes for the biases.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stockholm University](https://md.research.elamur.ai/free/en/universities/stockholm-university/) — University, Stockholm, Sweden · 21 works · 1,832 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Murathan Kurfali and Robert Östling. 2025. Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kurfali2025conflicting,
  title = {{Conflicting Needles in a Haystack: How LLMs behave when faced with contradictory information}},
  author = {Murathan Kurfali and Robert Östling},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1742},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1742}
}
```

- [← previous BiMediX2 : Bio-Medical EXpert LMM for Diverse Medical Modalities](https://md.research.elamur.ai/free/en/papers/mullappilly-2025-bimedix2-bio-medical-expert-lmm/)
- [next → EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/zou-2025-eifbench-extremely-complex-instruction-following/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. [L-eval: Instituting standardized evaluation for long context language models](https://md.research.elamur.ai/free/en/papers/an-2023-l-eval-instituting-standardized-evaluation/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, and 1 others. 2024. [Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin…
3. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. [Benchmarking large language models in retrieval-augmented generation](https://md.research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762.
4. Roger Fletcher. 2000. Practical methods of optimization. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=Roger+Fletcher%2E+2000%2E+Practical+methods+of+optimization%2E+John+Wiley+%26+Sons%2E)
5. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. [Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/). arXiv preprint arXiv:2312.10997, 2:1.
6. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
7. Jennifer Hu and Roger Levy. 2023. [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060.
8. Gregory Kamradt. 2023. Needle in a haystack - pressure testing llms. [https://github.com/gkamradt/](https://github.com/gkamradt/) needle-in-a-haystack. GitHub repository. [↗](https://github.com/gkamradt/)
9. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A “novel” challenge for long-context language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17048–17085. [find](https://scholar.google.com/scholar?q=Marzena+Karpinska%2C+Katherine+Thai%2C+Kyle+Lo%2C+Tanya+Goyal%2C+and+Mohit+Iyyer%2E+2024%2E+One+thousand+and+one+pairs%3A+A+%E2%80%9Cnovel%E2%80%9D+challenge+for+long%2Dcontext+language+models%2E+In+Proceedings+of+the+2024+Conference+)
10. Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86. [find](https://scholar.google.com/scholar?q=Solomon+Kullback+and+Richard+A+Leibler%2E+1951%2E+On+information+and+sufficiency%2E+The+annals+of+mathematical+statistics%2C+22%281%29%3A79%E2%80%9386%2E)
11. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in neural information processing systems, 33:9459–…
12. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. [Lost in the middle: How language models use long contexts](https://md.research.elamur.ai/free/en/papers/liu-2023-lost-middle-language-models-use/). Transactions of the Association for Computational Linguistics, 12:157–173.
13. Chenyang Lyu, Minghao Wu, and Alham Aji. 2024. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 109–131. [find](https://scholar.google.com/scholar?q=Chenyang+Lyu%2C+Minghao+Wu%2C+and+Alham+Aji%2E+2024%2E+Beyond+probabilities%3A+Unveiling+the+misalignment+in+evaluating+large+language+models%2E+In+Proceedings+of+the+1st+Workshop+on+Towards+Knowledgeable+Languag)
14. Meta. 2024. Llama 3.3 Model Card. [https://github.com/meta-llama/llama-models/](https://github.com/meta-llama/llama-models/) blob/main/models/llama3_3/MODEL_CARD.md. Accessed: February 2025. [↗](https://github.com/meta-llama/llama-models/)
15. Ella Neeman, Roee Aharoni, Or Honnovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. [Disentqa: Disentangling parametric and contextual knowledge with counterfactual question answering](https://md.research.elamur.ai/free/en/papers/neeman-2023-disentqa-disentangling-parametric-contextual-knowledge/). In Annual Meeting of the Association for Computational Linguistics.
16. Quang Pham, Hoang Ngo, Luu Anh Tuan, and Dat Quoc Nguyen. 2024. Who’s who: Large language models meet knowledge conflicts in practice. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10142–10151. [find](https://scholar.google.com/scholar?q=Quang+Pham%2C+Hoang+Ngo%2C+Luu+Anh+Tuan%2C+and+Dat+Quoc+Nguyen%2E+2024%2E+Who%E2%80%99s+who%3A+Large+language+models+meet+knowledge+conflicts+in+practice%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMN)
17. Siyuan Song, Jennifer Hu, and Kyle Mahowald. 2025. [Language models fail to introspect about their knowledge of language](https://md.research.elamur.ai/free/en/papers/song-2025-language-models-fail-introspect-about/). arXiv e-prints, pages arXiv–2503.
18. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. [Long range arena: A benchmark for efficient transformers](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/). arXiv preprint arXiv:2011.04006.
19. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, and 179… [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Ta)
20. Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Rottger, and Barbara Plank. 2024a. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Xinpeng+Wang%2C+Chengzhi+Hu%2C+Bolei+Ma%2C+Paul+Rottger%2C+and+Barbara+Plank%2E+2024a%2E+Look+at+the+text%3A+Instruction%2Dtuned+language+models+are+more+robust+multiple+choice+selectors+than+you+think%2E+In+First+Conf)
21. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024b. [“my answer is c”: First-token probabilities do not match text answers in instruction-tuned language models](https://md.research.elamur.ai/free/en/papers/wang-2024-my-answer-is-c-first/). In Findings of the Association for Computational Linguistics ACL 2024…
22. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. [Retrieval meets long context large language models](https://md.research.elamur.ai/free/en/papers/xu-2023-retrieval-meets-long-context-large/). In The Twelfth International Conference on Learning Representations.

*Updated 31 October 2026*