---
title: "TruthTrap — paper by Shafiei et al. (2026) — summary and ideas"
description: "LLMs are often used to answer factual, information-seeking questions (ISQs). 63 references."
authors: ["Mohammadamin Shafiei", "Hamidreza Saffari", "Mohammad Taher Pilehvar", "Alessandro Raganato"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.155/"
doi: "10.18653/v1/2026.findings-eacl.155"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shafiei-2026-truthtrap-bilingual-benchmark-evaluating-factually/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shafiei-2026-truthtrap-bilingual-benchmark-evaluating-factually/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / TruthTrap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering

# TruthTrap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering

**Authors:** Mohammadamin Shafiei, Hamidreza Saffari, [Mohammad Taher Pilehvar](https://md.research.elamur.ai/free/en/authors/mohammad-taher-pilehvar/) and [Alessandro Raganato](https://md.research.elamur.ai/free/en/authors/alessandro-raganato/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.155/) · [PDF](https://aclanthology.org/2026.findings-eacl.155.pdf) · [DOI 10.18653/v1/2026.findings-eacl.155](https://doi.org/10.18653/v1/2026.findings-eacl.155) · [OpenAlex](https://openalex.org/W7140137854)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLMs are often used to answer factual, information-seeking questions (ISQs). Earlier robustness research has mostly tested how models react to false or contradictory context. This paper looks at a less-studied risk: statements that are true but off-target and still pull a model toward a wrong answer. The authors built TruthTrap, a manually verified benchmark of Iran-related questions drawn from Farsi Wikipedia, in Farsi with a post-edited English translation. Each item has four answer options, an explanation that supports the correct answer, and a persuasive hint that is accurate but irrelevant to the question. Nine proprietary and open-source LLMs were tested on multiple-choice QA and on factuality classification of the explanations and hints. Adding the misleading hint lowered accuracy by about 25% on average, and models frequently picked the option the hint pointed to. Models also often labelled true statements as false or uncertain when shown alone, yet still relied on them when answering. The work shows how partial truths can mislead models and adds a rare challenging evaluation resource for Farsi.

## Key points
- **Dataset:** TruthTrap has 1,000 items in Farsi plus 1,000 English counterparts. Each item has four options, a correct explanation, and a true-but-misleading hint, and the items span ten categories (e.g., History, Geography, Sports, Food).
- **Effect of hints:** Average baseline QA accuracy was about 48%. With the hint added, it fell to roughly 23–25%. Hint-aligned answers rose from about 22% to over 65%, reaching up to 77% for some models.
- **Effect of explanations:** Adding the correct explanation raised accuracy above 90% for almost all models in both languages, which confirms the questions are answerable.
- **Factuality classification:** Fewer than half of the true statements were labelled True. Answers shifted more often when a model had judged the added statement False or Uncertain (about 50%) than when it had judged it True (about 35%).
- **Model and language differences:** Open-source models (e.g., LLaMA, Qwen, DeepSeek) were more vulnerable to the hints, and GPT-4o held up comparatively well. Cross-lingual gaps reached about 10%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Cardiff University](https://md.research.elamur.ai/free/en/universities/cardiff-university/) — University, Cardiff, United Kingdom · 22 works · 2,444 citations
- [University of Milano-Bicocca](https://md.research.elamur.ai/free/en/universities/university-of-milano-bicocca/) — University, Milan, Italy · 15 works · 4,946 citations
- [University of Milan](https://md.research.elamur.ai/free/en/universities/university-of-milan/) — University, Milan, Italy · 6 works · 38,727 citations
- [Politecnico di Milano](https://md.research.elamur.ai/free/en/universities/politecnico-di-milano/) — University, Milan, Italy · 5 works · 258 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 577 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works

## Cite

Cite as: Mohammadamin Shafiei, Hamidreza Saffari, Mohammad Taher Pilehvar and Alessandro Raganato. 2026. TruthTrap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shafiei2026truthtrap,
  title = {{TruthTrap: A Bilingual Benchmark for Evaluating Factually Correct Yet Misleading Information in Question Answering}},
  author = {Mohammadamin Shafiei, Hamidreza Saffari, Mohammad Taher Pilehvar and Alessandro Raganato},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.155},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.155}
}
```

- [← previous TruthSplit: Operationalizing Conditional Validity in Arguments Through Multi-Perspective Reasoning](https://md.research.elamur.ai/free/en/papers/stieger-2026-truthsplit-operationalizing-conditional-validity-arguments/)
- [next → TryggLLM: A Benchmark for Evaluating LLM Safety in Norwegian](https://md.research.elamur.ai/free/en/papers/touileb-2026-tryggllm-benchmark-evaluating-llm-safety/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/)

## References

63 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sotiris Anagnostidis and Jannis Bulian. 2024. How susceptible are LLMs to influence in prompts? In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Sotiris+Anagnostidis+and+Jannis+Bulian%2E+2024%2E+How+susceptible+are+LLMs+to+influence+in+prompts%3F+In+First+Conference+on+Language+Modeling%2E)
2. Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar. 2024. Leancontext: Cost-efficient domain-specific question answering using llms. Natural Language Processing Journal, 7:100065. [find](https://scholar.google.com/scholar?q=Md+Adnan+Arefeen%2C+Biplob+Debnath%2C+and+Srimat+Chakradhar%2E+2024%2E+Leancontext%3A+Cost%2Defficient+domain%2Dspecific+question+answering+using+llms%2E+Natural+Language+Processing+Journal%2C+7%3A100065%2E)
3. Akari Asai and Eunsol Choi. 2021. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… [find](https://scholar.google.com/scholar?q=Akari+Asai+and+Eunsol+Choi%2E+2021%2E+Challenges+in+information%2Dseeking+QA%3A+Unanswerable+questions+and+paragraph+retrieval%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+L)
4. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+and+1+others%2E+2024%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longte)
5. Zahra Bokaei, Walid Magdy, and Bonnie Webber. 2025. Culture matters in toxic language detection in Persian. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9290–9304, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zahra+Bokaei%2C+Walid+Magdy%2C+and+Bonnie+Webber%2E+2025%2E+Culture+matters+in+toxic+language+detection+in+Persian%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28)
6. Carlos Carrasco-Farre. 2024. Large language models are as persuasive as humans, but how? about the cognitive effort and moral-emotional language of llm arguments. arXiv preprint arXiv:2404.09329. — [arXiv:2404.09329](https://arxiv.org/abs/2404.09329) [find](https://scholar.google.com/scholar?q=Carlos+Carrasco%2DFarre%2E+2024%2E+Large+language+models+are+as+persuasive+as+humans%2C+but+how%3F+about+the+cognitive+effort+and+moral%2Demotional+language+of+llm+arguments%2E+arXiv+preprint+arXiv%3A2404%2E09329%2E+%E2%80%94+%5Ba)
7. Tyler A Chang, Catherine Arnett, Abdelrahman Eldesokey, Abdelrahman Sadallah, Abeer Kashar, Abolade Daud, Abosede Grace Olanihun, Adamu Labaran Mohammed, Adeyemi Praise, Adhikarinayum Meerajita Sharma, and 1 others. 2025. Global piqa: Evaluating physical commonsense reasoning across 100+ languages a… — [arXiv:2510.24081](https://arxiv.org/abs/2510.24081) [find](https://scholar.google.com/scholar?q=Tyler+A+Chang%2C+Catherine+Arnett%2C+Abdelrahman+Eldesokey%2C+Abdelrahman+Sadallah%2C+Abeer+Kashar%2C+Abolade+Daud%2C+Abosede+Grace+Olanihun%2C+Adamu+Labaran+Mohammed%2C+Adeyemi+Praise%2C+Adhikarinayum+Meerajita+Sharma)
8. Gobinda Chowdhury and Sudatta Chowdhury. 2024. Ai-and llm-driven search tools: A paradigm shift in information access for education and research. Journal of Information Science, page 01655515241284046. [find](https://scholar.google.com/scholar?q=Gobinda+Chowdhury+and+Sudatta+Chowdhury%2E+2024%2E+Ai%2Dand+llm%2Ddriven+search+tools%3A+A+paradigm+shift+in+information+access+for+education+and+research%2E+Journal+of+Information+Science%2C+page+01655515241284046)
9. Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seeking question answering in ty pologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470… [find](https://scholar.google.com/scholar?q=Jonathan+H+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+2020%2E+Tydi+qa%3A+A+benchmark+for+information%2Dseeking+question+answering+in+ty+pol)
10. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. [A dataset of information-seeking questions and answers anchored in research papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). arXiv preprint arXiv:2105.03011.
11. Pedro Henrique Luz de Araujo and Benjamin Roth. 2024. Helpful assistant or fruitful facilitator? investigating how personas affect language model behavior. arXiv preprint arXiv:2407.02099. — [arXiv:2407.02099](https://arxiv.org/abs/2407.02099) [find](https://scholar.google.com/scholar?q=Pedro+Henrique+Luz+de+Araujo+and+Benjamin+Roth%2E+2024%2E+Helpful+assistant+or+fruitful+facilitator%3F+investigating+how+personas+affect+language+model+behavior%2E+arXiv+preprint+arXiv%3A2407%2E02099%2E+%E2%80%94+%5BarXiv%3A24)
12. Karen de Souza, Alexandre Nikolaev, and Maarit Koponen. 2025. Generative ai for technical writing: Comparing human and llm assessments of generated content. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDa… [find](https://scholar.google.com/scholar?q=Karen+de+Souza%2C+Alexandre+Nikolaev%2C+and+Maarit+Koponen%2E+2025%2E+Generative+ai+for+technical+writing%3A+Comparing+human+and+llm+assessments+of+generated+content%2E+In+Proceedings+of+the+Joint+25th+Nordic+Con)
13. Saba Emami and Maedeh Mosharraf. 2023. Farcqa: A farsi community dataset for question classification and answer selection. In 2023 13th International Conference on Computer and Knowledge Engineering (ICCKE), pages 567–572. [find](https://scholar.google.com/scholar?q=Saba+Emami+and+Maedeh+Mosharraf%2E+2023%2E+Farcqa%3A+A+farsi+community+dataset+for+question+classification+and+answer+selection%2E+In+2023+13th+International+Conference+on+Computer+and+Knowledge+Engineering+%28)
14. E Eskola. 1998. University students’ information seeking behaviour in a changing learning environment. how are students’ information needs, seeking and use affected by new teaching methods. Information Research, 4(2):4–2. [find](https://scholar.google.com/scholar?q=E+Eskola%2E+1998%2E+University+students%E2%80%99+information+seeking+behaviour+in+a+changing+learning+environment%2E+how+are+students%E2%80%99+information+needs%2C+seeking+and+use+affected+by+new+teaching+methods%2E+Informatio)
15. Marcos Fernández-Pichel, Juan C Pichel, and David E Losada. 2024. Search engines, llms or both? evaluating information seeking strategies for answering health questions. arXiv preprint arXiv:2407.12468. — [arXiv:2407.12468](https://arxiv.org/abs/2407.12468) [find](https://scholar.google.com/scholar?q=Marcos+Fern%C3%A1ndez%2DPichel%2C+Juan+C+Pichel%2C+and+David+E+Losada%2E+2024%2E+Search+engines%2C+llms+or+both%3F+evaluating+information+seeking+strategies+for+answering+health+questions%2E+arXiv+preprint+arXiv%3A2407%2E1246)
16. Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (persianmmlu): Is your llm truly wise to the persian language? Preprint, arXiv:2404.06644. — [arXiv:2404.06644](https://arxiv.org/abs/2404.06644) [find](https://scholar.google.com/scholar?q=Omid+Ghahroodi%2C+Marzia+Nouri%2C+Mohammad+Vali+Sanian%2C+Alireza+Sahebi%2C+Doratossadat+Dastgheib%2C+Ehsaneddin+Asgari%2C+Mahdieh+Soleymani+Baghshah%2C+and+Mohammad+Hossein+Rohban%2E+2024%2E+Khayyam+challenge+%28persian)
17. Lotem Golany, Filippo Galgani, Maya Mamo, Nimrod Parasol, Omer Vandsburger, Nadav Bar, and Ido Dagan. 2024. Efficient data generation for source-grounded information-seeking dialogs: A use case for meeting transcripts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1… [find](https://scholar.google.com/scholar?q=Lotem+Golany%2C+Filippo+Galgani%2C+Maya+Mamo%2C+Nimrod+Parasol%2C+Omer+Vandsburger%2C+Nadav+Bar%2C+and+Ido+Dagan%2E+2024%2E+Efficient+data+generation+for+source%2Dgrounded+information%2Dseeking+dialogs%3A+A+use+case+for+me)
18. Sara Bourbour Hosseinbeigi, Behnam Rohani, Mostafa Masoudi, Mehrnoush Shamsfard, Zahra Saaberi, Mostafa Karimi Manesh, and Mohammad Amin Abbasi. 2025. Advancing Persian LLM evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2711–2727, Albuquerque, New Mexico.… [find](https://scholar.google.com/scholar?q=Sara+Bourbour+Hosseinbeigi%2C+Behnam+Rohani%2C+Mostafa+Masoudi%2C+Mehrnoush+Shamsfard%2C+Zahra+Saaberi%2C+Mostafa+Karimi+Manesh%2C+and+Mohammad+Amin+Abbasi%2E+2025%2E+Advancing+Persian+LLM+evaluation%2E+In+Findings+of+)
19. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
20. Jia Hua Jeng, Gloria Anne Babile Kasangu, Alain D Starke, Erik Knudsen, and Christoph Trattner. 2024. Negativity sells? using an llm to affectively reframe news articles in a recommender system. In ACM Conference on Recommender Systems (RecSys’ 24). [find](https://scholar.google.com/scholar?q=Jia+Hua+Jeng%2C+Gloria+Anne+Babile+Kasangu%2C+Alain+D+Starke%2C+Erik+Knudsen%2C+and+Christoph+Trattner%2E+2024%2E+Negativity+sells%3F+using+an+llm+to+affectively+reframe+news+articles+in+a+recommender+system%2E+In+AC)
21. Robin Jia and Percy Liang. 2017. [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
22. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. [HoVer: A dataset for many-hop fact extraction and claim verification](https://md.research.elamur.ai/free/en/papers/jiang-2020-hover-dataset-many-hop-fact/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460, Online. Association for Computational Ling…
23. Chuhao Jin, Kening Ren, Lingzhen Kong, Xiting Wang, Ruihua Song, and Huan Chen. 2024. Persuading across diverse domains: a dataset and persuasion large language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1678– 1706… [find](https://scholar.google.com/scholar?q=Chuhao+Jin%2C+Kening+Ren%2C+Lingzhen+Kong%2C+Xiting+Wang%2C+Ruihua+Song%2C+and+Huan+Chen%2E+2024%2E+Persuading+across+diverse+domains%3A+a+dataset+and+persuasion+large+language+model%2E+In+Proceedings+of+the+62nd+Annua)
24. Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307.16883. — [arXiv:2307.16883](https://arxiv.org/abs/2307.16883) [find](https://scholar.google.com/scholar?q=Ehsan+Kamalloo%2C+Aref+Jafari%2C+Xinyu+Zhang%2C+Nandan+Thakur%2C+and+Jimmy+Lin%2E+2023%2E+Hagrid%3A+A+human%2Dllm+collaborative+dataset+for+generative+information%2Dseeking+with+attribution%2E+arXiv+preprint+arXiv%3A2307%2E1)
25. Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadeg… — [ParsiNLU: A Suite of Language Understanding Challenges for Persian](https://md.research.elamur.ai/free/en/papers/khashabi-2021-parsinlu-suite-language-understanding-challenges/)
26. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, and 1 others. 2019. [Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). Transactions of the Association for Computational Lin…
27. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. [Mlqa: Evaluating cross-lingual extractive question answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330.
28. Bryan Li, Aleksey Panasyuk, and Chris Callison-Burch. 2024. Uncovering differences in persuasive language in Russian versus English Wikipedia. In Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia, pages 21–35, Miami, Florida, USA. Association for Computational… [find](https://scholar.google.com/scholar?q=Bryan+Li%2C+Aleksey+Panasyuk%2C+and+Chris+Callison%2DBurch%2E+2024%2E+Uncovering+differences+in+persuasive+language+in+Russian+versus+English+Wikipedia%2E+In+Proceedings+of+the+First+Workshop+on+Advancing+Natural)
29. Louise Limberg and Olof Sundin. 2006. Teaching information seeking: relating information literacy education to theories of information behaviour. Information Research: an international electronic journal, 12(1):n1. [find](https://scholar.google.com/scholar?q=Louise+Limberg+and+Olof+Sundin%2E+2006%2E+Teaching+information+seeking%3A+relating+information+literacy+education+to+theories+of+information+behaviour%2E+Information+Research%3A+an+international+electronic+jour)
30. Siyi Liu, Qiang Ning, Kishaloy Halder, Zheng Qi, Wei Xiao, Phu Mon Htut, Yi Zhang, Neha Anna John, Bonan Min, Yassine Benajiba, and Dan Roth. 2025. Open domain question answering with conflicting contexts. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1838– 1854, Al… [find](https://scholar.google.com/scholar?q=Siyi+Liu%2C+Qiang+Ning%2C+Kishaloy+Halder%2C+Zheng+Qi%2C+Wei+Xiao%2C+Phu+Mon+Htut%2C+Yi+Zhang%2C+Neha+Anna+John%2C+Bonan+Min%2C+Yassine+Benajiba%2C+and+Dan+Roth%2E+2025%2E+Open+domain+question+answering+with+conflicting+cont)
31. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
32. Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. 2024. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association, 31(9):1964–1975. [find](https://scholar.google.com/scholar?q=Mary+M+Lucas%2C+Justin+Yang%2C+Jon+K+Pomeroy%2C+and+Christopher+C+Yang%2E+2024%2E+Reasoning+with+large+language+models+for+medical+question+answering%2E+Journal+of+the+American+Medical+Informatics+Association%2C+31)
33. Nishith Reddy Mannuru, Aashrith Mannuru, and Brady Lund. 2024. Large language models (llms) as a tool to facilitate information seeking behavior. InfoS-cience Trends, 1(3):34–42. [find](https://scholar.google.com/scholar?q=Nishith+Reddy+Mannuru%2C+Aashrith+Mannuru%2C+and+Brady+Lund%2E+2024%2E+Large+language+models+%28llms%29+as+a+tool+to+facilitate+information+seeking+behavior%2E+InfoS%2Dcience+Trends%2C+1%283%29%3A34%E2%80%9342%2E)
34. AI Meta, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783, 2.
35. Jyoti Mishra, David Allen, and Alan Pearman. 2015. Information seeking, use, and decision making. Journal of the association for information science and technology, 66(4):662–673. [find](https://scholar.google.com/scholar?q=Jyoti+Mishra%2C+David+Allen%2C+and+Alan+Pearman%2E+2015%2E+Information+seeking%2C+use%2C+and+decision+making%2E+Journal+of+the+association+for+information+science+and+technology%2C+66%284%29%3A662%E2%80%93673%2E)
36. Joao Monteiro, Pierre-Andre Noel, Etienne Marcotte, Sai Rajeswar Mudumba, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Chris Pal, and Perouz Taslakian. 2024. Repliqa: A question-answering dataset for benchmarking llms on unseen reference content. Advances in Neural Information Processing… [find](https://scholar.google.com/scholar?q=Joao+Monteiro%2C+Pierre%2DAndre+Noel%2C+Etienne+Marcotte%2C+Sai+Rajeswar+Mudumba%2C+Valentina+Zantedeschi%2C+David+Vazquez%2C+Nicolas+Chapados%2C+Chris+Pal%2C+and+Perouz+Taslakian%2E+2024%2E+Repliqa%3A+A+question%2Danswering+d)
37. Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, and Mohammad Taher Pilehvar. 2025. PerCul: A story-driven cultural evaluation of LLMs in Persian. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Erfan+Moosavi+Monazzah%2C+Vahid+Rahimzadeh%2C+Yadollah+Yaghoobzadeh%2C+Azadeh+Shakery%2C+and+Mohammad+Taher+Pilehvar%2E+2025%2E+PerCul%3A+A+story%2Ddriven+cultural+evaluation+of+LLMs+in+Persian%2E+In+Proceedings+of+the)
38. S Narayanan, William Bailey, Juee Tendulkar, Karen Wilson, Raymond Daley, and Daniel Pliske. 1999. Modeling real-world information seeking in a corporate environment. Human Factors and Ergonomics in Manufacturing & Service Industries, 9(2):203– 229. [find](https://scholar.google.com/scholar?q=S+Narayanan%2C+William+Bailey%2C+Juee+Tendulkar%2C+Karen+Wilson%2C+Raymond+Daley%2C+and+Daniel+Pliske%2E+1999%2E+Modeling+real%2Dworld+information+seeking+in+a+corporate+environment%2E+Human+Factors+and+Ergonomics+in+M)
39. Yishai Ofran, Ora Paltiel, Dan Pelleg, Jacob M Rowe, and Elad Yom-Tov. 2012. Patterns of information-seeking for cancer on the internet: an analysis of real world data. [find](https://scholar.google.com/scholar?q=Yishai+Ofran%2C+Ora+Paltiel%2C+Dan+Pelleg%2C+Jacob+M+Rowe%2C+and+Elad+Yom%2DTov%2E+2012%2E+Patterns+of+information%2Dseeking+for+cancer+on+the+internet%3A+an+analysis+of+real+world+data%2E)
40. Chaoxu Pang, Yixuan Cao, Chunhao Yang, and Ping Luo. 2024. Uncovering limitations of large language models in information seeking from tables. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1388–1409, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chaoxu+Pang%2C+Yixuan+Cao%2C+Chunhao+Yang%2C+and+Ping+Luo%2E+2024%2E+Uncovering+limitations+of+large+language+models+in+information+seeking+from+tables%2E+In+Findings+of+the+Association+for+Computational+Linguist)
41. Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Faviq: Fact verification from information-seeking questions. arXiv preprint arXiv:2107.02153. [find](https://scholar.google.com/scholar?q=Jungsoo+Park%2C+Sewon+Min%2C+Jaewoo+Kang%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Faviq%3A+Fact+verification+from+information%2Dseeking+questions%2E+arXiv+preprint+arXiv%3A2107%2E02153%2E)
42. J Perchik. 2023. Does chatgpt pass the lirads test? comparing quality of ai generated impressions to human reports. J Gastro Hepato, 10(5):1–5. [find](https://scholar.google.com/scholar?q=J+Perchik%2E+2023%2E+Does+chatgpt+pass+the+lirads+test%3F+comparing+quality+of+ai+generated+impressions+to+human+reports%2E+J+Gastro+Hepato%2C+10%285%29%3A1%E2%80%935%2E)
43. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
44. Till Raphael Saenger, Musashi Hinck, Justin Grimmer, and Brandon M. Stewart. 2024. AutoPersuade: A framework for evaluating and explaining persuasive arguments. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16325–16342, Miami, Florida, USA. Associat… [find](https://scholar.google.com/scholar?q=Till+Raphael+Saenger%2C+Musashi+Hinck%2C+Justin+Grimmer%2C+and+Brandon+M%2E+Stewart%2E+2024%2E+AutoPersuade%3A+A+framework+for+evaluating+and+explaining+persuasive+arguments%2E+In+Proceedings+of+the+2024+Conference+o)
45. Hamidreza Saffari, Mohammadamin Shafiei, Donya Rooein, and Debora Nozza. 2025a. Measuring gender bias in language models in Farsi. In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 228–241, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hamidreza+Saffari%2C+Mohammadamin+Shafiei%2C+Donya+Rooein%2C+and+Debora+Nozza%2E+2025a%2E+Measuring+gender+bias+in+language+models+in+Farsi%2E+In+Proceedings+of+the+6th+Workshop+on+Gender+Bias+in+Natural+Language)
46. Hamidreza Saffari, Mohammadamin Shafiei, Donya Rooein, Francesco Pierri, and Debora Nozza. 2025b. Can I introduce my boyfriend to my grandmother? evaluating large language models capabilities on Iranian social norm classification. In Findings of the Association for Computational Linguistics: NAACL 2… [find](https://scholar.google.com/scholar?q=Hamidreza+Saffari%2C+Mohammadamin+Shafiei%2C+Donya+Rooein%2C+Francesco+Pierri%2C+and+Debora+Nozza%2E+2025b%2E+Can+I+introduce+my+boyfriend+to+my+grandmother%3F+evaluating+large+language+models+capabilities+on+Irani)
47. Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2023. In-context impersonation reveals large language models’ strengths and biases. Advances in neural information processing systems, 36:72044–72057. [find](https://scholar.google.com/scholar?q=Leonard+Salewski%2C+Stephan+Alaniz%2C+Isabel+Rio%2DTorto%2C+Eric+Schulz%2C+and+Zeynep+Akata%2E+2023%2E+In%2Dcontext+impersonation+reveals+large+language+models%E2%80%99+strengths+and+biases%2E+Advances+in+neural+information+pr)
48. Tefko Saracevic, Paul Kantor, Alice Y Chamis, and Donna Trivison. 1988. A study of information seeking and retrieving. i. background and methodology. Journal of the American Society for Information science, 39(3):161–176. [find](https://scholar.google.com/scholar?q=Tefko+Saracevic%2C+Paul+Kantor%2C+Alice+Y+Chamis%2C+and+Donna+Trivison%2E+1988%2E+A+study+of+information+seeking+and+retrieving%2E+i%2E+background+and+methodology%2E+Journal+of+the+American+Society+for+Information+sc)
49. Joshua M Scacco and Ashley Muddiman. 2020. The curiosity effect: Information seeking in the contemporary news environment. New Media & Society, 22(3):429–448. [find](https://scholar.google.com/scholar?q=Joshua+M+Scacco+and+Ashley+Muddiman%2E+2020%2E+The+curiosity+effect%3A+Information+seeking+in+the+contemporary+news+environment%2E+New+Media+%26+Society%2C+22%283%29%3A429%E2%80%93448%2E)
50. Mohammadamin Shafiei, Hamidreza Saffari, and Nafise Sadat Moosavi. 2025. MultiHoax: A dataset of multi-hop false-premise questions. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10169–10187, Vienna, Austria. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mohammadamin+Shafiei%2C+Hamidreza+Saffari%2C+and+Nafise+Sadat+Moosavi%2E+2025%2E+MultiHoax%3A+A+dataset+of+multi%2Dhop+false%2Dpremise+questions%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+20)
51. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. [Large language models can be easily distracted by irrelevant context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). In International Conference on Machine Learning, pages 31210–31227. PMLR.
52. Alexander Rogiers, Sander Noels, Maarten Buyl, and Tijl De Bie. 2024. Persuasion with [large language models: a survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/). arXiv preprint arXiv:2411.06837. — [arXiv:2411.06837](https://arxiv.org/abs/2411.06837)
53. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [FEVER: a large-scale dataset for fact extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/). In NAACL-HLT.
54. Nikta Gohari Sadr, Sahar Heidariasl, Karine Megerdoomian, Laleh Seyyed-Kalantari, and Ali Emami. 2025. We politely insist: Your llm must learn the persian art of taarof. arXiv preprint arXiv:2509.01035. [find](https://scholar.google.com/scholar?q=Nikta+Gohari+Sadr%2C+Sahar+Heidariasl%2C+Karine+Megerdoomian%2C+Laleh+Seyyed%2DKalantari%2C+and+Ali+Emami%2E+2025%2E+We+politely+insist%3A+Your+llm+must+learn+the+persian+art+of+taarof%2E+arXiv+preprint+arXiv%3A2509%2E0103)
55. Lieke LF van Lieshout, Floris P de Lange, and Roshan Cools. 2020. Why so curious? quantifying mechanisms of information seeking. Current Opinion in Behavioral Sciences, 35:112–117. [find](https://scholar.google.com/scholar?q=Lieke+LF+van+Lieshout%2C+Floris+P+de+Lange%2C+and+Roshan+Cools%2E+2020%2E+Why+so+curious%3F+quantifying+mechanisms+of+information+seeking%2E+Current+Opinion+in+Behavioral+Sciences%2C+35%3A112%E2%80%93117%2E)
56. Krithik Vishwanath, Anton Alyakin, Daniel Alexander Alber, Jin Vivian Lee, Douglas Kondziolka, and Eric Karl Oermann. 2025. Medical large language models are easily distracted. arXiv preprint arXiv:2504.01201. — [arXiv:2504.01201](https://arxiv.org/abs/2504.01201) [find](https://scholar.google.com/scholar?q=Krithik+Vishwanath%2C+Anton+Alyakin%2C+Daniel+Alexander+Alber%2C+Jin+Vivian+Lee%2C+Douglas+Kondziolka%2C+and+Eric+Karl+Oermann%2E+2025%2E+Medical+large+language+models+are+easily+distracted%2E+arXiv+preprint+arXiv%3A25)
57. Yaqing Wang, Jiepu Jiang, Mingyang Zhang, Cheng Li, Yi Liang, Qiaozhu Mei, and Michael Bendersky. 2023. Automated evaluation of personalized text generation using large language models. arXiv preprint arXiv:2310.11593. category as well as the number of times when the hint targets the correct answer… — [arXiv:2310.11593](https://arxiv.org/abs/2310.11593) [find](https://scholar.google.com/scholar?q=Yaqing+Wang%2C+Jiepu+Jiang%2C+Mingyang+Zhang%2C+Cheng+Li%2C+Yi+Liang%2C+Qiaozhu+Mei%2C+and+Michael+Bendersky%2E+2023%2E+Automated+evaluation+of+personalized+text+generation+using+large+language+models%2E+arXiv+preprint)
58. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+and+1+others%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2E)
59. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
60. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. [How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms](https://md.research.elamur.ai/free/en/papers/zeng-2024-how-johnny-can-persuade-llms/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
61. Count Date or Time Location Number Person Other proper and common nouns Group or Organization Other Event Artwork 198 178 165 151 109 105 42 27 25 Sum *Table 5: Distribution of question classes in our final set of 1,000 curated QA items. Each question is assigned a class (e.g., Date/Time, Location,… [find](https://scholar.google.com/scholar?q=Count+Date+or+Time+Location+Number+Person+Other+proper+and+common+nouns+Group+or+Organization+Other+Event+Artwork+198+178+165+151+109+105+42+27+25+Sum+%2ATable+5%3A+Distribution+of+question+classes+in+our)
62. Kamyar Zeinalipour, Neda Jamshidi, Fahimeh Akbari, Marco Maggini, Monica Bianchini, and Marco Gori. 2025. PersianMCQ-instruct: A comprehensive resource for generating multiple-choice questions in Persian. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 344–3… [find](https://scholar.google.com/scholar?q=Kamyar+Zeinalipour%2C+Neda+Jamshidi%2C+Fahimeh+Akbari%2C+Marco+Maggini%2C+Monica+Bianchini%2C+and+Marco+Gori%2E+2025%2E+PersianMCQ%2Dinstruct%3A+A+comprehensive+resource+for+generating+multiple%2Dchoice+questions+in+Pers)
63. Question class Target About an option other than the answer About the answer Count 956 44 [find](https://scholar.google.com/scholar?q=Question+class+Target+About+an+option+other+than+the+answer+About+the+answer+Count+956+44)

*Updated 1 November 2026*