---
title: "AbstentionBench — paper by Kirichenko et al. (2025)"
description: "The authors argue that declining or hedging in these cases matters as much as accuracy, but no… 92 references, cited by 33 papers here."
authors: ["Polina Kirichenko", "Mark Ibrahim", "Kamalika Chaudhuri", "Samuel J. Bell"]
year: 2025
venue: "arXiv:2506.09038"
source: "https://arxiv.org/abs/2506.09038"
doi: "10.48550/arxiv.2506.09038"
arxiv: "2506.09038"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kirichenko-2025-abstentionbench-reasoning-llms-fail-unanswerable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kirichenko-2025-abstentionbench-reasoning-llms-fail-unanswerable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/) / AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions

# AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions

**Authors:** Polina Kirichenko, [Mark Ibrahim](https://md.research.elamur.ai/free/en/authors/mark-ibrahim/), Kamalika Chaudhuri and Samuel J. Bell · **Year:** 2025 · **Venue:** [arXiv:2506.09038](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2025](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2025/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2506.09038) · [PDF](https://arxiv.org/pdf/2506.09038) · [DOI 10.48550/arxiv.2506.09038](https://doi.org/10.48550/arxiv.2506.09038) · [arXiv 2506.09038](https://arxiv.org/abs/2506.09038) · [OpenAlex](https://openalex.org/W4417258166)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [33](https://md.research.elamur.ai/free/en/papers/kirichenko-2025-abstentionbench-reasoning-llms-fail-unanswerable/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Large language models often receive questions that have no reliable answer. A request may omit key details, rest on a false assumption, depend on opinion, or concern facts that are outdated or unknown. The authors argue that declining or hedging in these cases matters as much as accuracy, but no broad evaluation of this ability existed. They built AbstentionBench, which combines 20 datasets chosen through a systematic review. These include new underspecified versions of GSM8K, GPQA and MMLU. Responses are scored automatically by an LLM judge whose quality was checked. Tests on 20 frontier models, both open and closed, show that knowing when to abstain is largely unsolved and barely improves with model size. Models handle only questions with genuinely unknown answers well. Reasoning-oriented fine-tuning makes abstention worse, even in the math and science domains it targets. Longer thinking budgets raise accuracy but further hurt abstention. A tailored system prompt helps somewhat but does not fix the underlying gap. The benchmark therefore gives reliability research a goal beyond accuracy.

## Key points

- AbstentionBench contains over 35k unanswerable questions across six scenario types: underspecified intent, underspecified context, false premise, stale data, subjective questions, and answer unknown. This is about 35× more prompts than CoCoNot.
- Sources were filtered from a shortlist of 82 candidate datasets. They include BBQ, BIG-Bench tasks, CoCoNot, FalseQA, FreshQA, KUQ, MediQ, MoralChoice, Musique, QASPER, SituatedQA (Geo), SQuAD 2.0, WorldSense and UMWP, plus the new GSM8K-Abstain, GPQA-Abstain and MMLU-Abstain.
- DeepSeek R1 (Distill Llama 70B) and s1 abstain about 24% less on average than their non-reasoning counterparts. They often invent the missing context and give a firm final answer even when their reasoning traces express doubt.
- For s1.1, raising the reasoning budget from 512 to 4096 tokens improves accuracy, while abstention recall stays flat or declines.
- Code and data are available at github.com/facebookresearch/AbstentionBench.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 233 works · 104,005 citations
- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 101 works · 54,571 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,621 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,948 works

## Cite

Cite as: Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri and Samuel J. Bell. 2025. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{kirichenko2025abstentionbench,
  title = {{AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions}},
  author = {Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri and Samuel J. Bell},
  year = {2025},
  doi = {10.48550/arxiv.2506.09038},
  eprint = {2506.09038},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.09038}
}
```

- [← previous WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/wei-2025-webagent-r1-training-web-agents/)
- [next → Benchmarking Distributional Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/meister-2025-benchmarking-distributional-alignment-large-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/4/)

## References

92 references, 53 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 29(8):1930–1940, 2023. [find](https://scholar.google.com/scholar?q=Arun+James+Thirunavukarasu%2C+Darren+Shu+Jeng+Ting%2C+Kabilan+Elangovan%2C+Laura+Gutierrez%2C+Ting+Fang+Tan%2C+and+Daniel+Shu+Wei+Ting%2E+Large+language+models+in+medicine%2E+Nature+Medicine%2C+29%288%29%3A1930%E2%80%931940%2C+2023%2E)
2. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. [Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/). Advances in Neural Information Processing…
3. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. [BloombergGPT: A Large Language Model for Finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). (arXiv:2303.17564), 2023.
4. Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, and Mark Ibrahim. Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations. (arXiv:2404.10960), 2024. — [arXiv:2404.10960](https://arxiv.org/abs/2404.10960) [find](https://scholar.google.com/scholar?q=Christian+Tomani%2C+Kamalika+Chaudhuri%2C+Ivan+Evtimov%2C+Daniel+Cremers%2C+and+Mark+Ibrahim%2E+Uncertainty%2DBased+Abstention+in+LLMs+Improves+Safety+and+Reduces+Hallucinations%2E+%28arXiv%3A2404%2E10960%29%2C+2024%2E+%E2%80%94+%5BarXi)
5. Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, and Masoud Hashemi. [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://md.research.elamur.ai/free/en/papers/madhusudhan-2025-do-llms-know-when-not/). In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proc…
6. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Transactions on Information Systems, 2023.
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training Verifiers to Solve Math Word Problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). (arXiv:2110.14168), 2021.
8. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling, 2024. URL [https://openreview.net/forum?id=Ti67584b98\#](https://openreview.net/forum?id=Ti67584b98\#) discussion. [↗](https://openreview.net/forum?id=Ti67584b98\#)
9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=d7KBjmI3GmQ](https://openreview.net/forum?id=d7KBjmI3GmQ). [↗](https://openreview.net/forum?id=d7KBjmI3GmQ)
10. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
11. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://md.research.elamur.ai/free/en/papers/ahn-2024-large-language-models-mathematical-reasoning/). (arXiv:2402.00157), 2024.
12. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL [https://openreview.net/forum?id=YfZ4ZPt8zd](https://openreview.net/forum?id=YfZ4ZPt8zd). [↗](https://openreview.net/forum?id=YfZ4ZPt8zd)
13. DeepSeek-AI. [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). (arXiv:2501.12948), 2025.
14. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. [S1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). (arXiv:2501.19393), 2025.
15. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do Large Language Models Know What They Don‘t Know? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665. Association for… [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Jiawen+Wu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+Do+Large+Language+Models+Know+What+They+Don%E2%80%98t+Know%3F+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki%2C+editors%2C+Fi)
16. Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen, and William Wang. [Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models](https://md.research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/). (arXiv:2305.13712), 2024.
17. Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on… [find](https://scholar.google.com/scholar?q=Aviv+Slobodkin%2C+Omer+Goldman%2C+Avi+Caciularu%2C+Ido+Dagan%2C+and+Shauli+Ravfogel%2E+The+Curious+Case+of+Hallucinatory+%28Un%29answerability%3A+Finding+Truths+in+the+Hidden+States+of+Over%2DConfident+Large+Language+M)
18. Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. [CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2024-clamber-benchmark-identifying-clarifying-ambiguous/). In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of t…
19. Belinda Z. Li, Been Kim, and Zi Wang. QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? (arXiv:2503.22674), 2025. — [arXiv:2503.22674](https://arxiv.org/abs/2503.22674) [find](https://scholar.google.com/scholar?q=Belinda+Z%2E+Li%2C+Been+Kim%2C+and+Zi+Wang%2E+QuestBench%3A+Can+LLMs+ask+the+right+question+to+acquire+information+in+reasoning+tasks%3F+%28arXiv%3A2503%2E22674%29%2C+2025%2E+%E2%80%94+%5BarXiv%3A2503%2E22674%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2503%2E2)
20. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching Models to Express Their Uncertainty in Words. (arXiv:2205.14334), 2022. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+Teaching+Models+to+Express+Their+Uncertainty+in+Words%2E+%28arXiv%3A2205%2E14334%29%2C+2022%2E)
21. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. (arXiv:2305.14975), 2023. [find](https://scholar.google.com/scholar?q=Katherine+Tian%2C+Eric+Mitchell%2C+Allan+Zhou%2C+Archit+Sharma%2C+Rafael+Rafailov%2C+Huaxiu+Yao%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+Just+Ask+for+Calibration%3A+Strategies+for+Eliciting+Calibrated+Confidenc)
22. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. Benchmarking Uncertainty Quantification Me… — [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/)
23. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. (arXiv:2306.13063), 2024. [find](https://scholar.google.com/scholar?q=Miao+Xiong%2C+Zhiyuan+Hu%2C+Xinyang+Lu%2C+Yifei+Li%2C+Jie+Fu%2C+Junxian+He%2C+and+Bryan+Hooi%2E+Can+LLMs+Express+Their+Uncertainty%3F+An+Empirical+Evaluation+of+Confidence+Elicitation+in+LLMs%2E+%28arXiv%3A2306%2E13063%29%2C+202)
24. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large Language Models Must Be Taught to Know What They Don’t Know. (arXiv:2406.08391), 2024. [find](https://scholar.google.com/scholar?q=Sanyam+Kapoor%2C+Nate+Gruver%2C+Manley+Roberts%2C+Katherine+Collins%2C+Arka+Pal%2C+Umang+Bhatt%2C+Adrian+Weller%2C+Samuel+Dooley%2C+Micah+Goldblum%2C+and+Andrew+Gordon+Wilson%2E+Large+Language+Models+Must+Be+Taught+to+Kn)
25. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+)
26. Lida Chen, Zujie Liang, Xintao Wang, Jiaqing Liang, Yanghua Xiao, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han, and Wei Wang. Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals. (arXiv:2406.10881), 2024. [find](https://scholar.google.com/scholar?q=Lida+Chen%2C+Zujie+Liang%2C+Xintao+Wang%2C+Jiaqing+Liang%2C+Yanghua+Xiao%2C+Feng+Wei%2C+Jinglei+Chen%2C+Zhenghong+Hao%2C+Bing+Han%2C+and+Wei+Wang%2E+Teaching+Large+Language+Models+to+Express+Knowledge+Boundary+from+Their)
27. Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et al. [The art of saying no: Contextual noncompliance in language models](https://md.research.elamur.ai/free/en/papers/brahman-2024-art-saying-no-contextual-noncompliance/). Advances in Neural Information Processing Systems, 37: 49…
28. Yang Deng, Yong Zhao, Moxin Li, See-Kiong Ng, and Tat-Seng Chua. Don‘t Just Say “I don‘t know”! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Me… [find](https://scholar.google.com/scholar?q=Yang+Deng%2C+Yong+Zhao%2C+Moxin+Li%2C+See%2DKiong+Ng%2C+and+Tat%2DSeng+Chua%2E+Don%E2%80%98t+Just+Say+%E2%80%9CI+don%E2%80%98t+know%E2%80%9D%21+Self%2Daligning+Large+Language+Models+for+Responding+to+Unknown+Questions+with+Explanations%2E+In+Yaser+Al%2DO)
29. Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. Know Your Limits: A Survey of Abstention in Large Language Models. (arXiv:2407.18418), 2025. [find](https://scholar.google.com/scholar?q=Bingbing+Wen%2C+Jihan+Yao%2C+Shangbin+Feng%2C+Chenjun+Xu%2C+Yulia+Tsvetkov%2C+Bill+Howe%2C+and+Lucy+Lu+Wang%2E+Know+Your+Limits%3A+A+Survey+of+Abstention+in+Large+Language+Models%2E+%28arXiv%3A2407%2E18418%29%2C+2025%2E)
30. Felix B. Mueller, Rebekka Görge, Anna K. Bernzen, Janna C. Pirk, and Maximilian Poretschkin. Llms and memorization: On quality and specificity of copyright compliance. In Sanmay Das, Brian Patrick Green, Kush Varshney, Marianna Ganapini, and Andrea Renda, editors, Proceedings of the Seventh AAAI/ACM… — [doi:10.1609/aies.v7i1.31697](https://doi.org/10.1609/aies.v7i1.31697) [find](https://scholar.google.com/scholar?q=Felix+B%2E+Mueller%2C+Rebekka+G%C3%B6rge%2C+Anna+K%2E+Bernzen%2C+Janna+C%2E+Pirk%2C+and+Maximilian+Poretschkin%2E+Llms+and+memorization%3A+On+quality+and+specificity+of+copyright+compliance%2E+In+Sanmay+Das%2C+Brian+Patrick+Gre)
31. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/). (arXiv:2402.04249), 2024.
32. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. [Hallucination is Inevitable: An Innate Limitation of Large Language Models](https://md.research.elamur.ai/free/en/papers/xu-2024-hallucination-inevitable-innate-limitation-large/). (arXiv:2401.11817), 2025.
33. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. [HalluLens: LLM Hallucination Benchmark](https://md.research.elamur.ai/free/en/papers/bang-2025-hallulens-llm-hallucination-benchmark/). (arXiv:2504.17550), 2025.
34. Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don’t Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association… [find](https://scholar.google.com/scholar?q=Shangbin+Feng%2C+Weijia+Shi%2C+Yike+Wang%2C+Wenxuan+Ding%2C+Vidhisha+Balachandran%2C+and+Yulia+Tsvetkov%2E+Don%E2%80%99t+Hallucinate%2C+Abstain%3A+Identifying+LLM+Knowledge+Gaps+via+Multi%2DLLM+Collaboration%2E+In+Lun%2DWei+Ku%2C+An)
35. Daniel D. Johnson, Daniel Tarlow, David Duvenaud, and Chris J. Maddison. Experts Don’t Cheat: Learning What You Don’t Know By Predicting Pairs. (arXiv:2402.08733), 2024. — [arXiv:2402.08733](https://arxiv.org/abs/2402.08733) [find](https://scholar.google.com/scholar?q=Daniel+D%2E+Johnson%2C+Daniel+Tarlow%2C+David+Duvenaud%2C+and+Chris+J%2E+Maddison%2E+Experts+Don%E2%80%99t+Cheat%3A+Learning+What+You+Don%E2%80%99t+Know+By+Predicting+Pairs%2E+%28arXiv%3A2402%2E08733%29%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E08733%5D%28https%3A%2F%2Fa)
36. Ziwei Ji, Lei Yu, Yeskendir Koishekenov, Yejin Bang, Anthony Hartshorn, Alan Schelten, Cheng Zhang, Pascale Fung, and Nicola Cancedda. [Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations](https://md.research.elamur.ai/free/en/papers/ji-2025-calibrating-verbal-uncertainty-as-linear/). (arXiv:2503.14477), 2025.
37. Yasin Abbasi Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating LLM Hallucinations via Conformal Abstention. (arXiv:2405.01563), 2024. — [arXiv:2405.01563](https://arxiv.org/abs/2405.01563) [find](https://scholar.google.com/scholar?q=Yasin+Abbasi+Yadkori%2C+Ilja+Kuzborskij%2C+David+Stutz%2C+Andr%C3%A1s+Gy%C3%B6rgy%2C+Adam+Fisch%2C+Arnaud+Doucet%2C+Iuliya+Beloshapka%2C+Wei%2DHung+Weng%2C+Yao%2DYuan+Yang%2C+Csaba+Szepesv%C3%A1ri%2C+Ali+Taylan+Cemgil%2C+and+Nenad+Tomasev%2E+M)
38. Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Sun, Luke Zettlemoyer, Gargi Ghosh, and Wen-tau Yih. Improving Factuality with Explicit Working Memory. (arXiv:2412.18069), 2025. [find](https://scholar.google.com/scholar?q=Mingda+Chen%2C+Yang+Li%2C+Karthik+Padthe%2C+Rulin+Shao%2C+Alicia+Sun%2C+Luke+Zettlemoyer%2C+Gargi+Ghosh%2C+and+Wen%2Dtau+Yih%2E+Improving+Factuality+with+Explicit+Working+Memory%2E+%28arXiv%3A2412%2E18069%29%2C+2025%2E)
39. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). Advances in Neural Information Processing Systems, 35:15476–15488, 2022.
40. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). (arXiv:2308.09583), 2025.
41. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuanjing Huang, and Xipeng Qiu. Reasoning in flux: Enhancing large language models reasoning through uncertainty-aware adaptive guidance. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings o… [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Zhiyuan+Zeng%2C+Xiaonan+Li%2C+Junqi+Dai%2C+Qinyuan+Cheng%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+Reasoning+in+flux%3A+Enhancing+large+language+models+reasoning+through+uncertain)
42. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. [Solving Quantitative Reasoning Problems with Language Models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Info…
43. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). (arXiv:…
44. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). (arXiv:2402.03300), 2024.
45. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). In The Twelfth International Conference on Learning Representations, 2023. URL https://openr…
46. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring Mathematical Problem Solving With the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). (arXiv:2103.03874), 2021.
47. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations, 2023. URL — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/) [https://openreview.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi). [↗](https://openreview.net/forum?id=v8L0pN6EOi)
48. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Bar… — [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://md.research.elamur.ai/free/en/papers/glazer-2024-frontiermath-benchmark-evaluating-advanced-mathematical/)
49. Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. [Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models](https://md.research.elamur.ai/free/en/papers/sun-2026-challenging-boundaries-reasoning-olympiad-level/). (arXiv:2503.21380), 2025.
50. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. [Matharena: Evaluating llms on uncontaminated math competitions](https://md.research.elamur.ai/free/en/papers/balunovic-2025-matharena-evaluating-llms-uncontaminated-math/). 2025. URL [https://matharena.ai/](https://matharena.ai/). [↗](https://matharena.ai/)
51. Jingyuan Ma, Damai Dai, Zihang Yuan, Rui li, Weilin Luo, Bin Wang, Qun Liu, Lei Sha, and Zhifang Sui. Large Language Models Struggle with Unreasonability in Math Problems. (arXiv:2403.19346), 2025. — [arXiv:2403.19346](https://arxiv.org/abs/2403.19346) [find](https://scholar.google.com/scholar?q=Jingyuan+Ma%2C+Damai+Dai%2C+Zihang+Yuan%2C+Rui+li%2C+Weilin+Luo%2C+Bin+Wang%2C+Qun+Liu%2C+Lei+Sha%2C+and+Zhifang+Sui%2E+Large+Language+Models+Struggle+with+Unreasonability+in+Math+Problems%2E+%28arXiv%3A2403%2E19346%29%2C+2025%2E+%E2%80%94+)
52. A. M. Muntasir Rahman, Junyi Ye, Wei Yao, Sierra S. Liu, Jesse Yu, Jonathan Yu, Wenpeng Yin, and Guiling Wang. From Blind Solvers to Logical Thinkers: Benchmarking LLMs’ Logical Integrity on Faulty Mathematical Problems. (arXiv:2410.[18921), 2025](https://md.research.elamur.ai/free/en/papers/rahman-2024-blind-solvers-logical-thinkers-benchmarking/).
53. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://md.research.elamur.ai/free/en/papers/shi-2023-large-language-models-easily-distracted/). In Proceedings of the 40th International Conference on Machine Learning, pages 31210–31227. PMLR, 2023.
54. Jialin Ouyang. [TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluation](https://md.research.elamur.ai/free/en/papers/ouyang-2025-treecut-synthetic-unanswerable-math-word/). (arXiv:2502.13442), 2025.
55. Zihao Zhou, Shudong Liu, Maizhen Ning, Wei Liu, Jindong Wang, Derek F. Wong, Xiaowei Huang, Qiufeng Wang, and Kaizhu Huang. Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist. (arXiv:2407.08733), 2024. — [arXiv:2407.08733](https://arxiv.org/abs/2407.08733) [find](https://scholar.google.com/scholar?q=Zihao+Zhou%2C+Shudong+Liu%2C+Maizhen+Ning%2C+Wei+Liu%2C+Jindong+Wang%2C+Derek+F%2E+Wong%2C+Xiaowei+Huang%2C+Qiufeng+Wang%2C+and+Kaizhu+Huang%2E+Is+Your+Model+Really+A+Good+Math+Reasoner%3F+Evaluating+Mathematical+Reasoning)
56. Asir Saadat, Tasmia Binte Sogir, Md Taukir Azam Chowdhury, and Syem Aziz. When Not to Answer: Evaluating Prompts on GPT Models for Effective Abstention in Unanswerable Math Word Problems. (arXiv:2410.13029), 2024. — [arXiv:2410.13029](https://arxiv.org/abs/2410.13029) [find](https://scholar.google.com/scholar?q=Asir+Saadat%2C+Tasmia+Binte+Sogir%2C+Md+Taukir+Azam+Chowdhury%2C+and+Syem+Aziz%2E+When+Not+to+Answer%3A+Evaluating+Prompts+on+GPT+Models+for+Effective+Abstention+in+Unanswerable+Math+Word+Problems%2E+%28arXiv%3A2410%2E)
57. YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. Benchmarking Hallucination in Large Language Models Based on Unanswerable Math Word Problem. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the… [find](https://scholar.google.com/scholar?q=YuHong+Sun%2C+Zhangyue+Yin%2C+Qipeng+Guo%2C+Jiawen+Wu%2C+Xipeng+Qiu%2C+and+Hui+Zhao%2E+Benchmarking+Hallucination+in+Large+Language+Models+Based+on+Unanswerable+Math+Word+Problem%2E+In+Nicoletta+Calzolari%2C+Min%2DYen+)
58. Xunjian Yin, Baizhou Huang, and Xiaojun Wan. ALCUNA: Large Language Models Meet New Knowledge. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1397–1414. Association for Computational Linguistics, 20… [find](https://scholar.google.com/scholar?q=Xunjian+Yin%2C+Baizhou+Huang%2C+and+Xiaojun+Wan%2E+ALCUNA%3A+Large+Language+Models+Meet+New+Knowledge%2E+In+Houda+Bouamor%2C+Juan+Pino%2C+and+Kalika+Bali%2C+editors%2C+Proceedings+of+the+2023+Conference+on+Empirical+Me)
59. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. [BBQ: A hand-built bias benchmark for question answering](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational…
60. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, . . . , Zijie J. Wang, Zirui Wang, and Ziyi Wu. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) uyTL5B… [↗](https://openreview.net/forum?id=)
61. Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. Won‘t Get Fooled Again: Answering Questions with False Premises. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Shengding+Hu%2C+Yifan+Luo%2C+Huadong+Wang%2C+Xingyi+Cheng%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Won%E2%80%98t+Get+Fooled+Again%3A+Answering+Questions+with+False+Premises%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okaza)
62. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/). (arXiv:2310.03214), 2023.
63. Shuyue S. Li, Vidhisha Balachandran, Shangbin Feng, Jonathan S. Ilgen, Emma Pierson, Pang W. Koh, and Yulia Tsvetkov. [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://md.research.elamur.ai/free/en/papers/li-2024-mediq-question-asking-llms-benchmark/). Advances in Neural Information Processing Systems, 37:28858–28888, 2024.
64. Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. [Evaluating the Moral Beliefs Encoded in LLMs](https://md.research.elamur.ai/free/en/papers/scherrer-2023-evaluating-moral-beliefs-encoded-llms/). Advances in Neural Information Processing Systems, 36:51778–51809, 2023.
65. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. (QA)^2: Question Answering with Questionable Assumptions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… [find](https://scholar.google.com/scholar?q=Najoung+Kim%2C+Phu+Mon+Htut%2C+Samuel+R%2E+Bowman%2C+and+Jackson+Petty%2E+%28QA%29%5E2%3A+Question+Answering+with+Questionable+Assumptions%2E+In+Anna+Rogers%2C+Jordan+Boyd%2DGraber%2C+and+Naoaki+Okazaki%2C+editors%2C+Proceedings+o)
66. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/). In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakra…
67. Michael Zhang and Eunsol Choi. SituatedQA: Incorporating Extra-Linguistic Contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387. Association for… [find](https://scholar.google.com/scholar?q=Michael+Zhang+and+Eunsol+Choi%2E+SituatedQA%3A+Incorporating+Extra%2DLinguistic+Contexts+into+QA%2E+In+Marie%2DFrancine+Moens%2C+Xuanjing+Huang%2C+Lucia+Specia%2C+and+Scott+Wen%2Dtau+Yih%2C+editors%2C+Proceedings+of+the+20)
68. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don‘t Know: Unanswerable Questions for SQuAD. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. Association for Comp… [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Robin+Jia%2C+and+Percy+Liang%2E+Know+What+You+Don%E2%80%98t+Know%3A+Unanswerable+Questions+for+SQuAD%2E+In+Iryna+Gurevych+and+Yusuke+Miyao%2C+editors%2C+Proceedings+of+the+56th+Annual+Meeting+of+the+Ass)
69. Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. [WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/benchekroun-2023-worldsense-synthetic-benchmark-grounded-reasoning/). (arXiv:2311.15930), 2023.
70. OpenAI. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). (arXiv:2410.21276), 2024.
71. OpenAI. [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). (arXiv:2412.16720), 2024.
72. Gemini Team. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). (arXiv:2403.05530), 2024.
73. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, . . . , Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). (arXiv:2407.21783), 2024.
74. Qwen Team. Qwen2.5: A Party of Foundation Models! Qwen, 2024. URL [https://qwenlm.github.io/](https://qwenlm.github.io/) blog/qwen2.5/. [↗](https://qwenlm.github.io/)
75. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
76. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, W… [find](https://scholar.google.com/scholar?q=Dirk+Groeneveld%2C+Iz+Beltagy%2C+Pete+Walsh%2C+Akshita+Bhagia%2C+Rodney+Kinney%2C+Oyvind+Tafjord%2C+Ananya+Harsh+Jha%2C+Hamish+Ivison%2C+Ian+Magnusson%2C+Yizhong+Wang%2C+Shane+Arora%2C+David+Atkinson%2C+Russell+Authur%2C+Khyat)
77. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… — [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
78. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levi…
79. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. (arXiv:2406.12624), 2025. [find](https://scholar.google.com/scholar?q=Aman+Singh+Thakur%2C+Kartik+Choudhary%2C+Venkat+Srinik+Ramayapally%2C+Sankaran+Vaidyanathan%2C+and+Dieuwke+Hupkes%2E+Judging+the+Judges%3A+Evaluating+Alignment+and+Vulnerabilities+in+LLMs%2Das%2DJudges%2E+%28arXiv%3A2406%2E1)
80. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
81. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. (arXiv:1707.06347), 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+Policy+Optimization+Algorithms%2E+%28arXiv%3A1707%2E06347%29%2C+2017%2E)
82. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). (arXiv:2305.18290), 2023.
83. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning Models Don’t Always Say What They Think. (arXiv:2505.[05410), 2025](https://md.research.elamur.ai/free/en/papers/chen-2025-reasoning-models-don-t-always/).
84. Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research. In Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL [https://openreview.net/](https://openreview.net/) forum?id=zN… [↗](https://openreview.net/)
85. Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? (arXiv:2006.07159), 2020. — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) [find](https://scholar.google.com/scholar?q=Lucas+Beyer%2C+Olivier+J%2E+H%C3%A9naff%2C+Alexander+Kolesnikov%2C+Xiaohua+Zhai%2C+and+A%C3%A4ron+van+den+Oord%2E+Are+we+done+with+ImageNet%3F+%28arXiv%3A2006%2E07159%29%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E07159%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E07159%29)
86. Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? (arXiv:1902.10811), 2019. [find](https://scholar.google.com/scholar?q=Benjamin+Recht%2C+Rebecca+Roelofs%2C+Ludwig+Schmidt%2C+and+Vaishaal+Shankar%2E+Do+ImageNet+Classifiers+Generalize+to+ImageNet%3F+%28arXiv%3A1902%2E10811%29%2C+2019%2E)
87. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, pages 611–626.…
88. Tianyu Gao, Xingcheng Yao, and Danqi Chen. [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894– 6910. Asso…
89. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. — [♫ MuSiQue: Multihop Questions via Single-hop Question Composition](https://md.research.elamur.ai/free/en/papers/trivedi-2022-musique-multihop-questions-single-hop/)
90. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questi… — [Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
91. Victor Wang, Michael J. Q. Zhang, and Eunsol Choi. Improving LLM-as-a-Judge Inference with the Judgment Distribution. (arXiv:2503.03064), 2025. [find](https://scholar.google.com/scholar?q=Victor+Wang%2C+Michael+J%2E+Q%2E+Zhang%2C+and+Eunsol+Choi%2E+Improving+LLM%2Das%2Da%2DJudge+Inference+with+the+Judgment+Distribution%2E+%28arXiv%3A2503%2E03064%29%2C+2025%2E)
92. Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. [Dynamic Early Exit in Reasoning Models](https://md.research.elamur.ai/free/en/papers/yang-2025-dynamic-early-exit-reasoning-models/). (arXiv:2504.15895), 2025.

*Updated 3 November 2026*