---
title: "R-Tuning: Instructing Large Language Models to Say ‘I Don't Know'"
description: "They propose R-Tuning (Refusal-Aware Instruction Tuning), a method consisting of two stages. 75 references, cited by 74 papers here."
authors: ["Hanning Zhang", "Shizhe Diao", "Yong Lin", "Yi Fung", "Qing Lian", "Xingyao Wang", "Yangyi Chen", "Heng Ji", "Tong Zhang"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.394/"
doi: "10.18653/v1/2024.naacl-long.394"
arxiv: "2311.09677"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-r-tuning-instructing-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-r-tuning-instructing-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / R-Tuning: Instructing Large Language Models to Say ‘I Don't Know'

# R-Tuning: Instructing Large Language Models to Say ‘I Don't Know'

**Authors:** [Hanning Zhang](https://md.research.elamur.ai/free/en/authors/hanning-zhang/), [Shizhe Diao](https://md.research.elamur.ai/free/en/authors/shizhe-diao/), [Yong Lin](https://md.research.elamur.ai/free/en/authors/yong-lin/), [Yi Fung](https://md.research.elamur.ai/free/en/authors/yi-fung/), [Qing Lian](https://md.research.elamur.ai/free/en/authors/qing-lian/), [Xingyao Wang](https://md.research.elamur.ai/free/en/authors/xingyao-wang/), [Yangyi Chen](https://md.research.elamur.ai/free/en/authors/yangyi-chen/), [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/) and [Tong Zhang](https://md.research.elamur.ai/free/en/authors/tong-zhang/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.394/) · [PDF](https://aclanthology.org/2024.naacl-long.394.pdf) · [DOI 10.18653/v1/2024.naacl-long.394](https://doi.org/10.18653/v1/2024.naacl-long.394) · [arXiv 2311.09677](https://arxiv.org/abs/2311.09677) · [OpenAlex](https://openalex.org/W4401042338)

**Citations:** 25 citations (OpenAlex, 2026-10-14) · [74](https://md.research.elamur.ai/free/en/papers/zhang-2024-r-tuning-instructing-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines hallucinations in large language models (LLMs). The authors attribute them to the fact that standard instruction tuning forces the model to answer every question, even when the required fact is absent from its parametric knowledge. The model thus learns to guess and cannot admit that it does not know something. They propose R-Tuning (Refusal-Aware Instruction Tuning), a method consisting of two stages. First, the pretrained model answers all training questions once. If the answer matches the reference, the example goes into the "certain" part D1; if not, into the "uncertain" part D0. Next, each example keeps the correct answer, followed by a question about confidence and the label "I am sure" or "I am unsure", and the model is fine-tuned on these data. On nine datasets, R-Tuning outperforms standard fine-tuning in AP score. The model answers what it knows more accurately and refuses to answer more often when it does not know. This ability also transfers to tasks outside the training domain, so the authors call it a meta-skill. The method offers a simple way to make a model more honest about the limits of its knowledge.

## Key points
- **Data and models.** QA-type tasks: ParaRel, HotpotQA, SelfAware, HaluEval, FalseQA, NEC. Multiple-choice tasks: MMLU, WiCE, FEVER. Models: OpenLLaMA-3B, LLaMA-7B and LLaMA-13B. Experiments were conducted in two settings: single-task and multi-task.
- **Training and evaluation.** The cross-entropy loss is computed only over the answer and the confidence expression; the question is not included in it. At inference, the model's confidence is defined as a weighted combination of the answer probability and the probability of the words "sure"/"unsure". The AP score is computed from this value.
- **Transfer to other tasks.** Training on multiple tasks strengthens the refusal ability, and it generalizes better to out-of-domain datasets.
- **Variant without reference answers.** The model answers the same question several times. Consistent answers are treated as certain, divergent ones as uncertain. This variant also proved effective. A separate finding: a model that learned to estimate uncertainty during training is better calibrated than an approach that filters by uncertainty only at test time.
- **Limitations and code.** Confidence is only binary (sure/unsure). Whether the model possesses knowledge was determined only by checking its answers and estimating uncertainty, not by comparison with the pretraining corpus. Code: github.com/shizhediao/R-Tuning.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 209 works · 71,394 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji and Tong Zhang. 2024. R-Tuning: Instructing Large Language Models to Say ‘I Don't Know'. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024r,
  title = {{R-Tuning: Instructing Large Language Models to Say ‘I Don't Know'}},
  author = {Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji and Tong Zhang},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.394},
  eprint = {2311.09677},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.394}
}
```

- [← previous Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://md.research.elamur.ai/free/en/papers/ong-2024-optimizing-instructions-demonstrations-multi-stage/)
- [next → Unsupervised Real-Time Hallucination Detection based on the Internal States of Large Language Models](https://md.research.elamur.ai/free/en/papers/su-2024-unsupervised-real-time-hallucination-detection/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

75 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. [Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/).
2. Amos Azaria and Tom Mitchell. 2023. [The internal state of an llm knows when it’s lying](https://md.research.elamur.ai/free/en/papers/azaria-2023-internal-state-llm-knows-when/).
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. 2023. [Purr: Efficiently editing language model hallucinations by denoising language model corruptions](https://md.research.elamur.ai/free/en/papers/chen-2023-purr-efficiently-editing-language-model/).
5. Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. 2022. A close look into the calibration of pre-trained language models. arXiv preprint arXiv:2211.00151. [find](https://scholar.google.com/scholar?q=Yangyi+Chen%2C+Lifan+Yuan%2C+Ganqu+Cui%2C+Zhiyuan+Liu%2C+and+Heng+Ji%2E+2022%2E+A+close+look+into+the+calibration+of+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2211%2E00151%2E)
6. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. 2023. [Dola: Decoding by contrasting layers improves factuality in large language models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/).
7. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. 2023. [Lm vs lm: Detecting factual errors via cross examination](https://md.research.elamur.ai/free/en/papers/cohen-2023-lm-vs-lm-detecting-factual/). arXiv preprint arXiv:2305.13281.
8. Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2023a. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Rui+Pan%2C+Hanze+Dong%2C+Ka+Shun+Shum%2C+Jipeng+Zhang%2C+Wei+Xiong%2C+and+Tong+Zhang%2E+2023a%2E+Lmflow%3A+An+extensible+toolkit+for+finetuning+and+inference+of+large+foundation+models%2E)
9. Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023b. Active prompting with chain-of-thought for large language models. [find](https://scholar.google.com/scholar?q=Shizhe+Diao%2C+Pengcheng+Wang%2C+Yong+Lin%2C+and+Tong+Zhang%2E+2023b%2E+Active+prompting+with+chain%2Dof%2Dthought+for+large+language+models%2E)
10. Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Zhifang Sui, and Lei Li. 2023. Statistical knowledge assessment for large language models. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Qingxiu+Dong%2C+Jingjing+Xu%2C+Lingpeng+Kong%2C+Zhifang+Sui%2C+and+Lei+Li%2E+2023%2E+Statistical+knowledge+assessment+for+large+language+models%2E+In+Thirty%2Dseventh+Conference+on+Neural+Information+Processing+Syste)
11. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325.
12. Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022. [On the origin of hallucinations in conversational models: Is it the datasets or the models?](https://md.research.elamur.ai/free/en/papers/dziri-2022-origin-hallucinations-conversational-models-is/) In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
13. Mohamed Elaraby, Mengyin Lu, Jacob Dunn, Xueying Zhang, Yu Wang, and Shizhu Liu. 2023. [Halo: Estimation and reduction of hallucinations in open-source weak large language models](https://md.research.elamur.ai/free/en/papers/elaraby-2023-halo-estimation-reduction-hallucinations-open/). arXiv preprint arXiv:2308.11764.
14. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+improving+consistency+in+pretrained+language+models%2E)
15. Mark Everingham, Luc van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. 2010. [find](https://scholar.google.com/scholar?q=Mark+Everingham%2C+Luc+van+Gool%2C+Christopher+K%2E+I%2E+Williams%2C+John+Winn%2C+and+Andrew+Zisserman%2E+2010%2E)
16. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303– 338. [find](https://scholar.google.com/scholar?q=The+pascal+visual+object+classes+%28voc%29+challenge%2E+International+Journal+of+Computer+Vision%2C+88%282%29%3A303%E2%80%93+338%2E)
17. Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama. [find](https://scholar.google.com/scholar?q=Xinyang+Geng+and+Hao+Liu%2E+2023%2E+Openllama%3A+An+open+reproduction+of+llama%2E)
18. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. [Critic: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). arXiv preprint arXiv:2305.11738.
19. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Geoff+Pleiss%2C+Yu+Sun%2C+and+Kilian+Q+Weinberger%2E+2017%2E+On+calibration+of+modern+neural+networks%2E+In+International+conference+on+machine+learning%2C+pages+1321%E2%80%931330%2E+PMLR%2E)
20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/).
21. Shengding Hu, Yi-Xiao Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. Won’t get fooled again: Answering questions with false premises. ArXiv, abs/2307.02394. [find](https://scholar.google.com/scholar?q=Shengding+Hu%2C+Yi%2DXiao+Luo%2C+Huadong+Wang%2C+Xingyi+Cheng%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Won%E2%80%99t+get+fooled+again%3A+Answering+questions+with+false+premises%2E+ArXiv%2C+abs%2F2307%2E02394%2E)
22. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023a. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798.
23. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023b. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/).
24. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
25. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+)
26. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. [Wice: Real-world entailment for claims in wikipedia](https://md.research.elamur.ai/free/en/papers/kamoi-2023-wice-real-world-entailment-claims/).
27. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. [Large language models struggle to learn long-tail knowledge](https://md.research.elamur.ai/free/en/papers/kandpal-2022-large-language-models-struggle-learn/). In International Conference on Machine Learning, pages 15696–15707. PMLR.
28. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, and Bryan Catanzaro. 2023. [Factuality enhanced language models for open-ended text generation](https://md.research.elamur.ai/free/en/papers/lee-2022-factuality-enhanced-language-models-open/).
29. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. [Large language models with controllable working memory](https://md.research.elamur.ai/free/en/papers/li-2023-large-language-models-controllable-working/).
30. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a. [Halueval: A large-scale hallucination evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/).
31. Miaoran Li, Baolin Peng, and Zhu Zhang. 2023b. [Self-checker: Plug-and-play modules for fact-checking with large language models](https://md.research.elamur.ai/free/en/papers/li-2024-self-checker-plug-play-modules/). arXiv preprint arXiv:2305.14623.
32. Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung, Charles Yu, Joel Tetreault, Eduard Hovy, and Heng Ji. 2023c. Defining a new NLP playground. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11932–11951, Singapore. Association for Computatio… [find](https://scholar.google.com/scholar?q=Sha+Li%2C+Chi+Han%2C+Pengfei+Yu%2C+Carl+Edwards%2C+Manling+Li%2C+Xingyao+Wang%2C+Yi+Fung%2C+Charles+Yu%2C+Joel+Tetreault%2C+Eduard+Hovy%2C+and+Heng+Ji%2E+2023c%2E+Defining+a+new+NLP+playground%2E+In+Findings+of+the+Association)
33. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
34. Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. 2023. Prudent silence or foolish babble? examining large language models’ responses to the unknown. [find](https://scholar.google.com/scholar?q=Genglin+Liu%2C+Xingyao+Wang%2C+Lifan+Yuan%2C+Yangyi+Chen%2C+and+Hao+Peng%2E+2023%2E+Prudent+silence+or+foolish+babble%3F+examining+large+language+models%E2%80%99+responses+to+the+unknown%2E)
35. Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. 2024. Examining llms’ uncertainty expression towards questions outside parametric knowledge. [find](https://scholar.google.com/scholar?q=Genglin+Liu%2C+Xingyao+Wang%2C+Lifan+Yuan%2C+Yangyi+Chen%2C+and+Hao+Peng%2E+2024%2E+Examining+llms%E2%80%99+uncertainty+expression+towards+questions+outside+parametric+knowledge%2E)
36. Adam Livska, Tom’avs Kovcisk’y, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. 2022. Streamingqa: A benchmark for adaptation to new kn… [find](https://scholar.google.com/scholar?q=Adam+Livska%2C+Tom%E2%80%99avs+Kovcisk%E2%80%99y%2C+Elena+Gribovskaya%2C+Tayfun+Terzi%2C+Eren+Sezener%2C+Devang+Agrawal%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Tim+Scholtes%2C+Manzil+Zaheer%2C+Susannah+Young%2C+Ellen+Gilsenan%2DMcMahon%2C+Sophia+A)
37. Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. [Augmented large language models with parametric knowledge guiding](https://md.research.elamur.ai/free/en/papers/luo-2023-augmented-large-language-models-parametric/). arXiv preprint arXiv:2305.04757.
38. Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022. [Time waits for no one! analysis and challenges of temporal misalignment](https://md.research.elamur.ai/free/en/papers/luu-2022-time-waits-no-one-analysis/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec…
39. Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado… — [Discovering Language Model Behaviors with Model-Written Evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
40. Xiaozhong Lyu, Stefan Grafberger, Samantha Biegel, Shaopeng Wei, Meng Cao, Sebastian Schelter, and Ce Zhang. 2023. Improving retrieval-augmented large language models via data importance learning. [find](https://scholar.google.com/scholar?q=Xiaozhong+Lyu%2C+Stefan+Grafberger%2C+Samantha+Biegel%2C+Shaopeng+Wei%2C+Meng+Cao%2C+Sebastian+Schelter%2C+and+Ce+Zhang%2E+2023%2E+Improving+retrieval%2Daugmented+large+language+models+via+data+importance+learning%2E)
41. Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896. [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+JF+Gales%2E+2023%2E+Selfcheckgpt%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+arXiv+preprint+arXiv%3A2303%2E08896%2E)
42. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020a. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). arXiv preprint arXiv:2005.00661.
43. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020b. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
44. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
45. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29. [find](https://scholar.google.com/scholar?q=Mahdi+Pakdaman+Naeini%2C+Gregory+Cooper%2C+and+Milos+Hauskrecht%2E+2015%2E+Obtaining+well+calibrated+probabilities+using+bayesian+binning%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+vol)
46. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Alessandro+Cappelli%2C+Hamza+Alobeidli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
47. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). arXiv preprint arXiv:2302.12813.
48. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Ansh Radhakr… [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Sam+Ringer%2C+Kamile%CC%87+Luko%C5%A1iu%CC%84te%CC%87%2C+Karina+Nguyen%2C+Edwin+Chen%2C+Scott+Heiner%2C+Craig+Pettit%2C+Catherine+Olsson%2C+Sandipan+Kundu%2C+Saurav+Kadavath%2C+Andy+Jones%2C+Anna+Chen%2C+Ben+Mann%2C+Brian+Israel%2C+B)
49. Revanth Gangi Reddy, Yi R. Fung, Qi Zeng, Manling Li, Ziqi Wang, Paul Sullivan, and Heng Ji. 2023. Smartbook: Ai-assisted situation report generation. [find](https://scholar.google.com/scholar?q=Revanth+Gangi+Reddy%2C+Yi+R%2E+Fung%2C+Qi+Zeng%2C+Manling+Li%2C+Ziqi+Wang%2C+Paul+Sullivan%2C+and+Heng+Ji%2E+2023%2E+Smartbook%3A+Ai%2Dassisted+situation+report+generation%2E)
50. Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. 2023. [Investigating the factual knowledge boundary of large language models with retrieval augmentation](https://md.research.elamur.ai/free/en/papers/ren-2025-investigating-factual-knowledge-boundary-large/).
51. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. [Fever: a large-scale dataset for fact extraction and verification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/).
52. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
53. Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. [A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation](https://md.research.elamur.ai/free/en/papers/varshney-2023-stitch-time-saves-nine-detecting/).
54. Yixin Wan, Fanyou Wu, Weijie Xu, and Srinivasan H. Sengamedu. 2024. Sequence-level certainty reduces hallucination in knowledge-grounded dialogue generation. [find](https://scholar.google.com/scholar?q=Yixin+Wan%2C+Fanyou+Wu%2C+Weijie+Xu%2C+and+Srinivasan+H%2E+Sengamedu%2E+2024%2E+Sequence%2Dlevel+certainty+reduces+hallucination+in+knowledge%2Dgrounded+dialogue+generation%2E)
55. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. [Towards understanding chain-of-thought prompting: An empirical study of what matters](https://md.research.elamur.ai/free/en/papers/wang-2023-understanding-chain-thought-prompting-empirical/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag…
56. Bi, Freda Shi, and Shuming Shi. 2023c. [Siren’s song in the ai ocean: A survey on hallucination in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/).
57. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on Machine Learning Research.
58. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2E)
59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/).
60. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023b. [Simple synthetic data reduces sycophancy in large language models](https://md.research.elamur.ai/free/en/papers/wei-2023-simple-synthetic-data-reduces-sycophancy/).
61. Haoyang Wen, Ying Lin, Tuan Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu, Alexander Dong, Zhenhailong Wang, Yi Fung, Piyush Mishra, Qing Lyu, Dídac Surís, Brian Chen, Susan Windisch Brown, Martha Palmer, Chris Callison-Burch, Carl Vondrick, Jiaw… [find](https://scholar.google.com/scholar?q=Haoyang+Wen%2C+Ying+Lin%2C+Tuan+Lai%2C+Xiaoman+Pan%2C+Sha+Li%2C+Xudong+Lin%2C+Ben+Zhou%2C+Manling+Li%2C+Haoyu+Wang%2C+Hongming+Zhang%2C+Xiaodong+Yu%2C+Alexander+Dong%2C+Zhenhailong+Wang%2C+Yi+Fung%2C+Piyush+Mishra%2C+Qing+Lyu%2C+D%C3%ADd)
62. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge clashes. [find](https://scholar.google.com/scholar?q=Jian+Xie%2C+Kai+Zhang%2C+Jiangjie+Chen%2C+Renze+Lou%2C+and+Yu+Su%2E+2023%2E+Adaptive+chameleon+or+stubborn+sloth%3A+Unraveling+the+behavior+of+large+language+models+in+knowledge+clashes%2E)
63. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/).
64. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Jiawen+Wu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2023%2E+Do+large+language+models+know+what+they+don%E2%80%99t+know%3F)
65. Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. [find](https://scholar.google.com/scholar?q=Xiang+Yue%2C+Boshi+Wang%2C+Kai+Zhang%2C+Ziru+Chen%2C+Yu+Su%2C+and+Huan+Sun%2E+2023%2E+Automatic+evaluation+of+attribution+by+large+language+models%2E)
66. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023a. [How language model hallucinations can snowball](https://md.research.elamur.ai/free/en/papers/zhang-2023-language-model-hallucinations-snowball/).
67. Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2023b. Mitigating language model hallucination with interactive question-knowledge alignment. [find](https://scholar.google.com/scholar?q=Shuo+Zhang%2C+Liangming+Pan%2C+Junzhou+Zhao%2C+and+William+Yang+Wang%2E+2023b%2E+Mitigating+language+model+hallucination+with+interactive+question%2Dknowledge+alignment%2E)
68. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Dataset ParaRel (Elazar et al., 2021) [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Yafu+Li%2C+Leyang+Cui%2C+Deng+Cai%2C+Lemao+Liu%2C+Tingchen+Fu%2C+Xinting+Huang%2C+Enbo+Zhao%2C+Yu+Zhang%2C+Yulong+Chen%2C+Longyue+Wang%2C+Anh+Tuan+Luu%2C+Wei+Dataset+ParaRel+%28Elazar+et+al%2E%2C+2021%29)
69. MMLU (Hendrycks et al., 2021) [find](https://scholar.google.com/scholar?q=MMLU+%28Hendrycks+et+al%2E%2C+2021%29)
70. HotpotQA (Yang et al., 2018) [find](https://scholar.google.com/scholar?q=HotpotQA+%28Yang+et+al%2E%2C+2018%29)
71. FEVER (Thorne et al., 2018) [find](https://scholar.google.com/scholar?q=FEVER+%28Thorne+et+al%2E%2C+2018%29)
72. SelfAware (Yin et al., 2023) [find](https://scholar.google.com/scholar?q=SelfAware+%28Yin+et+al%2E%2C+2023%29)
73. HaluEval (Li et al., 2023a) [find](https://scholar.google.com/scholar?q=HaluEval+%28Li+et+al%2E%2C+2023a%29)
74. Example (Our Format) Question: Which country is Georgi Parvanov a citizen of? Answer: Bulgaria Question: Which of the following did the post-war welfare state of 1948 not aim to provide: (A) free health care and education for all (B) a minimum wage (C) full employment (D) universal welfare. Answer:… [find](https://scholar.google.com/scholar?q=Example+%28Our+Format%29+Question%3A+Which+country+is+Georgi+Parvanov+a+citizen+of%3F+Answer%3A+Bulgaria+Question%3A+Which+of+the+following+did+the+post%2Dwar+welfare+state+of+1948+not+aim+to+provide%3A+%28A%29+free+heal)
75. Original Size Actual Size Used Total data: 253448 Training data: 5575 ID test data: 5584 OOD test data: 13974 Total data: 14033 Training data: 2448 ID test data: 2439 OOD test data: 9155 Training data: 3470 Dev data: 949 Test data: 958 Training data: 3470 Test data: 958 Training data: 99564 Dev data… [find](https://scholar.google.com/scholar?q=Original+Size+Actual+Size+Used+Total+data%3A+253448+Training+data%3A+5575+ID+test+data%3A+5584+OOD+test+data%3A+13974+Total+data%3A+14033+Training+data%3A+2448+ID+test+data%3A+2439+OOD+test+data%3A+9155+Training+data)

*Updated 2 November 2026*