---
title: "Alignment for Honesty — paper by Yang et al. (2023)"
description: "The authors therefore propose several supervised fine-tuning schemes trained on automatically… 83 references, cited by 81 papers here."
authors: ["Yuqing Yang", "Ethan Chern", "Xipeng Qiu", "Graham Neubig", "Pengfei Liu"]
year: 2023
venue: "arXiv:2312.07000"
source: "https://arxiv.org/abs/2312.07000"
doi: "10.48550/arxiv.2312.07000"
arxiv: "2312.07000"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2023-alignment-honesty/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2023-alignment-honesty/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Alignment for Honesty

# Alignment for Honesty

**Authors:** [Yuqing Yang](https://md.research.elamur.ai/free/en/authors/yuqing-yang/), [Ethan Chern](https://md.research.elamur.ai/free/en/authors/ethan-chern/), [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and [Pengfei Liu](https://md.research.elamur.ai/free/en/authors/pengfei-liu/) · **Year:** 2023 · **Venue:** [arXiv:2312.07000](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.07000) · [PDF](https://arxiv.org/pdf/2312.07000) · [DOI 10.48550/arxiv.2312.07000](https://doi.org/10.48550/arxiv.2312.07000) · [arXiv 2312.07000](https://arxiv.org/abs/2312.07000) · [OpenAlex](https://openalex.org/W4415796979)

**Citations:** 9 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/yang-2023-alignment-honesty/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM alignment work targets helpfulness and harmlessness. This paper focuses on honesty instead, drawing on a saying from the Analects of Confucius: a model should answer what it actually knows and openly decline what it does not, without becoming needlessly cautious. A model's knowledge boundary cannot be observed directly, so the authors infer it from behaviour. Each response is labelled correct, wrong, or an "I don't know" (idk) reply, and a question counts as known if the model can answer it correctly. Comparing a model before and after alignment produces "evolutionary" metrics, including a prudence score for appropriate refusals and an over-conservativeness score for unnecessary ones. Prompting alone was not enough. The authors therefore propose several supervised fine-tuning schemes trained on automatically synthesised honesty-oriented data, such as CONFIDENCE-VERB and MULTISAMPLE. On knowledge-intensive QA, including generalisation tests, the aligned models become clearly more honest, and helpfulness on unrelated tasks barely changes. The work provides a definition, metrics and open tooling for reducing hallucination by teaching models to admit ignorance.

## Key points
- Responses fall into three types: idk (contains refusal markers such as "I'm not able to"), correct (the gold answer appears as a substring), or wrong.
- Honesty is evaluated by tracking how each answer's type changes from model M_t to aligned M_t+1. Prudence and over-conservativeness scores are among the derived metrics.
- A FINE-TUNED BASELINE trained on data where about 25% of questions exceed the model's knowledge lost roughly 10 accuracy points. This supports the view that SFT can induce hallucination.
- Alignment tax was measured on the Eval-P⁻ helpfulness set, with AUTO-J and GPT-4 (gpt-4-0613) as judges on a 1–10 scale. Scores were 5.56 / 8.62 for the unaligned model, 5.54 / 8.61 for CONFIDENCE-VERB and 5.52 / 8.56 for MULTISAMPLE.
- All resources are released at github.com/GAIR-NLP/alignment-for-honesty. The authors list better knowledge-boundary estimation, calibrated confidence and representation-level methods as future directions.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🤔 Philosophy](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/philosophy/) · 287 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 52 works

## Cite

Cite as: Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig and Pengfei Liu. 2023. Alignment for Honesty. arXiv:2312.07000.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2023alignment,
  title = {{Alignment for Honesty}},
  author = {Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig and Pengfei Liu},
  year = {2023},
  doi = {10.48550/arxiv.2312.07000},
  eprint = {2312.07000},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.07000}
}
```

- [← previous AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation](https://md.research.elamur.ai/free/en/papers/huang-2023-agentcoder-multi-agent-based/)
- [next → AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models](https://md.research.elamur.ai/free/en/papers/zhu-2023-autodan-interpretable-gradient-based-adversarial/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

83 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allen-Zhu, Z. and Li, Y. (2023). [Physics of language models: Part 3.2, knowledge manipulation](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3-2/). CoRR, abs/2309.14402.
2. Amayuelas, A., Pan, L., Chen, W., and Wang, W. Y. (2023). [Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models](https://md.research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/). CoRR, abs/2305.13712.
3. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., Chu, E., Clark, J. H., Shafey, L. E., Huang, Y., Meier-Hellstern, K., Mishra, G., Moreira, E., Omernick, M., Robinson, K., Ruder, S., Tay, Y., Xiao, K., Xu, Y., Zhang, Y., Ábrego, G… — [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
4. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. (2021). A general lan… [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+DasSarma%2C+N%2E%2C+Elhage%2C+N%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Hernandez%2C+D%2E%2C+Kernion%2C+J%2E%2C+Ndousse%2C+K%2E%2C+Olsson%2C+C%2E%2C)
5. Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang,… — [Qwen Technical Report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
6. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., Showk, S. E., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.,… [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E%2C+Askell%2C+A%2E%2C+Chen%2C+A%2E%2C+DasSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Fort%2C+S%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Joseph%2C+N%2E%2C+Kadavath%2C+S%2E%2C+Kernion%2C+J%2E%2C+Conerly%2C+T%2E%2C+Showk%2C+S%2E+E%2E%2C+Elhage%2C+N%2E%2C+Hatfi)
7. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosiute,… — [Constitutional AI: Harmlessness from AI Feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
8. Baichuan (2023). [Baichuan 2: Open large-scale language models](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/). arXiv preprint arXiv:2309.10305.
9. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
10. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2023). Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Burns%2C+C%2E%2C+Ye%2C+H%2E%2C+Klein%2C+D%2E%2C+and+Steinhardt%2C+J%2E+%282023%29%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+ICLR)
11. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. (2023). Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Ippolito%2C+D%2E%2C+Jagielski%2C+M%2E%2C+Lee%2C+K%2E%2C+Tram%C3%A8r%2C+F%2E%2C+and+Zhang%2C+C%2E+%282023%29%2E+Quantifying+memorization+across+neural+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Repres)
12. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T. B., Song, D., Erlingsson, Ú., Oprea, A., and Raffel, C. (2021). [Extracting training data from large language models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). In Bailey, M. D. and Greenstadt, R., editors, 30th USENIX Security Symposium, US…
13. Chern, I., Chern, S., Chen, S., Yuan, W., Feng, K., Zhou, C., He, J., Neubig, G., and Liu, P. (2023). [Factool: Factuality detection in generative AI - A tool augmented framework for multi-task and multi-domain scenarios](https://md.research.elamur.ai/free/en/papers/chern-2023-factool-factuality-detection-generative-ai/). CoRR, abs/2307.13528.
14. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Narang, S., Mishra, G., Yu, A., Zhao, V. Y., Huang, Y., Dai, A. M., Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J.,… [find](https://scholar.google.com/scholar?q=Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpre%2C+S%2E%2C+Zoph%2C+B%2E%2C+Tay%2C+Y%2E%2C+Fedus%2C+W%2E%2C+Li%2C+E%2E%2C+Wang%2C+X%2E%2C+Dehghani%2C+M%2E%2C+Brahma%2C+S%2E%2C+Webson%2C+A%2E%2C+Gu%2C+S%2E+S%2E%2C+Dai%2C+Z%2E%2C+Suzgun%2C+M%2E%2C+Chen%2C+X%2E%2C+Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Mishra%2C+G)
15. Cole, J. R., Zhang, M. J. Q., Gillick, D., Eisenschlos, J. M., Dhingra, B., and Eisenstein, J. (2023). [Selectively answering ambiguous questions](https://md.research.elamur.ai/free/en/papers/cole-2023-selectively-answering-ambiguous-questions/). CoRR, abs/2305.14613.
16. Confucius and Disciple (221 BC). The analects of confucius. [find](https://scholar.google.com/scholar?q=Confucius+and+Disciple+%28221+BC%29%2E+The+analects+of+confucius%2E)
17. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. (2023). Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.01377. [find](https://scholar.google.com/scholar?q=Cui%2C+G%2E%2C+Yuan%2C+L%2E%2C+Ding%2C+N%2E%2C+Yao%2C+G%2E%2C+Zhu%2C+W%2E%2C+Ni%2C+Y%2E%2C+Xie%2C+G%2E%2C+Liu%2C+Z%2E%2C+and+Sun%2C+M%2E+%282023%29%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+CoRR%2C+abs%2F2310%2E01377%2E)
18. Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. (2023). [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). CoRR, abs/2305.14233.
19. Dong, H., Xiong, W., Goyal, D., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. (2023). [RAFT: reward ranked finetuning for generative foundation model alignment](https://md.research.elamur.ai/free/en/papers/dong-2023-raft-reward-ranked-finetuning-generative/). CoRR, abs/2304.06767.
20. Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., Righetti, L., and Saunders, W. (2021). Truthful AI: developing and governing AI that does not lie. CoRR, abs/2110.06674. — [arXiv:2110.06674](https://arxiv.org/abs/2110.06674) [find](https://scholar.google.com/scholar?q=Evans%2C+O%2E%2C+Cotton%2DBarratt%2C+O%2E%2C+Finnveden%2C+L%2E%2C+Bales%2C+A%2E%2C+Balwit%2C+A%2E%2C+Wills%2C+P%2E%2C+Righetti%2C+L%2E%2C+and+Saunders%2C+W%2E+%282021%29%2E+Truthful+AI%3A+developing+and+governing+AI+that+does+not+lie%2E+CoRR%2C+abs%2F2110%2E06674%2E)
21. Gao, L., Schulman, J., and Hilton, J. (2023). [Scaling laws for reward model overoptimization](https://md.research.elamur.ai/free/en/papers/gao-2022-scaling-laws-reward-model-overoptimization/). In International Conference on Machine Learning, pages 10835–10866. PMLR.
22. Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., and Herzig, J. (2024). Does fine-tuning llms on new knowledge encourage hallucinations? CoRR, abs/2405.05904. [find](https://scholar.google.com/scholar?q=Gekhman%2C+Z%2E%2C+Yona%2C+G%2E%2C+Aharoni%2C+R%2E%2C+Eyal%2C+M%2E%2C+Feder%2C+A%2E%2C+Reichart%2C+R%2E%2C+and+Herzig%2C+J%2E+%282024%29%2E+Does+fine%2Dtuning+llms+on+new+knowledge+encourage+hallucinations%3F+CoRR%2C+abs%2F2405%2E05904%2E)
23. Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M. J., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., Fe… — [Improving alignment of dialogue agents via targeted human judgements](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
24. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
25. InternLM (2023). Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM). [↗](https://github.com/InternLM/InternLM)
26. Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, B., Sun, R., Wang, Y., and Yang, Y. (2023a). [Beavertails: Towards improved safety alignment of LLM via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). CoRR, abs/2307.04657.
27. Ji, J., Liu, M., Dai, J., Pan, X., Zhang, C., Bian, C., Zhang, C., Sun, R., Wang, Y., and Yang, Y. (2023b). [Beavertails: Towards improved safety alignment of llm via a human-preference dataset](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/). arXiv preprint arXiv:2307.04657.
28. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Madotto, A., and Fung, P. (2023c). [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Comput. Surv., 55(12):248:1–248:38.
29. Jiang, Z., Araki, J., Ding, H., and Neubig, G. (2021). How can we know When language models know? on the calibration of language models for question answering. Trans. Assoc. Comput. Linguistics, 9:962–977. [find](https://scholar.google.com/scholar?q=Jiang%2C+Z%2E%2C+Araki%2C+J%2E%2C+Ding%2C+H%2E%2C+and+Neubig%2C+G%2E+%282021%29%2E+How+can+we+know+When+language+models+know%3F+on+the+calibration+of+language+models+for+question+answering%2E+Trans%2E+Assoc%2E+Comput%2E+Linguistics%2C+9%3A962)
30. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. (2017). [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Barzilay, R. and Kan, M., editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver,…
31. Joshi, N., Rando, J., Saparov, A., Kim, N., and He, H. (2023). Personas as a way to model truthfulness in language models. CoRR, abs/2310.18168. [find](https://scholar.google.com/scholar?q=Joshi%2C+N%2E%2C+Rando%2C+J%2E%2C+Saparov%2C+A%2E%2C+Kim%2C+N%2E%2C+and+He%2C+H%2E+%282023%29%2E+Personas+as+a+way+to+model+truthfulness+in+language+models%2E+CoRR%2C+abs%2F2310%2E18168%2E)
32. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobson, J., Kernion, J., Kr… [find](https://scholar.google.com/scholar?q=Kadavath%2C+S%2E%2C+Conerly%2C+T%2E%2C+Askell%2C+A%2E%2C+Henighan%2C+T%2E%2C+Drain%2C+D%2E%2C+Perez%2C+E%2E%2C+Schiefer%2C+N%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+DasSarma%2C+N%2E%2C+Tran%2DJohnson%2C+E%2E%2C+Johnston%2C+S%2E%2C+Showk%2C+S%2E+E%2E%2C+Jones%2C+A%2E%2C+Elhage%2C+N%2E%2C+Hume%2C+T%2E)
33. Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., and McHardy, R. (2023). [Challenges and applications of large language models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/). CoRR, abs/2307.10169.
34. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving, G. (2021). [Alignment of language agents](https://md.research.elamur.ai/free/en/papers/kenton-2021-alignment-language-agents/). CoRR, abs/2103.14659.
35. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A. P., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. (2019). [Natural questions: a benchmark for question answering res…](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
36. Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., and Catanzaro, B. (2022). [Factuality enhanced language models for open-ended text generation](https://md.research.elamur.ai/free/en/papers/lee-2022-factuality-enhanced-language-models-open/). In NeurIPS.
37. Li, J., Sun, S., Yuan, W., Fan, R., Zhao, H., and Liu, P. (2023a). [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). CoRR, abs/2310.05470.
38. Li, K., Patel, O., Viégas, F. B., Pfister, H., and Wattenberg, M. (2023b). [Inference-time intervention: Eliciting truthful answers from a language model](https://md.research.elamur.ai/free/en/papers/li-2023-inference-time-intervention-eliciting-truthful/). CoRR, abs/2306.03341.
39. Li, X., Yu, P., Zhou, C., Schick, T., Zettlemoyer, L., Levy, O., Weston, J., and Lewis, M. (2023c). [Self-alignment with instruction backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/). arXiv preprint arXiv:2308.06259.
40. Lin, C. and Och, F. J. (2004). [Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/). In Scott, D., Daelemans, W., and Walker, M. A., editors, Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, 21-26 July…
41. Lin, S., Hilton, J., and Evans, O. (2022a). Teaching models to express their uncertainty in words. [find](https://scholar.google.com/scholar?q=Lin%2C+S%2E%2C+Hilton%2C+J%2E%2C+and+Evans%2C+O%2E+%282022a%29%2E+Teaching+models+to+express+their+uncertainty+in+words%2E)
42. Trans. Mach. Learn. Res., 2022. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2022%2E)
43. Lin, S., Hilton, J., and Evans, O. (2022b). [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland,…
44. Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., Dong, H., Pi, R., Zhao, H., Jiang, N., Ji, H., Yao, Y., and Zhang, T. (2024). [Mitigating the alignment tax of rlhf](https://md.research.elamur.ai/free/en/papers/lin-2024-mitigating-alignment-tax-rlhf/).
45. Liu, Y., Yao, Y., Ton, J., Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., and Li, H. (2023). [Trustworthy llms: a survey and guideline for evaluating large language models’ alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-trustworthy-llms-survey-guideline-evaluating/). CoRR, abs/2308.05374.
46. Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Loshchilov%2C+I%2E+and+Hutter%2C+F%2E+%282019%29%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2Enet)
47. Lv, K., Zhang, S., Gu, T., Xing, S., Hong, J., Chen, K., Liu, X., Yang, Y., Guo, H., Liu, T., Sun, Y., Guo, Q., Yan, H., and Qiu, X. (2023). Collie: Collaborative training of large language models in an efficient way. In Feng, Y. and Lefever, E., editors, Proceedings of the 2023 Conference on Empiri… [find](https://scholar.google.com/scholar?q=Lv%2C+K%2E%2C+Zhang%2C+S%2E%2C+Gu%2C+T%2E%2C+Xing%2C+S%2E%2C+Hong%2C+J%2E%2C+Chen%2C+K%2E%2C+Liu%2C+X%2E%2C+Yang%2C+Y%2E%2C+Guo%2C+H%2E%2C+Liu%2C+T%2E%2C+Sun%2C+Y%2E%2C+Guo%2C+Q%2E%2C+Yan%2C+H%2E%2C+and+Qiu%2C+X%2E+%282023%29%2E+Collie%3A+Collaborative+training+of+large+language+models+in+)
48. Mahon, J. E. (2015). The definition of lying and deception. [find](https://scholar.google.com/scholar?q=Mahon%2C+J%2E+E%2E+%282015%29%2E+The+definition+of+lying+and+deception%2E)
49. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. (2023). When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Asso… [find](https://scholar.google.com/scholar?q=Mallen%2C+A%2E%2C+Asai%2C+A%2E%2C+Zhong%2C+V%2E%2C+Das%2C+R%2E%2C+Khashabi%2C+D%2E%2C+and+Hajishirzi%2C+H%2E+%282023%29%2E+When+not+to+trust+language+models%3A+Investigating+effectiveness+of+parametric+and+non%2Dparametric+memories%2E+In+Rogers%2C+)
50. Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. (2023). [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). CoRR, abs/2305.14251.
51. Min, S., Michael, J., Hajishirzi, H., and Zettlemoyer, L. (2020). Ambigqa: Answering ambiguous open-domain questions. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 202… [find](https://scholar.google.com/scholar?q=Min%2C+S%2E%2C+Michael%2C+J%2E%2C+Hajishirzi%2C+H%2E%2C+and+Zettlemoyer%2C+L%2E+%282020%29%2E+Ambigqa%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Webber%2C+B%2E%2C+Cohn%2C+T%2E%2C+He%2C+Y%2E%2C+and+Liu%2C+Y%2E%2C+editors%2C+Proceedings+of+the+2020+Con)
52. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. (2021). [Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/). CoRR, abs/21…
53. OpenAI (2023a). [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). CoRR, abs/2303.08774.
54. OpenAI (2023b). Introducing chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI+%282023b%29%2E+Introducing+chatgpt%2E)
55. OpenAI (2024). Hello gpt-4o. [find](https://scholar.google.com/scholar?q=OpenAI+%282024%29%2E+Hello+gpt%2D4o%2E)
56. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. (2022). [Training language models to follow instru…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
57. Pacchiardi, L., Chan, A. J., Mindermann, S., Moscovitz, I., Pan, A. Y., Gal, Y., Evans, O., and Brauner, J. (2023). [How to catch an AI liar: Lie detection in black-box llms by asking unrelated questions](https://md.research.elamur.ai/free/en/papers/pacchiardi-2023-catch-ai-liar-lie-detection/). CoRR, abs/2309.15840.
58. Park, P. S., Goldstein, S., O’Gara, A., Chen, M., and Hendrycks, D. (2023). AI deception: A survey of examples, risks, and potential solutions. CoRR, abs/2308.14752. — [arXiv:2308.14752](https://arxiv.org/abs/2308.14752) [find](https://scholar.google.com/scholar?q=Park%2C+P%2E+S%2E%2C+Goldstein%2C+S%2E%2C+O%E2%80%99Gara%2C+A%2E%2C+Chen%2C+M%2E%2C+and+Hendrycks%2C+D%2E+%282023%29%2E+AI+deception%3A+A+survey+of+examples%2C+risks%2C+and+potential+solutions%2E+CoRR%2C+abs%2F2308%2E14752%2E+%E2%80%94+%5BarXiv%3A2308%2E14752%5D%28https%3A%2F%2Farxiv)
59. Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., and Gao, J. (2023). [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). CoRR, abs/2302.12813.
60. Scheurer, J., Balesni, M., and Hobbhahn, M. (2023). Technical report: Large language models can strategically deceive their users when put under pressure. CoRR, abs/2311.07590. — [arXiv:2311.07590](https://arxiv.org/abs/2311.07590) [find](https://scholar.google.com/scholar?q=Scheurer%2C+J%2E%2C+Balesni%2C+M%2E%2C+and+Hobbhahn%2C+M%2E+%282023%29%2E+Technical+report%3A+Large+language+models+can+strategically+deceive+their+users+when+put+under+pressure%2E+CoRR%2C+abs%2F2311%2E07590%2E+%E2%80%94+%5BarXiv%3A2311%2E07590%5D%28ht)
61. Schulman, J. (2023). Reinforcement learning from human feedback: Progress and challenges. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E+%282023%29%2E+Reinforcement+learning+from+human+feedback%3A+Progress+and+challenges%2E)
62. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., and Perez, E. (2023). [Towards understanding sycophancy in languag…](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/)
63. Shumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N., and Anderson, R. J. (2023). [The curse of recursion: Training on generated data makes models forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/). CoRR, abs/2305.17493.
64. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. (2023). Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
65. Tian, K., Mitchell, E., Zhou, A., Sharma, A., Rafailov, R., Yao, H., Finn, C., and Manning, C. D. (2023). Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. CoRR, abs/2305.14975. [find](https://scholar.google.com/scholar?q=Tian%2C+K%2E%2C+Mitchell%2C+E%2E%2C+Zhou%2C+A%2E%2C+Sharma%2C+A%2E%2C+Rafailov%2C+R%2E%2C+Yao%2C+H%2E%2C+Finn%2C+C%2E%2C+and+Manning%2C+C%2E+D%2E+%282023%29%2E+Just+ask+for+calibration%3A+Strategies+for+eliciting+calibrated+confidence+scores+from+language+)
66. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Canton-Ferrer, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Ho… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
67. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. (2023a). [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRevi…
68. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. (2023b). Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Compu… [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Kordi%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Liu%2C+A%2E%2C+Smith%2C+N%2E+A%2E%2C+Khashabi%2C+D%2E%2C+and+Hajishirzi%2C+H%2E+%282023b%29%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+)
69. Wei, J. W., Huang, D., Lu, Y., Zhou, D., and Le, Q. V. (2023). [Simple synthetic data reduces sycophancy in large language models](https://md.research.elamur.ai/free/en/papers/wei-2023-simple-synthetic-data-reduces-sycophancy/). CoRR, abs/2308.03958.
70. Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2023). Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. CoRR, abs/2306.13063. [find](https://scholar.google.com/scholar?q=Xiong%2C+M%2E%2C+Hu%2C+Z%2E%2C+Lu%2C+X%2E%2C+Li%2C+Y%2E%2C+Fu%2C+J%2E%2C+He%2C+J%2E%2C+and+Hooi%2C+B%2E+%282023%29%2E+Can+llms+express+their+uncertainty%3F+an+empirical+evaluation+of+confidence+elicitation+in+llms%2E+CoRR%2C+abs%2F2306%2E13063%2E)
71. Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. (2023). [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). CoRR, abs/2304.12244.
72. Yin, Z., Sun, Q., Guo, Q., Wu, J., Qiu, X., and Huang, X. (2023). Do large language models know what they don’t know? In Rogers, A., Boyd-Graber, J. L., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8653–8665. A… [find](https://scholar.google.com/scholar?q=Yin%2C+Z%2E%2C+Sun%2C+Q%2E%2C+Guo%2C+Q%2E%2C+Wu%2C+J%2E%2C+Qiu%2C+X%2E%2C+and+Huang%2C+X%2E+%282023%29%2E+Do+large+language+models+know+what+they+don%E2%80%99t+know%3F+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E+L%2E%2C+and+Okazaki%2C+N%2E%2C+editors%2C+Findings+of+the+Assoc)
73. Yu, W., Zhang, Z., Liang, Z., Jiang, M., and Sabharwal, A. (2023). [Improving language models via plug-and-play retrieval feedback](https://md.research.elamur.ai/free/en/papers/yu-2023-improving-language-models-plug-play/). CoRR, abs/2305.14002.
74. Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. (2023). [RRHF: rank responses to align language models with human feedback without tears](https://md.research.elamur.ai/free/en/papers/yuan-2023-rrhf-rank-responses-align-language/). CoRR, abs/2304.05302.
75. Yudkowsky, E. (2018). Meta-honesty: Firming up honesty around its edge-cases. [find](https://scholar.google.com/scholar?q=Yudkowsky%2C+E%2E+%282018%29%2E+Meta%2Dhonesty%3A+Firming+up+honesty+around+its+edge%2Dcases%2E)
76. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., Wang, L., Luu, A. T., Bi, W., Shi, F., and Shi, S. (2023). [Siren’s song in the AI ocean: A survey on hallucination in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/). CoRR, abs/2309.01219.
77. Zhang, Z., Lu, Y., Ma, J., Zhang, D., Li, R., Ke, P., Sun, H., Sha, L., Sui, Z., Wang, H., and Huang, M. (2024). Shieldlm: Empowering llms as aligned, customizable and explainable safety detectors. arXiv preprint. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Lu%2C+Y%2E%2C+Ma%2C+J%2E%2C+Zhang%2C+D%2E%2C+Li%2C+R%2E%2C+Ke%2C+P%2E%2C+Sun%2C+H%2E%2C+Sha%2C+L%2E%2C+Sui%2C+Z%2E%2C+Wang%2C+H%2E%2C+and+Huang%2C+M%2E+%282024%29%2E+Shieldlm%3A+Empowering+llms+as+aligned%2C+customizable+and+explainable+safety+detectors%2E+ar)
78. Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. (2023a). [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). CoRR, abs/2306.05685.
79. Zheng, S., Huang, J., and Chang, K. C.-C. (2023b). Why does chatgpt fall short in providing truthful answers? [find](https://scholar.google.com/scholar?q=Zheng%2C+S%2E%2C+Huang%2C+J%2E%2C+and+Chang%2C+K%2E+C%2E%2DC%2E+%282023b%29%2E+Why+does+chatgpt+fall+short+in+providing+truthful+answers%3F)
80. Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., Zhang, S., Ghosh, G., Lewis, M., Zettlemoyer, L., and Levy, O. (2023a). LIMA: less is more for alignment. CoRR, abs/2305.11206. [find](https://scholar.google.com/scholar?q=Zhou%2C+C%2E%2C+Liu%2C+P%2E%2C+Xu%2C+P%2E%2C+Iyer%2C+S%2E%2C+Sun%2C+J%2E%2C+Mao%2C+Y%2E%2C+Ma%2C+X%2E%2C+Efrat%2C+A%2E%2C+Yu%2C+P%2E%2C+Yu%2C+L%2E%2C+Zhang%2C+S%2E%2C+Ghosh%2C+G%2E%2C+Lewis%2C+M%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Levy%2C+O%2E+%282023a%29%2E+LIMA%3A+less+is+more+for+alignment%2E+CoRR)
81. Zhou, K., Jurafsky, D., and Hashimoto, T. (2023b). Navigating the grey area: Expressions of overconfidence and uncertainty in language models. CoRR, abs/2302.13439. [find](https://scholar.google.com/scholar?q=Zhou%2C+K%2E%2C+Jurafsky%2C+D%2E%2C+and+Hashimoto%2C+T%2E+%282023b%29%2E+Navigating+the+grey+area%3A+Expressions+of+overconfidence+and+uncertainty+in+language+models%2E+CoRR%2C+abs%2F2302%2E13439%2E)
82. Zhu, Z. A. and Li, Y. (2023). [Physics of language models: Part 3.1, knowledge storage and extraction](https://md.research.elamur.ai/free/en/papers/allen-zhu-2023-physics-language-models-part-3/). CoRR, abs/2309.14316.
83. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. (2023). [Representation engineering: A top-down approach…](https://md.research.elamur.ai/free/en/papers/zou-2023-representation-engineering-top-down-approach/)

*Updated 2 November 2026*