---
title: "LUQ: Long-text Uncertainty Quantification for LLMs (2024)"
description: "They then introduce LUQ, a black-box, sampling-based method. 60 references, cited by 31 papers here."
authors: ["Caiqi Zhang", "Fangyu Liu", "Marco Basaldella", "Nigel Collier"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.299/"
doi: "10.18653/v1/2024.emnlp-main.299"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / LUQ: Long-text Uncertainty Quantification for LLMs

# LUQ: Long-text Uncertainty Quantification for LLMs

**Authors:** [Caiqi Zhang](https://md.research.elamur.ai/free/en/authors/caiqi-zhang/), [Fangyu Liu](https://md.research.elamur.ai/free/en/authors/fangyu-liu/), [Marco Basaldella](https://md.research.elamur.ai/free/en/authors/marco-basaldella/) and [Nigel Collier](https://md.research.elamur.ai/free/en/authors/nigel-collier/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.299/) · [PDF](https://aclanthology.org/2024.emnlp-main.299.pdf) · [DOI 10.18653/v1/2024.emnlp-main.299](https://doi.org/10.18653/v1/2024.emnlp-main.299) · [OpenAlex](https://openalex.org/W4404784419)

**Citations:** 11 citations (OpenAlex, 2026-10-14) · [31](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Most uncertainty quantification (UQ) work for LLMs targets very short answers, usually under ten words, and often needs access to logits. Real use cases, however, involve long responses, and many leading models are closed-source and available only through APIs. The authors first show that existing UQ techniques break down on long outputs, where whole-response similarity becomes uniformly high. They then introduce LUQ, a black-box, sampling-based method. LUQ splits each sampled answer into sentences and uses an NLI model to check how well each sentence is entailed by the other samples. Low agreement among samples signals high uncertainty. Two variants, LUQ-Atomic and LUQ-Pair, are also proposed.

Experiments use six popular LLMs on FActScore and on a new medical-domain extension, FActScore-DIS. LUQ correlates more strongly, and negatively, with factuality than all baselines. A further method, LUQ-Ensemble, chooses the answer from the least uncertain model, which raises factuality. This gives a practical way to flag likely hallucinations without costly fact-checking.

## Key points

- LUQ correlates with factuality more strongly than prior methods, reaching a coefficient of −0.85 for Gemini Pro.
- Sentence-level consistency is scored with DeBERTa-v3-large fine-tuned on MultiNLI. Only the entailment and contradiction logits are used, and neutral predictions are ignored.
- Evaluation covers long answers of at least 100 words from six LLMs, including GPT-4 and Gemini 1.0 Pro. The data are FActScore plus the new human-validated, disease-focused FActScore-DIS.
- LUQ-Ensemble improves overall factuality by up to 5% over the best single model. The authors also apply selective answering, declining to answer high-uncertainty queries.
- Stated limitations: factuality is the only quality dimension measured, ambiguous or unanswerable questions (e.g., ASQA, SelfAware) are not tested, and the analysis targets overall model uncertainty rather than per-instance uncertainty.
- Code: github.com/caiqizh/LUQ

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 243 works · 121,396 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 112 works · 40,565 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Caiqi Zhang, Fangyu Liu, Marco Basaldella and Nigel Collier. 2024. LUQ: Long-text Uncertainty Quantification for LLMs. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2024luq,
  title = {{LUQ: Long-text Uncertainty Quantification for LLMs}},
  author = {Caiqi Zhang, Fangyu Liu, Marco Basaldella and Nigel Collier},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.299},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.299}
}
```

- [← previous Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels](https://md.research.elamur.ai/free/en/papers/pangakis-2024-knowledge-distillation-automated-annotation-supervised/)
- [next → Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models](https://md.research.elamur.ai/free/en/papers/ormazabal-2024-reka-core-flash-edge-series/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

60 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 01.ai. 2023. Building the next generation of open-source and bilingual llms. [https://huggingface.co/01-ai/Yi-34B-Chat](https://huggingface.co/01-ai/Yi-34B-Chat). Accessed: 2024-02-05. [↗](https://huggingface.co/01-ai/Yi-34B-Chat)
2. Anthropic. 2023. Introducing claude 2.1. Available from Anthropic: [https://www.anthropic.com/](https://www.anthropic.com/) news/claude-2-1. [↗](https://www.anthropic.com/)
3. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Nico+Daheim%2C+Evgenia+Ilia%2C+Dennis+Ulmer%2C+Haau%2DSing+Li%2C+Raquel+Fern%C3%A1ndez%2C+Barbara+Plank%2C+Rico+Sennrich%2C+Chrysoula+Zerva%2C+and+Wilker+Aziz%2E+2023%2E+Uncertainty+in+natural+language+generation%3A+F)
4. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ArXiv preprint, abs/2307.03109.
5. Jiuhai Chen and Jonas Mueller. 2023. [Quantifying uncertainty in answers from any language model and enhancing their trustworthiness](https://md.research.elamur.ai/free/en/papers/chen-2024-quantifying-uncertainty-answers-any-language/).
6. Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. 2024. See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses. [find](https://scholar.google.com/scholar?q=Yulong+Chen%2C+Yang+Liu%2C+Jianhao+Yan%2C+Xuefeng+Bai%2C+Ming+Zhong%2C+Yinghao+Yang%2C+Ziyi+Yang%2C+Chenguang+Zhu%2C+and+Yue+Zhang%2E+2024%2E+See+what+llms+cannot+answer%3A+A+self%2Dchallenge+framework+for+uncovering+llm+wea)
7. Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. [Selectively answering ambiguous questions](https://md.research.elamur.ai/free/en/papers/cole-2023-selectively-answering-ambiguous-questions/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 530–543, Singapore. Association for Computational Li…
8. Armen Der Kiureghian and Ove Ditlevsen. 2009. Aleatory or epistemic? does it matter? Structural safety, 31(2):105–112. [find](https://scholar.google.com/scholar?q=Armen+Der+Kiureghian+and+Ove+Ditlevsen%2E+2009%2E+Aleatory+or+epistemic%3F+does+it+matter%3F+Structural+safety%2C+31%282%29%3A105%E2%80%93112%2E)
9. Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2024. Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models. [find](https://scholar.google.com/scholar?q=Hanxing+Ding%2C+Liang+Pang%2C+Zihao+Wei%2C+Huawei+Shen%2C+and+Xueqi+Cheng%2E+2024%2E+Retrieve+only+when+it+needs%3A+Adaptive+retrieval+augmentation+for+hallucination+mitigation+in+large+language+models%2E)
10. Yijiang River Dong, Tiancheng Hu, and Nigel Collier. 2024. Can llm be a personalized judge? [find](https://scholar.google.com/scholar?q=Yijiang+River+Dong%2C+Tiancheng+Hu%2C+and+Nigel+Collier%2E+2024%2E+Can+llm+be+a+personalized+judge%3F)
11. Jinhao Duan, Hao Cheng, Shiqi Wang, Chenan Wang, Alex Zavalny, Renjing Xu, Bhavya Kailkhura, and 5252 9 [find](https://scholar.google.com/scholar?q=Jinhao+Duan%2C+Hao+Cheng%2C+Shiqi+Wang%2C+Chenan+Wang%2C+Alex+Zavalny%2C+Renjing+Xu%2C+Bhavya+Kailkhura%2C+and+5252+9)
12. Kaidi Xu. 2023. Shifting attention to relevance: Towards the uncertainty estimation of large language models. ArXiv preprint, abs/2307.01379. sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Kaidi+Xu%2E+2023%2E+Shifting+attention+to+relevance%3A+Towards+the+uncertainty+estimation+of+large+language+models%2E+ArXiv+preprint%2C+abs%2F2307%2E01379%2E+sharing%2E+In+The+Eleventh+International+Conference+on+Learn)
13. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncertainty estimation for language models. In Proceedings of the 202… [find](https://scholar.google.com/scholar?q=Ekaterina+Fadeeva%2C+Roman+Vashurin%2C+Akim+Tsvigun%2C+Artem+Vazhentsev%2C+Sergey+Petrakov%2C+Kirill+Fedyanin%2C+Daniil+Vasilev%2C+Elizaveta+Goncharova%2C+Alexander+Panchenko%2C+Maxim+Panov%2C+Timothy+Baldwin%2C+and+Artem+)
14. Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2017%2E+A+baseline+for+detecting+misclassified+and+out%2Dof%2Ddistribution+examples+in+neural+networks%2E+In+5th+International+Conference+on+Learning+Representations%2C+ICLR+2017)
15. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. [ELI5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
16. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Transactions of the Association for Computational Linguistics, 8:539–555.
17. Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. 2023. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513–1589. [find](https://scholar.google.com/scholar?q=Jakob+Gawlikowski%2C+Cedrique+Rovile+Njieutcheu+Tassi%2C+Mohsin+Ali%2C+Jongseok+Lee%2C+Matthias+Humt%2C+Jianxiang+Feng%2C+Anna+Kruspe%2C+Rudolph+Triebel%2C+Peter+Jung%2C+Ribana+Roscher%2C+et+al%2E+2023%2E+A+survey+of+uncerta)
18. Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4878–4887. [find](https://scholar.google.com/scholar?q=Yonatan+Geifman+and+Ran+El%2DYaniv%2E+2017%2E+Selective+classification+for+deep+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+30%3A+Annual+Conference+on+Neural+Information+Processing+S)
19. Gemini Team. 2023. [Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). Technical report, Google.
20. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2023. A survey of language model confidence estimation and calibration. ArXiv preprint, abs/2311.08298. [find](https://scholar.google.com/scholar?q=Jiahui+Geng%2C+Fengyu+Cai%2C+Yuxia+Wang%2C+Heinz+Koeppl%2C+Preslav+Nakov%2C+and+Iryna+Gurevych%2E+2023%2E+A+survey+of+language+model+confidence+estimation+and+calibration%2E+ArXiv+preprint%2C+abs%2F2311%2E08298%2E)
21. Stephen C Hora. 1996. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2-3):217–223. [find](https://scholar.google.com/scholar?q=Stephen+C+Hora%2E+1996%2E+Aleatory+and+epistemic+uncertainty+in+probability+elicitation+with+an+example+from+hazardous+waste+management%2E+Reliability+Engineering+%26+System+Safety%2C+54%282%2D3%29%3A217%E2%80%93223%2E)
22. Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natural language processing: Sources, quantification, and applications. [find](https://scholar.google.com/scholar?q=Mengting+Hu%2C+Zhen+Zhang%2C+Shiwan+Zhao%2C+Minlie+Huang%2C+and+Bingzhe+Wu%2E+2023%2E+Uncertainty+in+natural+language+processing%3A+Sources%2C+quantification%2C+and+applications%2E)
23. Xinmeng Huang, Shuo Li, Mengxin Yu, Matteo Sesia, Hamed Hassani, Insup Lee, Osbert Bastani, and Edgar Dobriban. 2024a. Uncertainty in language models: Assessment through rank-calibration. [find](https://scholar.google.com/scholar?q=Xinmeng+Huang%2C+Shuo+Li%2C+Mengxin+Yu%2C+Matteo+Sesia%2C+Hamed+Hassani%2C+Insup+Lee%2C+Osbert+Bastani%2C+and+Edgar+Dobriban%2E+2024a%2E+Uncertainty+in+language+models%3A+Assessment+through+rank%2Dcalibration%2E)
24. Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. 2023. Look before you leap: An exploratory study of uncertainty measurement for large language models. ArXiv preprint, abs/2307.10236. — [arXiv:2307.10236](https://arxiv.org/abs/2307.10236) [find](https://scholar.google.com/scholar?q=Yuheng+Huang%2C+Jiayang+Song%2C+Zhijie+Wang%2C+Huaming+Chen%2C+and+Lei+Ma%2E+2023%2E+Look+before+you+leap%3A+An+exploratory+study+of+uncertainty+measurement+for+large+language+models%2E+ArXiv+preprint%2C+abs%2F2307%2E10236)
25. Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024b. Calibrating long-form generations from large language models. [find](https://scholar.google.com/scholar?q=Yukun+Huang%2C+Yixin+Liu%2C+Raghuveer+Thirukovalluru%2C+Arman+Cohan%2C+and+Bhuwan+Dhingra%2E+2024b%2E+Calibrating+long%2Dform+generations+from+large+language+models%2E)
26. Eyke Hüllermeier and Willem Waegeman. 2021. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457–506. [find](https://scholar.google.com/scholar?q=Eyke+H%C3%BCllermeier+and+Willem+Waegeman%2E+2021%2E+Aleatoric+and+epistemic+uncertainty+in+machine+learning%3A+An+introduction+to+concepts+and+methods%2E+Machine+Learning%2C+110%3A457%E2%80%93506%2E)
27. Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. [Camels in a changing climate: Enhancing lm adaptation with tulu 2](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/).
28. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. [A survey on automated fact-checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/). Transactions of the Association for Computational Linguistics, 10:178–206.
29. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684– 5696, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Amita+Kamath%2C+Robin+Jia%2C+and+Percy+Liang%2E+2020%2E+Selective+question+answering+under+domain+shift%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+5684%E2%80%93)
30. Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. 2024. Towards uncertainty-aware language agent. In Findings of the Association for Computational Linguistics ACL 2024, pages 6662–6685, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jiuzhou+Han%2C+Wray+Buntine%2C+and+Ehsan+Shareghi%2E+2024%2E+Towards+uncertainty%2Daware+language+agent%2E+In+Findings+of+the+Association+for+Computational+Linguistics+ACL+2024%2C+pages+6662%E2%80%936685%2C+Bangkok%2C+Thailand)
31. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Lorenz+Kuhn%2C+Yarin+Gal%2C+and+Sebastian+Farquhar%2E+2023%2E+Semantic+uncertainty%3A+Linguistic+invariances+for+uncertainty+estimation+in+natural+language+generation%2E+In+The+Eleventh+International+Conference+o)
32. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. 5253 10 [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+2023%2E+Debertav3%3A+Improving+deberta+using+electra%2Dstyle+pre%2Dtraining+with+gradient%2Ddisentangled+embedding+J+Richard+Landis+and+Gary+G+Koch%2E+1977%2E+The+measur)
33. Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, and Ivan Vulić. 2023. On task performance and model calibration with supervised and self-ensembled in-context learning. [find](https://scholar.google.com/scholar?q=Chengzu+Li%2C+Han+Zhou%2C+Goran+Glava%C5%A1%2C+Anna+Korhonen%2C+and+Ivan+Vulic%CC%81%2E+2023%2E+On+task+performance+and+model+calibration+with+supervised+and+self%2Densembled+in%2Dcontext+learning%2E)
34. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. ArXiv preprint, abs/2205.14334. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+Teaching+models+to+express+their+uncertainty+in+words%2E+ArXiv+preprint%2C+abs%2F2205%2E14334%2E)
35. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. [find](https://scholar.google.com/scholar?q=Zhen+Lin%2C+Shubhendu+Trivedi%2C+and+Jimeng+Sun%2E+2023%2E+Generating+with+confidence%3A+Uncertainty+quantification+for+black%2Dbox+large+language+models%2E)
36. Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Andrey+Malinin+and+Mark+J%2E+F%2E+Gales%2E+2021%2E+Uncertainty+estimation+in+autoregressive+structured+prediction%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+2021%2C+Virtual+Event%2C+Austri)
37. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+Gales%2E+2023%2E+SelfCheckGPT%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empir)
38. Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022. Reducing conversational agents’ overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857–872. [find](https://scholar.google.com/scholar?q=Sabrina+J%2E+Mielke%2C+Arthur+Szlam%2C+Emily+Dinan%2C+and+Y%2DLan+Boureau%2E+2022%2E+Reducing+conversational+agents%E2%80%99+overconfidence+through+linguistic+calibration%2E+Transactions+of+the+Association+for+Computational+)
39. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
40. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 212–223, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kenton+Murray+and+David+Chiang%2E+2018%2E+Correcting+length+bias+in+neural+machine+translation%2E+In+Proceedings+of+the+Third+Conference+on+Machine+Translation%3A+Research+Papers%2C+pages+212%E2%80%93223%2C+Brussels%2C+Bel)
41. OpenAI. 2022. Chatgpt blog post. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Chatgpt+blog+post%2E)
42. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
43. Mauricio Rivera, Jean-François Godbout, Reihaneh Rabbany, and Kellin Pelrine. 2024. Combining confidence elicitation and sample-based methods for uncertainty quantification in misinformation mitigation. In Proceedings of the 1st Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pages 114–126, St… [find](https://scholar.google.com/scholar?q=Mauricio+Rivera%2C+Jean%2DFran%C3%A7ois+Godbout%2C+Reihaneh+Rabbany%2C+and+Kellin+Pelrine%2E+2024%2E+Combining+confidence+elicitation+and+sample%2Dbased+methods+for+uncertainty+quantification+in+misinformation+mitigatio)
44. Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia, 126(5):1763– 1768. [find](https://scholar.google.com/scholar?q=Patrick+Schober%2C+Christa+Boer%2C+and+Lothar+A+Schwarte%2E+2018%2E+Correlation+coefficients%3A+appropriate+use+and+interpretation%2E+Anesthesia+%26+analgesia%2C+126%285%29%3A1763%E2%80%93+1768%2E)
45. Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. [ASQA: Factoid questions meet long-form answers](https://md.research.elamur.ai/free/en/papers/stelmakh-2022-asqa-factoid-questions-meet-long/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
46. Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, and Artem Shelmanov. 2023. Efficient out-of-domain detection for sequence to sequence models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1430– 1454, T… [find](https://scholar.google.com/scholar?q=Artem+Vazhentsev%2C+Akim+Tsvigun%2C+Roman+Vashurin%2C+Sergey+Petrakov%2C+Daniil+Vasilev%2C+Maxim+Panov%2C+Alexander+Panchenko%2C+and+Artem+Shelmanov%2E+2023%2E+Efficient+out%2Dof%2Ddomain+detection+for+sequence+to+sequence)
47. Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. 2023. [Survey on factuality in large language models: Knowledge, retrieval and domain-specifi…](https://md.research.elamur.ai/free/en/papers/wang-2023-survey-factuality-large-language-models/)
48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
49. Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273–7284,… [find](https://scholar.google.com/scholar?q=Yuxin+Xiao%2C+Paul+Pu+Liang%2C+Umang+Bhatt%2C+Willie+Neiswanger%2C+Ruslan+Salakhutdinov%2C+and+Louis%2DPhilippe+Morency%2E+2022%2E+Uncertainty+quantification+with+pre%2Dtrained+language+models%3A+A+large%2Dscale+empirical+)
50. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. [find](https://scholar.google.com/scholar?q=Miao+Xiong%2C+Zhiyuan+Hu%2C+Xinyang+Lu%2C+Yifei+Li%2C+Jie+Fu%2C+Junxian+He%2C+and+Bryan+Hooi%2E+2023%2E+Can+llms+express+their+uncertainty%3F+an+empirical+evaluation+of+confidence+elicitation+in+llms%2E)
51. Qi Yang, Shreya Ravikumar, Fynn Schmitt-Ulms, Satvik Lolla, Ege Demir, Iaroslav Elistratov, Alex Lavaee, Sadhana Lolla, Elaheh Ahmadi, Daniela Rus, Alexander Amini, and Alejandro Perez. 2023. Uncertainty-aware language modeling for selective question answering. [find](https://scholar.google.com/scholar?q=Qi+Yang%2C+Shreya+Ravikumar%2C+Fynn+Schmitt%2DUlms%2C+Satvik+Lolla%2C+Ege+Demir%2C+Iaroslav+Elistratov%2C+Alex+Lavaee%2C+Sadhana+Lolla%2C+Elaheh+Ahmadi%2C+Daniela+Rus%2C+Alexander+Amini%2C+and+Alejandro+Perez%2E+2023%2E+Uncertai)
52. Yuxuan Yao, Han Wu, Zhijiang Guo, Biyan Zhou, Jiahui Gao, Sichun Luo, Hanxu Hou, Xiaojin Fu, and Linqi Song. 2024. Learning from correctness without prompting makes llm efficient reasoner. [find](https://scholar.google.com/scholar?q=Yuxuan+Yao%2C+Han+Wu%2C+Zhijiang+Guo%2C+Biyan+Zhou%2C+Jiahui+Gao%2C+Sichun+Luo%2C+Hanxu+Hou%2C+Xiaojin+Fu%2C+and+Linqi+Song%2E+2024%2E+Learning+from+correctness+without+prompting+makes+llm+efficient+reasoner%2E)
53. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In 5254 11 [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Jiawen+Wu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2023%2E+Do+large+language+models+know+what+they+don%E2%80%99t+know%3F+In+5254+11)
54. Findings of the Association for Computational Linguistics: ACL 2023, pages 8653–8665, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2023%2C+pages+8653%E2%80%938665%2C+Toronto%2C+Canada%2E+Association+for+Computational+Linguistics%2E)
55. Caiqi Zhang, Zhijiang Guo, and Andreas Vlachos. 2024. Do we need language-specific fact-checking models? the case of chinese. [find](https://scholar.google.com/scholar?q=Caiqi+Zhang%2C+Zhijiang+Guo%2C+and+Andreas+Vlachos%2E+2024%2E+Do+we+need+language%2Dspecific+fact%2Dchecking+models%3F+the+case+of+chinese%2E)
56. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
57. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. [Siren’s song in the ai ocean: A survey on hallucination in large language models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/). ArXiv preprint, abs/2309.01219.
58. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. [A survey of l…](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
59. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems 36: A…
60. Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine Heller, and Subhrajit Roy. 2023. Batch calibration: Rethinking calibration for in-context learning and prompt engineering. [find](https://scholar.google.com/scholar?q=Han+Zhou%2C+Xingchen+Wan%2C+Lev+Proleev%2C+Diana+Mincu%2C+Jilin+Chen%2C+Katherine+Heller%2C+and+Subhrajit+Roy%2E+2023%2E+Batch+calibration%3A+Rethinking+calibration+for+in%2Dcontext+learning+and+prompt+engineering%2E)

*Updated 30 October 2026*