---
title: "Revisiting Uncertainty Quantification Evaluation in Language"
description: "Uncertainty quantification (UQ) methods for language models are usually benchmarked by checking… 48 references, cited by 5 papers here."
authors: ["Andrea Santilli", "Adam Golinski", "Michael Kirchhof", "Federico Danieli", "Arno Blaas", "Miao Xiong", "Luca Zappella", "Sinead Williamson"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2025.acl-short.60/"
doi: "10.18653/v1/2025.acl-short.60"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/santilli-2025-revisiting-uncertainty-quantification-evaluation-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/santilli-2025-revisiting-uncertainty-quantification-evaluation-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results

# Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results

**Authors:** Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella and [Sinead Williamson](https://md.research.elamur.ai/free/en/authors/sinead-williamson/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-short.60/) · [PDF](https://aclanthology.org/2025.acl-short.60.pdf) · [DOI 10.18653/v1/2025.acl-short.60](https://doi.org/10.18653/v1/2025.acl-short.60) · [OpenAlex](https://openalex.org/W4412889513)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/santilli-2025-revisiting-uncertainty-quantification-evaluation-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Uncertainty quantification (UQ) methods for language models are usually benchmarked by checking, via AUROC, how well uncertainty scores separate correct from incorrect answers. Correctness itself is estimated automatically, for example with ROUGE-L. The authors ask whether this proxy can distort the comparison. They show formally that random, independent labeling errors only add noise and pull AUROC toward 0.5. When the errors of the correctness function and the UQ score depend on a common factor, however, rankings shift systematically.

To test this, they run generative QA experiments covering 7 correctness functions (lexical, embedding-based, and LM-as-a-judge), 4 datasets, 4 models, and 8 UQ methods. They also collect human annotations on 450 model outputs. Response length emerges as the shared confounder: lexical and embedding metrics are length-biased, as are probability-based UQ scores. This makes some methods look better than they really are. LM-as-a-judge shows the least length bias and agrees best with humans, which makes it the recommended basis for fairer UQ benchmarks.

## Key points
- **Theory:** independent errors in the correctness function leave AUROC noisy but unbiased. Errors that correlate with the uncertainty score (a "mutual bias") skew rankings systematically, and this holds for any shared confounder, not only length.
- **Ranking instability:** across ROUGE-1, ROUGE-L, SQuAD F1, BERTScore, SentenceBERT, AlignScore, and prompt-based LM-as-a-judge, both AUROC values and UQ method rankings change.
- **Length baselines expose the problem:**
  - Trivial token-length and character-length baselines look competitive under lexical and embedding metrics.
  - They fall to last place under LM-as-a-judge.
  - This could account for why Fadeeva et al. (2023) reported negative sequence probability as performing unusually well.
- **Length independence:** AlignScore and ROUGE-L (Recall) do not depend on answer length, but ROUGE-L (Recall) correlates less with human judgments, which yields noisier estimates.
- **Caveats:**
  - The study is limited to QA.
  - LM judges are computationally costly.
  - Judges should be validated against human annotations before use on new tasks.
  - Using the same LM as both judge and UQ component may itself create correlated errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 180 works · 34,304 citations
- [Sapienza University of Rome](https://md.research.elamur.ai/free/en/universities/sapienza-university-of-rome/) — University, Rome, Italy · 57 works · 9,564 citations
- [Apple](https://md.research.elamur.ai/free/en/companies/apple/) — Company, Cupertino, United States · 20 works · 2,146 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella and Sinead Williamson. 2025. Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{santilli2025revisiting,
  title = {{Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results}},
  author = {Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella and Sinead Williamson},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.60},
  url = {https://doi.org/10.18653/v1/2025.acl-short.60}
}
```

- [← previous Revisiting Common Assumptions about Arabic Dialects in NLP](https://md.research.elamur.ai/free/en/papers/keleg-2025-revisiting-common-assumptions-about-arabic/)
- [next → rStar2-Agent: Agentic Reasoning Technical Report](https://md.research.elamur.ai/free/en/papers/shang-2025-rstar2-agent-agentic-reasoning/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

48 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lukas Aichberger, Kajetan Schweighofer, Mykyta Ielanskyi, and Sepp Hochreiter. 2025. Improving uncertainty estimation through semantically diverse language generation. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lukas+Aichberger%2C+Kajetan+Schweighofer%2C+Mykyta+Ielanskyi%2C+and+Sepp+Hochreiter%2E+2025%2E+Improving+uncertainty+estimation+through+semantically+diverse+language+generation%2E+In+International+Conference+on+L)
2. Ron Artstein and Massimo Poesio. 2008. [Survey article: Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555–596.
3. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703. — [arXiv:2307.15703](https://arxiv.org/abs/2307.15703) [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Nico+Daheim%2C+Evgenia+Ilia%2C+Dennis+Ulmer%2C+Haau%2DSing+Li%2C+Raquel+Fern%C3%A1ndez%2C+Barbara+Plank%2C+Rico+Sennrich%2C+Chrysoula+Zerva%2C+and+Wilker+Aziz%2E+2023%2E+Uncertainty+in+natural+language+generation%3A+F)
4. Lalit R. Bahl, Frederick Jelinek, and Robert L. Mercer. 1983. A maximum likelihood approach to continuous speech recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-5(2):179– 190. [find](https://scholar.google.com/scholar?q=Lalit+R%2E+Bahl%2C+Frederick+Jelinek%2C+and+Robert+L%2E+Mercer%2E+1983%2E+A+maximum+likelihood+approach+to+continuous+speech+recognition%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+PAMI%2D5%282%29%3A)
5. Neil Band, Xuechen Li, Tengyu Ma, and Tatsunori Hashimoto. 2024. Linguistic calibration of long-form generations. In International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Neil+Band%2C+Xuechen+Li%2C+Tengyu+Ma%2C+and+Tatsunori+Hashimoto%2E+2024%2E+Linguistic+calibration+of+long%2Dform+generations%2E+In+International+Conference+on+Machine+Learning%2E)
6. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya… — [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/)
7. Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Evaluating question answering evaluation. In 2nd Workshop on Machine Reading for Question Answering, pages 119–124. [find](https://scholar.google.com/scholar?q=Anthony+Chen%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+Evaluating+question+answering+evaluation%2E+In+2nd+Workshop+on+Machine+Reading+for+Question+Answering%2C+pages+119%E2%80%93124%2E)
8. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024a. [find](https://scholar.google.com/scholar?q=Chao+Chen%2C+Kai+Liu%2C+Ze+Chen%2C+Yi+Gu%2C+Yue+Wu%2C+Mingyuan+Tao%2C+Zhihang+Fu%2C+and+Jieping+Ye%2E+2024a%2E)
9. INSIDE: LLMs’ internal states retain the power of hallucination detection. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=INSIDE%3A+LLMs%E2%80%99+internal+states+retain+the+power+of+hallucination+detection%2E+In+International+Conference+on+Learning+Representations%2E)
10. Xiaobo Guo and Soroush Vosoughi. 2023. Length does matter: Summary length can bias summarization metrics. In Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Xiaobo+Guo+and+Soroush+Vosoughi%2E+2023%2E+Length+does+matter%3A+Summary+length+can+bias+summarization+metrics%2E+In+Empirical+Methods+in+Natural+Language+Processing%2E)
11. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024b. [Humans or LLMs as the judge? a study on judgement bias](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-or-llms-as-judge/). In Empirical Methods in Natural Language Processing.
12. Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. In International Conference on Language Resources and Evaluation. [find](https://scholar.google.com/scholar?q=Arman+Cohan+and+Nazli+Goharian%2E+2016%2E+Revisiting+summarization+evaluation+for+scientific+articles%2E+In+International+Conference+on+Language+Resources+and+Evaluation%2E)
13. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
14. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Com… [find](https://scholar.google.com/scholar?q=Jinhao+Duan%2C+Hao+Cheng%2C+Shiqi+Wang%2C+Alex+Zavalny%2C+Chenan+Wang%2C+Renjing+Xu%2C+Bhavya+Kailkhura%2C+and+Kaidi+Xu%2E+2024%2E+Shifting+attention+to+relevance%3A+Towards+the+predictive+uncertainty+quantification+of+f)
15. Alexander R Fabbri, Wojciech Kryściński, Bryan Mc-Cann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. [find](https://scholar.google.com/scholar?q=Alexander+R+Fabbri%2C+Wojciech+Krys%CC%81cin%CC%81ski%2C+Bryan+Mc%2DCann%2C+Caiming+Xiong%2C+Richard+Socher%2C+and+Dragomir+Radev%2E+2021%2E+Summeval%3A+Re%2Devaluating+summarization+evaluation%2E+Transactions+of+the+Association+for)
16. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-polygraph: Uncertainty estimation for language models. In Empirical Methods in N… [find](https://scholar.google.com/scholar?q=Ekaterina+Fadeeva%2C+Roman+Vashurin%2C+Akim+Tsvigun%2C+Artem+Vazhentsev%2C+Sergey+Petrakov%2C+Kirill+Fedyanin%2C+Daniil+Vasilev%2C+Elizaveta+Goncharova%2C+Alexander+Panchenko%2C+Maxim+Panov%2C+Timothy+Baldwin%2C+and+Artem+)
17. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. [find](https://scholar.google.com/scholar?q=Sebastian+Farquhar%2C+Jannik+Kossen%2C+Lorenz+Kuhn%2C+and+Yarin+Gal%2E+2024%2E+Detecting+hallucinations+in+large+language+models+using+semantic+entropy%2E+Nature%2C+630%288017%29%3A625%E2%80%93630%2E)
18. Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P. Dickerson. 2025. Style outweighs substance: Failure modes of LLM judges in alignment benchmarking. Preprint, arXiv:2409.15268. — [arXiv:2409.15268](https://arxiv.org/abs/2409.15268) [find](https://scholar.google.com/scholar?q=Benjamin+Feuer%2C+Micah+Goldblum%2C+Teresa+Datta%2C+Sanjana+Nambiar%2C+Raz+Besaleli%2C+Samuel+Dooley%2C+Max+Cembalest%2C+and+John+P%2E+Dickerson%2E+2025%2E+Style+outweighs+substance%3A+Failure+modes+of+LLM+judges+in+alignm)
19. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Transactions of the Association for Computational Linguistics, 8:539–555.
20. Nuno M Guerreiro, Duarte M Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André FT Martins. 2023. [Hallucinations in large multilingual translation models](https://md.research.elamur.ai/free/en/papers/guerreiro-2023-hallucinations-large-multilingual-translation-models/). Transactions of the Association for Computational Linguistics, 11:1500– 1517.
21. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). ACM Trans. Inf. Syst., 43(2).
22. Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. Calibrating long-form generations from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024. [find](https://scholar.google.com/scholar?q=Yukun+Huang%2C+Yixin+Liu%2C+Raghuveer+Thirukovalluru%2C+Arman+Cohan%2C+and+Bhuwan+Dhingra%2E+2024%2E+Calibrating+long%2Dform+generations+from+large+language+models%2E+In+Findings+of+the+Association+for+Computational+)
23. Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger, and Sepp Hochreiter. 2025. Addressing pitfalls in the evaluation of uncertainty estimation methods for natural language generation. ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI. [find](https://scholar.google.com/scholar?q=Mykyta+Ielanskyi%2C+Kajetan+Schweighofer%2C+Lukas+Aichberger%2C+and+Sepp+Hochreiter%2E+2025%2E+Addressing+pitfalls+in+the+evaluation+of+uncertainty+estimation+methods+for+natural+language+generation%2E+ICLR+Works)
24. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
25. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Association for Computational Linguistics (Volume 1: Long Papers).
26. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El%2DShowk%2C+Andy+Jones%2C+)
27. Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. 2024. Large language models must be taught to know what they don’t know. In Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Sanyam+Kapoor%2C+Nate+Gruver%2C+Manley+Roberts%2C+Katherine+Collins%2C+Arka+Pal%2C+Umang+Bhatt%2C+Adrian+Weller%2C+Samuel+Dooley%2C+Micah+Goldblum%2C+and+Andrew+Gordon+Wilson%2E+2024%2E+Large+language+models+must+be+taught)
28. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lorenz+Kuhn%2C+Yarin+Gal%2C+and+Sebastian+Farquhar%2E+2023%2E+Semantic+uncertainty%3A+Linguistic+invariances+for+uncertainty+estimation+in+natural+language+generation%2E+International+Conference+on+Learning+Repre)
29. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. [Latent retrieval for weakly supervised open domain question answering](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/). In Association for Computational Linguistics.
30. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
31. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Transations on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Zhen+Lin%2C+Shubhendu+Trivedi%2C+and+Jimeng+Sun%2E+2024%2E+Generating+with+confidence%3A+Uncertainty+quantification+for+black%2Dbox+large+language+models%2E+Transations+on+Machine+Learning+Research%2E)
32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
33. Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650. [find](https://scholar.google.com/scholar?q=Andrey+Malinin+and+Mark+Gales%2E+2020%2E+Uncertainty+estimation+in+autoregressive+structured+prediction%2E+arXiv+preprint+arXiv%3A2002%2E07650%2E)
34. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only. Advances in Neural Information Processi… [find](https://scholar.google.com/scholar?q=Guilherme+Penedo%2C+Quentin+Malartic%2C+Daniel+Hesslow%2C+Ruxandra+Cojocaru%2C+Hamza+Alobeidli%2C+Alessandro+Cappelli%2C+Baptiste+Pannier%2C+Ebtesam+Almazrouei%2C+and+Julien+Launay%2E+2023%2E+The+refinedweb+dataset+for+f)
35. Xin Qiu and Risto Miikkulainen. 2024. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Xin+Qiu+and+Risto+Miikkulainen%2E+2024%2E+Semantic+density%3A+Uncertainty+quantification+for+large+language+models+through+confidence+measurement+in+semantic+space%2E+In+Neural+Information+Processing+Systems%2E)
36. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Empirical Methods in Natural Language Processing.
38. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
39. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. [find](https://scholar.google.com/scholar?q=Ehud+Reiter+and+Anja+Belz%2E+2009%2E+An+investigation+into+the+validity+of+some+metrics+for+automatically+evaluating+natural+language+generation+systems%2E+Computational+Linguistics%2C+35%284%29%3A529%E2%80%93558%2E)
40. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Kirill Grishchenkov, Sergey Petrakov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. [Benchmarking uncertainty quanti…](https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/)
41. Leandro Von Werra, Lewis Tunstall, Abhishek Thakur, Sasha Luccioni, Tristan Thrush, Aleksandra Piktus, Felix Marty, Nazneen Rajani, Victor Mustar, and Helen Ngo. 2022. Evaluate & evaluation on the hub: Better best practices for data and model measurements. In Empirical Methods in Natural Language Pr… [find](https://scholar.google.com/scholar?q=Leandro+Von+Werra%2C+Lewis+Tunstall%2C+Abhishek+Thakur%2C+Sasha+Luccioni%2C+Tristan+Thrush%2C+Aleksandra+Piktus%2C+Felix+Marty%2C+Nazneen+Rajani%2C+Victor+Mustar%2C+and+Helen+Ngo%2E+2022%2E+Evaluate+%26+evaluation+on+the+hub)
42. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Association for Computational Linguistics (Volume 1: Long Papers). [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Zefan+Cai%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Lingpeng+Kong%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+2024%2E+Large+language+models+are+not+fair+evaluators%2E+In+Association+f)
43. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. [Measuring short-form factuality in large language models](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/). Preprint, arXiv:2411.04368.
44. Yijun Xiao and William Yang Wang. 2021. [On hallucination and predictive uncertainty in conditional language generation](https://md.research.elamur.ai/free/en/papers/xiao-2021-hallucination-predictive-uncertainty-conditional-language/). In European Chapter of the Association for Computational Linguistics.
45. Ruihan Yang, Caiqi Zhang, Zhisong Zhang, Xinting Huang, Sen Yang, Nigel Collier, Dong Yu, and Deqing Yang. 2024. [Logu: Long-form generation with uncertainty expressions](https://md.research.elamur.ai/free/en/papers/yang-2025-logu-long-form-generation-uncertainty/). arXiv preprint arXiv:2410.14309.
46. Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [AlignScore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Association for Computational Linguistics (Volume 1: Long Papers).
47. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2E)
48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Neural Information Processing Systems.

*Updated 31 October 2026*