---
title: "Benchmarking Uncertainty Quantification Methods for Large"
description: "The authors build a unified benchmark on top of the LM-Polygraph framework. 2 references, cited by 48 papers here."
authors: ["Roman Vashurin", "Ekaterina Fadeeva", "Artem Vazhentsev", "Lyudmila Rvanova", "Daniil Vasilev", "Akim Tsvigun", "Sergey Petrakov", "Rui Xing", "Abdelrahman Sadallah", "Kirill Grishchenkov", "Alexander Panchenko", "Timothy Baldwin", "Preslav Nakov", "Maxim Panov", "Artem Shelmanov"]
year: 2025
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2025.tacl-1.11/"
doi: "10.1162/tacl_a_00737"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph

# Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph

**Authors:** Roman Vashurin, [Ekaterina Fadeeva](https://md.research.elamur.ai/free/en/authors/ekaterina-fadeeva/), Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, [Sergey Petrakov](https://md.research.elamur.ai/free/en/authors/sergey-petrakov/), Rui Xing, [Abdelrahman Sadallah](https://md.research.elamur.ai/free/en/authors/abdelrahman-sadallah/), Kirill Grishchenkov, [Alexander Panchenko](https://md.research.elamur.ai/free/en/authors/alexander-panchenko/), [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/), Maxim Panov and [Artem Shelmanov](https://md.research.elamur.ai/free/en/authors/artem-shelmanov/) · **Year:** 2025 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2025](https://md.research.elamur.ai/free/en/venues/tacl/2025/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.tacl-1.11/) · [PDF](https://aclanthology.org/2025.tacl-1.11.pdf) · [DOI 10.1162/tacl\_a\_00737](https://doi.org/10.1162/tacl_a_00737) · [OpenAlex](https://openalex.org/W4409218928)

**Citations:** 18 citations (OpenAlex, 2026-10-14) · 48 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Uncertainty quantification (UQ) can help flag hallucinated or low-quality outputs of large language models. Earlier work on UQ for LLMs used incompatible methods and evaluation setups, so findings were hard to compare.

The authors build a unified benchmark on top of the LM-Polygraph framework. It implements many UQ techniques under one evaluation protocol, grouped into information-based, sample-diversity, density-based and reflexive families, with both white-box and black-box variants. The benchmark covers three task types:
- selective question answering;
- selective generation (machine translation and text summarization);
- claim-level fact-checking, supported by a new automatic pipeline for English, Chinese, Arabic and Russian.

The authors also add a calibration-oriented metric. It measures how well confidence normalization methods turn raw uncertainty into bounded, interpretable scores, and they propose a normalization baseline to test against it.

Experiments span eleven datasets. No single method wins everywhere, because task type and output length strongly affect which family performs best. The open, MIT-licensed benchmark makes future UQ research more comparable and easier to begin.

## Key points
- Selective QA/generation results (PRR with AlignScore and COMET):
  - Sample-diversity methods, especially SAR, DegMat and Semantic Entropy, achieve the best mean results.
  - The MSP baseline lags behind but remains reasonable.
- Output length matters:
  - For very short answers (under 7 symbols), information-based scores such as MSP, Perplexity, CCP and TokenSAR perform best.
  - For longer outputs, NLI-based black-box methods lead.
  - SAR is the most stable across all lengths.
- Fact-checking setup: biographies of 100 people were generated with Mistral 7B, Jais 13B, Vikhr 7B and Yi 6B. Extracted claims per setting ranged from 1,031 (Arabic) to 3,104 (Russian).
- Fact-checking results: CCP gives the best ROC-AUC/PR-AUC in every setting except Chinese, where it ties with P(True).
- Normalization: binned and isotonic PCC give clearly better calibration (lower MSE against AlignScore) than linear or quantile normalization. Monotonic normalizers keep the PRR of the raw scores.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 60 works · 21,199 citations
- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 36 works · 4,534 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations
- [National Research University Higher School of Economics](https://md.research.elamur.ai/free/en/universities/national-research-university-higher-school-of-economics/) — University, Moscow, Russia · 16 works · 39,548 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov and Artem Shelmanov. 2025. Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{vashurin2025benchmarking,
  title = {{Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph}},
  author = {Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov and Artem Shelmanov},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.1162/tacl\_a\_00737},
  url = {https://doi.org/10.1162/tacl_a_00737}
}
```

- [← previous xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://md.research.elamur.ai/free/en/papers/zhang-2025-xlam-family-large-action-models/)
- [next → A Comprehensive Survey on Long Context Language Modeling](https://md.research.elamur.ai/free/en/papers/liu-2025-comprehensive-survey-long-context-language/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/)

## References

2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. 2019. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Arsenii+Ashukha%2C+Alexander+Lyzhov%2C+Dmitry+Molchanov%2C+and+Dmitry+Vetrov%2E+2019%2E+Pitfalls+of+in%2Ddomain+uncertainty+estimation+and+ensembling+in+deep+learning%2E+In+International+Conference+on+Learning+Repr)
2. Loı̈c Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. [Findings of the 2019 Conference on Machine Translatio…](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)

*Updated 24 October 2026*