---
title: "Unconditional Truthfulness — paper by Vazhentsev et al. (2025)"
description: "The authors argue that what matters is a token's unconditional chance of being correct. 61 references, cited by 8 papers here."
authors: ["Artem Vazhentsev", "Ekaterina Fadeeva", "Rui Xing", "Gleb Kuzmin", "Ivan Lazichny", "Alexander Panchenko", "Preslav Nakov", "Timothy Baldwin", "Maxim Panov", "Artem Shelmanov"]
year: 2025
venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2025.emnlp-main.1807/"
doi: "10.18653/v1/2025.emnlp-main.1807"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vazhentsev-2025-unconditional-truthfulness-learning-unconditional-uncertainty/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vazhentsev-2025-unconditional-truthfulness-learning-unconditional-uncertainty/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models

# Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models

**Authors:** [Artem Vazhentsev](https://md.research.elamur.ai/free/en/authors/artem-vazhentsev/), [Ekaterina Fadeeva](https://md.research.elamur.ai/free/en/authors/ekaterina-fadeeva/), [Rui Xing](https://md.research.elamur.ai/free/en/authors/rui-xing/), [Gleb Kuzmin](https://md.research.elamur.ai/free/en/authors/gleb-kuzmin/), [Ivan Lazichny](https://md.research.elamur.ai/free/en/authors/ivan-lazichny/), [Alexander Panchenko](https://md.research.elamur.ai/free/en/authors/alexander-panchenko/), [Preslav Nakov](https://md.research.elamur.ai/free/en/authors/preslav-nakov/), [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/), [Maxim Panov](https://md.research.elamur.ai/free/en/authors/maxim-panov/) and [Artem Shelmanov](https://md.research.elamur.ai/free/en/authors/artem-shelmanov/) · **Year:** 2025 · **Venue:** [Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2025](https://md.research.elamur.ai/free/en/venues/emnlp/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1807/) · [PDF](https://aclanthology.org/2025.emnlp-main.1807.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1807](https://doi.org/10.18653/v1/2025.emnlp-main.1807) · [OpenAlex](https://openalex.org/W4416035128)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [8](https://md.research.elamur.ai/free/en/papers/vazhentsev-2025-unconditional-truthfulness-learning-unconditional-uncertainty/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs score each new token assuming everything generated so far is correct. If an early token is wrong, the tokens that follow can still look highly confident, so uncertainty estimates built on raw probabilities understate the risk. The authors argue that what matters is a token's unconditional chance of being correct. Instead of hand-designing how doubt should carry forward, they learn this dependency from data. Their method, Trainable Attention-based Dependency (TAD), is a lightweight regression model with three inputs: the LLM's attention maps, the current step's probability, and uncertainty values computed recurrently for earlier tokens. Because those recurrent inputs come from the model itself, training happens in two stages: an intermediate first-stage model supplies the scores used in the second. The authors test TAD on selective generation with three open 7–9B LLMs and roughly ten QA and generation datasets. It generally outperforms both unsupervised methods such as SAR and supervised detectors, especially on long outputs, and it generalizes across domains. This allows cheap hallucination filtering without external knowledge bases or a second, larger LLM.

## Key points
- TAD's inputs are attention weights, current-step probabilities and recurrently propagated uncertainty from previous tokens. The ablations indicate that both the attention and recurrent features are needed, and that two-stage training prevents overfitting.
- The paper is inconsistent on the dataset count: the abstract says ten datasets, while the contributions and conclusion say nine. All three LLMs are in the 7–9B range, including Llama-3.1 8B.
- On one H100, sampling-based methods (DegMat, Lexical Similarity, Semantic Entropy, SAR) add about 420–580% runtime over plain MSP inference, whereas TAD adds only about 5%.
- In Llama-3.1 8B, a few heads focus strongly on the preceding token, for example head 25 in layer 30. On TruthfulQA, this attention stays sharp for correct answers but blurs on hallucinated tokens, which gives TAD a useful signal.
- Code is available at github.com/mbzuai-nlp/llm-tad-uncertainty.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 102 works · 22,618 citations
- [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) — University, Zurich, Switzerland · 84 works · 9,181 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov and Artem Shelmanov. 2025. Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vazhentsev2025unconditional,
  title = {{Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models}},
  author = {Artem Vazhentsev, Ekaterina Fadeeva, Rui Xing, Gleb Kuzmin, Ivan Lazichny, Alexander Panchenko, Preslav Nakov, Timothy Baldwin, Maxim Panov and Artem Shelmanov},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1807},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1807}
}
```

- [← previous TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data](https://md.research.elamur.ai/free/en/papers/liu-2025-transmi-framework-create-strong-baselines/)
- [next → WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning](https://md.research.elamur.ai/free/en/papers/mundada-2025-wildscore-benchmarking-mllms-wild-symbolic/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/)

## References

61 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC Bioinform., 20(1):511:1–511:23. [find](https://scholar.google.com/scholar?q=Asma+Ben+Abacha+and+Dina+Demner%2DFushman%2E+2019%2E+A+question%2Dentailment+approach+to+question+answering%2E+BMC+Bioinform%2E%2C+20%281%29%3A511%3A1%E2%80%93511%3A23%2E)
2. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. [Findings of the 2019 Conference on Machine Translation (…](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
3. Julia Belikova, Evegeniy Beliakin, and Vasily Konovalov. 2024. JellyBell at TextGraphs-17 shared task: Fusing large language models with external knowledge for enhanced question answering. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 154–160, Bangkok, T… [find](https://scholar.google.com/scholar?q=Julia+Belikova%2C+Evegeniy+Beliakin%2C+and+Vasily+Konovalov%2E+2024%2E+JellyBell+at+TextGraphs%2D17+shared+task%3A+Fusing+large+language+models+with+external+knowledge+for+enhanced+question+answering%2E+In+Proceedi)
4. Sky CH-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie. 2024. Do androids know they’re only dreaming of electric sheep? In Findings of the Association for Computational Linguistics: ACL 2024, pages 4401–4420, Bangkok, Thailand. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sky+CH%2DWang%2C+Benjamin+Van+Durme%2C+Jason+Eisner%2C+and+Chris+Kedzie%2E+2024%2E+Do+androids+know+they%E2%80%99re+only+dreaming+of+electric+sheep%3F+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2024%2C)
5. Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye. 2024. [INSIDE: LLMs’ internal states retain the power of hallucination detection](https://md.research.elamur.ai/free/en/papers/chen-2024-inside-llms-internal-states-retain/). In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
6. Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. 2023. Hallucination detection: Robustly discerning reliable answers in large language models. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Managemen… [find](https://scholar.google.com/scholar?q=Yuyan+Chen%2C+Qiang+Fu%2C+Yichen+Yuan%2C+Zhihao+Wen%2C+Ge+Fan%2C+Dayiheng+Liu%2C+Dongmei+Zhang%2C+Zhixu+Li%2C+and+Yanghua+Xiao%2E+2023%2E+Hallucination+detection%3A+Robustly+discerning+reliable+answers+in+large+language+mo)
7. Julius Cheng and Andreas Vlachos. 2024. Measuring uncertainty in neural machine translation with similarity-sensitive entropy. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2115–2128, St. Julian’s, Malta.… [find](https://scholar.google.com/scholar?q=Julius+Cheng+and+Andreas+Vlachos%2E+2024%2E+Measuring+uncertainty+in+neural+machine+translation+with+similarity%2Dsensitive+entropy%2E+In+Proceedings+of+the+18th+Conference+of+the+European+Chapter+of+the+Asso)
8. Amos Azaria and Tom Mitchell. 2023. [The internal state of an LLM knows when it’s lying](https://md.research.elamur.ai/free/en/papers/azaria-2023-internal-state-llm-knows-when/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 967–976, Singapore. Association for Computational Linguistics.
9. Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James R. Glass. 2024. [Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps](https://md.research.elamur.ai/free/en/papers/chuang-2024-lookback-lens-detecting-mitigating-contextual/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Proces…
10. Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. 2023. Uncertainty in natural language generation: From theory to applications. arXiv preprint arXiv:2307.15703. — [arXiv:2307.15703](https://arxiv.org/abs/2307.15703) [find](https://scholar.google.com/scholar?q=Joris+Baan%2C+Nico+Daheim%2C+Evgenia+Ilia%2C+Dennis+Ulmer%2C+Haau%2DSing+Li%2C+Raquel+Fern%C3%A1ndez%2C+Barbara+Plank%2C+Rico+Sennrich%2C+Chrysoula+Zerva%2C+and+Wilker+Aziz%2E+2023%2E+Uncertainty+in+natural+language+generation%3A+F)
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
12. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Com… [find](https://scholar.google.com/scholar?q=Jinhao+Duan%2C+Hao+Cheng%2C+Shiqi+Wang%2C+Alex+Zavalny%2C+Chenan+Wang%2C+Renjing+Xu%2C+Bhavya+Kailkhura%2C+and+Kaidi+Xu%2E+2024%2E+Shifting+attention+to+relevance%3A+Towards+the+predictive+uncertainty+quantification+of+f)
13. Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. 2024. [Fact-checking the output of large language models via token-level uncertainty quantificat…](https://md.research.elamur.ai/free/en/papers/fadeeva-2024-fact-checking-output-large-language/)
14. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2023. LM-Polygraph: Uncertainty estimation for language models. In Proceedings of the 202… [find](https://scholar.google.com/scholar?q=Ekaterina+Fadeeva%2C+Roman+Vashurin%2C+Akim+Tsvigun%2C+Artem+Vazhentsev%2C+Sergey+Petrakov%2C+Kirill+Fedyanin%2C+Daniil+Vasilev%2C+Elizaveta+Goncharova%2C+Alexander+Panchenko%2C+Maxim+Panov%2C+Timothy+Baldwin%2C+and+Artem+)
15. Javier Ferrando and Elena Voita. 2024. [Information flow routes: Automatically interpreting language models at scale](https://md.research.elamur.ai/free/en/papers/ferrando-2024-information-flow-routes-automatically-interpreting/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17432–17445, Miami, Florida, USA. Association for Computational Linguistics.
16. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Transactions of the Association for Computational Linguistics, 8:539–555.
17. Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050–1059, New York, New York, USA. PMLR. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+Dropout+as+a+Bayesian+approximation%3A+Representing+model+uncertainty+in+deep+learning%2E+In+Proceedings+of+The+33rd+International+Conference+on+Machine+Learning%2C+vo)
18. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. [A survey of confidence estimation and calibration in large language models](https://md.research.elamur.ai/free/en/papers/geng-2023-survey-confidence-estimation-calibration-large/). In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
19. Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. [SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization](https://md.research.elamur.ai/free/en/papers/gliwa-2019-samsum-corpus-human-annotated-dialogue/). In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70–79, Hong Kong, China. Association for Computational Linguistics.
20. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/)
21. Jianfeng He, Linlin Yu, Shuo Lei, Chang-Tien Lu, and Feng Chen. 2024a. Uncertainty estimation on sequential labeling via uncertainty transmission. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2823–2835, Mexico City, Mexico. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Jianfeng+He%2C+Linlin+Yu%2C+Shuo+Lei%2C+Chang%2DTien+Lu%2C+and+Feng+Chen%2E+2024a%2E+Uncertainty+estimation+on+sequential+labeling+via+uncertainty+transmission%2E+In+Findings+of+the+Association+for+Computational+Ling)
22. Jianfeng He, Xuchao Zhang, Shuo Lei, Zhiqian Chen, Fanglan Chen, Abdulaziz Alhamadani, Bei Xiao, and ChangTien Lu. 2020. Towards more accurate uncertainty estimation in text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8362–… [find](https://scholar.google.com/scholar?q=Jianfeng+He%2C+Xuchao+Zhang%2C+Shuo+Lei%2C+Zhiqian+Chen%2C+Fanglan+Chen%2C+Abdulaziz+Alhamadani%2C+Bei+Xiao%2C+and+ChangTien+Lu%2E+2020%2E+Towards+more+accurate+uncertainty+estimation+in+text+classification%2E+In+Proceed)
23. Jinwen He, Yujia Gong, Zijin Lin, Cheng’an Wei, Yue Zhao, and Kai Chen. 2024b. LLM factoscope: Uncovering LLMs’ factual discernment through measuring inner states. In Findings of the Association for Computational Linguistics ACL 2024, pages 10218– 10230, Bangkok, Thailand and virtual meeting. Associ… [find](https://scholar.google.com/scholar?q=Jinwen+He%2C+Yujia+Gong%2C+Zijin+Lin%2C+Cheng%E2%80%99an+Wei%2C+Yue+Zhao%2C+and+Kai+Chen%2E+2024b%2E+LLM+factoscope%3A+Uncovering+LLMs%E2%80%99+factual+discernment+through+measuring+inner+states%2E+In+Findings+of+the+Association+for+C)
24. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
25. Mykyta Ielanskyi, Kajetan Schweighofer, Lukas Aichberger, and Sepp Hochreiter. 2025. Addressing pitfalls in the evaluation of uncertainty estimation methods for natural language generation. In ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable A… [find](https://scholar.google.com/scholar?q=Mykyta+Ielanskyi%2C+Kajetan+Schweighofer%2C+Lukas+Aichberger%2C+and+Sepp+Hochreiter%2E+2025%2E+Addressing+pitfalls+in+the+evaluation+of+uncertainty+estimation+methods+for+natural+language+generation%2E+In+ICLR+Wo)
26. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM computing surveys, 55(12):1–38.
27. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver…
28. Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. Captum: A unified and generic model interpretability library for PyTorch. arXiv preprint arXiv:2009.07… — [arXiv:2009.07896](https://arxiv.org/abs/2009.07896) [find](https://scholar.google.com/scholar?q=Narine+Kokhlikyan%2C+Vivek+Miglani%2C+Miguel+Martin%2C+Edward+Wang%2C+Bilal+Alsallakh%2C+Jonathan+Reynolds%2C+Alexander+Melnikov%2C+Natalia+Kliushkina%2C+Carlos+Araya%2C+Siqi+Yan%2C+and+Orion+Reblitz%2DRichardson%2E+2020%2E+Ca)
29. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. [Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation](https://md.research.elamur.ai/free/en/papers/kuhn-2023-semantic-uncertainty-linguistic-invariances-uncertainty/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
30. Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. 2023. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Salem+Lahlou%2C+Moksh+Jain%2C+Hadi+Nekoei%2C+Victor+I+Butoi%2C+Paul+Bertin%2C+Jarrid+Rector%2DBrooks%2C+Maksym+Korablyov%2C+and+Yoshua+Bengio%2E+2023%2E+DEUP%3A+Direct+epistemic+uncertainty+prediction%2E+Transactions+on+Mach)
31. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
32. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Zhen+Lin%2C+Shubhendu+Trivedi%2C+and+Jimeng+Sun%2E+2024%2E+Generating+with+confidence%3A+Uncertainty+quantification+for+black%2Dbox+large+language+models%2E+Transactions+on+Machine+Learning+Research%2E)
33. Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. 2022. Learning confidence for transformer-based neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2353–2364, Dublin, Ireland. Association for… [find](https://scholar.google.com/scholar?q=Yu+Lu%2C+Jiali+Zeng%2C+Jiajun+Zhang%2C+Shuangzhi+Wu%2C+and+Mu+Li%2E+2022%2E+Learning+confidence+for+transformer%2Dbased+neural+machine+translation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+C)
34. Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. [find](https://scholar.google.com/scholar?q=Andrey+Malinin+and+Mark+J%2E+F%2E+Gales%2E+2021%2E+Uncertainty+estimation+in+autoregressive+structured+prediction%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+2021%2C+Virtual+Event%2C+Austri)
35. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. [FActScore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
36. Viktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, and Alexander Panchenko. 2025. [Adaptive retrieval without self-knowledge? Bringing uncertainty back home](https://md.research.elamur.ai/free/en/papers/moskvoretskii-2025-adaptive-retrieval-without-self-knowledge/). In Proceedings of the 63rd Annual Mee…
37. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As…
38. Alexander Nikitin, Jannik Kossen, Yarin Gal, and Pekka Marttinen. 2024. Kernel language entropy: Fine-grained uncertainty quantification for llms from semantic similarities. Advances in Neural Information Processing Systems, 37:8901–8929. [find](https://scholar.google.com/scholar?q=Alexander+Nikitin%2C+Jannik+Kossen%2C+Yarin+Gal%2C+and+Pekka+Marttinen%2E+2024%2E+Kernel+language+entropy%3A+Fine%2Dgrained+uncertainty+quantification+for+llms+from+semantic+similarities%2E+Advances+in+Neural+Informa)
39. Yookoon Park and David Blei. 2024. Density uncertainty layers for reliable uncertainty estimation. In International Conference on Artificial Intelligence and Statistics, pages 163–171. PMLR. [find](https://scholar.google.com/scholar?q=Yookoon+Park+and+David+Blei%2E+2024%2E+Density+uncertainty+layers+for+reliable+uncertainty+estimation%2E+In+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+pages+163%E2%80%93171%2E+PMLR%2E)
40. Xin Qiu and Risto Miikkulainen. 2024. Semantic density: Uncertainty quantification for large language models through confidence measurement in semantic space. In Advances in Neural Information Processing Systems, volume 37, pages 134507–134533. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Xin+Qiu+and+Risto+Miikkulainen%2E+2024%2E+Semantic+density%3A+Uncertainty+quantification+for+large+language+models+through+confidence+measurement+in+semantic+space%2E+In+Advances+in+Neural+Information+Process)
41. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
42. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. [COMET: A neural framework for MT evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
43. Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023. [Out-of-distribution detection and selective generation for conditional language models](https://md.research.elamur.ai/free/en/papers/ren-2022-out-distribution-detection-selective-generation/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May…
44. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Ni… [find](https://scholar.google.com/scholar?q=Morgane+Rivi%C3%A8re%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+Cassidy+Hardin%2C+Surya+Bhupatiraju%2C+L%C3%A9onard+Hussenot%2C+Thomas+Mesnard%2C+Bobak+Shahriari%2C+Alexandre+Ram%C3%A9%2C+Johan+Ferret%2C+Peter+Liu%2C+Pouya+Tafti%2C+Abe+Fri)
45. Andrea Santilli, Adam Golinski, Michael Kirchhof, Federico Danieli, Arno Blaas, Miao Xiong, Luca Zappella, and Sinead Williamson. 2025. [Revisiting uncertainty quantification evaluation in language models: Spurious interactions with response length bias results](https://md.research.elamur.ai/free/en/papers/santilli-2025-revisiting-uncertainty-quantification-evaluation-language/). In Proceedings of the 63rd Annual Meet…
46. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
47. Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu. 2024. [Unsupervised real-time hallucination detection based on the internal states of large language models](https://md.research.elamur.ai/free/en/papers/su-2024-unsupervised-real-time-hallucination-detection/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 14379–14391, Bangkok, Thai…
48. Artem Vazhentsev, Lyudmila Rvanova, Ivan Lazichny, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov. 2025. [Token-level density-based uncertainty quantification methods for eliciting truthfulness of large language models](https://md.research.elamur.ai/free/en/papers/vazhentsev-2025-token-level-density-based-uncertainty/). In Proceedings of the 2025 Conference of the Nations of t…
49. Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022. [Uncertainty estimation and reduction of pre-trained models for text regression](https://md.research.elamur.ai/free/en/papers/wang-2022-uncertainty-estimation-reduction-pre-trained/). Transactions of the Association for Computational Linguistics, 10:680–696.
50. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+2017%2E+Axiomatic+attribution+for+deep+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+ICML+2017%2C+Sydney%2C+NSW%2C+Australia)
51. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
52. Igor Tufanov, Karen Hambardzumyan, Javier Ferrando, and Elena Voita. 2024. LM transparency tool: Interactive tool for analyzing transformer language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 51–60, Bang… [find](https://scholar.google.com/scholar?q=Igor+Tufanov%2C+Karen+Hambardzumyan%2C+Javier+Ferrando%2C+and+Elena+Voita%2E+2024%2E+LM+transparency+tool%3A+Interactive+tool+for+analyzing+transformer+language+models%2E+In+Proceedings+of+the+62nd+Annual+Meeting+o)
53. Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Raphael+Tang%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2021%2E+The+art+of+abstention%3A+Selective+prediction+and+error+regularization+for+natural+language+processing%2E+In+Proceedings+of+the+59th+Annual+Meeting+)
54. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2025. [Benchmarking uncertainty quantificat…](https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/)
55. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, et al. 2024. Qwen2.5 technical report. arXiv preprint a… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
56. Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, and Leonid Zhukov. 2022. Uncertainty estimation of transformer predictions for misclassification detection. In Proceeding… [find](https://scholar.google.com/scholar?q=Artem+Vazhentsev%2C+Gleb+Kuzmin%2C+Artem+Shelmanov%2C+Akim+Tsvigun%2C+Evgenii+Tsymbalov%2C+Kirill+Fedyanin%2C+Maxim+Panov%2C+Alexander+Panchenko%2C+Gleb+Gusev%2C+Mikhail+Burtsev%2C+Manvel+Avetisian%2C+and+Leonid+Zhukov%2E+20)
57. Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burt-Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. [AlignScore: Evaluating factual consistency with a unified alignment function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/). In Proceedings of the 61st Annual Meeting of the Association for Computa…
58. Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. 2024. [LUQ: Long-text uncertainty quantification for LLMs](https://md.research.elamur.ai/free/en/papers/zhang-2024-luq-long-text-uncertainty-quantification/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5244–5262, Miami, Florida, USA. Association for Computational Linguistics.
59. Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023. Enhancing uncertainty-based hallucination detection with stronger focus. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 915–932,… [find](https://scholar.google.com/scholar?q=Tianhang+Zhang%2C+Lin+Qiu%2C+Qipeng+Guo%2C+Cheng+Deng%2C+Yue+Zhang%2C+Zheng+Zhang%2C+Chenghu+Zhou%2C+Xinbing+Wang%2C+and+Luoyi+Fu%2E+2023%2E+Enhancing+uncertainty%2Dbased+hallucination+detection+with+stronger+focus%2E+In+Pro)
60. Xuchao Zhang, Fanglan Chen, Chang-Tien Lu, and Naren Ramakrishnan. 2019. Mitigating uncertainty in document classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pape… [find](https://scholar.google.com/scholar?q=Xuchao+Zhang%2C+Fanglan+Chen%2C+Chang%2DTien+Lu%2C+and+Naren+Ramakrishnan%2E+2019%2E+Mitigating+uncertainty+in+document+classification%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+A)
61. Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. [Explainability for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/). ACM Transactions on Intelligent Systems and Technology, 15(2):1–38.

*Updated 2 November 2026*