---
title: "LM-Polygraph: Uncertainty Estimation for Language Models (2023)"
description: "The authors release LM-Polygraph, an open-source Python framework that gathers many UE… 42 references, cited by 55 papers here."
authors: ["Ekaterina Fadeeva", "Roman Vashurin", "Akim Tsvigun", "Artem Vazhentsev", "Sergey Petrakov", "Kirill Fedyanin", "Daniil Vasilev", "Elizaveta Goncharova", "Alexander Panchenko", "Maxim Panov", "Timothy Baldwin", "Artem Shelmanov"]
year: 2023
venue: "arXiv:2311.07383"
source: "https://arxiv.org/abs/2311.07383"
doi: "10.48550/arxiv.2311.07383"
arxiv: "2311.07383"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/fadeeva-2023-lm-polygraph-uncertainty-estimation-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/fadeeva-2023-lm-polygraph-uncertainty-estimation-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / LM-Polygraph: Uncertainty Estimation for Language Models

# LM-Polygraph: Uncertainty Estimation for Language Models

**Authors:** [Ekaterina Fadeeva](https://md.research.elamur.ai/free/en/authors/ekaterina-fadeeva/), [Roman Vashurin](https://md.research.elamur.ai/free/en/authors/roman-vashurin/), [Akim Tsvigun](https://md.research.elamur.ai/free/en/authors/akim-tsvigun/), [Artem Vazhentsev](https://md.research.elamur.ai/free/en/authors/artem-vazhentsev/), [Sergey Petrakov](https://md.research.elamur.ai/free/en/authors/sergey-petrakov/), Kirill Fedyanin, [Daniil Vasilev](https://md.research.elamur.ai/free/en/authors/daniil-vasilev/), Elizaveta Goncharova, [Alexander Panchenko](https://md.research.elamur.ai/free/en/authors/alexander-panchenko/), [Maxim Panov](https://md.research.elamur.ai/free/en/authors/maxim-panov/), [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/) and [Artem Shelmanov](https://md.research.elamur.ai/free/en/authors/artem-shelmanov/) · **Year:** 2023 · **Venue:** arXiv:2311.07383 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.07383) · [PDF](https://arxiv.org/pdf/2311.07383) · [DOI 10.48550/arxiv.2311.07383](https://doi.org/10.48550/arxiv.2311.07383) · [arXiv 2311.07383](https://arxiv.org/abs/2311.07383) · [OpenAlex](https://openalex.org/W4388685627)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [55](https://md.research.elamur.ai/free/en/papers/fadeeva-2023-lm-polygraph-uncertainty-estimation-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models often produce fluent but false statements, and users have no built-in signal for when to doubt an answer. Uncertainty estimation (UE) could provide that signal. However, UE for text generation is harder than for classification: token-level scores must be aggregated, the output space is enormous, and differently worded answers may mean the same thing. Prior work on the topic was also scattered and hard to compare.

The authors release LM-Polygraph, an open-source Python framework that gathers many UE techniques for generative LMs behind a single interface. It works with Huggingface models and, for methods that need only generated text, with API-based services. Alongside the library they provide an extendable benchmark for selective generation and a web demo in which chat replies carry confidence scores.

Experiments on summarization, QA and translation datasets show that no current method comes close to ideal performance. Results are especially weak on XSum and WMT14. The resource matters because it lets researchers compare UE methods on equal terms and helps practitioners flag unreliable LLM outputs.

## Key points
- About 19 UE methods are implemented. White-box families are information-based, meaning-diversity, ensembling, density-based and reflexive. Examples include Maximum sequence probability, Semantic entropy, Mahalanobis distance, RMD, HUQ and p(True). Black-box methods include NumSets, EigV, Deg, Ecc and LexSim.
- The framework is tested with BLOOMz, Dolly v2, Alpaca, LLaMA-2 and Flan-T5. It supports both seq2seq and decoder-only models, and black-box methods can be used with ChatGPT and GPT-4 through APIs.
- The benchmark covers AESLC, XSUM, CoQA, bAbiQA, WMT14 De-En and WMT14 Fr-En. Methods are scored with PRR, using ROUGE-L and BERTScore as quality metrics, on the Vicuna and LLaMA-2 models.
- Among black-box methods, Eccentricity had the best average result for Vicuna. For LLaMA-2, no black-box method was clearly ahead.
- Methods differ in compute and memory cost and in whether they need training data. Density-based methods are cheap to run but require access to training data, and they capture only epistemic uncertainty.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 104 works · 22,674 citations
- [National Research University Higher School of Economics](https://md.research.elamur.ai/free/en/universities/national-research-university-higher-school-of-economics/) — University, Moscow, Russia · 25 works · 39,640 citations
- [National University of Sciences and Technology](https://md.research.elamur.ai/free/en/universities/national-university-of-sciences-and-technology/) — University, Islamabad, Pakistan · 4 works · 55 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works

## Cite

Cite as: Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin and Artem Shelmanov. 2023. LM-Polygraph: Uncertainty Estimation for Language Models. arXiv:2311.07383.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{fadeeva2023lm,
  title = {{LM-Polygraph: Uncertainty Estimation for Language Models}},
  author = {Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin and Artem Shelmanov},
  year = {2023},
  doi = {10.48550/arxiv.2311.07383},
  eprint = {2311.07383},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.07383}
}
```

- [← previous DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents](https://md.research.elamur.ai/free/en/papers/nair-2023-dera-enhancing-large-language-model/)
- [next → Zoology: Measuring and Improving Recall in Efficient Language Models](https://md.research.elamur.ai/free/en/papers/arora-2023-zoology-measuring-improving-recall-efficient/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

42 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. [Findings of the 2014 workshop on statistical machine translation](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/). In Proceedings of the…
2. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang… [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+Albert+Webson%2C+Shixiang+Shane+Gu%2C+Zhuyun+Dai%2C+Mirac+Suzgun%2C+Xiny)
3. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct)
4. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander H. Miller, Arthur Szlam, and Jason Weston. 2016. [Evaluating prerequisite qualities for learning end-to-end dialog systems](https://md.research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/). In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, Ma…
5. Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022. [On the origin of hallucinations in conversational models: Is it the datasets or the models?](https://md.research.elamur.ai/free/en/papers/dziri-2022-origin-hallucinations-conversational-models-is/) In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
6. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Transactions of the Association for Computational Linguistics, 8:539–555.
7. Yarin Gal. 2016. Uncertainty in Deep Learning. Ph.D. thesis, University of Cambridge. [find](https://scholar.google.com/scholar?q=Yarin+Gal%2E+2016%2E+Uncertainty+in+Deep+Learning%2E+Ph%2ED%2E+thesis%2C+University+of+Cambridge%2E)
8. Jianfeng He, Xuchao Zhang, Shuo Lei, Zhiqian Chen, Fanglan Chen, Abdulaziz Alhamadani, Bei Xiao, and Chang-Tien Lu. 2020. Towards more accurate uncertainty estimation in text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online… [find](https://scholar.google.com/scholar?q=Jianfeng+He%2C+Xuchao+Zhang%2C+Shuo+Lei%2C+Zhiqian+Chen%2C+Fanglan+Chen%2C+Abdulaziz+Alhamadani%2C+Bei+Xiao%2C+and+Chang%2DTien+Lu%2E+2020%2E+Towards+more+accurate+uncertainty+estimation+in+text+classification%2E+In+Procee)
9. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [Deberta: decoding-enhanced bert with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
10. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El+Showk%2C+Andy+Jones%2C+)
11. Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. 2022. Nonparametric uncertainty quantification for single deterministic neural network. In Advances in Neural Information Processing Sy… [find](https://scholar.google.com/scholar?q=Nikita+Kotelevskii%2C+Aleksandr+Artemenkov%2C+Kirill+Fedyanin%2C+Fedor+Noskov%2C+Alexander+Fishkov%2C+Artem+Shelmanov%2C+Artem+Vazhentsev%2C+Aleksandr+Petiushko%2C+and+Maxim+Panov%2E+2022%2E+Nonparametric+uncertainty+qua)
12. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. [Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation](https://md.research.elamur.ai/free/en/papers/kuhn-2023-semantic-uncertainty-linguistic-invariances-uncertainty/). In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
13. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8… [find](https://scholar.google.com/scholar?q=Kimin+Lee%2C+Kibok+Lee%2C+Honglak+Lee%2C+and+Jinwoo+Shin%2E+2018%2E+A+simple+unified+framework+for+detecting+out%2Dof%2Ddistribution+samples+and+adversarial+attacks%2E+In+Advances+in+Neural+Information+Processing+Sys)
14. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. CoRR, abs/2305.19187. [find](https://scholar.google.com/scholar?q=Zhen+Lin%2C+Shubhendu+Trivedi%2C+and+Jimeng+Sun%2E+2023%2E+Generating+with+confidence%3A+Uncertainty+quantification+for+black%2Dbox+large+language+models%2E+CoRR%2C+abs%2F2305%2E19187%2E)
15. Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. 2022. [ParaDetox: Detoxification with parallel data](https://md.research.elamur.ai/free/en/papers/logacheva-2022-paradetox-detoxification-parallel-data/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo…
16. Andrey Malinin and Mark J. F. Gales. 2021. Uncertainty estimation in autoregressive structured prediction. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. [find](https://scholar.google.com/scholar?q=Andrey+Malinin+and+Mark+J%2E+F%2E+Gales%2E+2021%2E+Uncertainty+estimation+in+autoregressive+structured+prediction%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+2021%2C+Virtual+Event%2C+Austri)
17. Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales. 2017. Incorporating uncertainty into deep learning for spoken language assessment. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–50, Vancouver, Canada. Association… [find](https://scholar.google.com/scholar?q=Andrey+Malinin%2C+Anton+Ragni%2C+Kate+Knill%2C+and+Mark+Gales%2E+2017%2E+Incorporating+uncertainty+into+deep+learning+for+spoken+language+assessment%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association)
18. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. [Don’t give me the details, just the summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - Novemb…
19. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21:140:1–140:67.
21. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. [CoQA: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/). Transactions of the Association for Computational Linguistics, 7:249–266.
22. Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter J Liu. 2023. [Out-of-distribution detection and selective generation for conditional language models](https://md.research.elamur.ai/free/en/papers/ren-2022-out-distribution-detection-selective-generation/). In The Eleventh International Conference on Learning Representations.
23. Peter J Rousseeuw. 1984. Least median of squares regression. Journal of the American statistical association, 79(388):871–880. [find](https://scholar.google.com/scholar?q=Peter+J+Rousseeuw%2E+1984%2E+Least+median+of+squares+regression%2E+Journal+of+the+American+statistical+association%2C+79%28388%29%3A871%E2%80%93880%2E)
24. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighof… [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+Jonathan+Tow%2C+Alexander+M%2E+Rush%2C+Stel)
25. Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, and Maxim Panov. 2021. How certain is your Transformer? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1833–1840, Online. Asso… [find](https://scholar.google.com/scholar?q=Artem+Shelmanov%2C+Evgenii+Tsymbalov%2C+Dmitri+Puzyrev%2C+Kirill+Fedyanin%2C+Alexander+Panchenko%2C+and+Maxim+Panov%2E+2021%2E+How+certain+is+your+Transformer%3F+In+Proceedings+of+the+16th+Conference+of+the+European+)
26. Junya Takayama and Yuki Arase. 2019. Relevant and informative response generation using pointwise mutual information. In Proceedings of the First Workshop on NLP for Conversational AI, pages 133–138, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Junya+Takayama+and+Yuki+Arase%2E+2019%2E+Relevant+and+informative+response+generation+using+pointwise+mutual+information%2E+In+Proceedings+of+the+First+Workshop+on+NLP+for+Conversational+AI%2C+pages+133%E2%80%93138%2C+)
27. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
28. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
29. Liam van der Poel, Ryan Cotterell, and Clara Meister. 2022. Mutual information alleviates hallucinations in abstractive summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5956–5965, Abu Dhabi, United Arab Emirates. Association for Computat… [find](https://scholar.google.com/scholar?q=Liam+van+der+Poel%2C+Ryan+Cotterell%2C+and+Clara+Meister%2E+2022%2E+Mutual+information+alleviates+hallucinations+in+abstractive+summarization%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Nat)
30. Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, and Leonid Zhukov. 2022. Uncertainty estimation of transformer predictions for misclassification detection. In Proceeding… [find](https://scholar.google.com/scholar?q=Artem+Vazhentsev%2C+Gleb+Kuzmin%2C+Artem+Shelmanov%2C+Akim+Tsvigun%2C+Evgenii+Tsymbalov%2C+Kirill+Fedyanin%2C+Maxim+Panov%2C+Alexander+Panchenko%2C+Gleb+Gusev%2C+Mikhail+Burtsev%2C+Manvel+Avetisian%2C+and+Leonid+Zhukov%2E+20)
31. Artem Vazhentsev, Gleb Kuzmin, Akim Tsvigun, Alexander Panchenko, Maxim Panov, Mikhail Burtsev, and Artem Shelmanov. 2023a. Hybrid uncertainty quantification for selective text classification in ambiguous tasks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Artem+Vazhentsev%2C+Gleb+Kuzmin%2C+Akim+Tsvigun%2C+Alexander+Panchenko%2C+Maxim+Panov%2C+Mikhail+Burtsev%2C+and+Artem+Shelmanov%2E+2023a%2E+Hybrid+uncertainty+quantification+for+selective+text+classification+in+ambig)
32. Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexander Panchenko, and Artem Shelmanov. 2023b. Efficient out-of-domain detection for sequence to sequence models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1430– 1454,… [find](https://scholar.google.com/scholar?q=Artem+Vazhentsev%2C+Akim+Tsvigun%2C+Roman+Vashurin%2C+Sergey+Petrakov%2C+Daniil+Vasilev%2C+Maxim+Panov%2C+Alexander+Panchenko%2C+and+Artem+Shelmanov%2E+2023b%2E+Efficient+out%2Dof%2Ddomain+detection+for+sequence+to+sequenc)
33. Yuxia Wang, Daniel Beck, Timothy Baldwin, and Karin Verspoor. 2022. [Uncertainty estimation and reduction of pre-trained models for text regression](https://md.research.elamur.ai/free/en/papers/wang-2022-uncertainty-estimation-reduction-pre-trained/). Transactions of the Association for Computational Linguistics, 10:680–696.
34. Yijun Xiao and William Yang Wang. 2021. [On hallucination and predictive uncertainty in conditional language generation](https://md.research.elamur.ai/free/en/papers/xiao-2021-hallucination-predictive-uncertainty-conditional-language/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2734–2744, Online. Association for Computational…
35. Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu… [find](https://scholar.google.com/scholar?q=Ji+Xin%2C+Raphael+Tang%2C+Yaoliang+Yu%2C+and+Jimmy+Lin%2E+2021%2E+The+art+of+abstention%3A+Selective+prediction+and+error+regularization+for+natural+language+processing%2E+In+Proceedings+of+the+59th+Annual+Meeting+)
36. Weijia Xu, Sweta Agrawal, Eleftheria Briakou, Marianna J. Martindale, and Marine Carpuat. 2023. [Understanding and detecting hallucinations in neural machine translation via model introspection](https://md.research.elamur.ai/free/en/papers/xu-2023-understanding-detecting-hallucinations-neural-machine/). Transactions of the Association for Computational Linguistics, 11:546–564.
37. Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M. Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. 2023. [BLOOM+1: adding language sup…](https://md.research.elamur.ai/free/en/papers/yong-2023-bloom-1-adding-language-support/)
38. KiYoon Yoo, Jangho Kim, Jiho Jang, and Nojun Kwak. 2022. Detection of adversarial examples in text classification: Benchmark and baseline via robust density estimation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3656–3672, Dublin, Ireland. Association for Computati… [find](https://scholar.google.com/scholar?q=KiYoon+Yoo%2C+Jangho+Kim%2C+Jiho+Jang%2C+and+Nojun+Kwak%2E+2022%2E+Detection+of+adversarial+examples+in+text+classification%3A+Benchmark+and+baseline+via+robust+density+estimation%2E+In+Findings+of+the+Association+)
39. Rui Zhang and Joel R. Tetreault. 2019. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 446–… [find](https://scholar.google.com/scholar?q=Rui+Zhang+and+Joel+R%2E+Tetreault%2E+2019%2E+This+email+could+save+your+life%3A+Introducing+the+task+of+email+subject+line+generation%2E+In+Proceedings+of+the+57th+Conference+of+the+Association+for+Computationa)
40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
41. Xuchao Zhang, Fanglan Chen, Chang-Tien Lu, and Naren Ramakrishnan. 2019. Mitigating uncertainty in document classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pape… [find](https://scholar.google.com/scholar?q=Xuchao+Zhang%2C+Fanglan+Chen%2C+Chang%2DTien+Lu%2C+and+Naren+Ramakrishnan%2E+2019%2E+Mitigating+uncertainty+in+document+classification%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+A)
42. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). CoRR, abs/2306.05685.

*Updated 3 November 2026*