---
title: "Revisiting Epistemic Markers in Confidence Estimation: Can"
description: "They define marker confidence as the empirical accuracy of a model's answers that contain a given… 64 references, cited by 7 papers here."
authors: ["Jiayu Liu", "Qing Zong", "Weiqi Wang", "Yangqiu Song"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2025.acl-short.18/"
doi: "10.18653/v1/2025.acl-short.18"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2025-revisiting-epistemic-markers-confidence-estimation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2025-revisiting-epistemic-markers-confidence-estimation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/) / Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?

# Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?

**Authors:** [Jiayu Liu](https://md.research.elamur.ai/free/en/authors/jiayu-liu/), [Qing Zong](https://md.research.elamur.ai/free/en/authors/qing-zong/), [Weiqi Wang](https://md.research.elamur.ai/free/en/authors/weiqi-wang/) and [Yangqiu Song](https://md.research.elamur.ai/free/en/authors/yangqiu-song/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-short.18/) · [PDF](https://aclanthology.org/2025.acl-short.18.pdf) · [DOI 10.18653/v1/2025.acl-short.18](https://doi.org/10.18653/v1/2025.acl-short.18) · [OpenAlex](https://openalex.org/W4412889555)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/liu-2025-revisiting-epistemic-markers-confidence-estimation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether large language models use verbal epistemic markers, such as "fairly certain," in a way that consistently tracks how often they are actually right. Earlier work compared LLM markers with human interpretations. Here the authors test whether a model is consistent with its own usage. They define marker confidence as the empirical accuracy of a model's answers that contain a given marker. This is computed on a training split and then checked on test data from the same dataset and from other datasets. Seven LLMs are evaluated on seven QA benchmarks covering commonsense, math, medical, legal and general knowledge. Seven metrics, based on ECE, Pearson/Spearman correlation and coefficient of variation, are compared against a baseline that directly prompts for numerical confidence. Markers calibrate well within the same distribution but transfer poorly across datasets. Their confidence values cluster in a narrow range, and their relative ranking is unstable. Stronger models behave somewhat better. The authors conclude that verbal markers are not yet a dependable signal of LLM uncertainty, which matters for high-stakes use.

## Key points
- **Models:** Llama-3.1-8B-Instruct, Qwen2.5-7B/14B/32B-Instruct, Mistral-7B-Instruct-v0.3, GPT-4o and GPT-4o-mini.
- **Datasets:** BoolQ, StrategyQA, GSM8K, MMLU, CSQA, MedMCQA and CaseHOLD. Code (MarCon) is released under MIT on GitHub.
- **Metrics:** I-AvgECE, C-AvgECE, NumECE, MAC, MRC, I-AvgCV and C-AvgCV. Markers seen fewer than 10 times are filtered out. In the paper's example, GPT-4o's "fairly certain" on StrategyQA has a marker confidence of 20/31 ≈ 64.5%.
- **Main findings:**
  - In-domain marker ECE stays below numerical ECE.
  - Cross-domain ECE exceeds NumECE for 6 of 7 models.
  - Average C-AvgCV is about 23.4%.
  - MAC exceeds 0.7 for 5 of 7 models, meaning marker confidence simply follows dataset accuracy.
  - Low MRC values mean marker rankings shift from dataset to dataset.
- **Transfer across datasets:** Diverse, non-domain-specific datasets such as MMLU and StrategyQA make in-domain transfer harder. The study is limited to English markers in short QA answers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Hong Kong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/hong-kong-university-of-science-and-technology/) — University, Hong Kong, China · 142 works · 25,136 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Jiayu Liu, Qing Zong, Weiqi Wang and Yangqiu Song. 2025. Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2025revisiting,
  title = {{Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?}},
  author = {Jiayu Liu, Qing Zong, Weiqi Wang and Yangqiu Song},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-short.18},
  url = {https://doi.org/10.18653/v1/2025.acl-short.18}
}
```

- [← previous ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization](https://md.research.elamur.ai/free/en/papers/chen-2025-reform-reflective-autoformalization-prospective-bounded/)
- [next → rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset](https://md.research.elamur.ai/free/en/papers/liu-2025-rstar-coder-scaling-competitive/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/10/)

## References

64 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahul Alam, Sudeh Cheraghi-Sohi, Maria Panagi-oti, Aneez Esmail, Stephen Campbell, and Efharis Panagopoulou. 2017. Managing diagnostic uncertainty in primary care: a systematic critical review. BMC Family Practice, 18:1–13. [find](https://scholar.google.com/scholar?q=Rahul+Alam%2C+Sudeh+Cheraghi%2DSohi%2C+Maria+Panagi%2Doti%2C+Aneez+Esmail%2C+Stephen+Campbell%2C+and+Efharis+Panagopoulou%2E+2017%2E+Managing+diagnostic+uncertainty+in+primary+care%3A+a+systematic+critical+review%2E+BMC+Fa)
2. Mohammad Beigi, Ying Shen, Runing Yang, Zihao Lin, Qifan Wang, Ankith Mohan, Jianfeng He, Ming Jin, Chang-Tien Lu, and Lifu Huang. 2024. Inter-nalinspector i 2 : Robust confidence estimation in llms through internal states. In Findings of the Association for Computational Linguistics: EMNLP 2024, Mi… [find](https://scholar.google.com/scholar?q=Mohammad+Beigi%2C+Ying+Shen%2C+Runing+Yang%2C+Zihao+Lin%2C+Qifan+Wang%2C+Ankith+Mohan%2C+Jianfeng+He%2C+Ming+Jin%2C+Chang%2DTien+Lu%2C+and+Lifu+Huang%2E+2024%2E+Inter%2Dnalinspector+i+2+%3A+Robust+confidence+estimation+in+llms+t)
3. Catarina G. Belém, Markelle Kelly, Mark Steyvers, Sameer Singh, and Padhraic Smyth. 2024. Perceptions of linguistic uncertainty by language models and humans. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024,… [find](https://scholar.google.com/scholar?q=Catarina+G%2E+Bel%C3%A9m%2C+Markelle+Kelly%2C+Mark+Steyvers%2C+Sameer+Singh%2C+and+Padhraic+Smyth%2E+2024%2E+Perceptions+of+linguistic+uncertainty+by+language+models+and+humans%2E+In+Proceedings+of+the+2024+Conference+on+)
4. Viraj Bhise, Suja S Rajan, Dean F Sittig, Robert O Morgan, Pooja Chaudhary, and Hardeep Singh. 2018. Defining and measuring diagnostic uncertainty in medicine: a systematic review. Journal of general internal medicine, 33:103–115. [find](https://scholar.google.com/scholar?q=Viraj+Bhise%2C+Suja+S+Rajan%2C+Dean+F+Sittig%2C+Robert+O+Morgan%2C+Pooja+Chaudhary%2C+and+Hardeep+Singh%2E+2018%2E+Defining+and+measuring+diagnostic+uncertainty+in+medicine%3A+a+systematic+review%2E+Journal+of+general+)
5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
6. Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
7. Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2023. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. [find](https://scholar.google.com/scholar?q=Collin+Burns%2C+Haotian+Ye%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+2023%2E+Discovering+latent+knowledge+in+language+models+without+supervision%2E+In+The+Eleventh+International+Conference+on+Learning+Representatio)
8. Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. [Humans or llms as the judge? A study on judgement bias](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-or-llms-as-judge/). In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 8301–8327. Association…
9. Jiuhai Chen and Jonas Mueller. 2024. [Quantifying uncertainty in answers from any language model and enhancing their trustworthiness](https://md.research.elamur.ai/free/en/papers/chen-2024-quantifying-uncertainty-answers-any-language/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pa…
10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). CoRR, abs/2110.14168.
12. Joost C. F. de Winter, Samuel D. Gosling, and Jeff Potter. 2016. Comparing the pearson and spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological Methods, 21(3):273–290. [find](https://scholar.google.com/scholar?q=Joost+C%2E+F%2E+de+Winter%2C+Samuel+D%2E+Gosling%2C+and+Jeff+Potter%2E+2016%2E+Comparing+the+pearson+and+spearman+correlation+coefficients+across+distributions+and+sample+sizes%3A+A+tutorial+using+simulations+and+emp)
13. Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Com… [find](https://scholar.google.com/scholar?q=Jinhao+Duan%2C+Hao+Cheng%2C+Shiqi+Wang%2C+Alex+Zavalny%2C+Chenan+Wang%2C+Renjing+Xu%2C+Bhavya+Kailkhura%2C+and+Kaidi+Xu%2E+2024%2E+Shifting+attention+to+relevance%3A+Towards+the+predictive+uncertainty+quantification+of+f)
14. Ido Erev and Brent L Cohen. 1990. Verbal versus numerical probabilities: Efficiency, biases, and the preference paradox. Organizational Behavior and Human Decision Processes, 45(1):1–18. [find](https://scholar.google.com/scholar?q=Ido+Erev+and+Brent+L+Cohen%2E+1990%2E+Verbal+versus+numerical+probabilities%3A+Efficiency%2C+biases%2C+and+the+preference+paradox%2E+Organizational+Behavior+and+Human+Decision+Processes%2C+45%281%29%3A1%E2%80%9318%2E)
15. Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. [Unsupervised quality estimation for neural machine translation](https://md.research.elamur.ai/free/en/papers/fomicheva-2020-unsupervised-quality-estimation-neural-machine/). Trans. Assoc. Comput. Linguistics, 8:539–555.
16. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. 2024. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language… [find](https://scholar.google.com/scholar?q=Jiahui+Geng%2C+Fengyu+Cai%2C+Yuxia+Wang%2C+Heinz+Koeppl%2C+Preslav+Nakov%2C+and+Iryna+Gurevych%2E+2024%2E+A+survey+of+confidence+estimation+and+calibration+in+large+language+models%2E+In+Proceedings+of+the+2024+Confe)
17. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346–361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+A+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Trans%2E+Assoc%2E+Co)
18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
19. Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natural language processing: Sources, quantification, and applications. CoRR, abs/2306.04459. — [arXiv:2306.04459](https://arxiv.org/abs/2306.04459) [find](https://scholar.google.com/scholar?q=Mengting+Hu%2C+Zhen+Zhang%2C+Shiwan+Zhao%2C+Minlie+Huang%2C+and+Bingzhe+Wu%2E+2023%2E+Uncertainty+in+natural+language+processing%3A+Sources%2C+quantification%2C+and+applications%2E+CoRR%2C+abs%2F2306%2E04459%2E+%E2%80%94+%5BarXiv%3A2306%2E044)
20. Amanda Jaffe-Katz, David V Budescu, and Thomas S Wallsten. 1989. Timed magnitude comparisons of numerical and nonnumerical expressions of uncertainty. Memory & Cognition, 17(3):249–264. [find](https://scholar.google.com/scholar?q=Amanda+Jaffe%2DKatz%2C+David+V+Budescu%2C+and+Thomas+S+Wallsten%2E+1989%2E+Timed+magnitude+comparisons+of+numerical+and+nonnumerical+expressions+of+uncertainty%2E+Memory+%26+Cognition%2C+17%283%29%3A249%E2%80%93264%2E)
21. Zahra Jalilibal, Amirhossein Amiri, Philippe Castagli-ola, and Michael B. C. Khoo. 2021. Monitoring the coefficient of variation: A literature review. Comput. Ind. Eng., 161:107600. [find](https://scholar.google.com/scholar?q=Zahra+Jalilibal%2C+Amirhossein+Amiri%2C+Philippe+Castagli%2Dola%2C+and+Michael+B%2E+C%2E+Khoo%2E+2021%2E+Monitoring+the+coefficient+of+variation%3A+A+literature+review%2E+Comput%2E+Ind%2E+Eng%2E%2C+161%3A107600%2E)
22. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
23. Marie Juanchich, Amélie Gourdon-Kanhukamwe, and Miroslav Sirota. 2017. “i am uncertain” vs “it is uncertain”. how linguistic markers of the uncertainty source affect uncertainty communication. Judgment and Decision Making, 12(5):445–465. [find](https://scholar.google.com/scholar?q=Marie+Juanchich%2C+Am%C3%A9lie+Gourdon%2DKanhukamwe%2C+and+Miroslav+Sirota%2E+2017%2E+%E2%80%9Ci+am+uncertain%E2%80%9D+vs+%E2%80%9Cit+is+uncertain%E2%80%9D%2E+how+linguistic+markers+of+the+uncertainty+source+affect+uncertainty+communication%2E+Judgmen)
24. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield%2DDodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+Scott+Johnston%2C+Sheer+El+Showk%2C+Andy+Jones%2C+)
25. Peter Knapp, Peter H Gardner, and Elizabeth Woolf. 2016. Combined verbal and numerical expressions increase perceived risk of medicine side-effects: a randomized controlled trial of ema recommendations. Health Expectations, 19(2):264–274. [find](https://scholar.google.com/scholar?q=Peter+Knapp%2C+Peter+H+Gardner%2C+and+Elizabeth+Woolf%2E+2016%2E+Combined+verbal+and+numerical+expressions+increase+perceived+risk+of+medicine+side%2Deffects%3A+a+randomized+controlled+trial+of+ema+recommendation)
26. Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation. CoRR, abs/2410.20774. [find](https://scholar.google.com/scholar?q=Dongryeol+Lee%2C+Yerin+Hwang%2C+Yongil+Kim%2C+Joonsuk+Park%2C+and+Kyomin+Jung%2E+2024%2E+Are+llm%2Djudges+robust+to+expressions+of+uncertainty%3F+investigating+the+effect+of+epistemic+markers+on+llm%2Dbased+evaluation%2E)
27. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. 2020. [On faithfulness and factuality in abstractive summarization](https://md.research.elamur.ai/free/en/papers/maynez-2020-faithfulness-factuality-abstractive-summarization/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1906–1919. Association for Comput…
28. OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024a%2E+Gpt%2D4o+mini%3A+advancing+cost%2Defficient+intelligence%2E+OpenAI%2E)
29. OpenAI. 2024b. Hello gpt-4o. OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+Hello+gpt%2D4o%2E+OpenAI%2E)
30. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. [Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/). In Conference on Health, Inference, and Learning, CHIL 2022, 7-8 April 2022, Virtual Event, volume 174 of Proceedings of Machine Lear…
31. Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh, Soham Dan, and Prasanna Sattigeri. 2024. Large language model confidence estimation via black-box access. CoRR, abs/2406.04370. — [arXiv:2406.04370](https://arxiv.org/abs/2406.04370) [find](https://scholar.google.com/scholar?q=Tejaswini+Pedapati%2C+Amit+Dhurandhar%2C+Soumya+Ghosh%2C+Soham+Dan%2C+and+Prasanna+Sattigeri%2E+2024%2E+Large+language+model+confidence+estimation+via+black%2Dbox+access%2E+CoRR%2C+abs%2F2406%2E04370%2E+%E2%80%94+%5BarXiv%3A2406%2E04370%5D%28)
32. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024a. Think twice before assure: Confidence estimation for large language models through reflection on multiple answers. CoRR, abs/2403.09972. — [arXiv:2403.09972](https://arxiv.org/abs/2403.09972) [find](https://scholar.google.com/scholar?q=Moxin+Li%2C+Wenjie+Wang%2C+Fuli+Feng%2C+Fengbin+Zhu%2C+Qifan+Wang%2C+and+Tat%2DSeng+Chua%2E+2024a%2E+Think+twice+before+assure%3A+Confidence+estimation+for+large+language+models+through+reflection+on+multiple+answers%2E+)
33. Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua. 2024b. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November… [find](https://scholar.google.com/scholar?q=Moxin+Li%2C+Wenjie+Wang%2C+Fuli+Feng%2C+Fengbin+Zhu%2C+Qifan+Wang%2C+and+Tat%2DSeng+Chua%2E+2024b%2E+Think+twice+before+trusting%3A+Self%2Ddetection+for+large+language+models+through+comprehensive+answer+reflection%2E+In+F)
34. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+Teaching+models+to+express+their+uncertainty+in+words%2E)
35. Trans. Mach. Learn. Res., 2022. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2022%2E)
36. Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2024. Generating with confidence: Uncertainty quantification for black-box large language models. [find](https://scholar.google.com/scholar?q=Zhen+Lin%2C+Shubhendu+Trivedi%2C+and+Jimeng+Sun%2E+2024%2E+Generating+with+confidence%3A+Uncertainty+quantification+for+black%2Dbox+large+language+models%2E)
37. Trans. Mach. Learn. Res., 2024. [find](https://scholar.google.com/scholar?q=Trans%2E+Mach%2E+Learn%2E+Res%2E%2C+2024%2E)
38. Pia Pennekamp, Jamal K. Mansour, and Rhiannon J. Batstone. 2024. Variability in verbal eyewitness confidence. Applied Cognitive Psychology. [find](https://scholar.google.com/scholar?q=Pia+Pennekamp%2C+Jamal+K%2E+Mansour%2C+and+Rhiannon+J%2E+Batstone%2E+2024%2E+Variability+in+verbal+eyewitness+confidence%2E+Applied+Cognitive+Psychology%2E)
39. Limin Su and Huimin Li. 2021. Project procurement method decision-making with spearman rank correlation coefficient under uncertainty circumstances. Int. J. Decis. Support Syst. Technol., 13(2):16–44. [find](https://scholar.google.com/scholar?q=Limin+Su+and+Huimin+Li%2E+2021%2E+Project+procurement+method+decision%2Dmaking+with+spearman+rank+correlation+coefficient+under+uncertainty+circumstances%2E+Int%2E+J%2E+Decis%2E+Support+Syst%2E+Technol%2E%2C+13%282%29%3A16%E2%80%9344%2E)
40. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
41. Zhisheng Tang, Ke Shen, and Mayank Kejriwal. 2024. An evaluation of estimative uncertainty in large language models. arXiv preprint arXiv:2405.15185. — [arXiv:2405.15185](https://arxiv.org/abs/2405.15185) [find](https://scholar.google.com/scholar?q=Zhisheng+Tang%2C+Ke+Shen%2C+and+Mayank+Kejriwal%2E+2024%2E+An+evaluation+of+estimative+uncertainty+in+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E15185%2E+%E2%80%94+%5BarXiv%3A2405%2E15185%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E15)
42. Jiayu Liu, Junhao Tang, Hanwen Wang, Baixuan Xu, Haochen Shi, Weiqi Wang, and Yangqiu Song. 2024. Gprooft: A multi-dimension multi-round fact checking framework based on claim fact extraction. In Proceedings of the Seventh Fact Extraction and VERification Workshop (FEVER), pages 118–129. [find](https://scholar.google.com/scholar?q=Jiayu+Liu%2C+Junhao+Tang%2C+Hanwen+Wang%2C+Baixuan+Xu%2C+Haochen+Shi%2C+Weiqi+Wang%2C+and+Yangqiu+Song%2E+2024%2E+Gprooft%3A+A+multi%2Ddimension+multi%2Dround+fact+checking+framework+based+on+claim+fact+extraction%2E+In+Proc)
43. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). CoRR, ab…
44. Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. 2023. [Fairness-guided few-shot prompting for large language models](https://md.research.elamur.ai/free/en/papers/ma-2023-fairness-guided-few-shot-prompting/). In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processin…
45. Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Akim Tsvigun, Daniil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Lyudmila Rvanova, Sergey Petrakov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov. 2024. [Benchmarking uncertainty quantification methods for…](https://md.research.elamur.ai/free/en/papers/vashurin-2025-benchmarking-uncertainty-quantification-methods-large/)
46. Thomas S Wallsten. 1986. Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General, 115(4):348–365. [find](https://scholar.google.com/scholar?q=Thomas+S+Wallsten%2E+1986%2E+Measuring+the+vague+meanings+of+probability+terms%2E+Journal+of+Experimental+Psychology%3A+General%2C+115%284%29%3A348%E2%80%93365%2E)
47. Weiqi Wang, Limeng Cui, Xin Liu, Sreyashi Nag, Wenju Xu, Chen Luo, Sheikh Muhammad Sarwar, Yang Li, Hansu Gu, Hui Liu, Changlong Yu, Jiaxin Bai, Yifan Gao, Haiyang Zhang, Qi He, Shuiwang Ji, and Yangqiu Song. 2025. Ecomscriptbench: A multi-task benchmark for e-commerce script planning via step-wise… [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Limeng+Cui%2C+Xin+Liu%2C+Sreyashi+Nag%2C+Wenju+Xu%2C+Chen+Luo%2C+Sheikh+Muhammad+Sarwar%2C+Yang+Li%2C+Hansu+Gu%2C+Hui+Liu%2C+Changlong+Yu%2C+Jiaxin+Bai%2C+Yifan+Gao%2C+Haiyang+Zhang%2C+Qi+He%2C+Shuiwang+Ji%2C+and+Yangq)
48. Weiqi Wang, Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosselut. 2023a. CAR: conceptualization-augmented reasoner for zero-shot commonsense question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, p… [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Tianqing+Fang%2C+Wenxuan+Ding%2C+Baixuan+Xu%2C+Xin+Liu%2C+Yangqiu+Song%2C+and+Antoine+Bosselut%2E+2023a%2E+CAR%3A+conceptualization%2Daugmented+reasoner+for+zero%2Dshot+commonsense+question+answering%2E+In+Find)
49. Weiqi Wang, Tianqing Fang, Chunyang Li, Haochen Shi, Wenxuan Ding, Baixuan Xu, Zhaowei Wang, Jiaxin Bai, Xin Liu, Cheng Jiayang, Chunkit Chan, and Yangqiu Song. 2024a. CANDLE: iterative conceptualization and instantiation distillation from large language models for commonsense reasoning. In Proceedi… [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Tianqing+Fang%2C+Chunyang+Li%2C+Haochen+Shi%2C+Wenxuan+Ding%2C+Baixuan+Xu%2C+Zhaowei+Wang%2C+Jiaxin+Bai%2C+Xin+Liu%2C+Cheng+Jiayang%2C+Chunkit+Chan%2C+and+Yangqiu+Song%2E+2024a%2E+CANDLE%3A+iterative+conceptualizat)
50. Weiqi Wang, Tianqing Fang, Haochen Shi, Baixuan Xu, Wenxuan Ding, Liyu Zhang, Wei Fan, Jiaxin Bai, Haoran Li, Xin Liu, and Yangqiu Song. 2024b. On the role of entity and event level conceptualization in generalizable reasoning: A survey of tasks, methods, applications, and future directions. CoRR, a… [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Tianqing+Fang%2C+Haochen+Shi%2C+Baixuan+Xu%2C+Wenxuan+Ding%2C+Liyu+Zhang%2C+Wei+Fan%2C+Jiaxin+Bai%2C+Haoran+Li%2C+Xin+Liu%2C+and+Yangqiu+Song%2E+2024b%2E+On+the+role+of+entity+and+event+level+conceptualization+)
51. Weiqi Wang, Tianqing Fang, Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023b. CAT: A contextualized conceptualization and instantiation framework for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers… [find](https://scholar.google.com/scholar?q=Weiqi+Wang%2C+Tianqing+Fang%2C+Baixuan+Xu%2C+Chun+Yi+Louis+Bo%2C+Yangqiu+Song%2C+and+Lei+Chen%2E+2023b%2E+CAT%3A+A+contextualized+conceptualization+and+instantiation+framework+for+commonsense+reasoning%2E+In+Proceeding)
52. Weiqi Wang and Yangqiu Song. 2024. MARS: benchmarking the metaphysical reasoning abilities of language models with a multi-task evaluation dataset. CoRR, abs/2406.02106. [find](https://scholar.google.com/scholar?q=Weiqi+Wang+and+Yangqiu+Song%2E+2024%2E+MARS%3A+benchmarking+the+metaphysical+reasoning+abilities+of+language+models+with+a+multi%2Dtask+evaluation+dataset%2E+CoRR%2C+abs%2F2406%2E02106%2E)
53. Sanne J. W. Willems, Casper J. Albers, and Ionica Smeets. 2019. Variability in the interpretation of dutch probability phrases - a risk for miscommunication. Preprint, arXiv:1901.09686. — [arXiv:1901.09686](https://arxiv.org/abs/1901.09686) [find](https://scholar.google.com/scholar?q=Sanne+J%2E+W%2E+Willems%2C+Casper+J%2E+Albers%2C+and+Ionica+Smeets%2E+2019%2E+Variability+in+the+interpretation+of+dutch+probability+phrases+%2D+a+risk+for+miscommunication%2E+Preprint%2C+arXiv%3A1901%2E09686%2E+%E2%80%94+%5BarXiv%3A1901%2E)
54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
55. Hui Xiong, Shashi Shekhar, Pang-Ning Tan, and Vipin Kumar. 2004. Exploiting a support-based upper bound of pearson’s correlation coefficient for efficiently identifying strongly correlated pairs. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,… [find](https://scholar.google.com/scholar?q=Hui+Xiong%2C+Shashi+Shekhar%2C+Pang%2DNing+Tan%2C+and+Vipin+Kumar%2E+2004%2E+Exploiting+a+support%2Dbased+upper+bound+of+pearson%E2%80%99s+correlation+coefficient+for+efficiently+identifying+strongly+correlated+pairs%2E+In+P)
56. Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open… [find](https://scholar.google.com/scholar?q=Miao+Xiong%2C+Zhiyuan+Hu%2C+Xinyang+Lu%2C+Yifei+Li%2C+Jie+Fu%2C+Junxian+He%2C+and+Bryan+Hooi%2E+2024%2E+Can+llms+express+their+uncertainty%3F+an+empirical+evaluation+of+confidence+elicitation+in+llms%2E+In+The+Twelfth+In)
57. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
58. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. 2024. [Justice or prejudice? quantifying biases in llm-as-a-judge](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/). CoRR, abs/2410.02736.
59. Gal Yona, Roee Aharoni, and Mor Geva. 2024. [Can large language models faithfully express their intrinsic uncertainty in words?](https://md.research.elamur.ai/free/en/papers/yona-2024-can-large-language-models-faithfully/) In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 7752–7764. Association for…
60. Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. 2024. [Self-alignment for factuality: Mitigating hallucinations in llms via self-evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2024-self-alignment-factuality-mitigating-hallucinations/). In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L…
61. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When does pretraining help?: assessing self-supervised learning for law and the casehold dataset of 53, 000+ legal holdings. In ICAIL ’21: Eighteenth International Conference for Artificial Intelligence and Law, Sã… [find](https://scholar.google.com/scholar?q=Lucia+Zheng%2C+Neel+Guha%2C+Brandon+R%2E+Anderson%2C+Peter+Henderson%2C+and+Daniel+E%2E+Ho%2E+2021%2E+When+does+pretraining+help%3F%3A+assessing+self%2Dsupervised+learning+for+law+and+the+casehold+dataset+of+53%2C+000%2B+legal)
62. Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, and Maarten Sap. 2024. Relying on the unreliable: The impact of language models’ reluctance to express uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand,… [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Jena+D%2E+Hwang%2C+Xiang+Ren%2C+and+Maarten+Sap%2E+2024%2E+Relying+on+the+unreliable%3A+The+impact+of+language+models%E2%80%99+reluctance+to+express+uncertainty%2E+In+Proceedings+of+the+62nd+Annual+Meeting+of)
63. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages… [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Dan+Jurafsky%2C+and+Tatsunori+Hashimoto%2E+2023%2E+Navigating+the+grey+area%3A+How+expressions+of+uncertainty+and+overconfidence+affect+language+models%2E+In+Proceedings+of+the+2023+Conference+on+)
64. Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu Ren, and Yangqiu Song. 2024. Comparisonqa: Evaluating factuality robustness of llms through knowledge frequency control and uncertainty. CoRR, abs/2412.20251. [find](https://scholar.google.com/scholar?q=Qing+Zong%2C+Zhaowei+Wang%2C+Tianshi+Zheng%2C+Xiyu+Ren%2C+and+Yangqiu+Song%2E+2024%2E+Comparisonqa%3A+Evaluating+factuality+robustness+of+llms+through+knowledge+frequency+control+and+uncertainty%2E+CoRR%2C+abs%2F2412%2E202)

*Updated 31 October 2026*