---
title: "Knowledge of Knowledge — paper by Amayuelas et al. (2024)"
description: "The paper asks whether large language models can recognize questions that have no definitive… 31 references, cited by 51 papers here."
authors: ["Alfonso Amayuelas", "Kyle Wong", "Liangming Pan", "Wenhu Chen", "William Yang Wang"]
year: 2024
venue: "Findings of the Association for Computational Linguistics: ACL 2024"
source: "https://aclanthology.org/2024.findings-acl.383/"
doi: "10.18653/v1/2024.findings-acl.383"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models

# Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models

**Authors:** [Alfonso Amayuelas](https://md.research.elamur.ai/free/en/authors/alfonso-amayuelas/), Kyle Wong, [Liangming Pan](https://md.research.elamur.ai/free/en/authors/liangming-pan/), [Wenhu Chen](https://md.research.elamur.ai/free/en/authors/wenhu-chen/) and [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2024 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2024](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2024](https://md.research.elamur.ai/free/en/venues/findings/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.findings-acl.383/) · [PDF](https://aclanthology.org/2024.findings-acl.383.pdf) · [DOI 10.18653/v1/2024.findings-acl.383](https://doi.org/10.18653/v1/2024.findings-acl.383) · [OpenAlex](https://openalex.org/W4402683869)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [51](https://md.research.elamur.ai/free/en/papers/amayuelas-2024-knowledge-knowledge-exploring-known-unknowns/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether large language models can recognize questions that have no definitive answer, which it calls "known-unknowns," instead of answering them with false confidence. The authors found existing resources too small for this, so they built KUQ (Known-Unknown Questions). Crowd workers wrote unanswerable questions, each tagged with a reason for its uncertainty, using six categories such as future events, unsolved problems and false premises. Each was paired with a similar answerable question from established QA datasets. They then fine-tuned open-source Llama models on KUQ and tested open-ended question answering. Before tuning, the Llama models trailed GPT-3.5 and GPT-4 at separating known from unknown questions. After tuning, their F1 rose substantially and approached those models. The gain came with some loss in answering known questions correctly. The tuned models also transferred to the SelfAware dataset. Finally, a KUQ-tuned Llama2-7B-Chat improved multiagent debate results on several reasoning benchmarks. This suggests that teaching models to voice uncertainty can help downstream tasks and reduce hallucination-prone behaviour.

## Key points
- **KUQ composition:** KUQ contains 6,884 questions. Of these, 3,437 are crowd-sourced unknown questions. The remaining 3,447 are known questions drawn from SQuAD (1,928), TriviaQA (854) and HotPotQA (665).
- **Data collection:** Annotators received a target category with examples, then wrote a question and explained why it cannot be answered. The category and explanation were confirmed by a majority vote of three workers. Known counterparts were selected by sentence similarity with SimCSE.
- **Six uncertainty categories:** Future Unknown, Unsolved Problem, Controversial, Questions with False Assumption, Counterfactual and Ambiguous. Each category holds roughly 13–20% of the unknown questions.
- **Debate setup:** The multiagent debate experiment used 3 agents and 2 rounds. It was evaluated on MMLU, CommonSenseQA, ARC and BIG-Bench Chess State Tracking.
- **Debate results:** Responses from the fine-tuned Llama2-7B-Chat matched a list of uncertainty expressions more closely, scored by SimCSE. On CommonSenseQA, average similarity rose from 0.525 to 0.631, and the share of high-similarity responses roughly doubled (0.064 to 0.137). Human inspection still showed that the models express uncertainty inconsistently.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 44 works · 12,783 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 35 works · 9,287 citations
- [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) — Research institute, Toronto, Canada · 19 works · 4,281 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen and William Yang Wang. 2024. Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{amayuelas2024knowledge,
  title = {{Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Models}},
  author = {Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhu Chen and William Yang Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.383},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.383}
}
```

- [← previous Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://md.research.elamur.ai/free/en/papers/wang-2024-arithmetic-control-llms-diverse-user/)
- [next → MagicPIG: LSH Sampling for Efficient LLM Generation](https://md.research.elamur.ai/free/en/papers/chen-2024-magicpig-lsh-sampling-efficient-llm/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). ArXiv preprint, abs/2107.03374. \#\# Ethics Statement Peter Clark, Isaac Cowhey, O…
2. Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. [Selectively answering ambiguous questions](https://md.research.elamur.ai/free/en/papers/cole-2023-selectively-answering-ambiguous-questions/). ArXiv preprint, abs/2305.14613.
3. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. [Improving factuality and reasoning in language models through multiagent debate](https://md.research.elamur.ai/free/en/papers/du-2023-improving-factuality-reasoning-language-models/). arXiv preprint arXiv:2305.14325.
4. Richárd Farkas, Veronika Vincze, György Móra, János Csirik, and György Szarvas. 2010. [The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text](https://md.research.elamur.ai/free/en/papers/farkas-2010-conll-2010-shared-task-learning/). In Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task, pages 1– 12, Upp…
5. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. [Simcse: Simple contrastive learning of sentence embeddings](https://md.research.elamur.ai/free/en/papers/gao-2021-simcse-simple-contrastive-learning-sentence/). arXiv preprint arXiv:2104.08821.
6. Human evaluation was conducted through crowdsourcing using the Amazon Mechanical Turk platform. To ensure the quality of our experiments, we only considered workers with a HIT approval rating of at least 95% from the Mechanical Turk Masters pool. We compensated the workers at a rate of $0.25 per tas… [find](https://scholar.google.com/scholar?q=Human+evaluation+was+conducted+through+crowdsourcing+using+the+Amazon+Mechanical+Turk+platform%2E+To+ensure+the+quality+of+our+experiments%2C+we+only+considered+workers+with+a+HIT+approval+rating+of+at+le)
7. Ruth Garner and Patricia A Alexander. 1989. Metacognition: Answered and unanswered questions. Educational psychologist, 24(2):143–158. \#\# Acknowledgements Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models… [find](https://scholar.google.com/scholar?q=Ruth+Garner+and+Patricia+A+Alexander%2E+1989%2E+Metacognition%3A+Answered+and+unanswered+questions%2E+Educational+psychologist%2C+24%282%29%3A143%E2%80%93158%2E+%5C%23%5C%23+Acknowledgements+Zhengbao+Jiang%2C+Jun+Araki%2C+Haibo+Ding%2C+and+)
8. This work was supported by the National Science Foundation award \#2048122. The views expressed are those of the authors and do not reflect the official policy or position of the US government. [find](https://scholar.google.com/scholar?q=This+work+was+supported+by+the+National+Science+Foundation+award+%5C%232048122%2E+The+views+expressed+are+those+of+the+authors+and+do+not+reflect+the+official+policy+or+position+of+the+US+government%2E)
9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
11. Mengting Hu, Zhen Zhang, Shiwan Zhao, Minlie Huang, and Bingzhe Wu. 2023. Uncertainty in natural language processing: Sources, quantification, and applications. ArXiv preprint, abs/2306.04459. — [arXiv:2306.04459](https://arxiv.org/abs/2306.04459) [find](https://scholar.google.com/scholar?q=Mengting+Hu%2C+Zhen+Zhang%2C+Shiwan+Zhao%2C+Minlie+Huang%2C+and+Bingzhe+Wu%2E+2023%2E+Uncertainty+in+natural+language+processing%3A+Sources%2C+quantification%2C+and+applications%2E+ArXiv+preprint%2C+abs%2F2306%2E04459%2E+%E2%80%94+%5BarXi)
12. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
13. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. [Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). arXiv preprint arXiv:1705.03551.
14. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. ArXiv preprint, abs/2207.05221. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield+Dodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+2022%2E+Language+models+%28mostly%29+know+w)
15. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. ArXiv preprint, abs/2302.09664. [find](https://scholar.google.com/scholar?q=Lorenz+Kuhn%2C+Yarin+Gal%2C+and+Sebastian+Farquhar%2E+2023%2E+Semantic+uncertainty%3A+Linguistic+invariances+for+uncertainty+estimation+in+natural+language+generation%2E+ArXiv+preprint%2C+abs%2F2302%2E09664%2E)
16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. ArXiv preprint, abs/2205.14334. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+Teaching+models+to+express+their+uncertainty+in+words%2E+ArXiv+preprint%2C+abs%2F2205%2E14334%2E)
17. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
18. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv preprint arXiv:1606.05250.
20. Shahin Sharifi Noorian, Sihang Qiu, Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2022. What should you know? a human-in-the-loop approach to unknown unknowns characterization in image recognition. In Proceedings of the ACM Web Conference 2022, pages 882–892. [find](https://scholar.google.com/scholar?q=Shahin+Sharifi+Noorian%2C+Sihang+Qiu%2C+Ujwal+Gadiraju%2C+Jie+Yang%2C+and+Alessandro+Bozzon%2E+2022%2E+What+should+you+know%3F+a+human%2Din%2Dthe%2Dloop+approach+to+unknown+unknowns+characterization+in+image+recognition%2E)
21. György Szarvas, Veronika Vincze, Richárd Farkas, György Móra, and Iryna Gurevych. 2012. Cross-genre and cross-domain detection of semantic uncertainty. Computational Linguistics, 38(2):335–367. [find](https://scholar.google.com/scholar?q=Gy%C3%B6rgy+Szarvas%2C+Veronika+Vincze%2C+Rich%C3%A1rd+Farkas%2C+Gy%C3%B6rgy+M%C3%B3ra%2C+and+Iryna+Gurevych%2E+2012%2E+Cross%2Dgenre+and+cross%2Ddomain+detection+of+semantic+uncertainty%2E+Computational+Linguistics%2C+38%282%29%3A335%E2%80%93367%2E)
22. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). arXiv preprint arXiv:1811.00937.
23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
24. Zhen Wang, Peide Zhu, and Jie Yang. 2023. Controversialqa: Exploring controversy in question answering. ArXiv preprint, abs/2302.05061. [find](https://scholar.google.com/scholar?q=Zhen+Wang%2C+Peide+Zhu%2C+and+Jie+Yang%2E+2023%2E+Controversialqa%3A+Exploring+controversy+in+question+answering%2E+ArXiv+preprint%2C+abs%2F2302%2E05061%2E)
25. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv preprint, abs/2201.11903.
26. Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7273–7284,… [find](https://scholar.google.com/scholar?q=Yuxin+Xiao%2C+Paul+Pu+Liang%2C+Umang+Bhatt%2C+Willie+Neiswanger%2C+Ruslan+Salakhutdinov%2C+and+Louis%2DPhilippe+Morency%2E+2022%2E+Uncertainty+quantification+with+pre%2Dtrained+language+models%3A+A+large%2Dscale+empirical+)
27. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [Hotpotqa: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). arXiv preprint arXiv:1809.09600.
28. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don’t know? In Findings of Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Qipeng+Guo%2C+Jiawen+Wu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2023%2E+Do+large+language+models+know+what+they+don%E2%80%99t+know%3F+In+Findings+of+Association+for+Computational+Linguistics+%28ACL)
29. Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023. Navigating the grey area: Expressions of overconfidence and uncertainty in language models. ArXiv preprint, abs/2302.13439. [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Dan+Jurafsky%2C+and+Tatsunori+Hashimoto%2E+2023%2E+Navigating+the+grey+area%3A+Expressions+of+overconfidence+and+uncertainty+in+language+models%2E+ArXiv+preprint%2C+abs%2F2302%2E13439%2E)
30. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+volume+31%2E)
31. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). ArXiv preprint, abs/2206.04615.

*Updated 24 October 2026*