---
title: "Measuring and Modifying Factual Knowledge in Large Language"
description: "Pezeshkpour proposes information-theoretic scores instead. 22 references, cited by 13 papers here."
authors: ["Pouya Pezeshkpour"]
year: 2023
venue: "arXiv:2306.06264"
source: "https://arxiv.org/abs/2306.06264"
doi: "10.48550/arxiv.2306.06264"
arxiv: "2306.06264"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pezeshkpour-2023-measuring-modifying-factual-knowledge-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pezeshkpour-2023-measuring-modifying-factual-knowledge-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Measuring and Modifying Factual Knowledge in Large Language Models

# Measuring and Modifying Factual Knowledge in Large Language Models

**Authors:** [Pouya Pezeshkpour](https://md.research.elamur.ai/free/en/authors/pouya-pezeshkpour/) · **Year:** 2023 · **Venue:** arXiv:2306.06264 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.06264) · [PDF](https://arxiv.org/pdf/2306.06264) · [DOI 10.48550/arxiv.2306.06264](https://doi.org/10.48550/arxiv.2306.06264) · [arXiv 2306.06264](https://arxiv.org/abs/2306.06264) · [OpenAlex](https://openalex.org/W4392942881)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [13](https://md.research.elamur.ai/free/en/papers/pezeshkpour-2023-measuring-modifying-factual-knowledge-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Fact probing of LLMs usually checks how highly the correct answer ranks in fill-in-the-blank predictions. This treats knowledge as yes/no, depends heavily on prompt wording, and ignores most of the output distribution. Pezeshkpour proposes information-theoretic scores instead. A model is considered to know a fact if adding that fact barely changes its uncertainty about the blank. Knowledge is measured either as the change in entropy or as the KL-divergence between the vocabulary distributions before and after the fact is instilled. For APIs that return only top-k tokens, the distribution is approximated over the union of returned tokens plus an oov bucket.

Facts are instilled in two ways: explicitly, by stating them in the prompt, or implicitly, by fine-tuning on them. In a synthetic test with BERT and T5 on LAMA/T-REx, both new metrics judge knowledge levels much more accurately than ranking. Explicit instillation proves unreliable mostly for location- and language-type relations. The metrics are also applied to factual alignment and hallucination analysis in in-context learning models, offering a cheaper alternative to fine-tuning where that is impractical.

## Key points
- Synthetic accuracy (Table 1), ranking / entropy / KL-divergence: BERT 51.6 / 72.2 / 74.5; T5 30.9 / 66.4 / 67.8. The paper reports the gain as about 30% in one place and over 35% in another.
- Data: T-REx and LAMA, with 100 T-REx facts randomly sampled per LAMA relation for comparing instillation methods. Models: BERT and T5, plus InstructGPT (text-davinci-003) and FLAN-T5 (XL) for in-context experiments.
- In InstructGPT and FLAN-T5, roughly 80% of samples still mispredicted after explicit instillation (gold label outside the top 5) involved location or language queries.
- For relations with a location object, the model showed more knowledge of hallucinated facts than of correctly generated ones. Facts that did not appear in the output generally scored lowest.
- Code and data are available at github.com/rit-git/lm-know.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Pouya Pezeshkpour. 2023. Measuring and Modifying Factual Knowledge in Large Language Models. arXiv:2306.06264.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{pezeshkpour2023measuring,
  title = {{Measuring and Modifying Factual Knowledge in Large Language Models}},
  author = {Pouya Pezeshkpour},
  year = {2023},
  doi = {10.48550/arxiv.2306.06264},
  eprint = {2306.06264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.06264}
}
```

- [← previous Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/abdallah-2023-leveraging-data-collection-unsupervised-learning/)
- [next → N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition](https://md.research.elamur.ai/free/en/papers/talafha-2023-n-shot-benchmarking-whisper-diverse/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

22 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. 2022. [A review on language models as knowledge bases](https://md.research.elamur.ai/free/en/papers/alkhamissi-2022-review-language-models-knowledge-bases/). arXiv preprint arXiv:2204.06031.
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. [Knowledgeable or educated guess? revisiting language models as knowledge bases](https://md.research.elamur.ai/free/en/papers/cao-2021-knowledgeable-or-educated-guess-revisiting/). arXiv preprint arXiv:2106.09231.
4. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. [find](https://scholar.google.com/scholar?q=Saurav+Kadavath%2C+Tom+Conerly%2C+Amanda+Askell%2C+Tom+Henighan%2C+Dawn+Drain%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Zac+Hatfield+Dodds%2C+Nova+DasSarma%2C+Eli+Tran%2DJohnson%2C+et+al%2E+2022%2E+Language+models+%28mostly%29+know+w)
5. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+2022%2E+Teaching+models+to+express+their+uncertainty+in+words%2E+arXiv+preprint+arXiv%3A2205%2E14334%2E)
6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
7. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
9. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
10. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXi)
11. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. [Check your facts and try again: Improving large language models with external knowledge and automated feedback](https://md.research.elamur.ai/free/en/papers/peng-2023-check-facts-try-again-improving/). arXiv preprint arXiv:2302.12813.
12. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. [Crawling the internal knowledge-base of language models](https://md.research.elamur.ai/free/en/papers/cohen-2023-crawling-internal-knowledge-base-language/). arXiv preprint arXiv:2301.12810.
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
14. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. [T-rex: A large scale alignment of natural language with knowledge base triples](https://md.research.elamur.ai/free/en/papers/elsahar-2018-t-rex-large-scale-alignment/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluati…
15. Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 166–175. [find](https://scholar.google.com/scholar?q=Ben+Goodrich%2C+Vinay+Rao%2C+Peter+J+Liu%2C+and+Mohammad+Saleh%2E+2019%2E+Assessing+the+factual+accuracy+of+generated+text%2E+In+Proceedings+of+the+25th+ACM+SIGKDD+International+Conference+on+Knowledge+Discovery+)
16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38.
17. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) arXiv preprint arXiv:1909.01066.
18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
19. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. [Retrieval augmentation reduces hallucination in conversation](https://md.research.elamur.ai/free/en/papers/shuster-2021-retrieval-augmentation-reduces-hallucination-conversation/). arXiv preprint arXiv:2104.07567.
20. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
21. Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings. arXiv preprint arXiv:2203.00211. [find](https://scholar.google.com/scholar?q=Neeraj+Varshney%2C+Swaroop+Mishra%2C+and+Chitta+Baral%2E+2022%2E+Investigating+selective+prediction+approaches+across+several+tasks+in+iid%2C+ood%2C+and+adversarial+settings%2E+arXiv+preprint+arXiv%3A2203%2E00211%2E)
22. Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. [Probing pretrained language models for lexical semantics](https://md.research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/). arXiv preprint arXiv:2010.05731.

*Updated 2 November 2026*