--- title: "Knowledgeable or Educated Guess? Revisiting Language Models" description: "Cao et al. test this for three querying styles: bare relation prompts, prompts preceded by example… 47 references, cited by 70 papers here." authors: ["Boxi Cao", "Hongyu Lin", "Xianpei Han", "Le Sun", "Lingyong Yan", "Meng Liao", "Tong Xue", "Jin Xu"] year: 2021 venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)" source: "https://aclanthology.org/2021.acl-long.146/" doi: "10.18653/v1/2021.acl-long.146" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/cao-2021-knowledgeable-or-educated-guess-revisiting/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cao-2021-knowledgeable-or-educated-guess-revisiting/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases # Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases **Authors:** [Boxi Cao](https://md.research.elamur.ai/free/en/authors/boxi-cao/), [Hongyu Lin](https://md.research.elamur.ai/free/en/authors/hongyu-lin/), [Xianpei Han](https://md.research.elamur.ai/free/en/authors/xianpei-han/), [Le Sun](https://md.research.elamur.ai/free/en/authors/le-sun/), [Lingyong Yan](https://md.research.elamur.ai/free/en/authors/lingyong-yan/), [Meng Liao](https://md.research.elamur.ai/free/en/authors/meng-liao/), Tong Xue and [Jin Xu](https://md.research.elamur.ai/free/en/authors/jin-xu/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.acl-long.146/) · [PDF](https://aclanthology.org/2021.acl-long.146.pdf) · [DOI 10.18653/v1/2021.acl-long.146](https://doi.org/10.18653/v1/2021.acl-long.146) · [OpenAlex](https://openalex.org/W3173673636) **Citations:** 71 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/cao-2021-knowledgeable-or-educated-guess-revisiting/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Do masked language models such as BERT actually store retrievable facts, or do they mainly exploit regularities in how they are evaluated? Cao et al. test this for three querying styles: bare relation prompts, prompts preceded by example subject–object pairs (case-based analogy), and prompts with added external context. Most experiments use BERT-large, with RoBERTa-large in the appendix. They run on the T-REx portion of LAMA and on WIKI-UNI, a new Wikidata-derived dataset with a much flatter answer distribution. With prompts alone, the model's outputs barely change between the two datasets. They are largely fixed by the template even when the subject is removed. Strong LAMA scores therefore reflect how closely a prompt matches that dataset's answer skew. Example cases mainly steer the model toward objects of the right entity type, using types induced from the Wikidata taxonomy, but not toward the correct entity within that type. Contexts help mostly when they reveal the answer, either directly or in a recoverable form. The study casts doubt on treating MLMs as dependable knowledge bases and calls for de-biased probing benchmarks. ## Key points - On LAMA, the five most frequent gold objects account for about 48% of instances, against under 8% on WIKI-UNI. Yet the model's top-5 predictions cover roughly 58% and 52% of instances, so its outputs look alike on both datasets. - Prompt-based precision falls from 30.36 on LAMA to 16.47 on WIKI-UNI. Three prompt families were tested: manual (Petroni et al.), mined (Jiang et al.), and auto-searched (Shin et al.). - Adding context lifts LAMA precision from 30.36 to 41.44. With the gold answer masked inside the context, precision still reaches 35.66. - When the masked answer can be reconstructed from the remaining context (60.23% of cases), the gain is +21.24 points. Otherwise the gain is only +6.99. - Code and data, including WIKI-UNI, are released in the LANKA repository (github.com/c-box/LANKA). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tencent](https://md.research.elamur.ai/free/en/companies/tencent/) — Company, Shenzhen, China · 66 works · 17,945 citations - [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 68 works · 22,753 citations - [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 32 works · 7,615 citations - [Institute of Software](https://md.research.elamur.ai/free/en/labs/institute-of-software/) — Research institute, Beijing, China · 10 works · 4,807 citations - [State Key Laboratory of Computer Science](https://md.research.elamur.ai/free/en/labs/state-key-laboratory-of-computer-science/) — Research institute, Beijing, China · 4 works · 1,309 citations Also on the byline: [Mylan](https://ror.org/045sjkt56). 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 103 works ## Cite Cite as: Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue and Jin Xu. 2021. Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{cao2021knowledgeable, title = {{Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases}}, author = {Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue and Jin Xu}, booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)}, year = {2021}, doi = {10.18653/v1/2021.acl-long.146}, url = {https://doi.org/10.18653/v1/2021.acl-long.146} } ``` - [← previous Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://md.research.elamur.ai/free/en/papers/gu-2021-fully-non-autoregressive-neural-machine/) - [next → Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://md.research.elamur.ai/free/en/papers/ravichander-2021-probing-probing-paradigm-does-probing/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) ## References 47 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ning Bian, Xianpei Han, Bo Chen, and Le Sun. 2021. Benchmarking Knowledge-Enhanced Commonsense Question Answering via Knowledge-to-Text Transformation. arXiv:2101.00760 [cs]. — [arXiv:2101.00760](https://arxiv.org/abs/2101.00760) [find](https://scholar.google.com/scholar?q=Ning+Bian%2C+Xianpei+Han%2C+Bo+Chen%2C+and+Le+Sun%2E+2021%2E+Benchmarking+Knowledge%2DEnhanced+Commonsense+Question+Answering+via+Knowledge%2Dto%2DText+Transformation%2E+arXiv%3A2101%2E00760+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2101%2E00760%5D%28http) 2. Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposiu… [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose%CC%81+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2020%2E+Inducing+relational+knowledge+from+BERT%2E+In+The+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intelligence%2C+AAAI+2020%2C+The+Thirty%2DSeco) 3. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) 4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. [Reading Wikipedia to answer open-domain questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870– 1879, Vancouver, Canada. Association for Computational Li… 5. Stephen A. Cook. 1985. A taxonomy of problems with fast parallel algorithms. Information and Control, 64(1):2–22. [find](https://scholar.google.com/scholar?q=Stephen+A%2E+Cook%2E+1985%2E+A+taxonomy+of+problems+with+fast+parallel+algorithms%2E+Information+and+Control%2C+64%281%29%3A2%E2%80%9322%2E) 6. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. [Commonsense knowledge mining from pre-trained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–… 7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 8. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. [T-REx: A large scale alignment of natural language with knowledge base triples](https://md.research.elamur.ai/free/en/papers/elsahar-2018-t-rex-large-scale-alignment/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluati… 9. Allyson Ettinger. 2020. [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48. 10. Maxwell Forbes, Ari Holtzman, and Yejin Choi. 2019. Do Neural Language Representations Learn Physical Commonsense? arXiv:1908.02899 [cs]. [find](https://scholar.google.com/scholar?q=Maxwell+Forbes%2C+Ari+Holtzman%2C+and+Yejin+Choi%2E+2019%2E+Do+Neural+Language+Representations+Learn+Physical+Commonsense%3F+arXiv%3A1908%2E02899+%5Bcs%5D%2E) 11. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. [Making Pre-trained Language Models Better Few-shot Learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). arXiv:2012.15723 [cs]. 12. Yoav Goldberg. 2019. [Assessing BERT’s Syntactic Abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). arXiv:1901.05287 [cs]. 13. John Hewitt and Christopher D. Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129… 14. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do Attention Heads in BERT Track Syntactic Dependencies? arXiv:1911.12246 [cs]. — [arXiv:1911.12246](https://arxiv.org/abs/1911.12246) [find](https://scholar.google.com/scholar?q=Phu+Mon+Htut%2C+Jason+Phang%2C+Shikha+Bordia%2C+and+Samuel+R%2E+Bowman%2E+2019%2E+Do+Attention+Heads+in+BERT+Track+Syntactic+Dependencies%3F+arXiv%3A1911%2E12246+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A1911%2E12246%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E12) 15. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 633–644. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+Jordan+Boyd%2DGraber%2C+Leonardo+Claudino%2C+Richard+Socher%2C+and+Hal+Daume%CC%81+III%2E+2014%2E+A+neural+network+for+factoid+question+answering+over+paragraphs%2E+In+Proceedings+of+the+2014+conference+on+) 16. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943– 5959, Online. As… [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020a%2E+X%2DFACTR%3A+Multilingual+factual+knowledge+retrieval+from+pretrained+language+models%2E+In+Proceedings+of+the+2020+) 17. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438. 18. Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. [Are pretrained language models symbolic reasoners over knowledge?](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/) In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552–564, Online. Association for Computational Linguistics. 19. Nora Kassner and Hinrich Schütze. 2020a. BERT-kNN: Adding a kNN search component to pretrained language models for better QA. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3424–3430, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nora+Kassner+and+Hinrich+Schu%CC%88tze%2E+2020a%2E+BERT%2DkNN%3A+Adding+a+kNN+search+component+to+pretrained+language+models+for+better+QA%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C) 20. Nora Kassner and Hinrich Schütze. 2020b. [Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics. 21. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. [Neural architectures for named entity recognition](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/). In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San… 22. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190 [cs]. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2DTuning%3A+Optimizing+Continuous+Prompts+for+Generation%2E+arXiv%3A2101%2E00190+%5Bcs%5D%2E) 23. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. [Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862– 6868, Online. Asso… 24. Yongjie Lin, Yi Chern Tan, and Robert Frank. 2019. [Open sesame: Getting inside BERT’s linguistic knowledge](https://md.research.elamur.ai/free/en/papers/lin-2019-open-sesame-getting-inside-bert/). In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 241–253, Florence, Italy. Association for Computational Linguistics. 25. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. [Linguistic knowledge and transferability of contextual representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… 26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pretraining+Approach%2E+) 27. Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020. Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems. arXiv:2008.06239 [cs]. — [arXiv:2008.06239](https://arxiv.org/abs/2008.06239) [find](https://scholar.google.com/scholar?q=Andrea+Madotto%2C+Zihan+Liu%2C+Zhaojiang+Lin%2C+and+Pascale+Fung%2E+2020%2E+Language+Models+as+Few%2DShot+Learner+for+Task%2DOriented+Dialogue+Systems%2E+arXiv%3A2008%2E06239+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2008%2E06239%5D%28https%3A%2F%2Farxiv%2Eorg%2F) 28. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Ling… 29. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… 30. Fabio Petroni, Patrick S. H. Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. [How context affects language models’ factual predictions](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/). In Conference on Automated Knowledge Base Construction, AKBC 2020, Virtual, June 22-24, 2020. 31. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… 32. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nina+Poerner%2C+Ulli+Waltinger%2C+and+Hinrich+Schu%CC%88tze%2E+2020%2E+E%2DBERT%3A+Efficient%2Dyet%2Deffective+entity+embeddings+for+BERT%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+80) 33. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.… [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+learn+from+context%3F+probing+for+sentence+structure+in+contextualized+word+representations%2E+In+7th+In) 34. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu) 35. Timo Schick and Hinrich Schütze. 2020a. Few-Shot Text Generation with Pattern-Exploiting Training. arXiv:2012.11926 [cs]. — [arXiv:2012.11926](https://arxiv.org/abs/2012.11926) [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+2020a%2E+Few%2DShot+Text+Generation+with+Pattern%2DExploiting+Training%2E+arXiv%3A2012%2E11926+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2012%2E11926%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2012%2E11926%29) 36. Timo Schick and Hinrich Schütze. 2020b. [It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). arXiv:2009.07118 [cs]. 37. Timo Schick and Hinrich Schütze. 2021. [Exploiting cloze-questions for few-shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pag… 38. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin… 39. Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. “you are grounded!”: Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850–6861, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Rachel+Rudinger%2C+and+Oyvind+Tafjord%2E+2020%2E+%E2%80%9Cyou+are+grounded%21%E2%80%9D%3A+Latent+name+artifacts+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natur) 40. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78–85. [find](https://scholar.google.com/scholar?q=Denny+Vrandec%CC%8Cic%CC%81+and+Markus+Kro%CC%88tzsch%2E+2014%2E+Wikidata%3A+A+free+collaborative+knowledgebase%2E+Commun%2E+ACM%2C+57%2810%29%3A78%E2%80%9385%2E) 41. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… 42. Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language Models are Open Knowledge Graphs. arXiv:2010.11967 [cs]. — [arXiv:2010.11967](https://arxiv.org/abs/2010.11967) [find](https://scholar.google.com/scholar?q=Chenguang+Wang%2C+Xiao+Liu%2C+and+Dawn+Song%2E+2020%2E+Language+Models+are+Open+Knowledge+Graphs%2E+arXiv%3A2010%2E11967+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2010%2E11967%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E11967%29) 43. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. [Investigating BERT’s knowledge of language: Five analysis methods with N…](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/) 44. Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2019. Evaluating Commonsense in Pre-trained Language Models. arXiv:1911.11931 [cs]. [find](https://scholar.google.com/scholar?q=Xuhui+Zhou%2C+Yue+Zhang%2C+Leyang+Cui%2C+and+Dandan+Huang%2E+2019%2E+Evaluating+Commonsense+in+Pre%2Dtrained+Language+Models%2E+arXiv%3A1911%2E11931+%5Bcs%5D%2E) 45. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [Commonsenseqa: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 46. Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, and Louise Naudin. 2020. Pre-training is (almost) all you need: An application to commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878–3887, Online. Asso… [find](https://scholar.google.com/scholar?q=Alexandre+Tamborrino%2C+Nicola+Pellicano%CC%80%2C+Baptiste+Pannier%2C+Pascal+Voitot%2C+and+Louise+Naudin%2E+2020%2E+Pre%2Dtraining+is+%28almost%29+all+you+need%3A+An+application+to+commonsense+reasoning%2E+In+Proceedings+of+the) 47. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Patrick+Xia%2C+Berlin+Chen%2C+Alex+Wang%2C+Adam+Poliak%2C+R%2E+Thomas+McCoy%2C+Najoung+Kim%2C+Benjamin+Van+Durme%2C+Samuel+R%2E+Bowman%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+What+do+you) *Updated 24 October 2026*