---
title: "A Review on Language Models as Knowledge Bases (2022)"
description: "The authors do not propose a new dataset or model. 174 references, cited by 92 papers here."
authors: ["Badr AlKhamissi", "Millicent Li", "Asli Celikyilmaz", "Mona Diab", "Marjan Ghazvininejad"]
year: 2022
venue: "arXiv:2204.06031"
source: "https://arxiv.org/abs/2204.06031"
doi: "10.48550/arxiv.2204.06031"
arxiv: "2204.06031"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/alkhamissi-2022-review-language-models-knowledge-bases/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/alkhamissi-2022-review-language-models-knowledge-bases/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / A Review on Language Models as Knowledge Bases

# A Review on Language Models as Knowledge Bases

**Authors:** [Badr AlKhamissi](https://md.research.elamur.ai/free/en/authors/badr-alkhamissi/), [Millicent Li](https://md.research.elamur.ai/free/en/authors/millicent-li/), [Asli Celikyilmaz](https://md.research.elamur.ai/free/en/authors/asli-celikyilmaz/), [Mona Diab](https://md.research.elamur.ai/free/en/authors/mona-diab/) and [Marjan Ghazvininejad](https://md.research.elamur.ai/free/en/authors/marjan-ghazvininejad/) · **Year:** 2022 · **Venue:** arXiv:2204.06031 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2204.06031) · [PDF](https://arxiv.org/pdf/2204.06031) · [DOI 10.48550/arxiv.2204.06031](https://doi.org/10.48550/arxiv.2204.06031) · [arXiv 2204.06031](https://arxiv.org/abs/2204.06031) · [OpenAlex](https://openalex.org/W4223974161)

**Citations:** 58 citations (OpenAlex, 2026-10-14) · [92](https://md.research.elamur.ai/free/en/papers/alkhamissi-2022-review-language-models-knowledge-bases/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

This survey by researchers from Meta AI examines the idea that a large pretrained language model can serve as a knowledge base. When trained on large web corpora without any manual annotation, such models store many facts in their weights. Classical KBs such as Wikidata or ATOMIC, by contrast, store knowledge as explicit triples. This makes them easy to query and update, and their answers are predictable. The authors do not propose a new dataset or model. Instead, they describe five properties that an LM needs before it can fully replace a KB: access to knowledge, editing, consistency, reasoning, and explainability and interpretability. They organize recent work along these axes. Their main conclusion is that progress is clear, but models still answer paraphrased questions differently, are hard to update one fact at a time, and remain "black boxes." The survey is useful as a map of this fast-growing field and as a list of open problems. Until these problems are solved, LMs should not be used as a reliable source of facts.

## Key points

- **Five aspects of LMs-as-KBs:** Access, Edit, Consistency, Reasoning, Explainability/Interpretability. The authors treat Petroni et al. (2019) as the starting point of this research direction.
- **Access to knowledge.** Knowledge is extracted from BERT, GPT-3, BART, T5 through cloze queries such as "Albert Einstein was born in [MASK]", through prompting, or through fine-tuning. Different phrasings of the same query can produce different answers.
- **Editing.** Retraining the whole model is too expensive, and changing individual weights affects other facts the model has memorized. The survey describes methods for targeted editing (De Cao et al., Mitchell et al.) and causal tracing (Meng et al., 2022). Causal tracing builds on the finding by Geva et al. (2021) that feedforward layers behave like key–value memories.
- **Explainability.** The survey covers attention, influence functions, and explanations generated by the model itself (T5). Model-generated explanations often contradict one another (Camburu et al., 2020).
- **Open problems.** To answer paraphrases consistently, a model needs separate additional fine-tuning (Elazar et al., 2021). Models are sensitive to negation and word order, and they handle time, causality and rare facts poorly. The authors keep an up-to-date list of papers at https://bkhmsi.github.io/lms-as-kbs/.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 211 works

## Cite

Cite as: Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab and Marjan Ghazvininejad. 2022. A Review on Language Models as Knowledge Bases. arXiv:2204.06031.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{alkhamissi2022review,
  title = {{A Review on Language Models as Knowledge Bases}},
  author = {Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab and Marjan Ghazvininejad},
  year = {2022},
  doi = {10.48550/arxiv.2204.06031},
  eprint = {2204.06031},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.06031}
}
```

- [← previous Revisiting DocRED - Addressing the False Negative Problem in Relation Extraction](https://md.research.elamur.ai/free/en/papers/tan-2022-revisiting-docred-addressing-false-negative/)
- [next → MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://md.research.elamur.ai/free/en/papers/fan-2022-minedojo-building-open-ended-embodied/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

174 references, 84 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leonard Adolphs, Shehzaad Dhuliawala, and Thomas Hofmann. 2021. How to query language models? ArXiv, abs/2108.01928. — [arXiv:2108.01928](https://arxiv.org/abs/2108.01928) [find](https://scholar.google.com/scholar?q=Leonard+Adolphs%2C+Shehzaad+Dhuliawala%2C+and+Thomas+Hofmann%2E+2021%2E+How+to+query+language+models%3F+ArXiv%2C+abs%2F2108%2E01928%2E+%E2%80%94+%5BarXiv%3A2108%2E01928%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2108%2E01928%29)
2. Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. 2022. Cm3: A causal masked multimodal model of the internet. ArXiv, abs/2201.07520. [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Bernie+Huang%2C+Candace+Ross%2C+Vladimir+Karpukhin%2C+Hu+Xu%2C+Naman+Goyal%2C+Dmytro+Okhonko%2C+Mandar+Joshi%2C+Gargi+Ghosh%2C+Mike+Lewis%2C+and+Luke+Zettlemoyer%2E+2022%2E+Cm3%3A+A+causal+masked+multimodal)
3. Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. HTLM: Hyper-Text Pre-Training and Prompting of Language Models. arXiv:2107.06955 [cs]. ArXiv: 2107.06955. [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Dmytro+Okhonko%2C+Mike+Lewis%2C+Mandar+Joshi%2C+Hu+Xu%2C+Gargi+Ghosh%2C+and+Luke+Zettlemoyer%2E+2021%2E+HTLM%3A+Hyper%2DText+Pre%2DTraining+and+Prompting+of+Language+Models%2E+arXiv%3A2107%2E06955+%5Bcs%5D%2E+ArXiv)
4. Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2017. [A neural knowledge language model](https://md.research.elamur.ai/free/en/papers/ahn-2016-neural-knowledge-language-model/).
5. KimAllan Andersen and Daniele Pretolani. 2001. Easy cases of probabilistic satisfiability. Annals of Mathematics and Artificial Intelligence, 33:69–91. [find](https://scholar.google.com/scholar?q=KimAllan+Andersen+and+Daniele+Pretolani%2E+2001%2E+Easy+cases+of+probabilistic+satisfiability%2E+Annals+of+Mathematics+and+Artificial+Intelligence%2C+33%3A69%E2%80%9391%2E)
6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). CoRR, abs/1409.0473.
7. Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. 2020. Interpretability and analysis in neural NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 1–5, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Sebastian+Gehrmann%2C+and+Ellie+Pavlick%2E+2020%2E+Interpretability+and+analysis+in+neural+NLP%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+)
8. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference)
9. Rahul Bhagat and Eduard Hovy. 2013. What Is a Paraphrase? Computational Linguistics, 39(3):463–472. [find](https://scholar.google.com/scholar?q=Rahul+Bhagat+and+Eduard+Hovy%2E+2013%2E+What+Is+a+Paraphrase%3F+Computational+Linguistics%2C+39%283%29%3A463%E2%80%93472%2E)
10. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. 2020. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). ArXiv, abs/1908.05739.
11. S. Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the practical ability of recurrent neural networks to recognize hierarchical languages. ArXiv, abs/2011.03965. [find](https://scholar.google.com/scholar?q=S%2E+Bhattamishra%2C+Kabir+Ahuja%2C+and+Navin+Goyal%2E+2020%2E+On+the+practical+ability+of+recurrent+neural+networks+to+recognize+hierarchical+languages%2E+ArXiv%2C+abs%2F2011%2E03965%2E)
12. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
14. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
15. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In NeurIPS.
16. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. 2020. Make up your mind! adversarial generation of inconsistent natural language explanations. In ACL. [find](https://scholar.google.com/scholar?q=Oana%2DMaria+Camburu%2C+Brendan+Shillingford%2C+Pasquale+Minervini%2C+Thomas+Lukasiewicz%2C+and+Phil+Blunsom%2E+2020%2E+Make+up+your+mind%21+adversarial+generation+of+inconsistent+natural+language+explanations%2E+In+AC)
17. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. [Extracting Training Data from Large Language Models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/). arXiv:2012.07805 [cs]. ArXiv: 2012.07805.
18. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems. ACM SIGKDD Explorations Newsletter, 19(2):25–35. [find](https://scholar.google.com/scholar?q=Hongshen+Chen%2C+Xiaorui+Liu%2C+Dawei+Yin%2C+and+Jiliang+Tang%2E+2017%2E+A+survey+on+dialogue+systems%2E+ACM+SIGKDD+Explorations+Newsletter%2C+19%282%29%3A25%E2%80%9335%2E)
19. Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv, abs/1412.3555. [find](https://scholar.google.com/scholar?q=Junyoung+Chung%2C+%C3%87aglar+G%C3%BCl%C3%A7ehre%2C+Kyunghyun+Cho%2C+and+Yoshua+Bengio%2E+2014%2E+Empirical+evaluation+of+gated+recurrent+neural+networks+on+sequence+modeling%2E+ArXiv%2C+abs%2F1412%2E3555%2E)
20. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does bert look at? an analysis of bert’s attention. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Urvashi+Khandelwal%2C+Omer+Levy%2C+and+Christopher+D%2E+Manning%2E+2019%2E+What+does+bert+look+at%3F+an+analysis+of+bert%E2%80%99s+attention%2E)
21. Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2020. [Transformers as soft reasoners over language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/). ArXiv, abs/2002.05867.
22. Pedro Colon-Hernandez, Catherine Havasi, Jason Alonso, Matthew Huggins, and Cynthia Breazeal. 2021. Combining pre-trained language models and structured knowledge. [find](https://scholar.google.com/scholar?q=Pedro+Colon%2DHernandez%2C+Catherine+Havasi%2C+Jason+Alonso%2C+Matthew+Huggins%2C+and+Cynthia+Breazeal%2E+2021%2E+Combining+pre%2Dtrained+language+models+and+structured+knowledge%2E)
23. Leyang Cui, Sijie Cheng, Yu Wu, and Yue Zhang. 2021. On commonsense cues in BERT for solving commonsense tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 683–693, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Leyang+Cui%2C+Sijie+Cheng%2C+Yu+Wu%2C+and+Yue+Zhang%2E+2021%2E+On+commonsense+cues+in+BERT+for+solving+commonsense+tasks%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pages+683)
24. Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. 2021. Analyzing commonsense emergence in few-shot knowledge models. [find](https://scholar.google.com/scholar?q=Jeff+Da%2C+Ronan+Le+Bras%2C+Ximing+Lu%2C+Yejin+Choi%2C+and+Antoine+Bosselut%2E+2021%2E+Analyzing+commonsense+emergence+in+few%2Dshot+knowledge+models%2E)
25. Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020. [A Survey of the State of Explainable AI for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/danilevsky-2020-survey-state-explainable-ai-natural/). arXiv:2010.00711 [cs]. ArXiv: 2010.00711.
26. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. [Commonsense knowledge mining from pre-trained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–…
27. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing Factual Knowledge in Language Models. arXiv:2104.08164 [cs]. ArXiv: 2104.08164. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Wilker+Aziz%2C+and+Ivan+Titov%2E+2021%2E+Editing+Factual+Knowledge+in+Language+Models%2E+arXiv%3A2104%2E08164+%5Bcs%5D%2E+ArXiv%3A+2104%2E08164%2E)
28. Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William Cohen. 2021. Mention memory: incorporating textual knowledge into transformers through entity mention attention. [find](https://scholar.google.com/scholar?q=Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Nicholas+FitzGerald%2C+Fei+Sha%2C+and+William+Cohen%2E+2021%2E+Mention+memory%3A+incorporating+textual+knowledge+into+transformers+through+entity+mention+attention%2E)
29. Cyprien de Masson d’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In NeurIPS. [find](https://scholar.google.com/scholar?q=Cyprien+de+Masson+d%E2%80%99Autume%2C+Sebastian+Ruder%2C+Lingpeng+Kong%2C+and+Dani+Yogatama%2E+2019%2E+Episodic+memory+in+lifelong+language+learning%2E+In+NeurIPS%2E)
30. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
31. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. [Time-Aware Language Models as Temporal Knowledge Bases](https://md.research.elamur.ai/free/en/papers/dhingra-2022-time-aware-language-models-as/). arXiv:2106.15110 [cs]. ArXiv: 2106.15110.
32. Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. 2020. Differentiable reasoning over a virtual knowledge base. [find](https://scholar.google.com/scholar?q=Bhuwan+Dhingra%2C+Manzil+Zaheer%2C+Vidhisha+Balachandran%2C+Graham+Neubig%2C+Ruslan+Salakhutdinov%2C+and+William+W%2E+Cohen%2E+2020%2E+Differentiable+reasoning+over+a+virtual+knowledge+base%2E)
33. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Russell Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
34. Filip Karlo Došilović, Mario Brčić, and Nikica Hlupić. 2018. Explainable artificial intelligence: A survey. In 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pages 0210–0215. [find](https://scholar.google.com/scholar?q=Filip+Karlo+Do%C5%A1ilovic%CC%81%2C+Mario+Brc%CC%8Cic%CC%81%2C+and+Nikica+Hlupic%CC%81%2E+2018%2E+Explainable+artificial+intelligence%3A+A+survey%2E+In+2018+41st+International+Convention+on+Information+and+Communication+Technology%2C+Elect)
35. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and Improving Consistency in Pretrained Language Models. arXiv:2102.01017 [cs]. ArXiv: 2102.01017. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+Improving+Consistency+in+Pretrained+Language+Models%2E+arXiv%3A2102)
36. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Ja… [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
37. Allyson Ettinger. 2020. [What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48.
38. Fuli Feng, Jizhi Zhang, Xiangnan He, Hanwang Zhang, and Tat-Seng Chua. 2021. Empowering language understanding with counterfactual reasoning. ArXiv, abs/2106.03046. [find](https://scholar.google.com/scholar?q=Fuli+Feng%2C+Jizhi+Zhang%2C+Xiangnan+He%2C+Hanwang+Zhang%2C+and+Tat%2DSeng+Chua%2E+2021%2E+Empowering+language+understanding+with+counterfactual+reasoning%2E+ArXiv%2C+abs%2F2106%2E03046%2E)
39. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. [Entities as experts: Sparse memory access with entity supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/). CoRR, abs/2004.07202.
40. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. [Entities as experts: Sparse memory access with entity supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/). In EMNLP.
41. Chelsea Finn, P. Abbeel, and Sergey Levine. 2017. [Model-agnostic meta-learning for fast adaptation of deep networks](https://md.research.elamur.ai/free/en/papers/finn-2017-model-agnostic-meta-learning-fast/). In ICML.
42. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/).
43. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. [Realtoxicityprompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). ArXiv, abs/2009.11462.
44. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Roei+Schuster%2C+Jonathan+Berant%2C+and+Omer+Levy%2E+2021%2E+Transformer+feed%2Dforward+layers+are+key%2Dvalue+memories%2E)
45. Yoav Goldberg. 2019. [Assessing bert’s syntactic abilities](https://md.research.elamur.ai/free/en/papers/goldberg-2019-assessing-bert-s-syntactic-abilities/). ArXiv, abs/1901.05287.
46. Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Joseph Pal. 2020. Measuring systematic generalization in neural proof generation with transformers. ArXiv, abs/2009.14786. — [arXiv:2009.14786](https://arxiv.org/abs/2009.14786) [find](https://scholar.google.com/scholar?q=Nicolas+Gontier%2C+Koustuv+Sinha%2C+Siva+Reddy%2C+and+Christopher+Joseph+Pal%2E+2020%2E+Measuring+systematic+generalization+in+neural+proof+generation+with+transformers%2E+ArXiv%2C+abs%2F2009%2E14786%2E+%E2%80%94+%5BarXiv%3A2009%2E147)
47. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Greg+Wayne%2C+and+Ivo+Danihelka%2E+2014%2E+Neural+turing+machines%2E)
48. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. [REALM: Retrieval-Augmented Language Model Pre-Training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). arXiv:2002.08909 [cs]. ArXiv: 2002.08909.
49. David Ha, Andrew Dai, and Quoc V. Le. 2017. Hypernetworks. [find](https://scholar.google.com/scholar?q=David+Ha%2C+Andrew+Dai%2C+and+Quoc+V%2E+Le%2E+2017%2E+Hypernetworks%2E)
50. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. [find](https://scholar.google.com/scholar?q=Michael+Hahn%2E+2020%2E+Theoretical+limitations+of+self%2Dattention+in+neural+sequence+models%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A156%E2%80%93171%2E)
51. Frank R. Hampel. 1974. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 69(346):383–393. [find](https://scholar.google.com/scholar?q=Frank+R%2E+Hampel%2E+1974%2E+The+influence+curve+and+its+role+in+robust+estimation%2E+Journal+of+the+American+Statistical+Association%2C+69%28346%29%3A383%E2%80%93393%2E)
52. Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. [Explaining black box predictions and unveiling data artifacts through influence functions](https://md.research.elamur.ai/free/en/papers/han-2020-explaining-black-box-predictions-unveiling/).
53. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification. [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Weilin+Zhao%2C+Ning+Ding%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2021%2E+Ptr%3A+Prompt+tuning+with+rules+for+text+classification%2E)
54. Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. 2021. Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs. arXiv:2111.13654 [cs]. ArXiv: 2111.13654. [find](https://scholar.google.com/scholar?q=Peter+Hase%2C+Mona+Diab%2C+Asli+Celikyilmaz%2C+Xian+Li%2C+Zornitsa+Kozareva%2C+Veselin+Stoyanov%2C+Mohit+Bansal%2C+and+Srinivasan+Iyer%2E+2021%2E+Do+Language+Models+Have+Beliefs%3F+Methods+for+Detecting%2C+Updating%2C+and+Vi)
55. Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. 2020. BERT-MK: Integrating graph contextualized knowledge into pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2281–2290, Online. Association for Computati… [find](https://scholar.google.com/scholar?q=Bin+He%2C+Di+Zhou%2C+Jinghui+Xiao%2C+Xin+Jiang%2C+Qun+Liu%2C+Nicholas+Jing+Yuan%2C+and+Tong+Xu%2E+2020%2E+BERT%2DMK%3A+Integrating+graph+contextualized+knowledge+into+pre%2Dtrained+language+models%2E+In+Findings+of+the+Assoc)
56. Benjamin Heinzerling and Kentaro Inui. 2021. Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries. arXiv:2008.09036 [cs]. ArXiv: 2008.09036. [find](https://scholar.google.com/scholar?q=Benjamin+Heinzerling+and+Kentaro+Inui%2E+2021%2E+Language+Models+as+Knowledge+Bases%3A+On+Entity+Representations%2C+Storage+Capacity%2C+and+Paraphrased+Queries%2E+arXiv%3A2008%2E09036+%5Bcs%5D%2E+ArXiv%3A+2008%2E09036%2E)
57. Chadi Helwe, Chloé Clavel, and Fabian M. Suchanek. 2021. Reasoning with transformer-based models: Deep learning, but shallow reasoning. In 3rd Conference on Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Chadi+Helwe%2C+Chlo%C3%A9+Clavel%2C+and+Fabian+M%2E+Suchanek%2E+2021%2E+Reasoning+with+transformer%2Dbased+models%3A+Deep+learning%2C+but+shallow+reasoning%2E+In+3rd+Conference+on+Automated+Knowledge+Base+Construction%2E)
58. John Hewitt and Percy Liang. 2019. Designing and Interpreting Probes with Control Tasks. arXiv:1909.03368 [cs]. ArXiv: 1909.03368. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+Interpreting+Probes+with+Control+Tasks%2E+arXiv%3A1909%2E03368+%5Bcs%5D%2E+ArXiv%3A+1909%2E03368%2E)
59. John Hewitt and Christopher D. Manning. 2019. [A structural probe for finding syntax in word representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129…
60. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
61. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In ICML. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+de+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+nlp%2E+In+)
62. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. [Cosmos QA: Machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…
63. Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. 2022. [Language models as zero-shot planners: Extracting actionable knowledge for embodied agents](https://md.research.elamur.ai/free/en/papers/huang-2022-language-models-zero-shot-planners/). ArXiv, abs/2201.07207.
64. Sarthak Jain and Byron C. Wallace. 2019. [Attention is not explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/).
65. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. [What does BERT learn about the structure of language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics.
66. Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020. Language generation with multi-hop reasoning on commonsense knowledge graph. [find](https://scholar.google.com/scholar?q=Haozhe+Ji%2C+Pei+Ke%2C+Shaohan+Huang%2C+Furu+Wei%2C+Xiaoyan+Zhu%2C+and+Minlie+Huang%2E+2020%2E+Language+generation+with+multi%2Dhop+reasoning+on+commonsense+knowledge+graph%2E)
67. Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2021. A Survey on Knowledge Graphs: Representation, Acquisition and Applications. arXiv:2002.00388 [cs]. ArXiv: 2002.00388. [find](https://scholar.google.com/scholar?q=Shaoxiong+Ji%2C+Shirui+Pan%2C+Erik+Cambria%2C+Pekka+Marttinen%2C+and+Philip+S%2E+Yu%2E+2021%2E+A+Survey+on+Knowledge+Graphs%3A+Representation%2C+Acquisition+and+Applications%2E+arXiv%3A2002%2E00388+%5Bcs%5D%2E+ArXiv%3A+2002%2E00388%2E)
68. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943– 5959, Online. As… [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020a%2E+X%2DFACTR%3A+Multilingual+factual+knowledge+retrieval+from+pretrained+language+models%2E+In+Proceedings+of+the+2020+)
69. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) arXiv:1911.12543 [cs]. ArXiv: 1911.12543.
70. Nora Kassner and Hinrich Schütze. 2020. [Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/).
71. Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. 2021. BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief. arXiv:2109.14723 [cs]. ArXiv: 2109.14723. [find](https://scholar.google.com/scholar?q=Nora+Kassner%2C+Oyvind+Tafjord%2C+Hinrich+Sch%C3%BCtze%2C+and+Peter+Clark%2E+2021%2E+BeliefBank%3A+Adding+Memory+to+a+Pre%2DTrained+Language+Model+for+a+Systematic+Notion+of+Belief%2E+arXiv%3A2109%2E14723+%5Bcs%5D%2E+ArXiv%3A+2109%2E14)
72. Yoon Kim. 2014. [Convolutional neural networks for sentence classification](https://md.research.elamur.ai/free/en/papers/kim-2014-convolutional-neural-networks-sentence-classification/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751, Doha, Qatar. Association for Computational Linguistics.
73. Pang Wei Koh and Percy Liang. 2020. Understanding black-box predictions via influence functions. [find](https://scholar.google.com/scholar?q=Pang+Wei+Koh+and+Percy+Liang%2E+2020%2E+Understanding+black%2Dbox+predictions+via+influence+functions%2E)
74. Himabindu Lakkaraju, Dylan Slack, Yuxin Chen, Chenhao Tan, and Sameer Singh. 2022. Rethinking explainability as a dialogue: A practitioner’s perspective. ArXiv, abs/2202.01875. — [arXiv:2202.01875](https://arxiv.org/abs/2202.01875) [find](https://scholar.google.com/scholar?q=Himabindu+Lakkaraju%2C+Dylan+Slack%2C+Yuxin+Chen%2C+Chenhao+Tan%2C+and+Sameer+Singh%2E+2022%2E+Rethinking+explainability+as+a+dialogue%3A+A+practitioner%E2%80%99s+perspective%2E+ArXiv%2C+abs%2F2202%2E01875%2E+%E2%80%94+%5BarXiv%3A2202%2E01875%5D%28ht)
75. Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the Gap: Assessing Temporal Generalization in Neural Langu… [find](https://scholar.google.com/scholar?q=Angeliki+Lazaridou%2C+Adhiguna+Kuncoro%2C+Elena+Gribovskaya%2C+Devang+Agrawal%2C+Adam+Liska%2C+Tayfun+Terzi%2C+Mai+Gimenez%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Tomas+Kocisky%2C+Sebastian+Ruder%2C+Dani+Yogatama%2C+Kris+Cao%2C+Sus)
76. Tao Lei, Regina Barzilay, and T. Jaakkola. 2016. [Rationalizing neural predictions](https://md.research.elamur.ai/free/en/papers/lei-2016-rationalizing-neural-predictions/). In EMNLP.
77. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/).
78. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. [Zero-shot relation extraction via reading comprehension](https://md.research.elamur.ai/free/en/papers/levy-2017-zero-shot-relation-extraction-reading/). In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
79. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
80. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). arXiv:2005.11401 [cs]. ArXiv: 2005.114…
81. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E)
82. Zhongyang Li, Xiao Ding, Kuo Liao, Ting Liu, and Bing Qin. 2021. Causalbert: Injecting causal knowledge into pre-trained models with minimal supervision. ArXiv, abs/2107.09852. — [arXiv:2107.09852](https://arxiv.org/abs/2107.09852) [find](https://scholar.google.com/scholar?q=Zhongyang+Li%2C+Xiao+Ding%2C+Kuo+Liao%2C+Ting+Liu%2C+and+Bing+Qin%2E+2021%2E+Causalbert%3A+Injecting+causal+knowledge+into+pre%2Dtrained+models+with+minimal+supervision%2E+ArXiv%2C+abs%2F2107%2E09852%2E+%E2%80%94+%5BarXiv%3A2107%2E09852%5D%28ht)
83. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. [Towards understanding and mitigating social biases in language models](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/). In ICML.
84. Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2021a. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). ArXiv, abs/2109.07958.
85. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Ves Stoyanov, a… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
86. Zachary Chase Lipton. 2018. The mythos of model interpretability. Queue, 16:31 – 57. [find](https://scholar.google.com/scholar?q=Zachary+Chase+Lipton%2E+2018%2E+The+mythos+of+model+interpretability%2E+Queue%2C+16%3A31+%E2%80%93+57%2E)
87. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a. [What makes good in-context examples for gpt-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
88. Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. 2021b. [Probing across time: What does roberta know and when?](https://md.research.elamur.ai/free/en/papers/liu-2021-probing-across-time-what-does/)
89. Linlin Liu, Xin Li, Ruidan He, Lidong Bing, Shafiq Joty, and Luo Si. 2021c. Knowledge Based Multilingual Language Model. arXiv:2111.10962 [cs]. ArXiv: 2111.10962. — [arXiv:2111.10962](https://arxiv.org/abs/2111.10962) [find](https://scholar.google.com/scholar?q=Linlin+Liu%2C+Xin+Li%2C+Ruidan+He%2C+Lidong+Bing%2C+Shafiq+Joty%2C+and+Luo+Si%2E+2021c%2E+Knowledge+Based+Multilingual+Language+Model%2E+arXiv%3A2111%2E10962+%5Bcs%5D%2E+ArXiv%3A+2111%2E10962%2E+%E2%80%94+%5BarXiv%3A2111%2E10962%5D%28https%3A%2F%2Farxiv%2Eor)
90. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021d. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ArXiv, abs/2107.13586.
91. Qi Liu, Dani Yogatama, and Phil Blunsom. 2022. Relational memory augmented language models. ArXiv, abs/2201.09680. [find](https://scholar.google.com/scholar?q=Qi+Liu%2C+Dani+Yogatama%2C+and+Phil+Blunsom%2E+2022%2E+Relational+memory+augmented+language+models%2E+ArXiv%2C+abs%2F2201%2E09680%2E)
92. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. [K-bert: Enabling language representation with knowledge graph](https://md.research.elamur.ai/free/en/papers/liu-2019-k-bert-enabling-language-representation/). In AAAI.
93. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). ArXiv, abs/1907.11692.
94. Robert L Logan, Ivana Balavzevi’c, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. [Cutting down on prompts and parameters: Simple few-shot learning with language models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/). ArXiv, abs/2106.13353.
95. Robert L. Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack’s wife hillary: Using knowledge-graphs for fact-aware language modeling. [find](https://scholar.google.com/scholar?q=Robert+L%2E+Logan%2C+Nelson+F%2E+Liu%2C+Matthew+E%2E+Peters%2C+Matt+Gardner%2C+and+Sameer+Singh%2E+2019%2E+Barack%E2%80%99s+wife+hillary%3A+Using+knowledge%2Dgraphs+for+fact%2Daware+language+modeling%2E)
96. Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen tau Yih, Barlas Oğuz, Veselin Stoyanov, and Gargi Ghosh. 2021. Multi-task retrieval for knowledge-intensive tasks. In ACL/IJCNLP. [find](https://scholar.google.com/scholar?q=Jean+Maillard%2C+Vladimir+Karpukhin%2C+Fabio+Petroni%2C+Wen+tau+Yih%2C+Barlas+Og%CC%86uz%2C+Veselin+Stoyanov%2C+and+Gargi+Ghosh%2E+2021%2E+Multi%2Dtask+retrieval+for+knowledge%2Dintensive+tasks%2E+In+ACL%2FIJCNLP%2E)
97. John McCarthy. 1959. Programs with common sense. In Proceedings of the Teddington Conference on the Mechanization of Thought Processes, pages 75–91, London. Her Majesty’s Stationary Office. [find](https://scholar.google.com/scholar?q=John+McCarthy%2E+1959%2E+Programs+with+common+sense%2E+In+Proceedings+of+the+Teddington+Conference+on+the+Mechanization+of+Thought+Processes%2C+pages+75%E2%80%9391%2C+London%2E+Her+Majesty%E2%80%99s+Stationary+Office%2E)
98. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+knowledge+in+gpt%2E+arXiv+preprint+arXiv%3A2202%2E05262%2E)
99. William Cooper Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. [Provable limitations of acquiring meaning from ungrounded form: What will future language models understand?](https://md.research.elamur.ai/free/en/papers/merrill-2021-provable-limitations-acquiring-meaning-ungrounded/) Transactions of the Association for Computational Linguistics, 9:1047–1060.
100. Kostas S. Metaxiotis, Dimitris Askounis, and John E. Psarras. 2002. Expert systems in production planning and scheduling: A state-of-the-art survey. Journal of Intelligent Manufacturing, 13:253–260. [find](https://scholar.google.com/scholar?q=Kostas+S%2E+Metaxiotis%2C+Dimitris+Askounis%2C+and+John+E%2E+Psarras%2E+2002%2E+Expert+systems+in+production+planning+and+scheduling%3A+A+state%2Dof%2Dthe%2Dart+survey%2E+Journal+of+Intelligent+Manufacturing%2C+13%3A253%E2%80%93260%2E)
101. Paul Michel, Omer Levy, and Graham Neubig. 2019. [Are sixteen heads really better than one?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
102. Kanishka Misra. 2021. On semantic cognition, inductive generalization, and language models. ArXiv, abs/2111.02603. — [arXiv:2111.02603](https://arxiv.org/abs/2111.02603) [find](https://scholar.google.com/scholar?q=Kanishka+Misra%2E+2021%2E+On+semantic+cognition%2C+inductive+generalization%2C+and+language+models%2E+ArXiv%2C+abs%2F2111%2E02603%2E+%E2%80%94+%5BarXiv%3A2111%2E02603%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E02603%29)
103. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast Model Editing at Scale. arXiv:2110.11309 [cs]. ArXiv: 2110.11309. [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Charles+Lin%2C+Antoine+Bosselut%2C+Chelsea+Finn%2C+and+Christopher+D%2E+Manning%2E+2021%2E+Fast+Model+Editing+at+Scale%2E+arXiv%3A2110%2E11309+%5Bcs%5D%2E+ArXiv%3A+2110%2E11309%2E)
104. Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, and Balaraman Ravindran. 2020. Towards transparent and explainable attention models. [find](https://scholar.google.com/scholar?q=Akash+Kumar+Mohankumar%2C+Preksha+Nema%2C+Sharan+Narasimhan%2C+Mitesh+M%2E+Khapra%2C+Balaji+Vasan+Srinivasan%2C+and+Balaraman+Ravindran%2E+2020%2E+Towards+transparent+and+explainable+attention+models%2E)
105. Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. [Wt5?! training text-to-text models to explain their predictions](https://md.research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/). ArXiv, abs/2004.14546.
106. A. Newell and H. Simon. 1956. [The logic theory machine–a complex information processing system](https://md.research.elamur.ai/free/en/papers/newell-1956-logic-theory-machine-complex-information/). IRE Transactions on Information Theory, 2(3):61– 79.
107. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/). ArXiv, abs/2112.00114.
108. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. [A smorgasbord of features for statistical machine translation](https://md.research.elamur.ai/free/en/papers/och-2004-smorgasbord-features-statistical-machine-translation/). In Proceedings of the Human Language Techn…
109. Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Prompting contrastive explanations for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4179–4192, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Julian+Michael%2C+Marjan+Ghazvininejad%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2021%2E+Prompting+contrastive+explanations+for+commonsense+reasoning+tasks%2E+In+Findings+of+the+Associ)
110. Matthew E. Peters, Mark Neumann, Robert L. Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Robert+L%2E+Logan%2C+Roy+Schwartz%2C+Vidur+Joshi%2C+Sameer+Singh%2C+and+Noah+A%2E+Smith%2E+2019%2E+Knowledge+enhanced+contextual+word+representations%2E)
111. Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih. 2018. [Dissecting contextual word embeddings: Architecture and representation](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/). In EMNLP.
112. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
113. Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron C. Wallace. 2021a. Combining feature and instance attribution to detect artifacts. [find](https://scholar.google.com/scholar?q=Pouya+Pezeshkpour%2C+Sarthak+Jain%2C+Sameer+Singh%2C+and+Byron+C%2E+Wallace%2E+2021a%2E+Combining+feature+and+instance+attribution+to+detect+artifacts%2E)
114. Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, and Sameer Singh. 2021b. An empirical comparison of instance attribution methods for nlp. [find](https://scholar.google.com/scholar?q=Pouya+Pezeshkpour%2C+Sarthak+Jain%2C+Byron+C%2E+Wallace%2C+and+Sameer+Singh%2E+2021b%2E+An+empirical+comparison+of+instance+attribution+methods+for+nlp%2E)
115. David Picado-Muiño. 2011. Measuring and repairing inconsistency in probabilistic knowledge bases. Int. J. Approx. Reasoning, 52:828–840. [find](https://scholar.google.com/scholar?q=David+Picado%2DMui%C3%B1o%2E+2011%2E+Measuring+and+repairing+inconsistency+in+probabilistic+knowledge+bases%2E+Int%2E+J%2E+Approx%2E+Reasoning%2C+52%3A828%E2%80%93840%2E)
116. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. [E-BERT: Efficient-yet-effective entity embeddings for BERT](https://md.research.elamur.ai/free/en/papers/poerner-2019-e-bert-efficient-yet-effective/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818, Online. Association for Computational Linguistics.
117. Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. 2022. [Formal mathematics statement curriculum learning](https://md.research.elamur.ai/free/en/papers/polu-2022-formal-mathematics-statement-curriculum-learning/). ArXiv, abs/2202.01344.
118. Stanislas Polu and Ilya Sutskever. 2020. [Generative language modeling for automated theorem proving](https://md.research.elamur.ai/free/en/papers/polu-2020-generative-language-modeling-automated-theorem/). ArXiv, abs/2009.03393.
119. Guanghui Qin and Jason Eisner. 2021. [Learning how to ask: Querying lms with mixtures of soft prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/).
120. Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. [Erica: Improving entity and relation understanding for pre-trained language models via contrastive learning](https://md.research.elamur.ai/free/en/papers/qin-2021-erica-improving-entity-relation-understanding/). In ACL/IJCNLP.
121. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford+and+Karthik+Narasimhan%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
122. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
123. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). ArXiv, abs/1910.10683.
124. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. [On the systematicity of probing contextualized word representations: The case of hypernymy in BERT](https://md.research.elamur.ai/free/en/papers/ravichander-2020-systematicity-probing-contextualized-word-representations/). In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102,…
125. Ryokan Ri, Ikuya Yamada, and Yoshimasa Tsuruoka. 2021. mluke: The power of entity representations in multilingual pretrained language models. ArXiv, abs/2110.08151. [find](https://scholar.google.com/scholar?q=Ryokan+Ri%2C+Ikuya+Yamada%2C+and+Yoshimasa+Tsuruoka%2E+2021%2E+mluke%3A+The+power+of+entity+representations+in+multilingual+pretrained+language+models%2E+ArXiv%2C+abs%2F2110%2E08151%2E)
126. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A Primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). arXiv:2002.12327 [cs]. ArXiv: 2002.12327.
127. Corby Rosset, Chenyan Xiong, Minh Hieu Phan, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020. Knowledge-aware language model pretraining. ArXiv, abs/2007.00655. — [arXiv:2007.00655](https://arxiv.org/abs/2007.00655) [find](https://scholar.google.com/scholar?q=Corby+Rosset%2C+Chenyan+Xiong%2C+Minh+Hieu+Phan%2C+Xia+Song%2C+Paul+Bennett%2C+and+Saurabh+Tiwary%2E+2020%2E+Knowledge%2Daware+language+model+pretraining%2E+ArXiv%2C+abs%2F2007%2E00655%2E+%E2%80%94+%5BarXiv%3A2007%2E00655%5D%28https%3A%2F%2Farxiv%2Eorg)
128. Tara Safavi and Danai Koutra. 2021. Relational world knowledge representation in contextual language models: A review. [find](https://scholar.google.com/scholar?q=Tara+Safavi+and+Danai+Koutra%2E+2021%2E+Relational+world+knowledge+representation+in+contextual+language+models%3A+A+review%2E)
129. Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, and Mohit Bansal. 2020. [Prover: Proof generation for interpretable reasoning over rules](https://md.research.elamur.ai/free/en/papers/saha-2020-prover-proof-generation-interpretable-reasoning/). ArXiv, abs/2010.02830.
130. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3027–30… [find](https://scholar.google.com/scholar?q=Maarten+Sap%2C+Ronan+Le+Bras%2C+Emily+Allaway%2C+Chandra+Bhagavatula%2C+Nicholas+Lourie%2C+Hannah+Rashkin%2C+Brendan+Roof%2C+Noah+A+Smith%2C+and+Yejin+Choi%2E+2019%2E+Atomic%3A+An+atlas+of+machine+commonsense+for+if%2Dthen+r)
131. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. [Analysing mathematical reasoning abilities of neural models](https://md.research.elamur.ai/free/en/papers/saxton-2019-analysing-mathematical-reasoning-abilities-neural/). ArXiv, abs/1904.01557.
132. Timo Schick and Hinrich Schütze. 2021a. [Exploiting cloze questions for few shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/).
133. Timo Schick and Hinrich Schütze. 2021b. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/).
134. Patrick Schramowski, Felix Friedrich, Christopher Tauchmann, and Kristian Kersting. 2021. Interactively generating explanations for transformer language models. CoRR, abs/2110.02058. — [arXiv:2110.02058](https://arxiv.org/abs/2110.02058) [find](https://scholar.google.com/scholar?q=Patrick+Schramowski%2C+Felix+Friedrich%2C+Christopher+Tauchmann%2C+and+Kristian+Kersting%2E+2021%2E+Interactively+generating+explanations+for+transformer+language+models%2E+CoRR%2C+abs%2F2110%2E02058%2E+%E2%80%94+%5BarXiv%3A2110%2E020)
135. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany. Association for Computational L… [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computationa)
136. Sofia Serrano and Noah A. Smith. 2019. [Is attention interpretable?](https://md.research.elamur.ai/free/en/papers/serrano-2019-is-attention-interpretable/)
137. Taylor Shin, Yasaman Razeghi, Robert L. Logan, Eric Wallace, and Sameer Singh. 2020. [Autoprompt: Eliciting knowledge from language models with automatically generated prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/).
138. Haitian Sun, Pat Verga, Bhuwan Dhingra, Ruslan Salakhutdinov, and William W. Cohen. 2021. Reasoning over virtual knowledge bases with open predicate relations. CoRR, abs/2102.07043. [find](https://scholar.google.com/scholar?q=Haitian+Sun%2C+Pat+Verga%2C+Bhuwan+Dhingra%2C+Ruslan+Salakhutdinov%2C+and+William+W%2E+Cohen%2E+2021%2E+Reasoning+over+virtual+knowledge+bases+with+open+predicate+relations%2E+CoRR%2C+abs%2F2102%2E07043%2E)
139. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. [Ernie: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). ArXiv, abs/1904.09223.
140. Mujeen Sung, Jinhyuk Lee, Sean Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang. 2021. [Can language models be biomedical knowledge bases?](https://md.research.elamur.ai/free/en/papers/sung-2021-can-language-models-be-biomedical/)
141. Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. [ProofWriter: Generating implications, proofs, and abductive statements over natural language](https://md.research.elamur.ai/free/en/papers/tafjord-2021-proofwriter-generating-implications-proofs-abductive/). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3621–3634, Online. Association for Computational Linguistics.
142. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020a. [oLMpics – On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). arXiv:1912.13283 [cs]. ArXiv: 1912.13283.
143. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
144. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020b. [Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2020-leap-thought-teaching-pre-trained/). arXiv:2006.06609 [cs]. ArXiv: 2006.06609.
145. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. [find](https://scholar.google.com/scholar?q=Wilson+L+Taylor%2E+1953%2E+%E2%80%9Ccloze+procedure%E2%80%9D%3A+A+new+tool+for+measuring+readability%2E+Journalism+quarterly%2C+30%284%29%3A415%E2%80%93433%2E)
146. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. [Bert rediscovers the classical nlp pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/).
147. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). In International Conference on Lea…
148. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefine-dukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY,…
149. Pat Verga, Haitian Sun, Livio Baldini Soares, and William W. Cohen. 2020. Facts as experts: Adaptable and interpretable neural memory over symbolic knowledge. CoRR, abs/2007.00849. — [arXiv:2007.00849](https://arxiv.org/abs/2007.00849) [find](https://scholar.google.com/scholar?q=Pat+Verga%2C+Haitian+Sun%2C+Livio+Baldini+Soares%2C+and+William+W%2E+Cohen%2E+2020%2E+Facts+as+experts%3A+Adaptable+and+interpretable+neural+memory+over+symbolic+knowledge%2E+CoRR%2C+abs%2F2007%2E00849%2E+%E2%80%94+%5BarXiv%3A2007%2E00849)
150. Jesse Vig and Yonatan Belinkov. 2019. [Analyzing the structure of attention in a transformer language model](https://md.research.elamur.ai/free/en/papers/vig-2019-analyzing-structure-attention-transformer-language/).
151. Denny Vrandečić and Markus Krötzsch. 2014. [Wikidata: A free collaborative knowledgebase](https://md.research.elamur.ai/free/en/papers/vrandecic-2014-wikidata-free-collaborative-knowledgebase/). Commun. ACM, 57(10):78–85.
152. Jonas Wallat, Jaspreet Singh, and Avishek Anand. 2021. BERTnesia: Investigating the capture and forgetting of knowledge in BERT. arXiv:2106.02902 [cs]. ArXiv: 2106.02902. [find](https://scholar.google.com/scholar?q=Jonas+Wallat%2C+Jaspreet+Singh%2C+and+Avishek+Anand%2E+2021%2E+BERTnesia%3A+Investigating+the+capture+and+forgetting+of+knowledge+in+BERT%2E+arXiv%3A2106%2E02902+%5Bcs%5D%2E+ArXiv%3A+2106%2E02902%2E)
153. Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language Models are Open Knowledge Graphs. arXiv:2010.11967 [cs]. ArXiv: 2010.11967. [find](https://scholar.google.com/scholar?q=Chenguang+Wang%2C+Xiao+Liu%2C+and+Dawn+Song%2E+2020%2E+Language+Models+are+Open+Knowledge+Graphs%2E+arXiv%3A2010%2E11967+%5Bcs%5D%2E+ArXiv%3A+2010%2E11967%2E)
154. Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021a. Entailment as few-shot learner. [find](https://scholar.google.com/scholar?q=Sinong+Wang%2C+Han+Fang%2C+Madian+Khabsa%2C+Hanzi+Mao%2C+and+Hao+Ma%2E+2021a%2E+Entailment+as+few%2Dshot+learner%2E)
155. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juan-Zi Li, and Jian Tang. 2021b. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 9:176– 194. [find](https://scholar.google.com/scholar?q=Xiaozhi+Wang%2C+Tianyu+Gao%2C+Zhaocheng+Zhu%2C+Zhiyuan+Liu%2C+Juan%2DZi+Li%2C+and+Jian+Tang%2E+2021b%2E+Kepler%3A+A+unified+model+for+knowledge+embedding+and+pre%2Dtrained+language+representation%2E+Transactions+of+the+Ass)
156. Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021c. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation. Transactions of the Association for Computational Linguistics, 9:176–194. [find](https://scholar.google.com/scholar?q=Xiaozhi+Wang%2C+Tianyu+Gao%2C+Zhaocheng+Zhu%2C+Zhengyan+Zhang%2C+Zhiyuan+Liu%2C+Juanzi+Li%2C+and+Jian+Tang%2E+2021c%2E+KEPLER%3A+A+Unified+Model+for+Knowledge+Embedding+and+Pre%2Dtrained+Language+Representation%2E+Transact)
157. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). ArXiv, abs/2201.11903.
158. Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew Arnold. 2021. Knowledge enhanced pretrained language models: A compreshen-sive survey. [find](https://scholar.google.com/scholar?q=Xiaokai+Wei%2C+Shen+Wang%2C+Dejiao+Zhang%2C+Parminder+Bhatia%2C+and+Andrew+Arnold%2E+2021%2E+Knowledge+enhanced+pretrained+language+models%3A+A+compreshen%2Dsive+survey%2E)
159. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://md.research.elamur.ai/free/en/papers/west-2022-symbolic-knowledge-distillation-general-language/). arXiv:2110.07178 [cs]. ArXiv: 2110.07178.
160. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. [find](https://scholar.google.com/scholar?q=Jason+Weston%2C+Sumit+Chopra%2C+and+Antoine+Bordes%2E+2015%2E+Memory+networks%2E)
161. Sarah Wiegreffe and Yuval Pinter. 2019. [Attention is not not explanation](https://md.research.elamur.ai/free/en/papers/wiegreffe-2019-attention-is-not-not-explanation/).
162. John Wieting and Kevin Gimpel. 2018. [ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations](https://md.research.elamur.ai/free/en/papers/wieting-2018-paranmt-50m-pushing-limits-paraphrastic/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia. Ass…
163. Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep contextualized entity representations with entity-aware self-attention. In EMNLP. [find](https://scholar.google.com/scholar?q=Ikuya+Yamada%2C+Akari+Asai%2C+Hiroyuki+Shindo%2C+Hideaki+Takeda%2C+and+Yuji+Matsumoto%2E+2020%2E+Luke%3A+Deep+contextualized+entity+representations+with+entity%2Daware+self%2Dattention%2E+In+EMNLP%2E)
164. Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg-bert: Bert for knowledge graph completion. ArXiv, abs/1909.03193. [find](https://scholar.google.com/scholar?q=Liang+Yao%2C+Chengsheng+Mao%2C+and+Yuan+Luo%2E+2019%2E+Kg%2Dbert%3A+Bert+for+knowledge+graph+completion%2E+ArXiv%2C+abs%2F1909%2E03193%2E)
165. Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. [find](https://scholar.google.com/scholar?q=Michihiro+Yasunaga%2C+Hongyu+Ren%2C+Antoine+Bosselut%2C+Percy+Liang%2C+and+Jure+Leskovec%2E+2021%2E+Qa%2Dgnn%3A+Reasoning+with+language+models+and+knowledge+graphs+for+question+answering%2E)
166. Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong. 2021. Adaptive Semiparametric Language Models. arXiv:2102.02557 [cs]. ArXiv: 2102.02557. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+and+Lingpeng+Kong%2E+2021%2E+Adaptive+Semiparametric+Language+Models%2E+arXiv%3A2102%2E02557+%5Bcs%5D%2E+ArXiv%3A+2102%2E02557%2E)
167. Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2021. Differentiable prompt makes pre-trained language models better few-shot learners. ArXiv, abs/2108.13161. — [arXiv:2108.13161](https://arxiv.org/abs/2108.13161) [find](https://scholar.google.com/scholar?q=Ningyu+Zhang%2C+Luoqiu+Li%2C+Xiang+Chen%2C+Shumin+Deng%2C+Zhen+Bi%2C+Chuanqi+Tan%2C+Fei+Huang%2C+and+Huajun+Chen%2E+2021%2E+Differentiable+prompt+makes+pre%2Dtrained+language+models+better+few%2Dshot+learners%2E+ArXiv%2C+abs%2F2)
168. Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 188–193, Portland, Oregon, USA. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Yue+Zhang+and+Joakim+Nivre%2E+2011%2E+Transition%2Dbased+dependency+parsing+with+rich+non%2Dlocal+features%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+La)
169. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. CoRR, abs/1905.07129. [find](https://scholar.google.com/scholar?q=Zhengyan+Zhang%2C+Xu+Han%2C+Zhiyuan+Liu%2C+Xin+Jiang%2C+Maosong+Sun%2C+and+Qun+Liu%2E+2019%2E+ERNIE%3A+enhanced+language+representation+with+informative+entities%2E+CoRR%2C+abs%2F1905%2E07129%2E)
170. Sumu Zhao, Damian Pascual, Gino Brunner, and Roger Wattenhofer. 2021. Of non-linearity and commutativity in bert. [find](https://scholar.google.com/scholar?q=Sumu+Zhao%2C+Damian+Pascual%2C+Gino+Brunner%2C+and+Roger+Wattenhofer%2E+2021%2E+Of+non%2Dlinearity+and+commutativity+in+bert%2E)
171. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. [Factual probing is mask: Learning vs. learning to recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/).
172. Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. [Evaluating commonsense in pre-trained language models](https://md.research.elamur.ai/free/en/papers/zhou-2019-evaluating-commonsense-pre-trained-language/). Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9733– 9740.
173. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying Memories in Transformer Models. arXiv:2012.00363 [cs]. ArXiv: 2012.00363. [find](https://scholar.google.com/scholar?q=Chen+Zhu%2C+Ankit+Singh+Rawat%2C+Manzil+Zaheer%2C+Srinadh+Bhojanapalli%2C+Daliang+Li%2C+Felix+Yu%2C+and+Sanjiv+Kumar%2E+2020%2E+Modifying+Memories+in+Transformer+Models%2E+arXiv%3A2012%2E00363+%5Bcs%5D%2E+ArXiv%3A+2012%2E00363%2E)
174. Hugo Zylberajch, Piyawat Lertvittayakumjorn, and Francesca Toni. 2021. HILDIF: Interactive debugging of NLI models using influence functions. In Proceedings of the First Workshop on Interactive Learning for Natural Language Processing, pages 1–6, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hugo+Zylberajch%2C+Piyawat+Lertvittayakumjorn%2C+and+Francesca+Toni%2E+2021%2E+HILDIF%3A+Interactive+debugging+of+NLI+models+using+influence+functions%2E+In+Proceedings+of+the+First+Workshop+on+Interactive+Learni)

*Updated 2 November 2026*