---
title: "Multilingual LAMA — Kassner 2021"
description: "The paper asks whether factual knowledge in a multilingual pretrained model can be retrieved through cloze-style prompts in languages other than English…"
authors: ["Nora Kassner", "Philipp Dufter", "Hinrich Schütze"]
year: 2021
venue: "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume"
source: "https://aclanthology.org/2021.eacl-main.284/"
doi: "10.18653/v1/2021.eacl-main.284"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

# Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

**Authors:** [Nora Kassner](https://md.research.elamur.ai/free/en/authors/nora-kassner/), Philipp Dufter, [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) · **Year:** 2021 · **Venue:** Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.eacl-main.284/) · [PDF](https://aclanthology.org/2021.eacl-main.284.pdf) · [DOI 10.18653/v1/2021.eacl-main.284](https://doi.org/10.18653/v1/2021.eacl-main.284) · [OpenAlex](https://openalex.org/W3127729136)

**Citations:** 74 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks whether factual knowledge in a multilingual pretrained model can be retrieved through cloze-style prompts in languages other than English, as earlier LAMA work did only for English. The authors build mLAMA by machine-translating the TREx and GoogleRE relation templates with Google Translate and taking entity names from Wikidata and the Google Knowledge Graph. The resulting probe covers 53 languages and is used to test mBERT. They also propose typed querying (TyQ), which ranks only candidate objects of the type a relation requires instead of the whole vocabulary, and they score multi-token entities by averaging log probabilities over several mask slots. TyQ turns out to be a clearly better probe than untyped querying. mBERT's accuracy differs greatly between languages, and its answers lean toward entities tied to the query language. Taking a majority vote across languages beats both English-queried mBERT and monolingual BERT. The work offers an openly available multilingual benchmark and shows that mBERT does not store facts independently of language.

## Key points
- **Data:** the source LAMA contains 39,567 triples, with 34,039 from TREx (41 relations) and 5,528 from GoogleRE (3 relations). Only languages where more than 20% of triples could be translated were kept, giving 53 languages with an average coverage of 43%. Code and data are released at github.com/norakassner/mlama.
- **TyQ vs. UnTyQ:** for multi-token objects, mean p1 is .17 with typed querying and only .03 with untyped querying. Native speakers corrected the German, Hindi and Japanese templates, and this improved untyped results only slightly.
- **Uneven coverage:** performance relative to English is acceptable for 21 languages but below 60% for the other 32. Japanese, Thai, Georgian and Tamil are among the weakest. Ukrainian appears to fall in the weaker group according to Figure 3.
- **Language bias:** the query language shifts predictions toward entities associated with it. For example, Italian prompts tend to return Italy as the country of origin.
- **Pooling (p1):** a majority vote across languages scores 41.1 on LAMA, compared with 35.0 for mBERT[en] and 38.5 for BERT. On LAMA-UHN it scores 32.1, compared with 25.7 and 29.0.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/)

## Cited by

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 94 not yet.

- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
- [Challenges and Strategies in Cross-Cultural NLP](https://md.research.elamur.ai/free/en/papers/hershcovich-2022-challenges-strategies-cross-cultural-nlp/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [FIBER: A Multilingual Evaluation Resource for Factual Inference Bias](https://md.research.elamur.ai/free/en/papers/munis-2026-fiber-multilingual-evaluation-resource-factual/)
- [RelEx-PT: A Portuguese Sentence-Level Relation Extraction Dataset](https://md.research.elamur.ai/free/en/papers/pinto-2026-relex-pt-portuguese-sentence-level/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kassner2021multilingual,
  title = {{Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models}},
  author = {Kassner, Nora and Dufter, Philipp and Schütze, Hinrich},
  booktitle = {Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume},
  year = {2021},
  doi = {10.18653/v1/2021.eacl-main.284},
  url = {https://doi.org/10.18653/v1/2021.eacl-main.284}
}
```

- [← previous MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/zhu-2021-mediasum-large-scale-media-interview/)
- [next → Debiasing Pre-trained Contextualised Embeddings](https://md.research.elamur.ai/free/en/papers/kaneko-2021-debiasing-pre-trained-contextualised-embeddings/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/)

## References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
2. Zied Bouraoui, José Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In The Thirty-Fourth AAAI Conference phrased queries. Computing Research Repository, arXiv:2008.09036. on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020, pages 745… [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose%CC%81+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2020%2E+Inducing+relational+knowledge+from+BERT%2E+In+The+Thirty%2DFourth+AAAI+Conference+phrased+queries%2E+Computing+Research+Repository%2C+arXiv%3A)
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
4. Jonathan H. Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454– 47… [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Jennimaria+Palomaki%2C+Vitaly+Nikolaev%2C+Eunsol+Choi%2C+Dan+Garrette%2C+Michael+Collins%2C+and+Tom+Kwiatkowski%2E+2020%2E+TyDi+QA%3A+A+benchmark+for+information%2Dseeking+question+answering+in+typol) [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
5. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
7. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluati… [find](https://scholar.google.com/scholar?q=Hady+Elsahar%2C+Pavlos+Vougiouklis%2C+Arslen+Remaci%2C+Christophe+Gravier%2C+Jonathon+Hare%2C+Frederique+Laforest%2C+and+Elena+Simperl%2E+2018%2E+T%2DREx%3A+A+large+scale+alignment+of+natural+language+with+knowledge+base)
8. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
9. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, V… [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalisation%2E+In)
10. Gautier Izacard and E. Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. ArXiv, abs/2007.01282. [find](https://scholar.google.com/scholar?q=Gautier+Izacard+and+E%2E+Grave%2E+2020%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+ArXiv%2C+abs%2F2007%2E01282%2E) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/)
11. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943– 5959, Online. As… [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020a%2E+X%2DFACTR%3A+Multilingual+factual+knowledge+retrieval+from+pretrained+language+models%2E+In+Proceedings+of+the+2020+)
12. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know. Transactions of the Association for Computational Linguistics, 8:423–438. [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Frank+F%2E+Xu%2C+Jun+Araki%2C+and+Graham+Neubig%2E+2020b%2E+How+can+we+know+what+language+models+know%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+8%3A423%E2%80%93438%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/)
13. Nora Kassner and Hinrich Schütze. 2020. BERT-kNN: Adding a kNN search component to pretrained language models for better QA. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 3424–3430. Associati… [find](https://scholar.google.com/scholar?q=Nora+Kassner+and+Hinrich+Schu%CC%88tze%2E+2020%2E+BERT%2DkNN%3A+Adding+a+kNN+search+component+to+pretrained+language+models+for+better+QA%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Lang)
14. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nora+Kassner+and+Hinrich+Schu%CC%88tze%2E+2020%2E+Negated+and+misprimed+probes+for+pretrained+language+models%3A+Birds+can+talk%2C+but+cannot+fly%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+C) [our card](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/)
15. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: retrieval-augmented language model pre-training. Computing Research Repository, arXiv:2002.08909. [find](https://scholar.google.com/scholar?q=Kelvin+Guu%2C+Kenton+Lee%2C+Zora+Tung%2C+Panupong+Pasupat%2C+and+Ming%2DWei+Chang%2E+2020%2E+REALM%3A+retrieval%2Daugmented+language+model+pre%2Dtraining%2E+Computing+Research+Repository%2C+arXiv%3A2002%2E08909%2E)
16. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020a. Pre-training via paraphrasing. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual… [find](https://scholar.google.com/scholar?q=Mike+Lewis%2C+Marjan+Ghazvininejad%2C+Gargi+Ghosh%2C+Armen+Aghajanyan%2C+Sida+Wang%2C+and+Luke+Zettlemoyer%2E+2020a%2E+Pre%2Dtraining+via+paraphrasing%2E+In+Advances+in+Neural+Information+Processing+Systems+33%3A+Annual+) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/)
17. Benjamin Heinzerling and Kentaro Inui. 2020. Language models as knowledge bases: On entity representations, storage capacity, and para-Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020b. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of… [find](https://scholar.google.com/scholar?q=Benjamin+Heinzerling+and+Kentaro+Inui%2E+2020%2E+Language+models+as+knowledge+bases%3A+On+entity+representations%2C+storage+capacity%2C+and+para%2DPatrick+Lewis%2C+Barlas+Oguz%2C+Ruty+Rinott%2C+Sebastian+Riedel%2C+and+Ho) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
18. Shayne Longpre, Yi Lu, and Joachim Daiber. 2020. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. CoRR, abs/2007.15207. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Yi+Lu%2C+and+Joachim+Daiber%2E+2020%2E+MKQA%3A+A+linguistically+diverse+benchmark+for+multilingual+open+domain+question+answering%2E+CoRR%2C+abs%2F2007%2E15207%2E)
19. Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, pages 216–225. The Association for Computer Linguis… [find](https://scholar.google.com/scholar?q=Roberto+Navigli+and+Simone+Paolo+Ponzetto%2E+2010%2E+Babelnet%3A+Building+a+very+large+multilingual+semantic+network%2E+In+ACL+2010%2C+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational)
20. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Jingfei+Du%2C+William+Yang+Wang%2C+and+Veselin+Stoyanov%2E+2020%2E+Pretrained+encyclopedia%3A+Weakly+supervised+knowledge%2Dpretrained+language+model%2E+In+8th+International+Conference+on+Learning+Rep)
21. Yadollah Yaghoobzadeh and Hinrich Schütze. 2016. Intrinsic subspace evaluation of word embedding representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 236– 246, Berlin, Germany. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Yadollah+Yaghoobzadeh+and+Hinrich+Schu%CC%88tze%2E+2016%2E+Intrinsic+subspace+evaluation+of+word+embedding+representations%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
22. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
23. Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Automated Knowledge Base Construction. [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Patrick+Lewis%2C+Aleksandra+Piktus%2C+Tim+Rockta%CC%88schel%2C+Yuxiang+Wu%2C+Alexander+H%2E+Miller%2C+and+Sebastian+Riedel%2E+2020%2E+How+context+affects+language+models%E2%80%99+factual+predictions%2E+In+Automated+K) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2020-context-affects-language-models-factual/)
24. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference o… [find](https://scholar.google.com/scholar?q=Fabio+Petroni%2C+Tim+Rockta%CC%88schel%2C+Sebastian+Riedel%2C+Patrick+S%2E+H%2E+Lewis%2C+Anton+Bakhtin%2C+Yuxiang+Wu%2C+and+Alexander+H%2E+Miller%2E+2019%2E+Language+models+as+knowledge+bases%3F+In+Proceedings+of+the+2019+Confere) [our card](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
25. Nina Poerner, Ulli Waltinger, and Hinrich Schütze. 2020. E-BERT: Efficient-yet-effective entity embeddings for BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 803–818, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nina+Poerner%2C+Ulli+Waltinger%2C+and+Hinrich+Schu%CC%88tze%2E+2020%2E+E%2DBERT%3A+Efficient%2Dyet%2Deffective+entity+embeddings+for+BERT%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+80)
26. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444–4451. AAAI Press. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Proceedings+of+the+Thirty%2DFirst+AAAI+Conference+on+Artificial+Intelligence%2C+Fe)
27. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. Computing Research Repository, arXiv:2002.01808. [find](https://scholar.google.com/scholar?q=Ruize+Wang%2C+Duyu+Tang%2C+Nan+Duan%2C+Zhongyu+Wei%2C+Xuanjing+Huang%2C+Jianshu+Ji%2C+Guihong+Cao%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020%2E+K%2Dadapter%3A+Infusing+knowledge+into+pre%2Dtrained+models+with+adapters%2E+Computing+) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/)