---
title: "Cross-Lingual Consistency of Factual Knowledge in Multilingual"
description: "The authors introduce RankC, a metric inspired by MAP@K that compares the full rankings of answer… 40 references, cited by 77 papers here."
authors: ["Jirui Qi", "Raquel Fernández", "Arianna Bisazza"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.658/"
doi: "10.18653/v1/2023.emnlp-main.658"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/qi-2023-cross-lingual-consistency-factual-knowledge/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/qi-2023-cross-lingual-consistency-factual-knowledge/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models

# Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models

**Authors:** Jirui Qi, [Raquel Fernández](https://md.research.elamur.ai/free/en/authors/raquel-fernandez/) and [Arianna Bisazza](https://md.research.elamur.ai/free/en/authors/arianna-bisazza/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.658/) · [PDF](https://aclanthology.org/2023.emnlp-main.658.pdf) · [DOI 10.18653/v1/2023.emnlp-main.658](https://doi.org/10.18653/v1/2023.emnlp-main.658) · [OpenAlex](https://openalex.org/W4389520374)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [77](https://md.research.elamur.ai/free/en/papers/qi-2023-cross-lingual-consistency-factual-knowledge/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual PLMs know different facts depending on the language they are queried in. Instead of measuring how much each language "knows," this paper asks whether a model gives the same answer to the same question across languages, even when that answer is wrong. The authors introduce RankC, a metric inspired by MAP@K that compares the full rankings of answer candidates between two languages, so consistency is measured separately from correctness. They also build BMLAMA, a multi-parallel benchmark filtered from X-FACTR and MLAMA in which every prompt and candidate set exists in all languages. They apply both to XLM-RoBERTa-large, mT5-large and several BLOOM models. Consistency is generally low, and larger models do not become more consistent even when their accuracy improves. Language pairs sharing more vocabulary agree more often than genetically related pairs. A small ROME editing experiment suggests that facts may spread across languages mainly through shared tokens rather than language-neutral representations. This matters for giving speakers of different languages equal answers and for editing multilingual models.

## Key points
- **RankC metric:** Earlier work (Jiang et al., 2020) counted only overlapping correct answers, peaking at just 34% for English–Dutch in mBERT. RankC instead counts matching wrong answers too and weighs the whole candidate ranking.
- **BMLAMA benchmark:** Every language uses an identical query set. This makes scores comparable across language pairs, unlike the per-pair filtering used before. Code and data are available at github.com/Betswish/Cross-Lingual-Consistency.
- **Model size:** Scaling up the BLOOM series raises probing accuracy in most languages but leaves average consistency nearly unchanged. Hardware limits meant BLOOM-7.1b was the largest model tested.
- **Predictors of consistency:** Consistency correlates significantly with genetic similarity between languages. Vocabulary overlap is a clearly stronger predictor.
- **Editing case study:** The authors used ROME to insert six false facts in English into BLOOM-3b. The edits carried over to Spanish and Vietnamese, which have high RankC with English. Hungarian and Greek, which have low RankC with English, mostly kept preferring the original correct answers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Amsterdam](https://md.research.elamur.ai/free/en/universities/university-of-amsterdam/) — University, Amsterdam, Netherlands · 89 works · 81,337 citations
- [University of Groningen](https://md.research.elamur.ai/free/en/universities/university-of-groningen/) — University, Groningen, Netherlands · 49 works · 4,843 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 465 works

## Cite

Cite as: Jirui Qi, Raquel Fernández and Arianna Bisazza. 2023. Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{qi2023cross,
  title = {{Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models}},
  author = {Jirui Qi, Raquel Fernández and Arianna Bisazza},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.658},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.658}
}
```

- [← previous Aligning Language Models to User Opinions](https://md.research.elamur.ai/free/en/papers/hwang-2023-aligning-language-models-user-opinions/)
- [next → PLACES: Prompting Language Models for Social Conversation Synthesis](https://md.research.elamur.ai/free/en/papers/chen-2023-places-prompting-language-models-social/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Israa Alghanmi, Luis Espinosa Anke, and Steven Schockaert. 2021. Probing pre-trained language models for disease knowledge. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3023–3033, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Israa+Alghanmi%2C+Luis+Espinosa+Anke%2C+and+Steven+Schockaert%2E+2021%2E+Probing+pre%2Dtrained+language+models+for+disease+knowledge%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+202)
2. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeffrey Wang, Lu… [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Shruti+Bhosale%2C+Naman+Goyal%2C+Todor+Mihaylov%2C+Myle+Ott%2C+Sam+Shleifer%2C+Xi+Victoria+Lin%2C+Jingfei+Du%2C+Srinivasan+Iyer%2C+Ramakanth+Pasunuru%2C+Giridharan+Anantharaman%2C+Xian+Li%2C+Shuohui+Chen%2C+Ha)
3. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Terra+Blevins+and+Luke+Zettlemoyer%2E+2022%2E+Language+contamination+helps+explains+the+cross%2Dlingual+capabilities+of+English+pretrained+models%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+)
4. Constanza Fierro and Anders Søgaard. 2022. Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046–3052, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Constanza+Fierro+and+Anders+S%C3%B8gaard%2E+2022%2E+Factual+consistency+of+multilingual+pretrained+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL+2022%2C+pages+3046%E2%80%933052%2C+Dubl)
5. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456–7463. [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2020%2E+Inducing+relational+knowledge+from+bert%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume+34%2C+pages+7456%E2%80%937463)
6. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
8. Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing, pages 1–4. [find](https://scholar.google.com/scholar?q=Israel+Cohen%2C+Yiteng+Huang%2C+Jingdong+Chen%2C+Jacob+Benesty%2C+Jacob+Benesty%2C+Jingdong+Chen%2C+Yiteng+Huang%2C+and+Israel+Cohen%2E+2009%2E+Pearson+correlation+coefficient%2E+Noise+reduction+in+speech+processing%2C+pag)
9. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
10. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. [Commonsense knowledge mining from pre-trained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–…
11. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nicola+De+Cao%2C+Wilker+Aziz%2C+and+Ivan+Titov%2E+2021%2E+Editing+factual+knowledge+in+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+6491%E2%80%93+6)
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
13. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904. — [arXiv:2203.06904](https://arxiv.org/abs/2203.06904) [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yujia+Qin%2C+Guang+Yang%2C+Fuchao+Wei%2C+Zonghan+Yang%2C+Yusheng+Su%2C+Shengding+Hu%2C+Yulin+Chen%2C+Chi%2DMin+Chan%2C+Weize+Chen%2C+et+al%2E+2022%2E+Delta+tuning%3A+A+comprehensive+study+of+parameter+efficient+meth)
14. Harald Hammarström, Robert Forkel, and Martin Haspelmath. 2017. Glottolog 3.0. Max Planck Institute for the Science of Human History. [find](https://scholar.google.com/scholar?q=Harald+Hammarstr%C3%B6m%2C+Robert+Forkel%2C+and+Martin+Haspelmath%2E+2017%2E+Glottolog+3%2E0%2E+Max+Planck+Institute+for+the+Science+of+Human+History%2E)
15. Yifan Hou, Wenxiang Jiao, Meizhen Liu, Carl Allen, Zhaopeng Tu, and Mrinmaya Sachan. 2022. Adapters for enhanced modeling of multilingual knowledge and text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3902–3917, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Yifan+Hou%2C+Wenxiang+Jiao%2C+Meizhen+Liu%2C+Carl+Allen%2C+Zhaopeng+Tu%2C+and+Mrinmaya+Sachan%2E+2022%2E+Adapters+for+enhanced+modeling+of+multilingual+knowledge+and+text%2E+In+Findings+of+the+Association+for+Computa)
16. Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. [Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification](https://md.research.elamur.ai/free/en/papers/hu-2022-knowledgeable-prompt-tuning-incorporating-knowledge/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics…
17. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske,… [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Mario+Giulianelli%2C+Verna+Dankers%2C+Mikel+Artetxe%2C+Yanai+Elazar%2C+Tiago+Pimentel%2C+Christos+Christodoulopoulos%2C+Karim+Lasri%2C+Naomi+Saphra%2C+Arabella+Sinclair%2C+Dennis+Ulmer%2C+Florian+Schottma)
18. Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020. X-FACTR: Multilingual factual knowledge retrieval from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5943–5959, Online. Ass… [find](https://scholar.google.com/scholar?q=Zhengbao+Jiang%2C+Antonios+Anastasopoulos%2C+Jun+Araki%2C+Haibo+Ding%2C+and+Graham+Neubig%2E+2020%2E+X%2DFACTR%3A+Multilingual+factual+knowledge+retrieval+from+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2020+)
19. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. [Multilingual LAMA: Investigating knowledge in multilingual pretrained language models](https://md.research.elamur.ai/free/en/papers/kassner-2021-multilingual-lama-investigating-knowledge-multilingual/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258, Online. Associ…
20. Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),… [find](https://scholar.google.com/scholar?q=Tao+Li%2C+Vivek+Gupta%2C+Maitrey+Mehta%2C+and+Vivek+Srikumar%2E+2019%2E+A+logic%2Ddriven+framework+for+consistency+of+neural+models%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+)
21. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. [URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors](https://md.research.elamur.ai/free/en/papers/littell-2017-uriel-lang2vec-representing-languages-as/). In Proceedings of the 15th Conference of the European Chapter of the Association for Computational…
22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
23. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359–17372. [find](https://scholar.google.com/scholar?q=Kevin+Meng%2C+David+Bau%2C+Alex+Andonian%2C+and+Yonatan+Belinkov%2E+2022%2E+Locating+and+editing+factual+associations+in+gpt%2E+Advances+in+Neural+Information+Processing+Systems%2C+35%3A17359%E2%80%9317372%2E)
24. Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Proceedings of the 2022 Conference on Empirical Methods in N… [find](https://scholar.google.com/scholar?q=Eric+Mitchell%2C+Joseph+Noh%2C+Siyan+Li%2C+Will+Armstrong%2C+Ananth+Agarwal%2C+Patrick+Liu%2C+Chelsea+Finn%2C+and+Christopher+Manning%2E+2022%2E+Enhancing+self%2Dconsistency+and+performance+of+pre%2Dtrained+language+models)
25. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — [Crosslingual Generalization through Multitask Finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/)
26. Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2023. Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses. CoRR. [find](https://scholar.google.com/scholar?q=Xenia+Ohmer%2C+Elia+Bruni%2C+and+Dieuwke+Hupkes%2E+2023%2E+Separating+form+and+meaning%3A+Using+self%2Dconsistency+to+quantify+task+understanding+across+multiple+senses%2E+CoRR%2E)
27. Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. 2022. COPEN: Probing conceptual knowledge in pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5015–5035, Abu Dhabi, United Ara… [find](https://scholar.google.com/scholar?q=Hao+Peng%2C+Xiaozhi+Wang%2C+Shengding+Hu%2C+Hailong+Jin%2C+Lei+Hou%2C+Juanzi+Li%2C+Zhiyuan+Liu%2C+and+Qun+Liu%2E+2022%2E+COPEN%3A+Probing+conceptual+knowledge+in+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+2022+Co)
28. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
29. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ponti-2019-modeling-language-variation-universals-survey/). Computational Linguistics, 45(3):559–601.
30. Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Knowledge inheritance for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Yujia+Qin%2C+Yankai+Lin%2C+Jing+Yi%2C+Jiajie+Zhang%2C+Xu+Han%2C+Zhengyan+Zhang%2C+Yusheng+Su%2C+Zhiyuan+Liu%2C+Peng+Li%2C+Maosong+Sun%2C+and+Jie+Zhou%2E+2022%2E+Knowledge+inheritance+for+pre%2Dtrained+language+models%2E+In+Proce)
31. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
32. Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. 2023. [Inseq: An interpretability toolkit for sequence generation models](https://md.research.elamur.ai/free/en/papers/sarti-2023-inseq-interpretability-toolkit-sequence-generation/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 421–435, Toronto, Canada.…
33. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+Matthias+Gall%C3%A9%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter)
34. Hinrich Schutze, Christopher D Manning, and Prabhakar Raghavan. 2008. Introduction to information retrieval. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Hinrich+Schutze%2C+Christopher+D+Manning%2C+and+Prabhakar+Raghavan%2E+2008%2E+Introduction+to+information+retrieval%2E+Cambridge+University+Press%2E)
35. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
36. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
37. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
38. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mT5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling…
39. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoM-LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, Unite… [find](https://scholar.google.com/scholar?q=Da+Yin%2C+Hritik+Bansal%2C+Masoud+Monajatipoor%2C+Liunian+Harold+Li%2C+and+Kai%2DWei+Chang%2E+2022%2E+GeoM%2DLAMA%3A+Geo%2Ddiverse+commonsense+probing+on+multilingual+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+20)
40. Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. 2022. UDapter: Typology-based Language Adapters for Multilingual Dependency Parsing and Sequence Labeling. Computational Linguistics, 48(3):555–592. [find](https://scholar.google.com/scholar?q=Ahmet+%C3%9Cst%C3%BCn%2C+Arianna+Bisazza%2C+Gosse+Bouma%2C+and+Gertjan+van+Noord%2E+2022%2E+UDapter%3A+Typology%2Dbased+Language+Adapters+for+Multilingual+Dependency+Parsing+and+Sequence+Labeling%2E+Computational+Linguistics%2C+)

*Updated 24 October 2026*