---
title: "Probing Pretrained Language Models for Lexical — Vulić 2020"
description: "The paper asks how much type-level (context-independent) lexical knowledge pretrained Transformer LMs such as BERT contain, and how best to extract it."
authors: ["Ivan Vulić", "Edoardo Maria Ponti", "Robert Litschko", "Goran Glavaš", "Anna Korhonen"]
year: 2020
venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/2020.emnlp-main.586/"
doi: "10.18653/v1/2020.emnlp-main.586"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vulic-2020-probing-pretrained-language-models-lexical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/) / Probing Pretrained Language Models for Lexical Semantics

# Probing Pretrained Language Models for Lexical Semantics

**Authors:** [Ivan Vulić](https://md.research.elamur.ai/free/en/authors/ivan-vulic/), [Edoardo Maria Ponti](https://md.research.elamur.ai/free/en/authors/edoardo-maria-ponti/), [Robert Litschko](https://md.research.elamur.ai/free/en/authors/robert-litschko/), [Goran Glavaš](https://md.research.elamur.ai/free/en/authors/goran-glavas/), [Anna Korhonen](https://md.research.elamur.ai/free/en/authors/anna-korhonen/) · **Year:** 2020 · **Venue:** Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.586/) · [PDF](https://aclanthology.org/2020.emnlp-main.586.pdf) · [DOI 10.18653/v1/2020.emnlp-main.586](https://doi.org/10.18653/v1/2020.emnlp-main.586) · [OpenAlex](https://openalex.org/W3092733346)

**Citations:** 180 citations (openalex-snapshot · 2026-10-14) · [140](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper asks how much type-level (context-independent) lexical knowledge pretrained Transformer LMs such as BERT contain, and how best to extract it. The authors turn contextual encoders into static "BERT-based" word vectors and test them on five lexical tasks: LSIM, WA, BLI, CLIR, and RELP. The experiments cover six typologically varied languages (EN, DE, RU, FI, ZH, TR). They vary four parts of the extraction process: the source model (monolingual BERT vs. mBERT), the input context (isolated word vs. averaging over many sentences), the treatment of the [CLS] and [SEP] tokens, and single-layer vs. averaged-layer representations. The vectors are compared against fastText, and linear CKA is used to measure cross-lingual structural similarity.

The best setup differs by language and task, but some patterns recur. Monolingual models work better than mBERT, and multiple contexts help. Leaving out special tokens and averaging the lower layers also pays off. With these choices, static vectors from monolingual BERT match or beat fastText on LSIM, WA, and RELP. The study offers practical guidance for building lexical resources from LMs, including for cross-lingual use.

## Key points
- The four extraction components combine into 24 configurations. Vocabularies are the 100K most frequent words of each language's Wikipedia-trained fastText model.
- Context sentences come from 1M sentences sampled per language: Europarl for EN/DE/FI, the United Nations Parallel Corpus for RU/ZH, and WMT17 monolingual data for TR. LSIM is evaluated on Multi-SimLex.
- Peak scores come from layers L1–L5, and no single layer matches averaging over layers. Lexical knowledge is therefore concentrated in the lower layers but spread across several of them.
- Translation pairs receive similar representations even from independently trained monolingual BERTs. This similarity is stronger for closely related languages. CKA similarity correlates strongly with BLI performance (ρ = 1.0, 0.83, 0.99 for EN–FI, EN–DE, DE–FI).
- Scale helps somewhat. EN BERT-Large raises peak EN LSIM from 0.518 to 0.531. For EN–IT, an Italian BERT trained on 81GB instead of 13GB improves BLI from 0.548 to 0.572 and CLIR from 0.148 to 0.160.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Lexical semantics & word embeddings](https://md.research.elamur.ai/free/en/topics/lexical-semantics/) · [Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/)

## Cited by

Cited by

140 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 128 not yet.

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://md.research.elamur.ai/free/en/papers/liu-2021-self-alignment-pretraining-biomedical-entity/)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/inoue-2021-interplay-variant-size-task-type/)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://md.research.elamur.ai/free/en/papers/ruckle-2021-adapterdrop-efficiency-adapters-transformers/)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://md.research.elamur.ai/free/en/papers/arora-2023-probing-pre-trained-language-models/)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://md.research.elamur.ai/free/en/papers/lex-2021-hofmann-superbizarre-derivational/)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://md.research.elamur.ai/free/en/papers/miletic-2024-semantics-multiword-expressions-transformer-based/)
- [The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks](https://md.research.elamur.ai/free/en/papers/schlechtweg-2026-lscd-benchmark-testbed-diachronic-word/)
- [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/)
- [Few-shot Prompting or Supervised Tuning? A Comparative Study of LLMs for Linguistically Distant Language Pairs in BDI](https://md.research.elamur.ai/free/en/papers/naorem-2026-few-shot-prompting-or-supervised/)
- [Mechanistic Interpretability Meets Cognitive Linguistics: Modelling Locative Image Schemas in the Circuit Framework](https://md.research.elamur.ai/free/en/papers/proietti-2026-mechanistic-interpretability-meets-cognitive-linguistics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{vulic2020probing,
  title = {{Probing Pretrained Language Models for Lexical Semantics}},
  author = {Vulić, Ivan and Ponti, Edoardo Maria and Litschko, Robert and Glavaš, Goran and Korhonen, Anna},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.586},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.586}
}
```

- [← previous Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/)
- [next → PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable](https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/2/)

## References

66 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL, pages 789–798. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+and+Eneko+Agirre%2E+2018%2E+A+robust+self%2Dlearning+method+for+fully+unsupervised+cross%2Dlingual+mappings+of+word+embeddings%2E+In+Proceedings+of+ACL%2C+pages+789%E2%80%93798%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-vecmap-unsupervised/)
2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of ACL, pages 4623–4637. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+2020%2E+On+the+cross%2Dlingual+transferability+of+monolingual+representations%2E+In+Proceedings+of+ACL%2C+pages+4623%E2%80%934637%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
3. Yonatan Belinkov and James R. Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association of Computational Linguistics, 7:49–72. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov+and+James+R%2E+Glass%2E+2019%2E+Analysis+methods+in+neural+language+processing%3A+A+survey%2E+Transactions+of+the+Association+of+Computational+Linguistics%2C+7%3A49%E2%80%9372%2E) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Jauvin%2E+2003%2E+A+neural+probabilistic+language+model%2E+Journal+of+Machine+Learning+Research%2C+3%3A1137%E2%80%931155%2E)
5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the ACL, 5:135–146. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2017%2E+Enriching+word+vectors+with+subword+information%2E+Transactions+of+the+ACL%2C+5%3A135%E2%80%93146%2E)
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Transla… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Rajen+Chatterjee%2C+Christian+Federmann%2C+Yvette+Graham%2C+Barry+Haddow%2C+Shujian+Huang%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Qun+Liu%2C+Varvara+Logacheva%2C+Christof+Monz%2C+Matteo+Negri%2C+Matt+Post%2C+Raph) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/)
7. Danushka Bollegala and Cong Bao. 2018. Learning word meta-embeddings by autoencoding. In Proceedings of COLING, pages 1650–1661. [find](https://scholar.google.com/scholar?q=Danushka+Bollegala+and+Cong+Bao%2E+2018%2E+Learning+word+meta%2Dembeddings+by+autoencoding%2E+In+Proceedings+of+COLING%2C+pages+1650%E2%80%931661%2E)
8. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45. [find](https://scholar.google.com/scholar?q=In%CC%83igo+Casanueva%2C+Tadas+Temc%CC%8Cinas%2C+Daniela+Gerz%2C+Matthew+Henderson%2C+and+Ivan+Vulic%CC%81%2E+2020%2E+Efficient+intent+detection+with+dual+sentence+encoders%2E+In+Proceedings+of+the+2nd+Workshop+on+Natural+Languag) [our card](https://md.research.elamur.ai/free/en/papers/casanueva-2020-efficient-intent-detection-dual-sentence/)
9. Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? Explaining contextualized embeddings with natural language definition. In Proceedings of EMNLP-IJCNLP, pages 6064–6070. [find](https://scholar.google.com/scholar?q=Ting%2DYun+Chang+and+Yun%2DNung+Chen%2E+2019%2E+What+does+this+word+mean%3F+Explaining+contextualized+embeddings+with+natural+language+definition%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C+pages+6064%E2%80%936070%2E)
10. Ethan A. Chi, John Hewitt, and Christopher D. Manning. 2020. Finding universal grammatical relations in multilingual BERT. In Proceedings of ACL, pages 5564–5577. [find](https://scholar.google.com/scholar?q=Ethan+A%2E+Chi%2C+John+Hewitt%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Finding+universal+grammatical+relations+in+multilingual+BERT%2E+In+Proceedings+of+ACL%2C+pages+5564%E2%80%935577%2E)
11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440–8451. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
12. Daniel Edmiston. 2020. A systematic analysis of morphological content in BERT models for multiple languages. CoRR, abs/2004.03032. — [arXiv:2004.03032](https://arxiv.org/abs/2004.03032) [find](https://scholar.google.com/scholar?q=Daniel+Edmiston%2E+2020%2E+A+systematic+analysis+of+morphological+content+in+BERT+models+for+multiple+languages%2E+CoRR%2C+abs%2F2004%2E03032%2E+%E2%80%94+%5BarXiv%3A2004%2E03032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2004%2E03032%29)
13. Julian Eisenschlos, Sebastian Ruder, Piotr Czapla, Marcin Kardas, Sylvain Gugger, and Jeremy Howard. 2019. MultiFiT: Efficient multi-lingual language model fine-tuning. In Proceedings of EMNLP-IJCNLP, pages 5701–5706. [find](https://scholar.google.com/scholar?q=Julian+Eisenschlos%2C+Sebastian+Ruder%2C+Piotr+Czapla%2C+Marcin+Kardas%2C+Sylvain+Gugger%2C+and+Jeremy+Howard%2E+2019%2E+MultiFiT%3A+Efficient+multi%2Dlingual+language+model+fine%2Dtuning%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C)
14. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of EMNLP-IJCNLP, pages 55–65. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2019%2E+How+contextual+are+contextualized+word+representations%3F+Comparing+the+geometry+of+BERT%2C+ELMo%2C+and+GPT%2D2+embeddings%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C+pages+55%E2%80%9365%2E)
15. Christiane Fellbaum. 1998. WordNet. MIT Press. [find](https://scholar.google.com/scholar?q=Christiane+Fellbaum%2E+1998%2E+WordNet%2E+MIT+Press%2E)
16. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of EMNLP, pages 316–327. [find](https://scholar.google.com/scholar?q=Daniela+Gerz%2C+Ivan+Vulic%CC%81%2C+Edoardo+Maria+Ponti%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2018%2E+On+the+relation+between+linguistic+typology+and+%28limitations+of%29+multilingual+language+modeling%2E+In+Proceedings+o)
17. Goran Glavaš and Ivan Vulić. 2018. Discriminating between lexico-semantic relations with the specialization tensor model. In Proceedings of NAACL-HLT, pages 181–187. [find](https://scholar.google.com/scholar?q=Goran+Glavas%CC%8C+and+Ivan+Vulic%CC%81%2E+2018%2E+Discriminating+between+lexico%2Dsemantic+relations+with+the+specialization+tensor+model%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+181%E2%80%93187%2E)
18. Goran Glavaš, Robert Litschko, Sebastian Ruder, and Ivan Vulić. 2019. How to (properly) evaluate cross-lingual word embeddings: On strong baselines, comparative analyses, and some misconceptions. In Proceedings of ACL, pages 710–721. [find](https://scholar.google.com/scholar?q=Goran+Glavas%CC%8C%2C+Robert+Litschko%2C+Sebastian+Ruder%2C+and+Ivan+Vulic%CC%81%2E+2019%2E+How+to+%28properly%29+evaluate+cross%2Dlingual+word+embeddings%3A+On+strong+baselines%2C+comparative+analyses%2C+and+some+misconceptions%2E+In)
19. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS, pages 249– 256. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+Proceedings+of+AISTATS%2C+pages+249%E2%80%93+256%2E)
20. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. BilBOWA: Fast bilingual distributed representations without word alignments. In Proceedings of ICML, pages 748–756. [find](https://scholar.google.com/scholar?q=Stephan+Gouws%2C+Yoshua+Bengio%2C+and+Greg+Corrado%2E+2015%2E+BilBOWA%3A+Fast+bilingual+distributed+representations+without+word+alignments%2E+In+Proceedings+of+ICML%2C+pages+748%E2%80%93756%2E)
21. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL-HLT, pages 4129–4138. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Christopher+D%2E+Manning%2E+2019%2E+A+structural+probe+for+finding+syntax+in+word+representations%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+4129%E2%80%934138%2E) [our card](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)
22. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2015%2E+SimLex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2C+41%284%29%3A665%E2%80%93695%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+4171%E2%80%934186%2E)
24. Valentin Hofmann, Janet B. Pierrehumbert, and Hinrich Schütze. 2020. Generating derivational morphology with BERT. CoRR, abs/2005.00672. — [arXiv:2005.00672](https://arxiv.org/abs/2005.00672) [find](https://scholar.google.com/scholar?q=Valentin+Hofmann%2C+Janet+B%2E+Pierrehumbert%2C+and+Hinrich+Schu%CC%88tze%2E+2020%2E+Generating+derivational+morphology+with+BERT%2E+CoRR%2C+abs%2F2005%2E00672%2E+%E2%80%94+%5BarXiv%3A2005%2E00672%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2005%2E00672%29)
25. Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. 2016. Word embeddings, analogies, and machine learning: Beyond king - man + woman = queen. In Proceedings of COLING, pages 3519– 3530. [find](https://scholar.google.com/scholar?q=Aleksandr+Drozd%2C+Anna+Gladkova%2C+and+Satoshi+Matsuoka%2E+2016%2E+Word+embeddings%2C+analogies%2C+and+machine+learning%3A+Beyond+king+%2D+man+%2B+woman+%3D+queen%2E+In+Proceedings+of+COLING%2C+pages+3519%E2%80%93+3530%2E)
26. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+2020%2E+XTREME%3A+A+massively+multilingual+multi%2Dtask+benchmark+for+evaluating+cross%2Dlingual+generalization%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/)
27. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of ACL, pages 3651– 3657. [find](https://scholar.google.com/scholar?q=Ganesh+Jawahar%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Djame%CC%81+Seddah%2E+2019%2E+What+does+BERT+learn+about+the+structure+of+language%3F+In+Proceedings+of+ACL%2C+pages+3651%E2%80%93+3657%2E) [our card](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
28. Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018. Dynamic meta-embeddings for improved sentence representations. In Proceedings of EMNLP, pages 1466–1477. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Changhan+Wang%2C+and+Kyunghyun+Cho%2E+2018%2E+Dynamic+meta%2Dembeddings+for+improved+sentence+representations%2E+In+Proceedings+of+EMNLP%2C+pages+1466%E2%80%931477%2E)
29. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the 10th Machine Translation Summit (MT SUMMIT), pages 79–86. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+the+10th+Machine+Translation+Summit+%28MT+SUMMIT%29%2C+pages+79%E2%80%9386%2E)
30. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of ICML, pages 3519–3529. [find](https://scholar.google.com/scholar?q=Simon+Kornblith%2C+Mohammad+Norouzi%2C+Honglak+Lee%2C+and+Geoffrey+E%2E+Hinton%2E+2019%2E+Similarity+of+neural+network+representations+revisited%2E+In+Proceedings+of+ICML%2C+pages+3519%E2%80%933529%2E) [our card](https://md.research.elamur.ai/free/en/papers/kornblith-2019-similarity-neural-network-representations-revisited/)
31. Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of ACL, pages 4077–4091. [find](https://scholar.google.com/scholar?q=Artur+Kulmizev%2C+Vinit+Ravishankar%2C+Mostafa+Abdou%2C+and+Joakim+Nivre%2E+2020%2E+Do+neural+language+models+show+preferences+for+syntactic+formalisms%3F+In+Proceedings+of+ACL%2C+pages+4077%E2%80%934091%2E)
32. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Proceedings of NeurIPS, pages 3111– 3119. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Gregory+S%2E+Corrado%2C+and+Jeffrey+Dean%2E+2013b%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Proceedings+of+NeurIPS%2C+pages+3111)
33. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP, pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+EMNLP%2C+pages+1532%E2%80%93+1543%2E)
34. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+2227%E2%80%9322)
35. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vulic%CC%81%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+2020%2E+MAD%2DX%3A+An+adapter%2Dbased+framework+for+multi%2Dtask+cross%2Dlingual+transfer%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
36. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. Information-theoretic probing for linguistic structure. In Proceedings of ACL, pages 4609–4622. [find](https://scholar.google.com/scholar?q=Tiago+Pimentel%2C+Josef+Valvoda%2C+Rowan+Hall+Maudslay%2C+Ran+Zmigrod%2C+Adina+Williams%2C+and+Ryan+Cotterell%2E+2020%2E+Information%2Dtheoretic+probing+for+linguistic+structure%2E+In+Proceedings+of+ACL%2C+pages+4609%E2%80%93462) [our card](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/)
37. Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In Proceedings of SIGIR, pages 1253–1256. [find](https://scholar.google.com/scholar?q=Robert+Litschko%2C+Goran+Glavas%CC%8C%2C+Simone+Paolo+Ponzetto%2C+and+Ivan+Vulic%CC%81%2E+2018%2E+Unsupervised+cross%2Dlingual+information+retrieval+using+monolingual+data+only%2E+In+Proceedings+of+SIGIR%2C+pages+1253%E2%80%931256%2E)
38. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Edoardo+Maria+Ponti%2C+Goran+Glavas%CC%8C%2C+Olga+Majewska%2C+Qianchu+Liu%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2020%2E+XCOPA%3A+A+multilingual+dataset+for+causal+commonsense+reasoning%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
39. Robert Litschko, Goran Glavaš, Ivan Vulić, and Laura Dietz. 2019. Evaluating resource-lean cross-lingual embedding models in unsupervised retrieval. In Proceedings of SIGIR, pages 1109–1112. [find](https://scholar.google.com/scholar?q=Robert+Litschko%2C+Goran+Glavas%CC%8C%2C+Ivan+Vulic%CC%81%2C+and+Laura+Dietz%2E+2019%2E+Evaluating+resource%2Dlean+cross%2Dlingual+embedding+models+in+unsupervised+retrieval%2E+In+Proceedings+of+SIGIR%2C+pages+1109%E2%80%931112%2E)
40. Yifan Qiao, Chenyan Xiong, Zheng-Hao Liu, and Zhiyuan Liu. 2019. Understanding the behaviors of BERT in ranking. CoRR, abs/1904.07531. — [arXiv:1904.07531](https://arxiv.org/abs/1904.07531) [find](https://scholar.google.com/scholar?q=Yifan+Qiao%2C+Chenyan+Xiong%2C+Zheng%2DHao+Liu%2C+and+Zhiyuan+Liu%2E+2019%2E+Understanding+the+behaviors+of+BERT+in+ranking%2E+CoRR%2C+abs%2F1904%2E07531%2E+%E2%80%94+%5BarXiv%3A1904%2E07531%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1904%2E07531%29)
41. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of NAACL-HLT, pages 1073–1094. [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+E%2E+Peters%2C+and+Noah+A%2E+Smith%2E+2019a%2E+Linguistic+knowledge+and+transferability+of+contextual+representations%2E+In+Proceedings+of+NAACL%2DHLT%2C+pages+1) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
42. Qianchu Liu, Diana McCarthy, Ivan Vulić, and Anna Korhonen. 2019b. Investigating cross-lingual alignment methods for contextualized embeddings with token-level evaluation. In Proceedings of CoNLL, pages 33–43. [find](https://scholar.google.com/scholar?q=Qianchu+Liu%2C+Diana+McCarthy%2C+Ivan+Vulic%CC%81%2C+and+Anna+Korhonen%2E+2019b%2E+Investigating+cross%2Dlingual+alignment+methods+for+contextualized+embeddings+with+token%2Dlevel+evaluation%2E+In+Proceedings+of+CoNLL%2C+pa)
43. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019c%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+)
44. Timothee Mickus, Denis Paperno, Mathieu Constant, and Kees van Deemter. 2020. What do you mean, BERT? Assessing BERT as a distributional semantics model. Proceedings of the Society for Computation in Linguistics, 3(34). [find](https://scholar.google.com/scholar?q=Timothee+Mickus%2C+Denis+Paperno%2C+Mathieu+Constant%2C+and+Kees+van+Deemter%2E+2020%2E+What+do+you+mean%2C+BERT%3F+Assessing+BERT+as+a+distributional+semantics+model%2E+Proceedings+of+the+Society+for+Computation+in+)
45. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013a. Exploiting similarities among languages for machine translation. arXiv preprint, CoRR, abs/1309.4168. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Quoc+V%2E+Le%2C+and+Ilya+Sutskever%2E+2013a%2E+Exploiting+similarities+among+languages+for+machine+translation%2E+arXiv+preprint%2C+CoRR%2C+abs%2F1309%2E4168%2E)
46. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271. [find](https://scholar.google.com/scholar?q=Xipeng+Qiu%2C+Tianxiang+Sun%2C+Yige+Xu%2C+Yunfan+Shao%2C+Ning+Dai%2C+and+Xuanjing+Huang%2E+2020%2E+Pre%2Dtrained+models+for+natural+language+processing%3A+A+survey%2E+CoRR%2C+abs%2F2003%2E08271%2E) [our card](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
47. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2019%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
48. Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Proceedings of NeurIPS, pages 8594– 8603. [find](https://scholar.google.com/scholar?q=Emily+Reif%2C+Ann+Yuan%2C+Martin+Wattenberg%2C+Fernanda+B%2E+Viegas%2C+Andy+Coenen%2C+Adam+Pearce%2C+and+Been+Kim%2E+2019%2E+Visualizing+and+measuring+the+geometry+of+BERT%2E+In+Proceedings+of+NeurIPS%2C+pages+8594%E2%80%93+8603%2E) [our card](https://md.research.elamur.ai/free/en/papers/coenen-2019-visualizing-measuring-geometry-bert/)
49. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: what we know about how BERT works. Transactions of the ACL. [find](https://scholar.google.com/scholar?q=Anna+Rogers%2C+Olga+Kovaleva%2C+and+Anna+Rumshisky%2E+2020%2E+A+primer+in+BERTology%3A+what+we+know+about+how+BERT+works%2E+Transactions+of+the+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/)
50. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019. A survey of cross-lingual embedding models. Journal of Artificial Intelligence Research, 65:569– 631. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Ivan+Vulic%CC%81%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+A+survey+of+cross%2Dlingual+embedding+models%2E+Journal+of+Artificial+Intelligence+Research%2C+65%3A569%E2%80%93+631%2E)
51. Jasdeep Singh, Bryan McCann, Richard Socher, and Caiming Xiong. 2019. BERT is not an interlingua and the bias of tokenization. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 47–55. [find](https://scholar.google.com/scholar?q=Jasdeep+Singh%2C+Bryan+McCann%2C+Richard+Socher%2C+and+Caiming+Xiong%2E+2019%2E+BERT+is+not+an+interlingua+and+the+bias+of+tokenization%2E+In+Proceedings+of+the+2nd+Workshop+on+Deep+Learning+Approaches+for+Low%2DRe)
52. Samuel L. Smith, David H.P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of ICLR (Conference Track). [find](https://scholar.google.com/scholar?q=Samuel+L%2E+Smith%2C+David+H%2EP%2E+Turban%2C+Steven+Hamblin%2C+and+Nils+Y%2E+Hammerla%2E+2017%2E+Offline+bilingual+word+vectors%2C+orthogonal+transformations+and+the+inverted+softmax%2E+In+Proceedings+of+ICLR+%28Conference+) [our card](https://md.research.elamur.ai/free/en/papers/smith-2017-offline-bilingual-word-vectors-orthogonal/)
53. Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of ACL, pages 778–788. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2C+Sebastian+Ruder%2C+and+Ivan+Vulic%CC%81%2E+2018%2E+On+the+limitations+of+unsupervised+bilingual+dictionary+induction%2E+In+Proceedings+of+ACL%2C+pages+778%E2%80%93788%2E) [our card](https://md.research.elamur.ai/free/en/papers/sogaard-2018-limitations-unsupervised-bilingual-dictionary-induction/)
54. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP, pages 833–844. [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+2019%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+EMNLP%2C+pages+833%E2%80%93844%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
55. Wenpeng Yin and Hinrich Schütze. 2016. Learning word meta-embeddings. In Proceedings of ACL, pages 1351–1360. [find](https://scholar.google.com/scholar?q=Wenpeng+Yin+and+Hinrich+Schu%CC%88tze%2E+2016%2E+Learning+word+meta%2Dembeddings%2E+In+Proceedings+of+ACL%2C+pages+1351%E2%80%931360%2E)
56. Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Michal+Ziemski%2C+Marcin+Junczys%2DDowmunt%2C+and+Bruno+Pouliquen%2E+2016%2E+The+United+Nations+Parallel+Corpus+v1%2E0%2E+In+Proceedings+of+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/ziemski-2016-united-nations-parallel-corpus-v1/)
57. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of ACL, pages 4593–4601. [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+2019%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+ACL%2C+pages+4593%E2%80%934601%2E) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
58. Jörg Tiedemann. 2009. News from OPUS - A collection of multilingual parallel corpora with tools and interfaces. In Proceedings of RANLP, pages 237– 248. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+2009%2E+News+from+OPUS+%2D+A+collection+of+multilingual+parallel+corpora+with+tools+and+interfaces%2E+In+Proceedings+of+RANLP%2C+pages+237%E2%80%93+248%2E)
59. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proceedings EMNLP-IJCNLP, pages 3632–3636. [find](https://scholar.google.com/scholar?q=Henry+Tsai%2C+Jason+Riesa%2C+Melvin+Johnson%2C+Naveen+Arivazhagan%2C+Xin+Li%2C+and+Amelia+Archer%2E+2019%2E+Small+and+practical+BERT+models+for+sequence+labeling%2E+In+Proceedings+EMNLP%2DIJCNLP%2C+pages+3632%E2%80%933636%2E)
60. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 6000– 6010. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Proceedings+of+NeurIPS%2C+pages+6000%E2%80%93+60)
61. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Elena+Voita+and+Ivan+Titov%2E+2020%2E+Information%2Dtheoretic+probing+with+minimum+description+length%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/voita-2020-information-theoretic-probing-minimum-description/)
62. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020. Multi-Simlex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. Computational Lingui… [find](https://scholar.google.com/scholar?q=Ivan+Vulic%CC%81%2C+Simon+Baker%2C+Edoardo+Maria+Ponti%2C+Ulla+Petti%2C+Ira+Leviant%2C+Kelly+Wing%2C+Olga+Majewska%2C+Eden+Bar%2C+Matt+Malone%2C+Thierry+Poibeau%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2020%2E+Multi%2DSimlex%3A+A+large%2D)
63. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-Adapter: Infusing knowledge into pre-trained models with adapters. CoRR, abs/2002.01808. [find](https://scholar.google.com/scholar?q=Ruize+Wang%2C+Duyu+Tang%2C+Nan+Duan%2C+Zhongyu+Wei%2C+Xuanjing+Huang%2C+Jianshu+Ji%2C+Guihong+Cao%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020%2E+K%2DAdapter%3A+Infusing+knowledge+into+pre%2Dtrained+models+with+adapters%2E+CoRR%2C+abs%2F) [our card](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/)
64. Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of EMNLP-IJCNLP, pages 5721–5727. [find](https://scholar.google.com/scholar?q=Yuxuan+Wang%2C+Wanxiang+Che%2C+Jiang+Guo%2C+Yijia+Liu%2C+and+Ting+Liu%2E+2019%2E+Cross%2Dlingual+BERT+transformation+for+zero%2Dshot+dependency+parsing%2E+In+Proceedings+of+EMNLP%2DIJCNLP%2C+pages+5721%E2%80%935727%2E)
65. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pier%2Dric+Cistac%2C+Tim+Rault%2C+R%E2%80%99emi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+2019%2E+HuggingFace%E2%80%99s+Transformers%3A+St) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
66. Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of ACL, pages 6022–6034. [find](https://scholar.google.com/scholar?q=Shijie+Wu%2C+Alexis+Conneau%2C+Haoran+Li%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Emerging+cross%2Dlingual+structure+in+pretrained+language+models%2E+In+Proceedings+of+ACL%2C+pages+6022%E2%80%936034%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/)