---
title: "Rethinking embedding coupling in pre-trained — Chung 2020"
description: "Encoder-only pre-trained models such as mBERT and XLM-R inherit the habit of tying input and output embeddings, which forces both matrices to share one…"
authors: ["Hyung Won Chung", "Thibault Févry", "Henry Tsai", "Melvin Johnson", "Sebastian Ruder"]
year: 2020
venue: "arXiv:2010.12821"
source: "https://arxiv.org/abs/2010.12821"
doi: "10.48550/arxiv.2010.12821"
arxiv: "2010.12821"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Rethinking embedding coupling in pre-trained language models

# Rethinking embedding coupling in pre-trained language models

**Authors:** [Hyung Won Chung](https://md.research.elamur.ai/free/en/authors/hyung-won-chung/), [Thibault Févry](https://md.research.elamur.ai/free/en/authors/thibault-fevry/), Henry Tsai, [Melvin Johnson](https://md.research.elamur.ai/free/en/authors/melvin-johnson/), [Sebastian Ruder](https://md.research.elamur.ai/free/en/authors/sebastian-ruder/) · **Year:** 2020 · **Venue:** arXiv:2010.12821 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2010.12821) · [PDF](https://arxiv.org/pdf/2010.12821) · [DOI 10.48550/arxiv.2010.12821](https://doi.org/10.48550/arxiv.2010.12821) · [arXiv 2010.12821](https://arxiv.org/abs/2010.12821) · [OpenAlex](https://openalex.org/W3094152627)

**Citations:** 11 citations (openalex-snapshot · 2026-10-14) · [131](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Encoder-only pre-trained models such as mBERT and XLM-R inherit the habit of tying input and output embeddings, which forces both matrices to share one dimensionality. In multilingual models with huge vocabularies, these embeddings consume roughly half or more of all parameters, even though the output embedding is thrown away before fine-tuning. The authors test what happens when the two matrices are separated, using an mBERT-sized baseline evaluated on fine-tuning tasks from XTREME (XNLI, NER, PAWS-X, XQuAD, MLQA, TyDiQA-GoldP).

Simply untying the weights changes little. Untying the shapes, however, opens up two options:
- **Shrink the input embedding**, which is cheap in accuracy.
- **Enlarge the output embedding**, which costs nothing at fine-tuning time.

The parameters saved can be moved into deeper or wider Transformer layers. Analysis suggests that a larger output embedding keeps the top layers from overfitting to the masked-LM objective, so their representations transfer better across tasks and languages.

Combining these ideas, the authors build RemBERT, which beats XLM-R on XTREME. The finding matters for anyone budgeting parameters in multilingual encoders.

## Key points
- Embedding matrices account for 52% of parameters in mBERT, 71% in XLM-R Base and 47% in XLM-R Large, compared with only 9–21% in English BERT.
- Naïve decoupling at a fixed size raises the XTREME-task average only slightly, from 62.3 to 62.7.
- A model with E_in=128 and E_out=768 matches the reverse setup (average 60.2 vs 60.1) while using 77M fewer fine-tuning parameters (100M vs 177M).
- ALBERT-style shrinking of coupled embeddings costs 0.8 points on English MNLI but 2.6 points on multilingual XNLI, so that approach does not suit multilingual models.
- RemBERT has 32 layers, hidden size 1152, a 250k vocabulary, E_in=256 and E_out=1536. It was pre-trained on Wikipedia and Common Crawl covering 110 languages, using 3.5× fewer tokens than XLM-R. It has 995M parameters in pre-training and 575M in fine-tuning. With English-only fine-tuning, it reaches an XTREME average of 75.4, compared with 71.4 for XLM-R.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 109 not yet.

- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/adelani-2022-masakhaner-2-0-africa-centric/)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://md.research.elamur.ai/free/en/papers/alabi-2022-adapting-pre-trained-language-models/)
- [SemEval-2022 Task 6: iSarcasmEval, Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/farha-2022-semeval-2022-task-6-isarcasmeval/)
- [Quality-Aware Decoding for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/fernandes-2022-quality-aware-decoding-neural-machine/)
- [Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://md.research.elamur.ai/free/en/papers/freitag-2022-high-quality-rather-than-high/)
- [E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral-benchmarks/)
- [E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral/)
- [SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER)](https://md.research.elamur.ai/free/en/papers/malmasi-2022-semeval-2022-task-11-multilingual/)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/muhammad-2022-naijasenti-nigerian-twitter-sentiment-corpus/)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://md.research.elamur.ai/free/en/papers/ni-2021-sentence-t5-scalable-sentence-encoders/)
- [MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem](https://md.research.elamur.ai/free/en/papers/perrella-2022-matese-machine-translation-evaluation-as/)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://md.research.elamur.ai/free/en/papers/perrella-2024-beyond-correlation-interpretable-evaluation-machine/)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/)
- [Learning Compact Metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/)
- [COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://md.research.elamur.ai/free/en/papers/rei-2022-cometkiwi-ist-unbabel-2022-submission/)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://md.research.elamur.ai/free/en/papers/ruder-2021-xtreme-r-more-challenging-nuanced/)
- [mGPT: Few-Shot Learners Go Multilingual](https://md.research.elamur.ai/free/en/papers/shliazhko-2022-mgpt-few-shot-learners-go/)
- [Automatic Suggestions Help Extending Eventive Ontology: A Case Study on SynSemClass](https://md.research.elamur.ai/free/en/papers/strakova-2026-automatic-suggestions-help-extending-eventive/)
- [Findings of the WMT 2022 Shared Task on Quality Estimation](https://md.research.elamur.ai/free/en/papers/zerva-2022-findings-wmt-2022-shared-task/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chung2020rethinking,
  title = {{Rethinking embedding coupling in pre-trained language models}},
  author = {Chung, Hyung Won and Févry, Thibault and Tsai, Henry and Johnson, Melvin and Ruder, Sebastian},
  year = {2020},
  doi = {10.48550/arxiv.2010.12821},
  eprint = {2010.12821},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2010.12821}
}
```

- [← previous Resumption as a sluicing source in Saudi Arabic: Evidence from sluicing with prepositional phrases](https://md.research.elamur.ai/free/en/papers/alshaalan-2020-resumption-as-sluicing-source-saudi/)
- [next → Revealing the Myth of Higher-Order Inference in Coreference Resolution](https://md.research.elamur.ai/free/en/papers/xu-2020-revealing-myth-higher-order-inference/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

66 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively Multilingual Neural Machine Translation. In Proceedings of NAACL 2019, 2019. [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Melvin+Johnson%2C+and+Orhan+Firat%2E+Massively+Multilingual+Neural+Machine+Translation%2E+In+Proceedings+of+NAACL+2019%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
2. Asaf Amrami and Yoav Goldberg. Towards better substitution-based word sense induction. arXiv preprint arXiv:1905.12598, 2019. — [arXiv:1905.12598](https://arxiv.org/abs/1905.12598) [find](https://scholar.google.com/scholar?q=Asaf+Amrami+and+Yoav+Goldberg%2E+Towards+better+substitution%2Dbased+word+sense+induction%2E+arXiv+preprint+arXiv%3A1905%2E12598%2C+2019%2E+%E2%80%94+%5BarXiv%3A1905%2E12598%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1905%2E12598%29)
3. Anonymous. {VECO}: Variable encoder-decoder pre-training for cross-lingual understanding and generation. In Submitted to International Conference on Learning Representations, 2021. URL [https://openreview.net/forum?id=YjNv-hzM8BE](https://openreview.net/forum?id=YjNv-hzM8BE). under review. [↗](https://openreview.net/forum?id=YjNv-hzM8BE)
4. Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the ACL 2019, 2019. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+Massively+Multilingual+Sentence+Embeddings+for+Zero%2DShot+Cross%2DLingual+Transfer+and+Beyond%2E+Transactions+of+the+ACL+2019%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/)
5. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020, 2020. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Sebastian+Ruder%2C+and+Dani+Yogatama%2E+On+the+Cross%2Dlingual+Transferability+of+Monolingual+Representations%2E+In+Proceedings+of+ACL+2020%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/)
6. Simon Baker, Roi Reichart, and Anna Korhonen. An unsupervised model for instance level subcategorization acquisition. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 278–289, 2014. [find](https://scholar.google.com/scholar?q=Simon+Baker%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+An+unsupervised+model+for+instance+level+subcategorization+acquisition%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Pro)
7. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. Results of the wmt16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 199–231, 2016. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+Amir+Kamran%2C+and+Milos%CC%8C+Stanojevic%CC%81%2E+Results+of+the+wmt16+metrics+shared+task%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+Papers) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/)
8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
9. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. Multimodal distributional semantics. Journal of Artificial Intelligence Research, 49:1–47, 2014. [find](https://scholar.google.com/scholar?q=Elia+Bruni%2C+Nam%2DKhanh+Tran%2C+and+Marco+Baroni%2E+Multimodal+distributional+semantics%2E+Journal+of+Artificial+Intelligence+Research%2C+49%3A1%E2%80%9347%2C+2014%2E)
10. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. [find](https://scholar.google.com/scholar?q=Aylin+Caliskan%2C+Joanna+J+Bryson%2C+and+Arvind+Narayanan%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2C+2017%2E)
11. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. In Transactions of the Association of Computational Linguistics, 2020a. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Eunsol+Choi%2C+Michael+Collins%2C+Dan+Garrette%2C+Tom+Kwiatkowski%2C+Vitaly+Nikolaev%2C+and+Jennimaria+Palomaki%2E+TyDi+QA%3A+A+Benchmark+for+Information%2DSeeking+Question+Answering+in+Typological) [our card](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)
12. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In Proceedings of ICLR 2020, 2020b. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+ELECTRA%3A+Pre%2Dtraining+Text+Encoders+as+Discriminators+Rather+Than+Generators%2E+In+Proceedings+of+ICLR+2020%2C+2020b%2E)
13. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP 2018, pp. 2475–2485, 2018. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+EMNLP+201)
14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Com… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Unsupervised+cross%2Dlingual+re)
15. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6022–6034, Online, July 2020b. Association for Computational L… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Shijie+Wu%2C+Haoran+Li%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Emerging+cross%2Dlingual+structure+in+pretrained+language+models%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associati) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+A)
17. Yanai Elazar and Yoav Goldberg. oLMpics - On what Language Model Pre-training Captures. arXiv preprint arXiv:1912.13283, 2019. [find](https://scholar.google.com/scholar?q=Yanai+Elazar+and+Yoav+Goldberg%2E+oLMpics+%2D+On+what+Language+Model+Pre%2Dtraining+Captures%2E+arXiv+preprint+arXiv%3A1912%2E13283%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
18. Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding. arXiv preprint arXiv:2009.05166, 2020. — [arXiv:2009.05166](https://arxiv.org/abs/2009.05166) [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Siqi+Sun%2C+and+Jingjing+Liu%2E+FILTER%3A+An+Enhanced+Fusion+Method+for+Cross%2Dlingual+Language+Understanding%2E+arXiv+preprint+arXiv%3A2009%2E05166%2C+2020%2E+%E2%80%94+%5BarXiv%3A2009%2E05166%5D%28)
19. Orhan Firat, Baskaran Sankaran, Yaser Al-onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. Zero-Resource Translation with Multi-Lingual Neural Machine Translation. In Proceedings of EMNLP 2016, pp. 268–277, 2016. [find](https://scholar.google.com/scholar?q=Orhan+Firat%2C+Baskaran+Sankaran%2C+Yaser+Al%2Donaizan%2C+Fatos+T%2E+Yarman+Vural%2C+and+Kyunghyun+Cho%2E+Zero%2DResource+Translation+with+Multi%2DLingual+Neural+Machine+Translation%2E+In+Proceedings+of+EMNLP+2016%2C+pp%2E+2) [our card](https://md.research.elamur.ai/free/en/papers/firat-2016-zero-resource-translation-multi-lingual/)
20. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In Proceedings of ACL 2020, 2020. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Ana+Marasovic%CC%81%2C+Swabha+Swayamdipta%2C+Kyle+Lo%2C+Iz+Beltagy%2C+Doug+Downey%2C+and+Noah+A%2E+Smith%2E+Don%E2%80%99t+Stop+Pretraining%3A+Adapt+Language+Models+to+Domains+and+Tasks%2E+In+Proceedings+of+ACL+20)
21. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. Large-scale learning of word relatedness with constraints. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1406–1414, 2012. [find](https://scholar.google.com/scholar?q=Guy+Halawi%2C+Gideon+Dror%2C+Evgeniy+Gabrilovich%2C+and+Yehuda+Koren%2E+Large%2Dscale+learning+of+word+relatedness+with+constraints%2E+In+Proceedings+of+the+18th+ACM+SIGKDD+international+conference+on+Knowledge+d)
22. Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695, 2015. [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+Simlex%2D999%3A+Evaluating+semantic+models+with+%28genuine%29+similarity+estimation%2E+Computational+Linguistics%2C+41%284%29%3A665%E2%80%93695%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/)
23. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531, 2015. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+Distilling+the+Knowledge+in+a+Neural+Network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
24. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018, 2018. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+Language+Model+Fine%2Dtuning+for+Text+Classification%2E+In+Proceedings+of+ACL+2018%2C+2018%2E)
25. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020. [find](https://scholar.google.com/scholar?q=Junjie+Hu%2C+Sebastian+Ruder%2C+Aditya+Siddhant%2C+Graham+Neubig%2C+Orhan+Firat%2C+and+Melvin+Johnson%2E+XTREME%3A+A+Massively+Multilingual+Multi%2Dtask+Benchmark+for+Evaluating+Cross%2Dlingual+Generalization%2E+In+Proce) [our card](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/)
26. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. In Proceedings of ICLR 2017, 2017. [find](https://scholar.google.com/scholar?q=Hakan+Inan%2C+Khashayar+Khosravi%2C+and+Richard+Socher%2E+Tying+Word+Vectors+and+Word+Classifiers%3A+A+Loss+Framework+for+Language+Modeling%2E+In+Proceedings+of+ICLR+2017%2C+2017%2E)
27. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. Transactions of the ACL 2017,… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+Google%E2%80%99s+Multiling)
28. Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=HJeT3yrtDr](https://openreview.net/forum?id=HJeT3yrtDr). [↗](https://openreview.net/forum?id=HJeT3yrtDr)
29. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv e-prints, art. arXiv:2001.08361, January 2020. [find](https://scholar.google.com/scholar?q=Jared+Kaplan%2C+Sam+McCandlish%2C+Tom+Henighan%2C+Tom+B%2E+Brown%2C+Benjamin+Chess%2C+Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+Jeffrey+Wu%2C+and+Dario+Amodei%2E+Scaling+Laws+for+Neural+Language+Models%2E+arXiv+e%2Dprints%2C+)
30. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66–71, Brussels, Belgium, November 2018.… [find](https://scholar.google.com/scholar?q=Taku+Kudo+and+John+Richardson%2E+SentencePiece%3A+A+simple+and+language+independent+subword+tokenizer+and+detokenizer+for+neural+text+processing%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods) [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
31. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=H1eA7AEtvS](https://openreview.net/forum?id=H1eA7AEtvS). [↗](https://openreview.net/forum?id=H1eA7AEtvS)
32. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv e-prints, art. arXiv:2006.16668, June 2020. [find](https://scholar.google.com/scholar?q=Dmitry+Lepikhin%2C+HyoukJoong+Lee%2C+Yuanzhong+Xu%2C+Dehao+Chen%2C+Orhan+Firat%2C+Yanping+Huang%2C+Maxim+Krikun%2C+Noam+Shazeer%2C+and+Zhifeng+Chen%2E+GShard%3A+Scaling+Giant+Models+with+Conditional+Computation+and+Autom)
33. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating Cross-lingual Extractive Question Answering. In Proceedings of ACL 2020, 2020. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Barlas+Og%CC%86uz%2C+Ruty+Rinott%2C+Sebastian+Riedel%2C+and+Holger+Schwenk%2E+MLQA%3A+Evaluating+Cross%2Dlingual+Extractive+Question+Answering%2E+In+Proceedings+of+ACL+2020%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/)
34. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of NAACL 2019, 2019a. [find](https://scholar.google.com/scholar?q=Nelson+F%2E+Liu%2C+Matt+Gardner%2C+Yonatan+Belinkov%2C+Matthew+E%2E+Peters%2C+and+Noah+A%2E+Smith%2E+Linguistic+Knowledge+and+Transferability+of+Contextual+Representations%2E+In+Proceedings+of+NAACL+2019%2C+2019a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre-training Approach. arXiv preprint arXiv:1907.11692, 2019b. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+RoBERTa%3A+A+Robustly+Optimized+BERT+Pre%2Dtraining+Approach%2E+arXiv+)
36. Minh-Thang Luong, Richard Socher, and Christopher D Manning. Better word representations with recursive neural networks for morphology. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 104–113, 2013. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Richard+Socher%2C+and+Christopher+D+Manning%2E+Better+word+representations+with+recursive+neural+networks+for+morphology%2E+In+Proceedings+of+the+Seventeenth+Conference+on+Computational+Na) [our card](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/)
37. Gábor Melis, Tomáš Kočiský, and Phil Blunsom. Mogrifier LSTM. In Proceedings of ICLR 2020, 2020. [find](https://scholar.google.com/scholar?q=Ga%CC%81bor+Melis%2C+Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+and+Phil+Blunsom%2E+Mogrifier+LSTM%2E+In+Proceedings+of+ICLR+2020%2C+2020%2E)
38. Paul Michel, Omer Levy, and Graham Neubig. Are Sixteen Heads Really Better than One? In Proceedings of NeurIPS 2019, 2019. [find](https://scholar.google.com/scholar?q=Paul+Michel%2C+Omer+Levy%2C+and+Graham+Neubig%2E+Are+Sixteen+Heads+Really+Better+than+One%3F+In+Proceedings+of+NeurIPS+2019%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/)
39. Subhabrata Mukherjee and Ahmed Hassan Awadallah. XtremeDistil : Multi-stage Distillation for Massive Multilingual Models. In Proceedings of ACL 2020, pp. 2221–2234, 2020. [find](https://scholar.google.com/scholar?q=Subhabrata+Mukherjee+and+Ahmed+Hassan+Awadallah%2E+XtremeDistil+%3A+Multi%2Dstage+Distillation+for+Massive+Multilingual+Models%2E+In+Proceedings+of+ACL+2020%2C+pp%2E+2221%E2%80%932234%2C+2020%2E)
40. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. Universal dependencies 2.2. 2018. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Mitchell+Abrams%2C+Z%CC%8Celjko+Agic%CC%81%2C+Lars+Ahrenberg%2C+Lene+Antonsen%2C+Maria+Jesus+Aranzabe%2C+Gashaw+Arutie%2C+Masayuki+Asahara%2C+Luma+Ateyah%2C+Mohammed+Attia%2C+et+al%2E+Universal+dependencies+2%2E2%2E+2018)
41. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Proceedings of ACL 2017, pp. 1946–1958, 2017. [find](https://scholar.google.com/scholar?q=Xiaoman+Pan%2C+Boliang+Zhang%2C+Jonathan+May%2C+Joel+Nothman%2C+Kevin+Knight%2C+and+Heng+Ji%2E+Cross%2Dlingual+name+tagging+and+linking+for+282+languages%2E+In+Proceedings+of+ACL+2017%2C+pp%2E+1946%E2%80%931958%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/)
42. Jonas Pfeiffer, Ivan Vuli, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of EMNLP 2020, 2020. [find](https://scholar.google.com/scholar?q=Jonas+Pfeiffer%2C+Ivan+Vuli%2C+Iryna+Gurevych%2C+and+Sebastian+Ruder%2E+MAD%2DX%3A+An+Adapter%2Dbased+Framework+for+Multi%2Dtask+Cross%2Dlingual+Transfer%2E+In+Proceedings+of+EMNLP+2020%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/)
43. Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, Iacer Calixto, and Samuel R Bowman. English intermediate-task training improves zero-shot cross-lingual transfer too. arXiv preprint arXiv:2005.13013, 2020. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Phu+Mon+Htut%2C+Yada+Pruksachatkun%2C+Haokun+Liu%2C+Clara+Vania%2C+Katharina+Kann%2C+Iacer+Calixto%2C+and+Samuel+R+Bowman%2E+English+intermediate%2Dtask+training+improves+zero%2Dshot+cross%2Dlingual+transfer)
44. Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996–5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1493. URL https:… [find](https://scholar.google.com/scholar?q=Telmo+Pires%2C+Eva+Schlinger%2C+and+Dan+Garrette%2E+How+multilingual+is+multilingual+BERT%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pp%2E+4996%E2%80%935001%2C+Florence) [our card](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
45. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 157–163, Valencia, Spain, April 2017. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+Using+the+output+embedding+to+improve+language+models%2E+In+Proceedings+of+the+15th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Linguistics%3A+Volume+) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/2…](http://jmlr.org/papers/v21/2…) [↗](http://jmlr.org/papers/v21/2…)
47. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP 2016, 2016. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+SQuAD%3A+100%2C000%2B+Questions+for+Machine+Comprehension+of+Text%2E+In+Proceedings+of+EMNLP+2016%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
48. Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15–18, 2019. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Matthew+E+Peters%2C+Swabha+Swayamdipta%2C+and+Thomas+Wolf%2E+Transfer+learning+in+natural+language+processing%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Ass) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
49. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2C+2019%2E)
50. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. arXiv:1907.10597, 2019. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Jesse+Dodge%2C+Noah+A%2E+Smith%2C+and+Oren+Etzioni%2E+arXiv%3A1907%2E10597%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/schwartz-2019-green-ai/)
51. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-B… [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Youlong+Cheng%2C+Niki+Parmar%2C+Dustin+Tran%2C+Ashish+Vaswani%2C+Penporn+Koanantakool%2C+Peter+Hawkins%2C+HyoukJoong+Lee%2C+Mingsheng+Hong%2C+Cliff+Young%2C+Ryan+Sepassi%2C+and+Blake+Hechtman%2E+Mesh%2Dtensorfl) [our card](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/)
52. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv e-prints, art. arXiv:1909.08053, September 2019. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2DLM%3A+Training+Multi%2DBillion+Parameter+Language+Models+Using+Model+Parallelism%2E+arXiv+e%2Dprint)
53. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT : a Compact Task-Agnostic BERT for Resource-Limited Devices. In Proceedings of ACL 2020, pp. 2158–2170, 2020. [find](https://scholar.google.com/scholar?q=Zhiqing+Sun%2C+Hongkun+Yu%2C+Xiaodan+Song%2C+Renjie+Liu%2C+Yiming+Yang%2C+and+Denny+Zhou%2E+MobileBERT+%3A+a+Compact+Task%2DAgnostic+BERT+for+Resource%2DLimited+Devices%2E+In+Proceedings+of+ACL+2020%2C+pp%2E+2158%E2%80%932170%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
54. Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating Transferability in Pretrained Language Models. arXiv e-prints, art. arXiv:2004.14975, April 2020. [find](https://scholar.google.com/scholar?q=Alex+Tamkin%2C+Trisha+Singh%2C+Davide+Giovanardi%2C+and+Noah+Goodman%2E+Investigating+Transferability+in+Pretrained+Language+Models%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2004%2E14975%2C+April+2020%2E)
55. Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1452. URL h… [find](https://scholar.google.com/scholar?q=Ian+Tenney%2C+Dipanjan+Das%2C+and+Ellie+Pavlick%2E+BERT+rediscovers+the+classical+NLP+pipeline%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pp%2E+4593%E2%80%934601%2C+Flo) [our card](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
56. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. arXiv preprint arXiv:1908.08962, 2019. [find](https://scholar.google.com/scholar?q=Iulia+Turc%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Well%2DRead+Students+Learn+Better%3A+On+the+Importance+of+Pre%2Dtraining+Compact+Models%2E+arXiv+preprint+arXiv%3A1908%2E08962%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
57. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Syst… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallac)
58. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of ACL 2019, 2019. [find](https://scholar.google.com/scholar?q=Elena+Voita%2C+David+Talbot%2C+Fedor+Moiseev%2C+Rico+Sennrich%2C+and+Ivan+Titov%2E+Analyzing+Multi%2DHead+Self%2DAttention%3A+Specialized+Heads+Do+the+Heavy+Lifting%2C+the+Rest+Can+Be+Pruned%2E+In+Proceedings+of+ACL+2019) [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-analyzing-multi-head-self-attention/)
59. Adina Williams, Nikita Nangia, and Samuel R. Bowman. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018, 2018. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+A+Broad%2DCoverage+Challenge+Corpus+for+Sentence+Understanding+through+Inference%2E+In+Proceedings+of+NAACL%2DHLT+2018%2C+2018%2E)
60. Green AI. arXiv preprint Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv preprint arXiv:1910.03771, 2019. [find](https://scholar.google.com/scholar?q=Green+AI%2E+arXiv+preprint+Thomas+Wolf%2C+Lysandre+Debut%2C+Victor+Sanh%2C+Julien+Chaumond%2C+Clement+Delangue%2C+Anthony+Moi%2C+Pierric+Cistac%2C+Tim+Rault%2C+Re%CC%81mi+Louf%2C+Morgan+Funtowicz%2C+and+Jamie+Brew%2E+HuggingFace%E2%80%99) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
61. Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 833–844, Hong Kon… [find](https://scholar.google.com/scholar?q=Shijie+Wu+and+Mark+Dredze%2E+Beto%2C+bentz%2C+becas%3A+The+surprising+cross%2Dlingual+effectiveness+of+BERT%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th) [our card](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/)
62. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of EMNLP 2019, pp. 3685–3690, 2019. [find](https://scholar.google.com/scholar?q=Yinfei+Yang%2C+Yuan+Zhang%2C+Chris+Tar%2C+and+Jason+Baldridge%2E+PAWS%2DX%3A+A+cross%2Dlingual+adversarial+dataset+for+paraphrase+identification%2E+In+Proceedings+of+EMNLP+2019%2C+pp%2E+3685%E2%80%933690%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/)
63. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pp. 3320–3328, 2014. [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Yoshua+Bengio%2C+and+Hod+Lipson%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3320%E2%80%933328%2C+2014%2E)
64. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger, and Yoav Artzi. Revisiting Few-sample BERT Fine-tuning. arXiv e-prints, art. arXiv:2006.05987, June 2020. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Felix+Wu%2C+Arzoo+Katiyar%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+Revisiting+Few%2Dsample+BERT+Fine%2Dtuning%2E+arXiv+e%2Dprints%2C+art%2E+arXiv%3A2006%2E05987%2C+June+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
65. Xiyou Zhou, Zhiyu Chen, Xiaoyong Jin, and William Yang Wang. HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing. arXiv preprint arXiv:2002.05829, 2020. [find](https://scholar.google.com/scholar?q=Xiyou+Zhou%2C+Zhiyu+Chen%2C+Xiaoyong+Jin%2C+and+William+Yang+Wang%2E+HULK%3A+An+Energy+Efficiency+Benchmark+Platform+for+Responsible+Natural+Language+Processing%2E+arXiv+preprint+arXiv%3A2002%2E05829%2C+2020%2E)
66. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Using Comparable Corpora, pp. 39–42, 2018. [find](https://scholar.google.com/scholar?q=Pierre+Zweigenbaum%2C+Serge+Sharoff%2C+and+Reinhard+Rapp%2E+Overview+of+the+third+bucc+shared+task%3A+Spotting+parallel+sentences+in+comparable+corpora%2E+In+Proceedings+of+11th+Workshop+on+Building+and+Using+C)