---
title: "English Intermediate-Task Training Improves Zero-Shot"
description: "The authors test whether STILTs-style intermediate-task training, known to help in… 7.0 points. 55 references, cited by 54 papers here."
authors: ["Jason Phang", "Iacer Calixto", "Phu Mon Htut", "Yada Pruksachatkun", "Haokun Liu", "Clara Vania", "Katharina Kann", "Samuel R. Bowman"]
year: 2020
venue: "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing"
source: "https://aclanthology.org/2020.aacl-main.56/"
doi: "10.18653/v1/2020.aacl-main.56"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/phang-2020-english-intermediate-task-training-improves/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/phang-2020-english-intermediate-task-training-improves/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

# English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

**Authors:** [Jason Phang](https://md.research.elamur.ai/free/en/authors/jason-phang/), [Iacer Calixto](https://md.research.elamur.ai/free/en/authors/iacer-calixto/), [Phu Mon Htut](https://md.research.elamur.ai/free/en/authors/phu-mon-htut/), [Yada Pruksachatkun](https://md.research.elamur.ai/free/en/authors/yada-pruksachatkun/), [Haokun Liu](https://md.research.elamur.ai/free/en/authors/haokun-liu/), [Clara Vania](https://md.research.elamur.ai/free/en/authors/clara-vania/), [Katharina Kann](https://md.research.elamur.ai/free/en/authors/katharina-kann/) and [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2020 · **Venue:** [Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing](https://md.research.elamur.ai/free/en/venues/aacl/) ([AACL 2020](https://md.research.elamur.ai/free/en/venues/aacl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.aacl-main.56/) · [PDF](https://aclanthology.org/2020.aacl-main.56.pdf) · [DOI 10.18653/v1/2020.aacl-main.56](https://doi.org/10.18653/v1/2020.aacl-main.56) · [OpenAlex](https://openalex.org/W3030045039)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [54](https://md.research.elamur.ai/free/en/papers/phang-2020-english-intermediate-task-training-improves/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Multilingual pretrained encoders are typically fine-tuned on English task data and then applied to other languages without target-language labels. The authors test whether STILTs-style intermediate-task training, known to help in English-only settings, also helps in this zero-shot cross-lingual scenario. Starting from XLM-R Large, they add a training phase on one of nine English tasks (ANLI+, MNLI, QQP, SQuAD v1.1/v2.0, HellaSwag, CCG, Cosmos QA, CommonsenseQA) or on all nine jointly. They then fine-tune on English data for each of the nine XTREME target tasks and evaluate across up to 40 languages. Two variants aim to reduce forgetting of non-English knowledge: mixing multilingual MLM updates into the intermediate phase, and machine-translating intermediate data into German, Russian and Swahili. Gains are largest on the sentence retrieval tasks BUCC and Tatoeba, moderate on question answering, and absent on XNLI. Choosing the best intermediate model per task raises the XTREME average by 5.4 points, the state of the art in June 2020. This shows that English-only labeled resources can improve performance in other languages.

## Key points
- On the development set, intermediate training on SQuAD, MNLI and HellaSwag improves the average by 8.2, 7.5 and 7.0 points, respectively. Multi-task training on all nine tasks does slightly better, with an 8.7-point gain.
- Compared with the best published XLM-R baselines, the best models gain 5.9 points on BUCC and 23.9 points on Tatoeba. These two retrieval tasks have no training data of their own and are scored by representation similarity.
- With SQuAD as the intermediate task, QA scores improve by 2.1 F1 on XQuAD, 0.8 F1 on MLQA and 10.4 F1 on TyDiQA-GoldP. XNLI does not benefit.
- Neither variant consistently beats plain English intermediate training: adding multilingual MLM did not help reliably, and neither did machine-translated intermediate data (German, Russian, Swahili). Translated HellaSwag in Russian and Swahili scored near chance on validation, yet it still helped PAN-X and TyDiQA.
- The 5.4-point XTREME improvement over XLM-R Large was the state of the art as of June 2020. By September 2020, Fang et al. (2020) held the top spot with an orthogonal method.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 70 works · 39,977 citations
- [University of Colorado Boulder](https://md.research.elamur.ai/free/en/universities/university-of-colorado-boulder/) — University, Boulder, United States · 52 works · 49,647 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann and Samuel R. Bowman. 2020. English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{phang2020english,
  title = {{English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too}},
  author = {Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann and Samuel R. Bowman},
  booktitle = {Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.56},
  url = {https://doi.org/10.18653/v1/2020.aacl-main.56}
}
```

- [← previous "I'd rather just go to bed": Understanding Indirect Answers](https://md.research.elamur.ai/free/en/papers/louis-2020-i-d-rather-just-go/)
- [next → Towards Robustifying NLI Models Against Lexical Dataset Biases](https://md.research.elamur.ai/free/en/papers/zhou-2020-robustifying-nli-models-against-lexical/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

55 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Željko Agić. 2017. [Cross-lingual parser selection for low-resource languages](https://md.research.elamur.ai/free/en/papers/agic-2017-cross-lingual-parser-selection-low/). In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 1–10, Gothenburg, Sweden. Association for Computational Linguistics.
2. Željko Agić, Anders Johannsen, Barbara Plank, Héctor Martı́nez Alonso, Natalie Schluter, and Anders Søgaard. 2016. [Multilingual projection for parsing truly low-resource languages](https://md.research.elamur.ai/free/en/papers/agic-2016-multilingual-projection-parsing-truly-low/). Transactions of the Association for Computational Linguistics, 4:301– 312.
3. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. [Massively multilingual neural machine translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884…
4. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. [Translation artifacts in cross-lingual transfer learning](https://md.research.elamur.ai/free/en/papers/artetxe-2020-translation-artifacts-cross-lingual-transfer/). arXiv preprint arXiv:2004.04721.
5. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. [On the cross-lingual transferability of monolingual representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
6. Mikel Artetxe and Holger Schwenk. 2019. [Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). Transactions of the Association for Computational Linguistics, 7:597–610.
7. Samuel R. Bowman, Gabor Angeli, Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in)
8. Samuel R Bowman, Jennimaria Palomaki, Livio Baldini Soares, and Emily Pitler. 2020. Collecting entailment data for pretraining: New protocols and negative results. arXiv preprint arXiv:2004.11997. — [arXiv:2004.11997](https://arxiv.org/abs/2004.11997) [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Jennimaria+Palomaki%2C+Livio+Baldini+Soares%2C+and+Emily+Pitler%2E+2020%2E+Collecting+entailment+data+for+pretraining%3A+New+protocols+and+negative+results%2E+arXiv+preprint+arXiv%3A2004%2E11997%2E+%E2%80%94+%5B)
9. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. [Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/). Transactions of the Association for Computational Linguistics.
10. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association f… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzma%CC%81n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dling)
11. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 (NeurIPS), pages 7059–7069. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+H%2E+Wallach%2C+H%2E+Larochelle%2C+A%2E+Beygelzimer%2C+F%2E+d%27Alche%CC%81%2DBuc%2C+E%2E+Fox%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Neural+)
12. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Ruty+Rinott%2C+Guillaume+Lample%2C+Adina+Williams%2C+Samuel+Bowman%2C+Holger+Schwenk%2C+and+Veselin+Stoyanov%2E+2018%2E+XNLI%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+Proceedings+of+the)
13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
15. Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. 2020. FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding. arXiv e-prints, page arXiv:2009.05166. — [arXiv:2009.05166](https://arxiv.org/abs/2009.05166) [find](https://scholar.google.com/scholar?q=Yuwei+Fang%2C+Shuohang+Wang%2C+Zhe+Gan%2C+Siqi+Sun%2C+and+Jingjing+Liu%2E+2020%2E+FILTER%3A+An+Enhanced+Fusion+Method+for+Cross%2Dlingual+Language+Understanding%2E+arXiv+e%2Dprints%2C+page+arXiv%3A2009%2E05166%2E+%E2%80%94+%5BarXiv%3A2009%2E0)
16. Julia Hockenmaier and Mark Steedman. 2007. [CCGbank: A corpus of CCG derivations and dependency structures extracted from the Penn treebank](https://md.research.elamur.ai/free/en/papers/hockenmaier-2007-ccgbank-corpus-ccg-derivations-dependency/). Computational Linguistics, 33(3):355–396.
17. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. [Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/). Proceedings of the 37th International Conference on Machine Learning.
18. Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019a. [Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks](https://md.research.elamur.ai/free/en/papers/huang-2019-unicoder-universal-language-encoder-pre/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internation…
19. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019b. [Cosmos QA: Machine reading comprehension with contextual commonsense reasoning](https://md.research.elamur.ai/free/en/papers/huang-2019-cosmos-qa-machine-reading-comprehension/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langu…
20. Rebecca Hwa, Philip Resnik, Amy Weinberg, Clara Cabezas, and Okan Kolak. 2005. Bootstrapping parsers via syntactic projection across parallel texts. Nat. Lang. Eng., 11(3):311–325. [find](https://scholar.google.com/scholar?q=Rebecca+Hwa%2C+Philip+Resnik%2C+Amy+Weinberg%2C+Clara+Cabezas%2C+and+Okan+Kolak%2E+2005%2E+Bootstrapping+parsers+via+syntactic+projection+across+parallel+texts%2E+Nat%2E+Lang%2E+Eng%2E%2C+11%283%29%3A311%E2%80%93325%2E)
21. Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the As… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V%2E+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+Macduff+Hughes%2C+and+Jeffrey+Dean%2E+2017%2E+Google%E2%80%99s+mu)
22. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018. [Marian: Fast neural machine translation in C++](https://md.research.elamur.ai/free/en/papers/junczys-dowmunt-2018-marian-fast-neural-machine-translation/). In Proceedings of ACL 2018, Syste…
23. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
24. Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E Peters, Ashish Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. Proceedings of the 37th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Ronan+Le+Bras%2C+Swabha+Swayamdipta%2C+Chandra+Bhagavatula%2C+Rowan+Zellers%2C+Matthew+E+Peters%2C+Ashish+Sabharwal%2C+and+Yejin+Choi%2E+2020%2E+Adversarial+filters+of+dataset+biases%2E+Proceedings+of+the+37th+Internat)
25. Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. [MLQA: Evaluating cross-lingual extractive question answering](https://md.research.elamur.ai/free/en/papers/lewis-2019-mlqa-evaluating-cross-lingual-extractive/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315– 7330, Online. Association for Computational Lingu…
26. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. [Multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496, Florence, Italy. Association for Computational Linguistic…
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019b%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+)
28. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. [Cheap translation for cross-lingual named entity recognition](https://md.research.elamur.ai/free/en/papers/mayhew-2017-cheap-translation-cross-lingual-named/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2536–2545, Copenhagen, Denmark. Association for Computational Linguistics.
29. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. [Multi-source transfer of delexicalized dependency parsers](https://md.research.elamur.ai/free/en/papers/mcdonald-2011-multi-source-transfer-delexicalized-dependency/). In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 62–72, Edinburgh, Scotland, UK. Association for Computational Linguistics.
30. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+Proceedings+of+the+58th+Annual+Meetin)
31. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, and et al. 2018. Universal Dependencies 2.2. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Facu… [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Mitchell+Abrams%2C+Z%CC%8Celjko+Agic%CC%81%2C+Lars+Ahrenberg%2C+Lene+Antonsen%2C+Maria+Jesus+Aranzabe%2C+Gashaw+Arutie%2C+Masayuki+Asahara%2C+Luma+Ateyah%2C+Mohammed+Attia%2C+and+et+al%2E+2018%2E+Universal+Dependencies)
32. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. [Cross-lingual name tagging and linking for 282 languages](https://md.research.elamur.ai/free/en/papers/pan-2017-cross-lingual-name-tagging-linking/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. A…
33. [Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and…](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
34. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. [MAD-X: An adapter-based framework for multi-task cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP).
35. Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/). Unpublished manuscript available on arXiv.
36. Bowman. 2020. [Intermediate-task transfer learning with pretrained language models: When and why does it work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online. Association for Computational Linguistics.
37. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
38. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. [Know what you don’t know: Unanswerable questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Lin…
39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
40. Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Robert+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Thirty%2DFirst+AAAI+Conference+on+Artificial+Intelligence%2E)
41. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. [CommonsenseQA: A question answering challenge targeting commonsense knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2019-commonsenseqa-question-answering-challenge-targeting/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
42. Jörg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), Istanbul, Turkey. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann%2E+2012%2E+Parallel+Data%2C+Tools+and+Interfaces+in+OPUS%2E+In+Proceedings+of+the+Eight+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+Istanbul%2C+Turkey%2E+European+Lang)
43. Jason Phang, Phil Yeres, Jesse Swanson, Haokun Liu, Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Phu Mon Htut, , Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Edouard Grave, Najoung Kim, Thibault Févry, Berlin Chen, Nikita Nangia, Anh… [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Phil+Yeres%2C+Jesse+Swanson%2C+Haokun+Liu%2C+Alex+Wang%2C+Ian+F%2E+Tenney%2C+Yada+Pruksachatkun%2C+Phu+Mon+Htut%2C+%2C+Katherin+Yu%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghu+Pappagari%2C+Shuning+Jin%2C+R%2E+Thomas+McCoy%2C+Rom)
44. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Jo%CC%88rg+Tiedemann+and+Santhosh+Thottingal%2E+2020%2E+OPUS%2DMT+%E2%80%94+Building+open+translation+services+for+the+World%2E+In+Proceedings+of+the+22nd+Annual+Conferenec+of+the+European+Association+for+Machine+Translat)
45. [Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Tu Vu, Tong Wang, Tsend…](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
46. Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pag… [find](https://scholar.google.com/scholar?q=Shuning+Jin%2C+Berlin+Chen%2C+Benjamin+Van+Durme%2C+Edouard+Grave%2C+Ellie+Pavlick%2C+and+Samuel+R%2E+Bowman%2E+2019a%2E+Can+you+tell+me+how+to+get+past+sesame+street%3F+sentence%2Dlevel+pretraining+beyond+language+model)
47. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019b. Can you tell me how to get past sesame street? sentence-… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghavendra+Pappagari%2C+R%2E+Thomas+McCoy%2C+Roma+Patel%2C+Najoung+Kim%2C+Ian+Tenney%2C+Yinghui+Huang%2C+Katherin+Yu%2C+Shuning+Jin%2C+Berlin+Chen%2C+Benjamin+Van+Durme%2C+Edouard+Grave%2C+)
48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). Unpublished manuscript available on arXiv.
50. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. [PAWS-x: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin…
51. Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373. [find](https://scholar.google.com/scholar?q=Dani+Yogatama%2C+Cyprien+de+Masson+d%E2%80%99Autume%2C+Jerome+Connor%2C+Tomas+Kocisky%2C+Mike+Chrzanowski%2C+Lingpeng+Kong%2C+Angeliki+Lazaridou%2C+Wang+Ling%2C+Lei+Yu%2C+Chris+Dyer%2C+et+al%2E+2019%2E+Learning+and+evaluating+genera)
52. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93– 104, Brussels, Belgium. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Yonatan+Bisk%2C+Roy+Schwartz%2C+and+Yejin+Choi%2E+2018%2E+SWAG%3A+A+large%2Dscale+adversarial+dataset+for+grounded+commonsense+inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
53. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791– 4800, Florence, Italy. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Com)
54. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2017. Overview of the second BUCC shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pages 60–67, Vancouver, Canada. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=Pierre+Zweigenbaum%2C+Serge+Sharoff%2C+and+Reinhard+Rapp%2E+2017%2E+Overview+of+the+second+BUCC+shared+task%3A+Spotting+parallel+sentences+in+comparable+corpora%2E+In+Proceedings+of+the+10th+Workshop+on+Building+)
55. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third BUCC shared task:spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshopon Building and Using Comparable Corpora, pages 39–42. [find](https://scholar.google.com/scholar?q=Pierre+Zweigenbaum%2C+Serge+Sharoff%2C+and+Reinhard+Rapp%2E+2018%2E+Overview+of+the+third+BUCC+shared+task%3Aspotting+parallel+sentences+in+comparable+corpora%2E+In+Proceedings+of+11th+Workshopon+Building+and+Usi)

*Updated 24 October 2026*