---
title: "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — Raffel 2019"
description: "The authors from Google propose a unified view of transfer learning in NLP. 132 references, cited by 7,433 papers here."
authors: ["Colin Raffel", "Noam Shazeer", "Adam Roberts", "Katherine Lee", "Sharan Narang", "Michael Matena", "Yanqi Zhou", "Wei Li", "Peter J. Liu"]
year: 2019
venue: "arXiv:1910.10683"
source: "https://arxiv.org/abs/1910.10683"
doi: "10.48550/arxiv.1910.10683"
arxiv: "1910.10683"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

# Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

**Authors:** [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/), [Noam Shazeer](https://md.research.elamur.ai/free/en/authors/noam-shazeer/), [Adam Roberts](https://md.research.elamur.ai/free/en/authors/adam-roberts/), [Katherine Lee](https://md.research.elamur.ai/free/en/authors/katherine-lee/), [Sharan Narang](https://md.research.elamur.ai/free/en/authors/sharan-narang/), [Michael Matena](https://md.research.elamur.ai/free/en/authors/michael-matena/), [Yanqi Zhou](https://md.research.elamur.ai/free/en/authors/yanqi-zhou/), [Wei Li](https://md.research.elamur.ai/free/en/authors/wei-li/) and [Peter J. Liu](https://md.research.elamur.ai/free/en/authors/peter-j-liu/) · **Year:** 2019 · **Venue:** arXiv:1910.10683 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1910.10683) · [PDF](https://arxiv.org/pdf/1910.10683) · [DOI 10.48550/arxiv.1910.10683](https://doi.org/10.48550/arxiv.1910.10683) · [arXiv 1910.10683](https://arxiv.org/abs/1910.10683)

**Citations:** [7433](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

The authors from Google propose a unified view of transfer learning in NLP. They cast every language task as the conversion of input text into output text: translation, classification, question answering, summarization. This makes it possible to use a single model, a single loss function and the same hyperparameters for all tasks. On this basis, the authors compare existing approaches: pre-training objectives, Transformer architectures, unlabeled corpora, fine-tuning strategies and multi-task learning. The goal of the work is not a new method but a careful comparison of existing ones. For the experiments they assembled the C4 corpus (Colossal Clean Crawled Corpus): hundreds of gigabytes of cleaned English text from Common Crawl. They combined the best choices and scaled them up into the T5 model (Text-to-Text Transfer Transformer) with up to 11 billion parameters. It achieved new state-of-the-art results on many benchmarks in summarization, question answering and text classification. The corpus, code and pre-trained models have been released openly. As a result, the work has become a foundational resource for further research and for tasks where labeled data is scarce.

## Key points

- **Text-to-text format:** the task type is specified by a text prefix on the input, for example `translate English to German:`, `summarize:`, `cola sentence:`. Even regression (STS-B) is solved by the model generating a number as text.
- **Architecture:** an encoder-decoder Transformer, close to the original by Vaswani et al. There are two changes: a simplified layer normalization without an additive bias, and relative position embeddings in the form of a scalar added to the attention logits.
- **C4 corpus:** hundreds of GB of filtered English web text. The source is Common Crawl, which yields about 20 TB of text each month.
- **Comparison:** multi-task learning without subsequent fine-tuning generally underperforms the "unsupervised pre-training + fine-tuning" scheme. If, however, multi-task pre-training is carried out first and the model is then fine-tuned, the result is nearly the same. Under a fixed compute budget, increasing the model size or the training duration gives a noticeable gain: on SuperGLUE-like tasks this amounts to several points.
- **Resources:** the code, C4 and T5 checkpoints have been released openly: github.com/google-research/text-to-text-transfer-transformer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cite

Cite as: Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2019. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv:1910.10683.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{raffel2019exploring,
  title = {{Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer}},
  author = {Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu},
  year = {2019},
  doi = {10.48550/arxiv.1910.10683},
  eprint = {1910.10683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.10683}
}
```

- [← previous BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
- [next → A Survey on Bias and Fairness in Machine Learning](https://md.research.elamur.ai/free/en/papers/mehrabi-2019-survey-bias-fairness-machine-learning/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

132 references, 69 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Dokook+Choe%2C+Noah+Constant%2C+Mandy+Guo%2C+and+Llion+Jones%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+20)
2. Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimization for large-scale learning. arXiv preprint arXiv:1901.11150, 2019. — [arXiv:1901.11150](https://arxiv.org/abs/1901.11150) [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Vineet+Gupta%2C+Tomer+Koren%2C+and+Yoram+Singer%2E+Memory%2Defficient+adaptive+optimization+for+large%2Dscale+learning%2E+arXiv+preprint+arXiv%3A1901%2E11150%2C+2019%2E+%E2%80%94+%5BarXiv%3A1901%2E11150%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
3. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019, 2019. [our card](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/)
4. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
5. Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785, 2019. [find](https://scholar.google.com/scholar?q=Alexei+Baevski%2C+Sergey+Edunov%2C+Yinhan+Liu%2C+Luke+Zettlemoyer%2C+and+Michael+Auli%2E+Cloze%2Ddriven+pretraining+of+self%2Dattention+networks%2E+arXiv+preprint+arXiv%3A1903%2E07785%2C+2019%2E)
6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Third International Conference on Learning Representations, 2015. [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/)
7. Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint arXiv:1909.08478, 2019. [our card](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/)
8. Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. [our card](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/)
9. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translati… [our card](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/)
10. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, et al. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translat… [our card](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
11. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, 2016. [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/)
12. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015. [our card](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/)
13. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. N-gram counts and language models from the common crawl. In LREC, 2014. [our card](https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/)
14. Rich Caruana. Multitask learning. Machine learning, 28(1), 1997. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+Multitask+learning%2E+Machine+learning%2C+28%281%29%2C+1997%2E)
15. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
16. Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016. [our card](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/)
17. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. [our card](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
18. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V+Le%2C+and+Christopher+D+Manning%2E+Electra%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+arXiv+preprint+arXiv%3A2003%2E10555%2C+2020%2E)
19. Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449, 2018. [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-senteval-evaluation-toolkit-universal-sentence/)
20. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017. [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
21. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+2005%2E)
22. Andrew M. Dai and Quoc V. Le. Semi-supervised sequence learning. In Advances in neural information processing systems, 2015. [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai+and+Quoc+V%2E+Le%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+2015%2E)
23. Marie-Catherine De Marneff, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. In Sinn und Bedeutung 23, 2019. [our card](https://md.research.elamur.ai/free/en/papers/marneffe-2019-commitmentbank-investigating-projection-naturally/)
24. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 2009. [find](https://scholar.google.com/scholar?q=Jia+Deng%2C+Wei+Dong%2C+Richard+Socher%2C+Li%2DJia+Li%2C+Kai+Li%2C+and+Li+Fei%2DFei%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+In+2009+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+2009%2E)
25. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
26. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
27. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197, 2019. [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
28. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381, 2018. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+Understanding+back%2Dtranslation+at+scale%2E+arXiv+preprint+arXiv%3A1808%2E09381%2C+2018%2E)
29. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893, 2018. [our card](https://md.research.elamur.ai/free/en/papers/grave-2018-learning-word-vectors-157-languages/)
30. Alex Graves. Generating sequences with recurrent neural networks. arXiv:1308.0850, 2013. [find](https://scholar.google.com/scholar?q=Alex+Graves%2E+Generating+sequences+with+recurrent+neural+networks%2E+arXiv%3A1308%2E0850%2C+2013%2E)
31. Ivan Habernal, Omnia Zayed, and Iryna Gurevych. C4Corpus: Multilingual web-size corpus with free license. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 914–922, 2016. [find](https://scholar.google.com/scholar?q=Ivan+Habernal%2C+Omnia+Zayed%2C+and+Iryna+Gurevych%2E+C4Corpus%3A+Multilingual+web%2Dsize+corpus+with+free+license%2E+In+Proceedings+of+the+Tenth+International+Conference+on+Language+Resources+and+Evaluation+%28LRE)
32. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+2016%2E)
33. Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking ImageNet pre-training. arXiv preprint arXiv:1811.08883, 2018. — [arXiv:1811.08883](https://arxiv.org/abs/1811.08883) [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Ross+Girshick%2C+and+Piotr+Doll%C3%A1r%2E+Rethinking+ImageNet+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A1811%2E08883%2C+2018%2E+%E2%80%94+%5BarXiv%3A1811%2E08883%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1811%2E08883%29)
34. Pengcheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. A hybrid neural network model for commonsense reasoning. arXiv preprint arXiv:1907.11983, 2019. [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Xiaodong+Liu%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+A+hybrid+neural+network+model+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1907%2E11983%2C+2019%2E)
35. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, 2015. [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
36. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. [our card](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/)
37. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016. [our card](https://md.research.elamur.ai/free/en/papers/hill-2016-learning-distributed-representations-sentences-unlabelled/)
38. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
39. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. arXiv preprint arXiv:1902.00751, 2019. [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+Bruna+Morrone%2C+Quentin+De+Laroussilhe%2C+Andrea+Gesmundo%2C+Mona+Attariyan%2C+and+Sylvain+Gelly%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+arXiv+pre)
40. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2C+2018%2E)
41. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In Seventh International Conference on Learning Representations, 2018… [find](https://scholar.google.com/scholar?q=Cheng%2DZhi+Anna+Huang%2C+Ashish+Vaswani%2C+Jakob+Uszkoreit%2C+Ian+Simon%2C+Curtis+Hawthorne%2C+Noam+Shazeer%2C+Andrew+M%2E+Dai%2C+Matthew+D%2E+Hoffman%2C+Monica+Dinculescu%2C+and+Douglas+Eck%2E+Music+transformer%3A+Generating+m)
42. Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes ImageNet good for transfer learning? arXiv preprint arXiv:1608.08614, 2016. — [arXiv:1608.08614](https://arxiv.org/abs/1608.08614) [find](https://scholar.google.com/scholar?q=Minyoung+Huh%2C+Pulkit+Agrawal%2C+and+Alexei+A%2E+Efros%2E+What+makes+ImageNet+good+for+transfer+learning%3F+arXiv+preprint+arXiv%3A1608%2E08614%2C+2016%2E+%E2%80%94+%5BarXiv%3A1608%2E08614%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1608%2E08614%29)
43. Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First Quora dataset release: Question pairs. [https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs), 2017. [↗](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs)
44. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, 2014. [our card](https://md.research.elamur.ai/free/en/papers/jia-2014-caffe-convolutional-architecture-fast-feature/)
45. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351, 2019. [find](https://scholar.google.com/scholar?q=Xiaoqi+Jiao%2C+Yichun+Yin%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Xiao+Chen%2C+Linlin+Li%2C+Fang+Wang%2C+and+Qun+Liu%2E+TinyBERT%3A+Distilling+BERT+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1909%2E10351%2C+2019%2E)
46. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
47. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
48. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [find](https://scholar.google.com/scholar?q=Rafal+Jozefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+Exploring+the+limits+of+language+modeling%2E+arXiv+preprint+arXiv%3A1602%2E02410%2C+2016%2E)
49. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2014. [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2014-convolutional-neural-network-modelling-sentences/)
50. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019a. [our card](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
51. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286, 2019b. — [arXiv:1904.09286](https://arxiv.org/abs/1904.09286) [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Unifying+question+answering+and+text+classification+via+span+extraction%2E+arXiv+preprint+arXiv%3A1904%2E09286%2C+2019b%2E+%E2%80%94+%5BarXiv%3A1904%2E0)
52. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2018-looking-beyond-surface-challenge-set/)
53. Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, 2015. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R%2E+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+2015%2E)
54. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for Winograd schema challenge. arXiv preprint arXiv:1905.06290, 2019. — [arXiv:1905.06290](https://arxiv.org/abs/1905.06290) [find](https://scholar.google.com/scholar?q=Vid+Kocijan%2C+Ana%2DMaria+Cretu%2C+Oana%2DMaria+Camburu%2C+Yordan+Yordanov%2C+and+Thomas+Lukasiewicz%2E+A+surprisingly+robust+trick+for+Winograd+schema+challenge%2E+arXiv+preprint+arXiv%3A1905%2E06290%2C+2019%2E+%E2%80%94+%5BarXiv%3A19)
55. Jakub Konečnỳ, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed optimization beyond the datacenter. arXiv preprint arXiv:1511.03575, 2015. — [arXiv:1511.03575](https://arxiv.org/abs/1511.03575) [find](https://scholar.google.com/scholar?q=Jakub+Kone%C4%8Dny%CC%80%2C+Brendan+McMahan%2C+and+Daniel+Ramage%2E+Federated+optimization%3A+Distributed+optimization+beyond+the+datacenter%2E+arXiv+preprint+arXiv%3A1511%2E03575%2C+2015%2E+%E2%80%94+%5BarXiv%3A1511%2E03575%5D%28https%3A%2F%2Farxiv%2Eor)
56. Jakub Konečnỳ, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016. — [arXiv:1610.05492](https://arxiv.org/abs/1610.05492) [find](https://scholar.google.com/scholar?q=Jakub+Kone%C4%8Dny%CC%80%2C+H%2E+Brendan+McMahan%2C+Felix+X%2E+Yu%2C+Peter+Richt%C3%A1rik%2C+Ananda+Theertha+Suresh%2C+and+Dave+Bacon%2E+Federated+learning%3A+Strategies+for+improving+communication+efficiency%2E+arXiv+preprint+arXiv%3A16)
57. Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do better ImageNet models transfer better? arXiv preprint arXiv:1805.08974, 2018. — [arXiv:1805.08974](https://arxiv.org/abs/1805.08974) [find](https://scholar.google.com/scholar?q=Simon+Kornblith%2C+Jonathon+Shlens%2C+and+Quoc+V%2E+Le%2E+Do+better+ImageNet+models+transfer+better%3F+arXiv+preprint+arXiv%3A1805%2E08974%2C+2018%2E+%E2%80%94+%5BarXiv%3A1805%2E08974%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1805%2E08974%29)
58. Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2E+One+weird+trick+for+parallelizing+convolutional+neural+networks%2E+arXiv+preprint+arXiv%3A1404%2E5997%2C+2014%2E)
59. Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959, 2018. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/)
60. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. [our card](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
61. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2C+2019%2E)
62. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1909%2E11)
63. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+Winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+2012%2E)
64. Qi Li. Literature survey: domain adaptation algorithms for natural language processing. 2012. [find](https://scholar.google.com/scholar?q=Qi+Li%2E+Literature+survey%3A+domain+adaptation+algorithms+for+natural+language+processing%2E+2012%2E)
65. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, 2004. [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
66. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
67. Peter J. Liu, Yu-An Chung, and Jie Ren. SummAE: Zero-shot abstractive text summarization using length-agnostic auto-encoders. arXiv preprint arXiv:1910.00998, 2019a. — [arXiv:1910.00998](https://arxiv.org/abs/1910.00998) [find](https://scholar.google.com/scholar?q=Peter+J%2E+Liu%2C+Yu%2DAn+Chung%2C+and+Jie+Ren%2E+SummAE%3A+Zero%2Dshot+abstractive+text+summarization+using+length%2Dagnostic+auto%2Dencoders%2E+arXiv+preprint+arXiv%3A1910%2E00998%2C+2019a%2E+%E2%80%94+%5BarXiv%3A1910%2E00998%5D%28https%3A%2F%2Farxiv)
68. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational L… [our card](https://md.research.elamur.ai/free/en/papers/liu-2015-representation-learning-multi-task-deep/)
69. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019b. [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
70. Yang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318, 2019. — [arXiv:1903.10318](https://arxiv.org/abs/1903.10318) [find](https://scholar.google.com/scholar?q=Yang+Liu%2E+Fine%2Dtune+BERT+for+extractive+summarization%2E+arXiv+preprint+arXiv%3A1903%2E10318%2C+2019%2E+%E2%80%94+%5BarXiv%3A1903%2E10318%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1903%2E10318%29)
71. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019c. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+arXiv+p)
72. Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893, 2018. [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
73. Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. [find](https://scholar.google.com/scholar?q=Dhruv+Mahajan%2C+Ross+Girshick%2C+Vignesh+Ramanathan%2C+Kaiming+He%2C+Manohar+Paluri%2C+Yixuan+Li%2C+Ashwin+Bharambe%2C+and+Laurens+van+der+Maaten%2E+Exploring+the+limits+of+weakly+supervised+pretraining%2E+In+Proceedi)
74. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
75. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
76. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, 2013b. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
77. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence RNNs and beyond. arXiv preprint arXiv:1602.06023, 2016. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+Nogueira+dos+santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+RNNs+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2C+2)
78. Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014. [find](https://scholar.google.com/scholar?q=Maxime+Oquab%2C+Leon+Bottou%2C+Ivan+Laptev%2C+and+Josef+Sivic%2E+Learning+and+transferring+mid%2Dlevel+image+representations+using+convolutional+neural+networks%2E+In+Proceedings+of+the+IEEE+conference+on+compute)
79. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 2002. [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
80. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017. [our card](https://md.research.elamur.ai/free/en/papers/paulus-2017-deep-reinforced-model-abstractive-summarization/)
81. Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014. [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
82. Matthew Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987, 2019. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Sebastian+Ruder%2C+and+Noah+A%2E+Smith%2E+To+tune+or+not+to+tune%3F+adapting+pretrained+representations+to+diverse+tasks%2E+arXiv+preprint+arXiv%3A1903%2E05987%2C+2019%2E)
83. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
84. Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
85. Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating context-sensitive representations. arXiv preprint arXiv:1808.09121, 2018. [our card](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/)
86. Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771, 2018. [our card](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
87. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2C+2018%2E)
88. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2C+2019%2E)
89. Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: the Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, 2012. [our card](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/)
90. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
91. Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683, 2016. [our card](https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/)
92. Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. Snorkel MeTaL: Weak supervision for multi-task learning. In Proceedings of the Second Workshop on Data Management for End-To-End Machine Learning, 2018. [find](https://scholar.google.com/scholar?q=Alex+Ratner%2C+Braden+Hancock%2C+Jared+Dunnmon%2C+Roger+Goldman%2C+and+Christopher+R%C3%A9%2E+Snorkel+MeTaL%3A+Weak+supervision+for+multi%2Dtask+learning%2E+In+Proceedings+of+the+Second+Workshop+on+Data+Management+for+End)
93. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. [find](https://scholar.google.com/scholar?q=Melissa+Roemmele%2C+Cosmin+Adrian+Bejan%2C+and+Andrew+S+Gordon%2E+Choice+of+plausible+alternatives%3A+An+evaluation+of+commonsense+causal+reasoning%2E+In+2011+AAAI+Spring+Symposium+Series%2C+2011%2E)
94. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017. [our card](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
95. Sebastian Ruder. Neural transfer learning for natural language processing. PhD thesis, NUI Galway, 2019. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+Neural+transfer+learning+for+natural+language+processing%2E+PhD+thesis%2C+NUI+Galway%2C+2019%2E)
96. Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pages 15–18, 2019. [our card](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
97. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 2015. [our card](https://md.research.elamur.ai/free/en/papers/russakovsky-2014-imagenet-large-scale-visual-recognition/)
98. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Lysandre+Debut%2C+Julien+Chaumond%2C+and+Thomas+Wolf%2E+DistilBERT%2C+a+distilled+version+of+BERT%3A+smaller%2C+faster%2C+cheaper+and+lighter%2E+arXiv+preprint+arXiv%3A1910%2E01108%2C+2019%2E)
99. Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+arXiv+preprint+arXiv%3A1704%2E04368%2C+2017%2E)
100. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
101. Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E. Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018. — [arXiv:1811.03600](https://arxiv.org/abs/1811.03600) [find](https://scholar.google.com/scholar?q=Christopher+J+Shallue%2C+Jaehoon+Lee%2C+Joe+Antognini%2C+Jascha+Sohl%2DDickstein%2C+Roy+Frostig%2C+and+George+E%2E+Dahl%2E+Measuring+the+effects+of+data+parallelism+on+neural+network+training%2E+arXiv+preprint+arXiv%3A18)
102. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155, 2018. [our card](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/)
103. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1804%2E04235%2C+2018%2E)
104. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [find](https://scholar.google.com/scholar?q=Noam+Shazeer%2C+Azalia+Mirhoseini%2C+Krzysztof+Maziarz%2C+Andy+Davis%2C+Quoc+Le%2C+Geoffrey+Hinton%2C+and+Jeff+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2E+arXiv+preprint)
105. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, 2018. [our card](https://md.research.elamur.ai/free/en/papers/shazeer-2018-mesh-tensorflow-deep-learning-supercomputers/)
106. Jason R. Smith, Herve Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. Dirt cheap web-scale parallel text from the common crawl. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013. [our card](https://md.research.elamur.ai/free/en/papers/smith-2013-dirt-cheap-web-scale-parallel/)
107. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 2013. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Proce)
108. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450, 2019. [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/)
109. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 2014. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+The+Journal+of+Machine+Learning+Resear)
110. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079, 2018. — [arXiv:1804.00079](https://arxiv.org/abs/1804.00079) [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Christopher+J%2E+Pal%2E+Learning+general+purpose+distributed+sentence+representations+via+large+scale+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A1804)
111. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 2014. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V%2E+Le%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+2014%2E)
112. Richard S. Sutton. The bitter lesson. [http://www.incompleteideas.net/IncIdeas/BitterLesson.html](http://www.incompleteideas.net/IncIdeas/BitterLesson.html), 2019. [↗](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)
113. Wilson L. Taylor. “Cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 1953. [find](https://scholar.google.com/scholar?q=Wilson+L%2E+Taylor%2E+%E2%80%9CCloze+procedure%E2%80%9D%3A+A+new+tool+for+measuring+readability%2E+Journalism+Bulletin%2C+1953%2E)
114. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. [find](https://scholar.google.com/scholar?q=Trieu+H%2E+Trinh+and+Quoc+V%2E+Le%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2C+2018%2E)
115. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. NewsQA: A machine comprehension dataset. arXiv preprint arXiv:1611.09830, 2016. [our card](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/)
116. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017. [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
117. Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. arXiv preprint arXiv:1909.01380, 2019. [our card](https://md.research.elamur.ai/free/en/papers/voita-2019-bottom-up-evolution-representations-transformer/)
118. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
119. Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, et al. Can you tell me how to get past Sesame Street? Sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Assoc… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghavendra+Pappagari%2C+R%2E+Thomas+McCoy%2C+Roma+Patel%2C+Najoung+Kim%2C+Ian+Tenney%2C+Yinghui+Huang%2C+Katherin+Yu%2C+et+al%2E+Can+you+tell+me+how+to+get+past+Sesame+Street%3F+Sentenc)
120. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019b. [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
121. Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. StructBERT: Incorporating language structures into pre-training for deep language understanding. arXiv preprint arXiv:1908.04577, 2019c. [find](https://scholar.google.com/scholar?q=Wei+Wang%2C+Bin+Bi%2C+Ming+Yan%2C+Chen+Wu%2C+Zuyi+Bao%2C+Liwei+Peng%2C+and+Luo+Si%2E+StructBERT%3A+Incorporating+language+structures+into+pre%2Dtraining+for+deep+language+understanding%2E+arXiv+preprint+arXiv%3A1908%2E04577%2C)
122. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2C+2018%2E)
123. Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+arXiv+preprint+arXiv%3A1704%2E05426%2C+2017%2E)
124. Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1989. [find](https://scholar.google.com/scholar?q=Ronald+J%2E+Williams+and+David+Zipser%2E+A+learning+algorithm+for+continually+running+fully+recurrent+neural+networks%2E+Neural+computation%2C+1989%2E)
125. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+t)
126. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019. [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
127. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, 2014. [find](https://scholar.google.com/scholar?q=Jason+Yosinski%2C+Jeff+Clune%2C+Yoshua+Bengio%2C+and+Hod+Lipson%2E+How+transferable+are+features+in+deep+neural+networks%3F+In+Advances+in+neural+information+processing+systems%2C+2014%2E)
128. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. QAnet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541, 2018. [our card](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/)
129. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616, 2019. [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)
130. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. [our card](https://md.research.elamur.ai/free/en/papers/zhang-2018-record-bridging-gap-between-human/)
131. Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Thomas Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for language understanding. arXiv preprint arXiv:1909.11764, 2019. [our card](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)
132. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 2015. [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)