---
title: "Sentence Encoders on STILTs — Phang 2018"
description: "Large pretrained sentence encoders are usually fine-tuned directly on each target task."
authors: ["Jason Phang", "Thibault Févry", "Samuel R. Bowman"]
year: 2018
venue: "arXiv:1811.01088"
source: "https://arxiv.org/abs/1811.01088"
doi: "10.48550/arxiv.1811.01088"
arxiv: "1811.01088"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

# Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

**Authors:** [Jason Phang](https://md.research.elamur.ai/free/en/authors/jason-phang/), [Thibault Févry](https://md.research.elamur.ai/free/en/authors/thibault-fevry/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2018 · **Venue:** arXiv:1811.01088 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1811.01088) · [PDF](https://arxiv.org/pdf/1811.01088) · [DOI 10.48550/arxiv.1811.01088](https://doi.org/10.48550/arxiv.1811.01088) · [arXiv 1811.01088](https://arxiv.org/abs/1811.01088) · [OpenAlex](https://openalex.org/W2898700502)

**Citations:** 230 citations (openalex-snapshot · 2026-10-14) · [260](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Large pretrained sentence encoders are usually fine-tuned directly on each target task. That step can be fragile, especially when target data are scarce. The authors propose STILTs (Supplementary Training on Intermediate Labeled-data Tasks), which adds an extra fine-tuning stage on a large labeled task between unsupervised pretraining and target-task training. They apply it to BERT, GPT and an ELMo variant. The intermediate tasks are MNLI, SNLI, QQP and a synthetic fake-sentence-detection task built from BooksCorpus. Evaluation covers the nine GLUE tasks, including runs where target training sets are cut to 1k or 5k examples. Picking the best intermediate task for each target, BERT on STILTs reached a GLUE score of 81.8, the top leaderboard result at the time and 1.4 points above plain BERT. Sentence-pair tasks and small datasets gained most, while single-sentence tasks like CoLA and SST did not improve. The method also made 24-layer BERT fine-tuning noticeably more stable. It matters because it is cheap: only one more fine-tuning round using existing infrastructure.

## Key points
- Pipeline: (1) unsupervised pretraining, (2) supervised training on a data-rich intermediate task (MNLI, SNLI, QQP or BooksCorpus real/fake detection), (3) fine-tuning on the GLUE target task. GPT and ELMo were run with the jiant toolkit (AllenNLP/PyTorch), and BERT with the HuggingFace PyTorch port.
- The "Best of Each" BERT setup scored 81.8 on the GLUE test set (+1.4 over BERT). For BERT on RTE, MNLI as the intermediate task added about 8 points on the test set.
- Stability with 5k training examples (24-layer BERT): degenerate runs fell from 5 to 1 on SST and from 2 to 0 on STS. Variance across random restarts also dropped.
- Benefits were larger with downsampled training sets (1k/5k), reaching up to about 10 points for some task pairs. They were also larger for GPT and ELMo than for BERT.
- Compared with multitask training for GPT with RTE as the target, sequential STILTs did best:
  - plain fine-tuning: 54.2
  - MNLI→RTE: 70.4
  - joint {MNLI, RTE}: 68.6
  - joint {MNLI, RTE}→RTE: 67.5
- Evaluation details: WNLI was handled by predicting the majority label, and reported numbers use the older QNLIv1.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

260 papers of our library cite it (of the 140884 whose bibliographies we read); 35 of them have a page here, 225 not yet.

- [Uncertain Natural Language Inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2023-meditron-70b-scaling-medical-pretraining/)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/)
- [Transformers as Soft Reasoners over Language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/)
- [Making Pre-trained Language Models Better Few-shot Learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://md.research.elamur.ai/free/en/papers/he-2021-effectiveness-adapter-based-tuning-pretrained/)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://md.research.elamur.ai/free/en/papers/leonardelli-2021-agreeing-disagree-annotating-offensive-language/)
- [Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art](https://md.research.elamur.ai/free/en/papers/lewis-2020-pretrained-language-models-biomedical-clinical/)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/)
- [Merging Models with Fisher-Weighted Averaging](https://md.research.elamur.ai/free/en/papers/matena-2021-merging-models-fisher-weighted-averaging/)
- [SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning](https://md.research.elamur.ai/free/en/papers/mirzaee-2021-spartqa-textual-question-answering-benchmark/)
- [DisSent: Learning Sentence Representations from Explicit Discourse Relations](https://md.research.elamur.ai/free/en/papers/nie-2019-dissent-learning-sentence-representations-explicit/)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://md.research.elamur.ai/free/en/papers/pang-2022-quality-question-answering-long-input/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [AdapterHub: A Framework for Adapting Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterhub-framework-adapting-transformers/)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/)
- [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Neural Unsupervised Domain Adaptation in NLP–-A Survey](https://md.research.elamur.ai/free/en/papers/ramponi-2020-neural-unsupervised-domain-adaptation-nlp/)
- [SocialIQA: Commonsense Reasoning about Social Interactions](https://md.research.elamur.ai/free/en/papers/sap-2019-socialiqa-commonsense-reasoning-about-social/)
- [Mining Discourse Markers for Unsupervised Sentence Representation Learning](https://md.research.elamur.ai/free/en/papers/sileo-2019-mining-discourse-markers-unsupervised-sentence/)
- [A Pragmatics-Centered Evaluation Framework for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/sileo-2022-pragmatics-centered-evaluation-framework-natural/)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/)
- [Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/)
- [TIES-Merging: Resolving Interference When Merging Models](https://md.research.elamur.ai/free/en/papers/yadav-2023-ties-merging-resolving-interference-merging/)
- [IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions](https://md.research.elamur.ai/free/en/papers/zeng-2023-iekg-commonsense-knowledge-graph-idiomatic/)
- [Revisiting Few-sample BERT Fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{phang2018sentence,
  title = {{Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks}},
  author = {Phang, Jason and Févry, Thibault and Bowman, Samuel R.},
  year = {2018},
  doi = {10.48550/arxiv.1811.01088},
  eprint = {1811.01088},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01088}
}
```

- [← previous Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/)
- [next → SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/)

## References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. [find](https://scholar.google.com/scholar?q=Joachim+Bingel+and+Anders+S%C3%B8gaard%2E+2017%2E+Identifying+beneficial+task+relations+for+multi%2Dtask+learning+in+deep+neural+networks%2E+In+EACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/bingel-2017-identifying-beneficial-task-relations-multi/)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E)
3. Samuel R. Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, and Berlin Chen. 2019. Looking for ELMo’s friends: Sentence-level pretraining be… — [arXiv:1812.10860](https://arxiv.org/abs/1812.10860) [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Ellie+Pavlick%2C+Edouard+Grave%2C+Benjamin+Van+Durme%2C+Alex+Wang%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghavendra+Pappagari%2C+R%2E+Thomas+McCoy%2C+Roma+Patel%2C+Najoung+Kim%2C+Ian+Tenney%2C+Yinghui+Huang%2C+Kather)
4. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2017. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+SemEval%2D2017%2E)
5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. [find](https://scholar.google.com/scholar?q=Ciprian+Chelba%2C+Tomas+Mikolov%2C+Mike+Schuster%2C+Qi+Ge%2C+Thorsten+Brants%2C+Phillipp+Koehn%2C+and+Tony+Robinson%2E+2013%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
7. Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Germa%CC%81n+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+%24%26%21%5C%23%2A+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In+A) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
8. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classificati)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+1810%2E04805%2E)
10. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proc. International Workshop on Paraphrasing (IWP). [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proc%2E+International+Workshop+on+Paraphrasing+%28IWP%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
11. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In NAACL. [find](https://scholar.google.com/scholar?q=Manaal+Faruqui%2C+Jesse+Dodge%2C+Sujay+Kumar+Jauhar%2C+Chris+Dyer%2C+Eduard+Hovy%2C+and+Noah+A%2E+Smith%2E+2015%2E+Retrofitting+word+vectors+to+semantic+lexicons%2E+In+NAACL%2E)
12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. AllenNLP: A deep semantic natural language processing platform. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+AllenNLP%3A+A+deep+semantic+natural+language+process) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
13. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
14. Emma Kerinec, Chloé Braud, and Anders Søgaard. 2018. When does deep multi-task learning work for loosely related document classification tasks? In Proc. EMNLP Workshop BlackboxNLP. [find](https://scholar.google.com/scholar?q=Emma+Kerinec%2C+Chloe%CC%81+Braud%2C+and+Anders+S%C3%B8gaard%2E+2018%2E+When+does+deep+multi%2Dtask+learning+work+for+loosely+related+document+classification+tasks%3F+In+Proc%2E+EMNLP+Workshop+BlackboxNLP%2E)
15. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+schema+challenge%2E+In+Aaai+spring+symposium%3A+Logical+formalizations+of+commonsense+reasoning%2C+volume+46%2C+page+47%2E)
16. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
17. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+pytorch%2E)
18. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018a%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E)
19. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+2018b%2E+Dissecting+contextual+word+embeddings%3A+Architecture+and+representation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
20. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Unpublished manuscript accessible via the OpenAI Blog. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+Unpublished+manuscript+accessible+via+the+OpenAI+Blog%2E)
21. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
23. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In ICLR. [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Christopher+J%2E+Pal%2E+2018%2E+Learning+general+purpose+distributed+sentence+representations+via+large+scale+multi%2Dtask+learning%2E+In+ICLR%2E)
24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+180) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
26. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+1805%2E12471%2E)
27. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proc. Eighth International Joint Conference on Natural Language Processing. [find](https://scholar.google.com/scholar?q=Aaron+Steven+White%2C+Pushpendre+Rastogi%2C+Kevin+Duh%2C+and+Benjamin+Van+Durme%2E+2017%2E+Inference+is+everything%3A+Recasting+semantic+resources+into+a+unified+evaluation+framework%2E+In+Proc%2E+Eighth+Internationa) [our card](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/)
28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
29. Kelly Zhang and Samuel R. Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint 1809.10040. [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+R%2E+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+syntax+than+translation+does%3A+Lessons+learned+through+auxiliary+task+analysis%2E+arXiv+preprint+1809%2E10040%2E)
30. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015a. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015a%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)
31. Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015b. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint arXiv:1506.06724. [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Richard+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015b%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+mov) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)