---
title: "Multi-Task Deep Neural Networks for Natural — Liu 2019"
description: "The paper brings together two usually separate ways of learning text representations: multi-task learning and language model pre-training."
authors: ["Xiaodong Liu", "Pengcheng He", "Weizhu Chen", "Jianfeng Gao"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1441/"
doi: "10.18653/v1/p19-1441"
arxiv: "1901.11504"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / Multi-Task Deep Neural Networks for Natural Language Understanding

# Multi-Task Deep Neural Networks for Natural Language Understanding

**Authors:** [Xiaodong Liu](https://md.research.elamur.ai/free/en/authors/xiaodong-liu/), [Pengcheng He](https://md.research.elamur.ai/free/en/authors/pengcheng-he/), [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/), [Jianfeng Gao](https://md.research.elamur.ai/free/en/authors/jianfeng-gao/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1441/) · [PDF](https://aclanthology.org/P19-1441.pdf) · [DOI 10.18653/v1/p19-1441](https://doi.org/10.18653/v1/p19-1441) · [arXiv 1901.11504](https://arxiv.org/abs/1901.11504) · [OpenAlex](https://openalex.org/W2963854351)

**Citations:** 1154 citations (openalex-snapshot · 2026-10-14) · [347](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
The paper brings together two usually separate ways of learning text representations: multi-task learning and language model pre-training. The authors' MT-DNN builds on an earlier multi-task network from Liu et al. (2015) and uses a pre-trained BERT model as its shared encoder. Training has two stages. First, BERT is pre-trained in the usual way. Then the shared Transformer layers are refined jointly on all GLUE tasks, with a separate output head for each task type: single-sentence classification, text similarity, pairwise classification and relevance ranking. The pairwise head uses the multi-step answer module from SAN, and QNLI is treated as a ranking problem instead of binary classification. Each task can then be fine-tuned separately. MT-DNN set new state-of-the-art results on eight of the nine GLUE tasks and on SNLI and SciTail. Its representations also adapted to new domains with far fewer labelled examples than plain BERT. This matters wherever annotated data is scarce, for example in lower-resource languages.

## Key points
- MT-DNN reached a GLUE score of 82.7%, which is 2.2 points above BERT as of February 2019. The only GLUE task where it did not set a new best was WNLI, a dataset with known problems.
- After adaptation, MT-DNN LARGE scored 91.6% test accuracy on SNLI (+1.5 points) and 95.0% on SciTail (+6.7 points over the previous best).
- With only 0.1% of the training data, MT-DNN scored much higher than BERT on the dev sets: 82.1% vs 52.5% on SNLI (549 examples) and 81.9% vs 51.2% on SciTail (23 examples). The advantage shrinks as more data is used.
- A single-task ablation called ST-DNN beat BERT on MNLI, QQP, RTE and MRPC, which supports using the SAN answer module. Its large gain on QNLI shows how much the ranking formulation helps.
- The code and pre-trained models are published at github.com/namisan/mt-dnn.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

347 papers of our library cite it (of the 140884 whose bibliographies we read); 34 of them have a page here, 313 not yet.

- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://md.research.elamur.ai/free/en/papers/hardalov-2022-survey-stance-detection-mis-disinformation/)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://md.research.elamur.ai/free/en/papers/jiang-2020-smart-robust-efficient-fine-tuning/)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/)
- [Contextualized Embeddings for Connective Disambiguation in Shallow Discourse Parsing](https://md.research.elamur.ai/free/en/papers/knaebel-2020-contextualized-embeddings-connective-disambiguation-shallow/)
- [Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/li-2021-improving-stance-detection-multi-dataset/)
- [Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing](https://md.research.elamur.ai/free/en/papers/lin-2020-bridging-textual-tabular-data-cross/)
- [DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules](https://md.research.elamur.ai/free/en/papers/liu-2023-dada-dialect-adaptation-dynamic-aggregation/)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/)
- [True Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-adapterfusion-non-destructive-task-composition/)
- [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://md.research.elamur.ai/free/en/papers/roy-2020-efficient-content-based-sparse-attention/)
- [ZPR2: Joint Zero Pronoun Recovery and Resolution using Multi-Task Learning and BERT](https://md.research.elamur.ai/free/en/papers/song-2020-zpr2-joint-zero-pronoun-recovery/)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://md.research.elamur.ai/free/en/papers/storks-2019-recent-advances-natural-language-inference/)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/)
- [How to Fine-Tune BERT for Text Classification?](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://md.research.elamur.ai/free/en/papers/talmor-2019-multiqa-empirical-investigation-generalization-transfer/)
- [BERT Rediscovers the Classical NLP Pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
- [Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://md.research.elamur.ai/free/en/papers/warstadt-2019-investigating-bert-s-knowledge-language/)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
- [Revisiting Few-sample BERT Fine-tuning](https://md.research.elamur.ai/free/en/papers/zhang-2020-revisiting-few-sample-bert-fine/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2019multi,
  title = {{Multi-Task Deep Neural Networks for Natural Language Understanding}},
  author = {Liu, Xiaodong and He, Pengcheng and Chen, Weizhu and Gao, Jianfeng},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1441},
  eprint = {1901.11504},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1441}
}
```

- [← previous Multi-Relational Script Learning for Discourse Relations](https://md.research.elamur.ai/free/en/papers/lee-2019-multi-relational-script-learning-discourse/)
- [next → Multi-Task Stance Detection with Sentiment and Stance Lexicons](https://md.research.elamur.ai/free/en/papers/li-2019-multi-task-stance-detection-sentiment/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+2015a%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirical)
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015b. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015b%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empiric)
3. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96. ACM. [find](https://scholar.google.com/scholar?q=Chris+Burges%2C+Tal+Shaked%2C+Erin+Renshaw%2C+Ari+Lazier%2C+Matt+Deeds%2C+Nicole+Hamilton%2C+and+Greg+Hullender%2E+2005%2E+Learning+to+rank+using+gradient+descent%2E+In+Proceedings+of+the+22nd+international+conference+) [our card](https://md.research.elamur.ai/free/en/papers/burges-2005-learning-rank-using-gradient/)
4. J. Gao, M. Galley, and L. Li. 2018. Neural approaches to conversational AI. CoRR, abs/1809.08267. [find](https://scholar.google.com/scholar?q=J%2E+Gao%2C+M%2E+Galley%2C+and+L%2E+Li%2E+2018%2E+Neural+approaches+to+conversational+AI%2E+CoRR%2C+abs%2F1809%2E08267%2E)
5. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In The 56th Annual Meeting of the Association for Computational Linguistics (ACL), Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Max+Glockner%2C+Vered+Shwartz%2C+and+Yoav+Goldberg%2E+2018%2E+Breaking+nli+systems+with+sentences+that+require+simple+lexical+inferences%2E+In+The+56th+Annual+Meeting+of+the+Association+for+Computational+Lingui) [our card](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/)
6. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 687–697. [find](https://scholar.google.com/scholar?q=Han+Guo%2C+Ramakanth+Pasunuru%2C+and+Mohit+Bansal%2E+2018%2E+Soft+layer%2Dspecific+multi%2Dtask+summarization+with+entailment+and+question+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+)
7. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 2333–2338. ACM. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Xiaodong+He%2C+Jianfeng+Gao%2C+Li+Deng%2C+Alex+Acero%2C+and+Larry+Heck%2E+2013%2E+Learning+deep+structured+semantic+models+for+web+search+using+clickthrough+data%2E+In+Proceedings+of+the+22nd+ACM+inte)
8. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+SciTail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+AAAI%2E)
9. Seonhoon Kim, Jin-Hyuk Hong, Inho Kang, and Nojun Kwak. 2018. Semantic sentence matching with densely-connected recurrent and co-attentive information. arXiv preprint arXiv:1805.11360. — [arXiv:1805.11360](https://arxiv.org/abs/1805.11360) [find](https://scholar.google.com/scholar?q=Seonhoon+Kim%2C+Jin%2DHyuk+Hong%2C+Inho+Kang%2C+and+Nojun+Kwak%2E+2018%2E+Semantic+sentence+matching+with+densely%2Dconnected+recurrent+and+co%2Dattentive+information%2E+arXiv+preprint+arXiv%3A1805%2E11360%2E+%E2%80%94+%5BarXiv%3A1805%2E1)
10. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
11. Xiaodong Liu, Kevin Duh, and Jianfeng Gao. 2018a. Stochastic answer networks for natural language inference. arXiv preprint arXiv:1804.07888. — [arXiv:1804.07888](https://arxiv.org/abs/1804.07888) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Kevin+Duh%2C+and+Jianfeng+Gao%2E+2018a%2E+Stochastic+answer+networks+for+natural+language+inference%2E+arXiv+preprint+arXiv%3A1804%2E07888%2E+%E2%80%94+%5BarXiv%3A1804%2E07888%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1804%2E07888%29)
12. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1997%2E+Multitask+learning%2E+Machine+learning%2C+28%281%29%3A41%E2%80%9375%2E)
13. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computati… [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Jianfeng+Gao%2C+Xiaodong+He%2C+Li+Deng%2C+Kevin+Duh%2C+and+Ye%2DYi+Wang%2E+2015%2E+Representation+learning+using+multi%2Dtask+deep+neural+networks+for+semantic+classification+and+information+retrieval%2E+) [our card](https://md.research.elamur.ai/free/en/papers/liu-2015-representation-learning-multi-task-deep/)
14. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug):2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+Machine+Learning+Research%2C+12%28Aug%29%3A) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
15. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019%2E+Improving+multi%2Dtask+deep+neural+networks+via+knowledge+distillation+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1904%2E0948)
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2E)
17. Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018b. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Yelong+Shen%2C+Kevin+Duh%2C+and+Jianfeng+Gao%2E+2018b%2E+Stochastic+answer+networks+for+machine+reading+comprehension%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computatio)
18. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197. [find](https://scholar.google.com/scholar?q=Li+Dong%2C+Nan+Yang%2C+Wenhui+Wang%2C+Furu+Wei%2C+Xiaodong+Liu%2C+Yu+Wang%2C+Jianfeng+Gao%2C+Ming+Zhou%2C+and+Hsiao%2DWuen+Hon%2E+2019%2E+Unified+language+model+pre%2Dtraining+for+natural+language+understanding+and+generatio) [our card](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/)
19. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2015%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+arXiv+preprint+arXiv%3A1511%2E06114%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
20. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2E)
21. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
22. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+pages+2383%E2%80%932392%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
24. Sebastian Ruder12, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent multi-task architecture learning. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder12%2C+Joachim+Bingel%2C+Isabelle+Augenstein%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+Latent+multi%2Dtask+architecture+learning%2E)
25. Aarne Talman and Stergios Chatzikyriakidis. 2018. Testing the generalization power of neural network models across nli benchmarks. arXiv preprint arXiv:1810.09774. [find](https://scholar.google.com/scholar?q=Aarne+Talman+and+Stergios+Chatzikyriakidis%2E+2018%2E+Testing+the+generalization+power+of+neural+network+models+across+nli+benchmarks%2E+arXiv+preprint+arXiv%3A1810%2E09774%2E)
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+arXiv+preprint+arXiv%3A1706%2E03762%2E)
27. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2018%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
28. Yichong Xu, Xiaodong Liu, Yelong Shen, Jingjing Liu, and Jianfeng Gao. 2018. Multi-task learning for machine reading comprehension. arXiv preprint arXiv:1809.06963. — [arXiv:1809.06963](https://arxiv.org/abs/1809.06963) [find](https://scholar.google.com/scholar?q=Yichong+Xu%2C+Xiaodong+Liu%2C+Yelong+Shen%2C+Jingjing+Liu%2C+and+Jianfeng+Gao%2E+2018%2E+Multi%2Dtask+learning+for+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1809%2E06963%2E+%E2%80%94+%5BarXiv%3A1809%2E06963%5D%28https%3A%2F%2Farxiv%2E)
29. Yu Zhang and Qiang Yang. 2017. A survey on multi-task learning. arXiv preprint arXiv:1707.08114. — [arXiv:1707.08114](https://arxiv.org/abs/1707.08114) [find](https://scholar.google.com/scholar?q=Yu+Zhang+and+Qiang+Yang%2E+2017%2E+A+survey+on+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A1707%2E08114%2E+%E2%80%94+%5BarXiv%3A1707%2E08114%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E08114%29)