---
title: "BAM! Born-Again Multi-Task Networks for Natural — Clark 2019"
description: "Multi-task neural models in NLP often fail to match the accuracy of separately trained single-task models."
authors: ["Kevin Clark", "Minh-Thang Luong", "Urvashi Khandelwal", "Christopher D. Manning", "Quoc V. Le"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1595/"
doi: "10.18653/v1/p19-1595"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / BAM! Born-Again Multi-Task Networks for Natural Language Understanding

# BAM! Born-Again Multi-Task Networks for Natural Language Understanding

**Authors:** [Kevin Clark](https://md.research.elamur.ai/free/en/authors/kevin-clark/), [Minh-Thang Luong](https://md.research.elamur.ai/free/en/authors/minh-thang-luong/), Urvashi Khandelwal, [Christopher D. Manning](https://md.research.elamur.ai/free/en/authors/christopher-d-manning/), [Quoc V. Le](https://md.research.elamur.ai/free/en/authors/quoc-v-le/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1595/) · [PDF](https://aclanthology.org/P19-1595.pdf) · [DOI 10.18653/v1/p19-1595](https://doi.org/10.18653/v1/p19-1595) · [OpenAlex](https://openalex.org/W2952603081)

**Citations:** 208 citations (openalex-snapshot · 2026-10-14) · [71](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multi-task neural models in NLP often fail to match the accuracy of separately trained single-task models. This paper introduces Born-Again Multi-tasking (BAM), which uses knowledge distillation to address that gap. First, a separate BERT-Large model is fine-tuned for each task. These models then act as teachers for one shared multi-task student of identical architecture. The authors also add teacher annealing: the training target begins as the teacher's predicted distribution and shifts linearly toward the gold label. This lets the student learn from the richer soft signal early on and still exceed its teachers later.

The method is evaluated on the GLUE benchmark, and dev results are reported as medians over at least 20 random seeds. Single→Multi distillation matches or beats standard single-task and multi-task training on every task except STS-B, the only regression task. The result matters because a single model that handles all tasks reliably is simpler and cheaper to deploy than a collection of per-task models.

## Key points
- **Dev results:** average GLUE dev scores (excluding WNLI) were 84.0 for Single, 85.5 for Multi, and 86.0 for Single→Multi. Single→Multi also outperformed Single→Single and Multi→Multi distillation.
- **Test results:** BERT-Large + BAM reached 82.3 on the GLUE test leaderboard, compared with 80.5 for BERT-Large. This was achieved without ensembling and without single-task fine-tuning, which added only +0.1 on top of BAM.
- **Ablations:** dropping teacher annealing cost 0.5 points, and pure distillation (λ = 0) did no better than plain multi-task training. Removing task sampling (proportional to |D|^0.75) cost 0.4 points, and removing layer-wise learning rates cost 0.3.
- **RTE task combinations:** RTE alone scored 70.0. Adding unrelated tasks (QQP, CoLA, SST) raised it to 75.1, while adding MNLI raised it to 83.4. This points to both regularization and transfer between related tasks as sources of gain.
- **Parameter sharing:** under 0.01% of the parameters are task-specific, and repeating the distillation for extra rounds brought no significant improvement.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

71 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 60 not yet.

- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://md.research.elamur.ai/free/en/papers/goot-2021-massive-choice-ample-tasks-machamp/)
- [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
- [Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/li-2021-improving-stance-detection-multi-dataset/)
- [Distilling Calibrated Knowledge for Stance Detection](https://md.research.elamur.ai/free/en/papers/li-2023-distilling-calibrated-knowledge-stance-detection/)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://md.research.elamur.ai/free/en/papers/mahabadi-2021-parameter-efficient-multi-task-fine/)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://md.research.elamur.ai/free/en/papers/turc-2019-well-read-students-learn-better/)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
- [Free the Plural: Unrestricted Split-Antecedent Anaphora Resolution](https://md.research.elamur.ai/free/en/papers/yu-2020-free-plural-unrestricted-split-antecedent/)
- [Multitask Learning-Based Neural Bridging Reference Resolution](https://md.research.elamur.ai/free/en/papers/yu-2020-multitask-learning-based-neural-bridging/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{clark2019bam,
  title = {{BAM! Born-Again Multi-Task Networks for Natural Language Understanding}},
  author = {Clark, Kevin and Luong, Minh-Thang and Khandelwal, Urvashi and Manning, Christopher D. and Le, Quoc V.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1595},
  url = {https://doi.org/10.18653/v1/p19-1595}
}
```

- [← previous Automating Dictionary Production: a Tagalog-English-Korean Dictionary from Scratch](https://md.research.elamur.ai/free/en/papers/baisa-2019-automating-dictionary-production/)
- [next → Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://md.research.elamur.ai/free/en/papers/yin-2019-benchmarking-zero-shot-text-classification/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

44 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In NIPS. [find](https://scholar.google.com/scholar?q=Jimmy+Ba+and+Rich+Caruana%2E+2014%2E+Do+deep+nets+really+need+to+be+deep%3F+In+NIPS%2E)
2. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. [find](https://scholar.google.com/scholar?q=Joachim+Bingel+and+Anders+S%C3%B8gaard%2E+2017%2E+Identifying+beneficial+task+relations+for+multi%2Dtask+learning+in+deep+neural+networks%2E+In+EACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/bingel-2017-identifying-beneficial-task-relations-multi/)
3. Samuel R Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, et al. 2018. Looking for ELMo’s friends: Sentence-level pretraining beyond language modeling. arXiv preprint arXiv:1812.10860. — [arXiv:1812.10860](https://arxiv.org/abs/1812.10860) [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Ellie+Pavlick%2C+Edouard+Grave%2C+Benjamin+Van+Durme%2C+Alex+Wang%2C+Jan+Hula%2C+Patrick+Xia%2C+Raghavendra+Pappagari%2C+R+Thomas+McCoy%2C+Roma+Patel%2C+et+al%2E+2018%2E+Looking+for+ELMo%E2%80%99s+friends%3A+Sentenc)
4. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In SIGKDD. [find](https://scholar.google.com/scholar?q=Cristian+Bucilu%2C+Rich+Caruana%2C+and+Alexandru+Niculescu%2DMizil%2E+2006%2E+Model+compression%2E+In+SIGKDD%2E)
5. Rich Caruana. 1997. Multitask learning. Machine Learning. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1997%2E+Multitask+learning%2E+Machine+Learning%2E)
6. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval@ACL. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Cer%2C+Mona+T%2E+Diab%2C+Eneko+Agirre%2C+In%CC%83igo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+SemEval%40AC)
7. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+deep+neural+networks+with+multitask+learning%2E+In+ICML%2E)
8. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+NIPS%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2DHLT%2E)
10. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In IWP@IJCNLP. [find](https://scholar.google.com/scholar?q=William+B%2E+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+IWP%40IJCNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
11. Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In ICML. [find](https://scholar.google.com/scholar?q=Tommaso+Furlanello%2C+Zachary+C+Lipton%2C+Michael+Tschannen%2C+Laurent+Itti%2C+and+Anima+Anandkumar%2E+2018%2E+Born+again+neural+networks%2E+In+ICML%2E)
12. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. 2007. The third pascal recognizing textual entailment challenge. In ACL-PASCAL@ACL. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+William+B%2E+Dolan%2E+2007%2E+The+third+pascal+recognizing+textual+entailment+challenge%2E+In+ACL%2DPASCAL%40ACL%2E)
13. Braden Hancock, Clara McCreery, Ines Chami, Vincent Chen, Sen Wu, Jared Dunnmon, Paroma Varma, Max Lam, and Chris R. 2019. Massive multi-task learning with snorkel metal: Bringing more supervision to bear. [find](https://scholar.google.com/scholar?q=Braden+Hancock%2C+Clara+McCreery%2C+Ines+Chami%2C+Vincent+Chen%2C+Sen+Wu%2C+Jared+Dunnmon%2C+Paroma+Varma%2C+Max+Lam%2C+and+Chris+R%2E+2019%2E+Massive+multi%2Dtask+learning+with+snorkel+metal%3A+Bringing+more+supervision+to+)
14. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In EMNLP. [find](https://scholar.google.com/scholar?q=Kazuma+Hashimoto%2C+Caiming+Xiong%2C+Yoshimasa+Tsuruoka%2C+and+Richard+Socher%2E+2017%2E+A+joint+many%2Dtask+model%3A+Growing+a+neural+network+for+multiple+NLP+tasks%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2015%2E+Distilling+the+knowledge+in+a+neural+network%2E+arXiv+preprint+arXiv%3A1503%2E02531%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
16. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics. [find](https://scholar.google.com/scholar?q=Sture+Holm%2E+1979%2E+A+simple+sequentially+rejective+multiple+test+procedure%2E+Scandinavian+journal+of+statistics%2E)
17. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
18. Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. 2017. First quora dataset release: Question pairs. [find](https://scholar.google.com/scholar?q=Shankar+Iyer%2C+Nikhil+Dandekar%2C+and+Kornl+Csernai%2E+2017%2E+First+quora+dataset+release%3A+Question+pairs%2E)
19. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286. — [arXiv:1904.09286](https://arxiv.org/abs/1904.09286) [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2019%2E+Unifying+question+answering+and+text+classification+via+span+extraction%2E+arXiv+preprint+arXiv%3A1904%2E09286%2E+%E2%80%94+%5BarXiv%3A1904%2E09)
20. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP. [find](https://scholar.google.com/scholar?q=Yoon+Kim+and+Alexander+M%2E+Rush%2E+2016%2E+Sequence%2Dlevel+knowledge+distillation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
21. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one mst parser. In EMNLP. [find](https://scholar.google.com/scholar?q=Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+Lingpeng+Kong%2C+Chris+Dyer%2C+and+Noah+A%2E+Smith%2E+2016%2E+Distilling+an+ensemble+of+greedy+dependency+parsers+into+one+mst+parser%2E+In+EMNLP%2E)
22. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2017%2E+Adversarial+multi%2Dtask+learning+for+text+classification%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2017-adversarial-multi-task-learning-text/)
23. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019a%2E+Improving+multi%2Dtask+deep+neural+networks+via+knowledge+distillation+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1904%2E094)
24. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Pengcheng+He%2C+Weizhu+Chen%2C+and+Jianfeng+Gao%2E+2019b%2E+Multi%2Dtask+deep+neural+networks+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1901%2E11504%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/)
25. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2016%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
26. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+Nitish+Shirish+Keskar%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2018%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
27. Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Zhi Jin. 2016. Distilling word embeddings: An encoding approach. In CIKM. [find](https://scholar.google.com/scholar?q=Lili+Mou%2C+Ran+Jia%2C+Yan+Xu%2C+Ge+Li%2C+Lu+Zhang%2C+and+Zhi+Jin%2E+2016%2E+Distilling+word+embeddings%3A+An+encoding+approach%2E+In+CIKM%2E)
28. Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. 2016. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR. [find](https://scholar.google.com/scholar?q=Emilio+Parisotto%2C+Jimmy+Ba%2C+and+Ruslan+Salakhutdinov%2E+2016%2E+Actor%2Dmimic%3A+Deep+multitask+and+transfer+reinforcement+learning%2E+In+ICLR%2E)
29. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+NAACL%2DHLT%2E)
30. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+STILTs%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
31. Barbara Plank and Héctor Martı́nez Alonso. 2017. When is multitask learning effective? Semantic sequence prediction under varying data conditions. In EACL. [find](https://scholar.google.com/scholar?q=Barbara+Plank+and+He%CC%81ctor+Mart%C4%B1%CC%81nez+Alonso%2E+2017%2E+When+is+multitask+learning+effective%3F+Semantic+sequence+prediction+under+varying+data+conditions%2E+In+EACL%2E)
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+S%2E+Liang%2E+2016%2E+Squad%3A+100%2C+000%2B+questions+for+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
33. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2E+2017%2E+An+overview+of+multi%2Dtask+learning+in+deep+neural+networks%2E+arXiv+preprint+arXiv%3A1706%2E05098%2E) [our card](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
34. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent multi-task architecture learning. In AAAI. [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Joachim+Bingel%2C+Isabelle+Augenstein%2C+and+Anders+S%C3%B8gaard%2E+2019%2E+Latent+multi%2Dtask+architecture+learning%2E+In+AAAI%2E)
35. Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. In AAAI. [find](https://scholar.google.com/scholar?q=Victor+Sanh%2C+Thomas+Wolf%2C+and+Sebastian+Ruder%2E+2019%2E+A+hierarchical+multi%2Dtask+approach+for+learning+embeddings+from+semantic+tasks%2E+In+AAAI%2E)
36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL%2E)
37. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D%2E+Manning%2C+Andrew+Y%2E+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E)
38. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard+and+Yoav+Goldberg%2E+2016%2E+Deep+multi%2Dtask+learning+with+low+level+tasks+supervised+at+lower+layers%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sogaard-2016-deep-multi-task-learning-low/)
39. Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR. [find](https://scholar.google.com/scholar?q=Xu+Tan%2C+Yi+Ren%2C+Di+He%2C+Tao+Qin%2C+and+Tie%2DYan+Liu%2E+2019%2E+Multilingual+neural+machine+translation+with+knowledge+distillation%2E+In+ICLR%2E)
40. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. 2017. Distral: Robust multitask reinforcement learning. In NIPS. [find](https://scholar.google.com/scholar?q=Yee+Teh%2C+Victor+Bapst%2C+Wojciech+M+Czarnecki%2C+John+Quan%2C+James+Kirkpatrick%2C+Raia+Hadsell%2C+Nicolas+Heess%2C+and+Razvan+Pascanu%2E+2017%2E+Distral%3A+Robust+multitask+reinforcement+learning%2E+In+NIPS%2E)
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
42. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+2019%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
43. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2E)
44. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2DHLT%2E)