---
title: "Identifying beneficial task relations for — Bingel 2017"
description: "Multi-task learning (MTL) with shared hidden layers often helps NLP models, but results vary, and it is unclear which task pairings help."
authors: ["Joachim Bingel", "Anders Søgaard"]
year: 2017
venue: "Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers"
source: "https://aclanthology.org/E17-2026/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bingel-2017-identifying-beneficial-task-relations-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bingel-2017-identifying-beneficial-task-relations-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/) / Identifying beneficial task relations for multi-task learning in deep neural networks

# Identifying beneficial task relations for multi-task learning in deep neural networks

**Authors:** [Joachim Bingel](https://md.research.elamur.ai/free/en/authors/joachim-bingel/), [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/) · **Year:** 2017 · **Venue:** Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/E17-2026/) · [PDF](https://aclanthology.org/E17-2026.pdf) · [OpenAlex](https://openalex.org/W2592170186)

**Citations:** 206 citations (openalex-snapshot · 2026-10-14) · [99](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Multi-task learning (MTL) with shared hidden layers often helps NLP models, but results vary, and it is unclear which task pairings help. Bingel and Søgaard study this for sequence labeling. They train a bi-LSTM with hard parameter sharing on ten English tasks, including CCG supertagging, chunking, sentence compression, FrameNet frames, POS tagging of tweets, hyperlink prediction, keyphrase detection, MWE detection and two super-sense tasks. Every ordered main/auxiliary pairing (90 in total) is compared with a single-task baseline, reusing the single-task hyperparameters unchanged. They then describe each pair with dataset statistics and features taken from the single-task loss curves, and fit a logistic regression that predicts whether MTL helps. MTL improved the main task in 40 of the 90 pairings. The meta-classifier predicted the outcome correctly in about 78.5% of cases. The most informative signals came from learning-curve shape rather than raw data properties. This gives a cheap way to screen auxiliary tasks before running expensive joint training.

## Key points
- **Architecture and training:**
  - The shared model is a single bi-LSTM layer of 100 dimensions over 100-d GloVe embeddings. The embeddings are updated during training and also shared.
  - Each task has its own dense output layer.
  - At each step the model picks one task uniformly at random and trains on a batch of 32 of its examples, using Adadelta.
  - Single-task models run for 25,000 batches and MTL models for 50,000.
- **Datasets:** CCGBank, Penn Treebank chunking, Google Compression, FrameNet 1.5, Universal POS-tagged tweets, the Spitkovsky et al. hypertext corpus, SemEval 2017 Task 10, Streusle (MWE and super-senses) and Semcor.
- **Pairwise results:**
  - Chunking and super-sense tagging were the most useful helpers. Hyperlink prediction did not significantly help any other task.
  - MWE detection and hyperlink detection benefited most as main tasks.
  - POS and CCG tagging helped each other, and so did MWE and compression.
- **Meta-prediction results:**
  - Each pair was described by 42 features: 14 for the main task, 14 for the auxiliary task, and their ratios.
  - Logistic regression reached 0.785 accuracy, against a majority baseline of 0.555.
  - Using dataset features alone gave a positive-class F1 of 0.542, worse than the baseline's 0.615.
- **Most predictive features:**
  - The strongest predictors were loss-curve gradients of the main task at 20–30% of training.
  - Next came main-task label count, auxiliary label entropy and main-task OOV rate.
  - Train/test JSD and dataset size mattered little.

## Interesting ideas
- **Predict MTL success before paying for it.** Training cheap single-task models and fitting a small classifier on their statistics can rank candidate auxiliary tasks or datasets before joint training. This would be useful, for example, when choosing which extra annotated resources to add for a low-resource language.
- **Use loss-curve shape as features.** The authors take gradients at fixed points of training and fit a logarithmic curve to the losses. This turns training dynamics into numbers that a classifier can use, and it could also help in comparing datasets or spotting tasks that converge early.
- **Pair stalled tasks with still-learning tasks.** MTL helped most when the main task's curve flattened early while the auxiliary task kept improving. This suggests the auxiliary signal can pull the main model out of poor local minima, which is a useful heuristic for choosing helper tasks.
- **Fix hyperparameters to isolate the effect.** Reusing single-task settings unchanged in the multi-task model removes tuning as a confounder. This makes gains attributable to the task pairing, and it is a clean protocol for any ablation-style comparison.
- **Model every directed pair.** Treating main/auxiliary as asymmetric across all ordered pairs exposes one-sided and mutual benefits that a single combined model would hide.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

99 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 87 not yet.

- [Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modeling](https://md.research.elamur.ai/free/en/papers/acl-2019-zalmout-adversarial-multitask-learning-joint-multi/)
- [Multi-task learning for historical text normalization: Size matters](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-bollmann-multi-task-learning-historical-text/)
- [Few-Shot and Zero-Shot Learning for Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/aclbulk-2019-bollmann-few-shot-zero-shot-learning/)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)
- [Scheduled Multi-Task Learning: From Syntax to Translation](https://md.research.elamur.ai/free/en/papers/kiperwasser-2018-scheduled-multi-task-learning-syntax/)
- [Joint Prediction of Morphosyntactic Categories for Fine-Grained Arabic Part-of-Speech Tagging Exploiting Tag Dictionary Information](https://md.research.elamur.ai/free/en/papers/lingbulk-2017-inoue-joint-prediction-morphosyntactic-categories-fine/)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
- [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/strubell-2018-linguistically-informed-self-attention-semantic/)
- [Exploring and Predicting Transferability across NLP Tasks](https://md.research.elamur.ai/free/en/papers/vu-2020-exploring-predicting-transferability-across-nlp/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bingel2017identifying,
  title = {{Identifying beneficial task relations for multi-task learning in deep neural networks}},
  author = {Bingel, Joachim and Søgaard, Anders},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.18653/v1/e17-2026},
  url = {https://doi.org/10.18653/v1/e17-2026}
}
```

- [← previous How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT](https://md.research.elamur.ai/free/en/papers/morey-2017-how-much-progress-have-we/)
- [next → Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/chen-2017-improved-neural-machine-translation-syntax/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/2/)

## References

19 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Nathan Schneider and Noah A Smith. 2015. A corpus and model integrating multiword expressions and supersenses. Proc. of NAACL-HLT. Denver, Colorado, USA. [find](https://scholar.google.com/scholar?q=Nathan+Schneider+and+Noah+A+Smith%2E+2015%2E+A+corpus+and+model+integrating+multiword+expressions+and+supersenses%2E+Proc%2E+of+NAACL%2DHLT%2E+Denver%2C+Colorado%2C+USA%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2015-schneider-corpus-model-integrating-multiword-expressions/)
2. Jonathan Baxter. 2000. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12:149–198. [find](https://scholar.google.com/scholar?q=Jonathan+Baxter%2E+2000%2E+A+model+of+inductive+bias+learning%2E+Journal+of+Artificial+Intelligence+Research%2C+12%3A149%E2%80%93198%2E)
3. Anders Søgaard and Yoav Goldberg. 2016. Deep multitask learning with low level tasks supervised at lower layers. In ACL. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard+and+Yoav+Goldberg%2E+2016%2E+Deep+multitask+learning+with+low+level+tasks+supervised+at+lower+layers%2E+In+ACL%2E)
4. Shai Ben-David and Reba Borberly. 2003. A notion of task relatedness yielding provable multiple-task learning guarantees. Machine Learning, 73:273– 287. [find](https://scholar.google.com/scholar?q=Shai+Ben%2DDavid+and+Reba+Borberly%2E+2003%2E+A+notion+of+task+relatedness+yielding+provable+multiple%2Dtask+learning+guarantees%2E+Machine+Learning%2C+73%3A273%E2%80%93+287%2E)
5. Valentin I Spitkovsky, Daniel Jurafsky, and Hiyan Alshawi. 2010. Profiting from mark-up: Hyper-text annotations for guided parsing. In ACL. [find](https://scholar.google.com/scholar?q=Valentin+I+Spitkovsky%2C+Daniel+Jurafsky%2C+and+Hiyan+Alshawi%2E+2010%2E+Profiting+from+mark%2Dup%3A+Hyper%2Dtext+annotations+for+guided+parsing%2E+In+ACL%2E)
6. Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017. Multitask learning for mental health conditions with limited social media data. In EACL. [find](https://scholar.google.com/scholar?q=Adrian+Benton%2C+Margaret+Mitchell%2C+and+Dirk+Hovy%2E+2017%2E+Multitask+learning+for+mental+health+conditions+with+limited+social+media+data%2E+In+EACL%2E)
7. Marcel Bollman and Anders Søgaard. 2016. Improving historical spelling normalization with bidirectional lstms and multi-task learning. In COL-ING. [find](https://scholar.google.com/scholar?q=Marcel+Bollman+and+Anders+S%C3%B8gaard%2E+2016%2E+Improving+historical+spelling+normalization+with+bidirectional+lstms+and+multi%2Dtask+learning%2E+In+COL%2DING%2E)
8. Chloe Braud, Barbara Plank, and Anders Søgaard. 2016. Multi-view and multi-task training of rst discourse parser. In COLING. [find](https://scholar.google.com/scholar?q=Chloe+Braud%2C+Barbara+Plank%2C+and+Anders+S%C3%B8gaard%2E+2016%2E+Multi%2Dview+and+multi%2Dtask+training+of+rst+discourse+parser%2E+In+COLING%2E)
9. Rich Caruana. 1993. Multitask learning: a knowledge-based source of inductive bias. In ICML. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1993%2E+Multitask+learning%3A+a+knowledge%2Dbased+source+of+inductive+bias%2E+In+ICML%2E)
10. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+ICML%2E)
11. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In EMNLP, pages 1481–1491. [find](https://scholar.google.com/scholar?q=Katja+Filippova+and+Yasemin+Altun%2E+2013%2E+Overcoming+the+lack+of+parallel+data+in+sentence+compression%2E+In+EMNLP%2C+pages+1481%E2%80%931491%2E)
12. Julia Hockenmaier and Mark Steedman. 2007. Ccgbank: A corpus of ccg derivations and dependency structures extracted from the penn treebank. Comput. Linguist., 33(3):355–396, September. [find](https://scholar.google.com/scholar?q=Julia+Hockenmaier+and+Mark+Steedman%2E+2007%2E+Ccgbank%3A+A+corpus+of+ccg+derivations+and+dependency+structures+extracted+from+the+penn+treebank%2E+Comput%2E+Linguist%2E%2C+33%283%29%3A355%E2%80%93396%2C+September%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-2007-hockenmaier-ccgbank-corpus-ccg-derivations-dependency/)
13. Sigrid Klerke, Yoav Goldberg, and Anders Søgaard. 2016. Improving sentence compression by learning to predict gaze. In NAACL. [find](https://scholar.google.com/scholar?q=Sigrid+Klerke%2C+Yoav+Goldberg%2C+and+Anders+S%C3%B8gaard%2E+2016%2E+Improving+sentence+compression+by+learning+to+predict+gaze%2E+In+NAACL%2E)
14. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+2016%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
15. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+Marcus%2C+Mary+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+English%3A+the+Penn+Treebank%2E+Computational+Linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
16. Héctor Martı́nez Alonso and Barbara Plank. 2017. Multitask learning for semantic sequence prediction under varying data conditions. In EACL. [find](https://scholar.google.com/scholar?q=He%CC%81ctor+Mart%C4%B1%CC%81nez+Alonso+and+Barbara+Plank%2E+2017%2E+Multitask+learning+for+semantic+sequence+prediction+under+varying+data+conditions%2E+In+EACL%2E)
17. Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2016. How transferable are neural networks in nlp applications? In EMNLP. [find](https://scholar.google.com/scholar?q=Lili+Mou%2C+Zhao+Meng%2C+Rui+Yan%2C+Ge+Li%2C+Yan+Xu%2C+Lu+Zhang%2C+and+Zhi+Jin%2E+2016%2E+How+transferable+are+neural+networks+in+nlp+applications%3F+In+EMNLP%2E)
18. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. CoRR abs/1104.2086. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2C+Dipanjan+Das%2C+and+Ryan+McDonald%2E+2011%2E+A+universal+part%2Dof%2Dspeech+tagset%2E+CoRR+abs%2F1104%2E2086%2E)
19. Barbara Plank. 2016. Keystroke dynamics as signal for shallow syntactic parsing. In COLING. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2E+2016%2E+Keystroke+dynamics+as+signal+for+shallow+syntactic+parsing%2E+In+COLING%2E)