---
title: "Deep multi-task learning with low level tasks — Søgaard 2016"
description: "Earlier deep multi-task learning (MTL) systems in NLP, such as Collobert et al."
authors: ["Anders Søgaard", "Yoav Goldberg"]
year: 2016
venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P16-2038/"
doi: "10.18653/v1/p16-2038"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sogaard-2016-deep-multi-task-learning-low/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sogaard-2016-deep-multi-task-learning-low/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Deep multi-task learning with low level tasks supervised at lower layers

# Deep multi-task learning with low level tasks supervised at lower layers

**Authors:** [Anders Søgaard](https://md.research.elamur.ai/free/en/authors/anders-sogaard/), [Yoav Goldberg](https://md.research.elamur.ai/free/en/authors/yoav-goldberg/) · **Year:** 2016 · **Venue:** Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P16-2038/) · [PDF](https://aclanthology.org/P16-2038.pdf) · [DOI 10.18653/v1/p16-2038](https://doi.org/10.18653/v1/p16-2038) · [OpenAlex](https://openalex.org/W2516255829)

**Citations:** 445 citations (openalex-snapshot · 2026-10-14) · [139](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Earlier deep multi-task learning (MTL) systems in NLP, such as Collobert et al. (2011), trained every task from the same top layer of the network. Søgaard and Goldberg ask whether a linguistic hierarchy, in which low-level analyses inform higher-level ones, should instead shape where each task is supervised. They use a greedy sequence tagger built on a 3-layer bi-LSTM, implemented with the CNN library and initialized with pre-trained embeddings. Each task gets its own softmax classifier attached to a chosen layer. POS tagging serves as the auxiliary task for syntactic chunking and CCG supertagging, with data from the English Penn Treebank. Training POS at the innermost layer and the main task at the outermost layer beats both a single-task model and conventional top-layer MTL. The same placement also helps domain adaptation on OntoNotes 4.0, where only POS labels exist in the target domains. Together, these results offer a simple way to merge cascaded pipelines with shared-representation learning, and they show that layer placement is a meaningful design choice.

## Key points
- **Main results.** Each figure lists, in order: single-task, both tasks on layer 3, and POS on layer 1 with the main task on layer 3.
  - Chunking F1 was 95.28, 95.30 and 95.56. The best score exceeds Suzuki and Isozaki (2008) at 95.15 and Collobert et al. (2011) MTL at 94.10.
  - CCG supertagging accuracy was 91.04, 92.94 and 93.26. The best score edges past Xu et al. (2015) at 93.00.
- **Data splits.** All data come from the Penn Treebank.
  - POS and CCG train on sections 0–18; chunking trains on sections 15–18.
  - Section 19 is used for development.
  - Section 20 is the chunking test set; section 23 is the CCG test set.
  - CCG is scored only on the 425 most frequent supertags.
- **Domain adaptation.** WSJ newswire is the source domain, and four OntoNotes 4.0 genres are the targets: broadcast, broadcast news, magazines and weblogs.
  - Lower-layer POS supervision gave the highest chunking F1 in every target genre, with gains of about 0.2–1.4 points over the baseline.
  - Supervising POS at the top layer gave little or no benefit.
- **Rademacher complexity.** This measures how well a model fits randomly relabeled chunking data, reported relative to chance. Values were 1.298 (single-task), 1.034 (top-layer MTL) and 0.990 (lower-layer POS), which suggests the auxiliary task acts as a regularizer.
- **Tasks that did not benefit.** Adding NER (CoNLL 2003), SemCor supersenses, or the Streusle corpus (MWE and supersense annotation) brought no gains. The authors take this as evidence that MTL helps mainly when the tasks are closely related, such as syntactic ones.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

139 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 120 not yet.

- [What can we learn from Semantic Tagging?](https://md.research.elamur.ai/free/en/papers/abdou-2018-what-can-we-learn-semantic/)
- [Adversarial Multitask Learning for Joint Multi-Feature and Multi-Dialect Morphological Modeling](https://md.research.elamur.ai/free/en/papers/acl-2019-zalmout-adversarial-multitask-learning-joint-multi/)
- [Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/)
- [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/)
- [Neural End-to-End Learning for Computational Argumentation Mining](https://md.research.elamur.ai/free/en/papers/eger-2017-neural-end-end-learning-computational/)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://md.research.elamur.ai/free/en/papers/goot-2021-massive-choice-ample-tasks-machamp/)
- [A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
- [Scheduled Multi-Task Learning: From Syntax to Translation](https://md.research.elamur.ai/free/en/papers/kiperwasser-2018-scheduled-multi-task-learning-syntax/)
- [Empower Sequence Labeling with Task-Aware Neural Language Model](https://md.research.elamur.ai/free/en/papers/liu-2017-empower-sequence-labeling-task-aware/)
- [Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation](https://md.research.elamur.ai/free/en/papers/liu-2023-annotation-inspired-implicit-discourse-relation/)
- [DyNet: The Dynamic Neural Network Toolkit](https://md.research.elamur.ai/free/en/papers/neubig-2017-dynet-dynamic-neural-network-toolkit/)
- [Semi-supervised sequence tagging with bidirectional language models](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
- [Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging](https://md.research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://md.research.elamur.ai/free/en/papers/ruder-2017-overview-multi-task-learning-deep/)
- [Semantic Role Labeling as Syntactic Dependency Parsing](https://md.research.elamur.ai/free/en/papers/shi-2020-semantic-role-labeling-as-syntactic/)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/strubell-2018-linguistically-informed-self-attention-semantic/)
- [Investigating NP-Chunking with Universal Dependencies for English](https://md.research.elamur.ai/free/en/papers/ud-2018-lacroix-investigating-np-chunking-universal-dependencies/)
- [Integrating Graph-Based and Transition-Based Dependency Parsers in the Deep Contextualized Era](https://md.research.elamur.ai/free/en/papers/ud-2020-falenska-integrating-graph-based-transition-based/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sogaard2016deep,
  title = {{Deep multi-task learning with low level tasks supervised at lower layers}},
  author = {Søgaard, Anders and Goldberg, Yoav},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-2038},
  url = {https://doi.org/10.18653/v1/p16-2038}
}
```

- [← previous Deep Biaffine Attention for Neural Dependency Parsing](https://md.research.elamur.ai/free/en/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
- [next → Deep Reinforcement Learning for Dialogue Generation](https://md.research.elamur.ai/free/en/papers/li-2016-deep-reinforcement-learning-dialogue-generation/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Junho Yim, Heechul Jung, ByungIn Yoo amd Changkyu Choi, Dusik Park, and Junmo Kim. 2015. Rotating Your Face Using Multi-task Deep Neural Network. In CVPR. [find](https://scholar.google.com/scholar?q=Junho+Yim%2C+Heechul+Jung%2C+ByungIn+Yoo+amd+Changkyu+Choi%2C+Dusik+Park%2C+and+Junmo+Kim%2E+2015%2E+Rotating+Your+Face+Using+Multi%2Dtask+Deep+Neural+Network%2E+In+CVPR%2E)
2. Rich Caruana. 1993. Multitask learning: a knowledge-based source of inductive bias. In ICML. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1993%2E+Multitask+learning%3A+a+knowledge%2Dbased+source+of+inductive+bias%2E+In+ICML%2E)
3. Kyunghyun Cho. 2015. Natural language understanding with distributed representation. CoRR, abs/1511.07916. — [arXiv:1511.07916](https://arxiv.org/abs/1511.07916) [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2E+2015%2E+Natural+language+understanding+with+distributed+representation%2E+CoRR%2C+abs%2F1511%2E07916%2E+%E2%80%94+%5BarXiv%3A1511%2E07916%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1511%2E07916%29)
4. Cha Zhang and Zhengyou Zhang. 2014. Improving Multiview Face Detection with Multi-Task Deep Convolutional Neural Networks. In WACV. [find](https://scholar.google.com/scholar?q=Cha+Zhang+and+Zhengyou+Zhang%2E+2014%2E+Improving+Multiview+Face+Detection+with+Multi%2DTask+Deep+Convolutional+Neural+Networks%2E+In+WACV%2E)
5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A2) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
6. Jerry Zhu, Timothy Rogers, and Bryan Gibson. 2009. Human Rademacher complexity. In NIPS. [find](https://scholar.google.com/scholar?q=Jerry+Zhu%2C+Timothy+Rogers%2C+and+Bryan+Gibson%2E+2009%2E+Human+Rademacher+complexity%2E+In+NIPS%2E)
7. Jeffrey L. Elman. 1990. Finding Structure in Time. Cognitive Science, 14(2):179–211, March. [find](https://scholar.google.com/scholar?q=Jeffrey+L%2E+Elman%2E+1990%2E+Finding+Structure+in+Time%2E+Cognitive+Science%2C+14%282%29%3A179%E2%80%93211%2C+March%2E)
8. Yoav Goldberg. 2015. A primer on neural network models for natural language processing. CoRR, abs/1510.00726. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2E+2015%2E+A+primer+on+neural+network+models+for+natural+language+processing%2E+CoRR%2C+abs%2F1510%2E00726%2E)
9. Sepp Hochreiter and Juergen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Juergen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%3A1735%E2%80%931780%2E)
10. Ozan Irsoy and Claire Cardie. 2014. Opinion Mining with Deep Recurrent Neural Networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 720–728, Doha, Qatar, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ozan+Irsoy+and+Claire+Cardie%2E+2014%2E+Opinion+Mining+with+Deep+Recurrent+Neural+Networks%2E+In+Proceedings+of+the+2014+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+720%E2%80%93728)
11. M.-T. Luong, Q. V. Le, I. Sutskever, O. Vinyals, and L. Kaiser. 2015. Multi-task Sequence to Sequence Learning. ArXiv e-prints, November. [find](https://scholar.google.com/scholar?q=M%2E%2DT%2E+Luong%2C+Q%2E+V%2E+Le%2C+I%2E+Sutskever%2C+O%2E+Vinyals%2C+and+L%2E+Kaiser%2E+2015%2E+Multi%2Dtask+Sequence+to+Sequence+Learning%2E+ArXiv+e%2Dprints%2C+November%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
12. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Johan+Hall%2C+Sandra+Ku%CC%88bler%2C+Ryan+Mc%2DDonald%2C+Jens+Nilsson%2C+Sebastian+Riedel%2C+and+Deniz+Yuret%2E+2007%2E)
13. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+CoNLL+2007+shared+task+on+dependency+parsing%2E+In+Proceedings+of+the+CoNLL+Shared+Task+Session+of+EMNLP%2DCoNLL+2007%2C+pages+915%E2%80%93932%2C+Prague%2C+Czech+Republic%2C+June%2E+Association+for+Computational+Lingui) [our card](https://md.research.elamur.ai/free/en/papers/nivre-2007-conll-2007-shared-task-dependency/)