---
title: "What can we learn from Semantic Tagging? — Abdou 2018"
description: "The authors test whether semantic tagging helps other NLP tasks when used as an auxiliary task in multi-task learning (MTL)."
authors: ["Mostafa Abdou", "Artur Kulmizev", "Vinit Ravishankar", "Lasha Abzianidze", "Johan Bos"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1526/"
doi: "10.18653/v1/d18-1526"
arxiv: "1808.09716"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/abdou-2018-what-can-we-learn-semantic/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/abdou-2018-what-can-we-learn-semantic/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/4/) / What can we learn from Semantic Tagging?

# What can we learn from Semantic Tagging?

**Authors:** [Mostafa Abdou](https://md.research.elamur.ai/free/authors/mostafa-abdou/), Artur Kulmizev, [Vinit Ravishankar](https://md.research.elamur.ai/free/authors/vinit-ravishankar/), [Lasha Abzianidze](https://md.research.elamur.ai/free/authors/lasha-abzianidze/), [Johan Bos](https://md.research.elamur.ai/free/authors/johan-bos/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1526/) · [PDF](https://aclanthology.org/D18-1526.pdf) · [DOI 10.18653/v1/d18-1526](https://doi.org/10.18653/v1/d18-1526) · [arXiv 1808.09716](https://arxiv.org/abs/1808.09716) · [OpenAlex](https://openalex.org/W2888872222)

**Citations:** 7 citations (openalex-snapshot · 2026-10-14) · 4 in our library, of the 140673 papers whose bibliographies we read

## Summary
The authors test whether semantic tagging helps other NLP tasks when used as an auxiliary task in multi-task learning (MTL). Semantic tagging is the annotation of words with language-neutral semantic categories from the Parallel Meaning Bank project. There are three main tasks: UPOS tagging, UD parsing and NLI. For each, a single-task model is compared with three ways of sharing parameters. The first is a fully shared network (FSN). The second is a partially shared network (PSN). The third is the Learning What to Share (LWS) scheme: each task has its own and a shared subspace, and a learned sigmoid gate determines how much information flows between them in each direction. The base models are a bi-LSTM tagger, a biaffine parser based on Dozat and Manning, and ESIM. For parsing and NLI, semantic tags were assigned by an automatic tagger trained on sem-PMB. LWS performs best on all four datasets. Full parameter sharing helps the syntactic tasks but hurts NLI. Thus, word-level semantic annotation is useful even for sentence-level tasks, provided the model itself chooses what to take from the auxiliary task. The authors also propose a general framework for assessing the usefulness of any auxiliary task.

## Key points
- The semantic tag set contains 80 tags in 13 coarse classes. It distinguishes negation, modality, quantifier types, named entity types, and verb tense and aspect.
- Data: UD 2.0 English for UPOS and parsing, sem-PMB 0.1.0 (silver for training, gold for testing), SNLI and SICK-E for NLI. For UD, SNLI and SICK-E, semantic tags were assigned automatically by the Stanford POS Tagger trained on sem-PMB.
- Results, single-task model → LWS: UPOS 92.12 → 95.54, UD DEP (LAS/UAS) 80.24/84.87 → 81.39/86.00, SNLI 87.01 → 87.51, SICK-E 81.30 → 84.57.
- On NLI, the FSN scheme performs worse than the single-task model. On the small SICK-E, accuracy drops to 56.69. For NLI, only LWS yields an improvement, and it is largest on the smaller SICK-E dataset.
- Control experiment on SNLI: the architecture was kept unchanged, but the auxiliary task was given zero weight. PSN and LWS then fall to the level of the single-task model. Hence the gain comes from the semantic tag signal itself, not from the more complex architecture.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/topics/part-of-speech-tagging/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{abdou2018what,
  title = {{What can we learn from Semantic Tagging?}},
  author = {Abdou, Mostafa and Kulmizev, Artur and Ravishankar, Vinit and Abzianidze, Lasha and Bos, Johan},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1526},
  eprint = {1808.09716},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1526}
}
```

- [← previous We Usually Don't Like Going to the Dentist: Using Common Sense to Detect Irony on Twitter](https://md.research.elamur.ai/free/papers/hee-2018-we-usually-don-t-like/)
- [next → What do RNN Language Models Learn about Filler-Gap Dependencies?](https://md.research.elamur.ai/free/papers/wilcox-2018-rnn-language-models-learn-about/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/4/)

## References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The parallel meaning bank: Towards a multilingual corpus of translations annotated with compositional meaning representations. In Proceedings of the 15th Conference of… [find](https://scholar.google.com/scholar?q=Lasha+Abzianidze%2C+Johannes+Bjerva%2C+Kilian+Evang%2C+Hessel+Haagsma%2C+Rik+van+Noord%2C+Pierre+Ludmann%2C+Duc%2DDuy+Nguyen%2C+and+Johan+Bos%2E+2017%2E+The+parallel+meaning+bank%3A+Towards+a+multilingual+corpus+of+transla) [our card](https://md.research.elamur.ai/free/papers/abzianidze-2017-parallel-meaning-bank-multilingual-corpus/)
2. Lasha Abzianidze and Johan Bos. 2017. Towards universal semantic tagging. In Proceedings of the 12th International Conference on Computational Semantics (IWCS 2017) – Short Papers, pages 1–6, Montpellier, France. [find](https://scholar.google.com/scholar?q=Lasha+Abzianidze+and+Johan+Bos%2E+2017%2E+Towards+universal+semantic+tagging%2E+In+Proceedings+of+the+12th+International+Conference+on+Computational+Semantics+%28IWCS+2017%29+%E2%80%93+Short+Papers%2C+pages+1%E2%80%936%2C+Montpell)
3. Jonathan Baxter et al. 2000. A model of inductive bias learning. J. Artif. Intell. Res.(JAIR), 12(149-198):3. [find](https://scholar.google.com/scholar?q=Jonathan+Baxter+et+al%2E+2000%2E+A+model+of+inductive+bias+learning%2E+J%2E+Artif%2E+Intell%2E+Res%2E%28JAIR%29%2C+12%28149%2D198%29%3A3%2E)
4. Yonatan Belinkov, Lluı́s Màrquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2018. Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. arXiv preprint arXiv:1801.07772. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Llu%C4%B1%CC%81s+Ma%CC%80rquez%2C+Hassan+Sajjad%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+and+James+Glass%2E+2018%2E+Evaluating+layers+of+representation+in+neural+machine+translation+on+part%2Dof%2Dspeech+and+semantic+ta) [our card](https://md.research.elamur.ai/free/papers/belinkov-2018-evaluating-layers-representation-neural-machine/)
5. Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic tagging with deep residual networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531–3541. [find](https://scholar.google.com/scholar?q=Johannes+Bjerva%2C+Barbara+Plank%2C+and+Johan+Bos%2E+2016%2E+Semantic+tagging+with+deep+residual+networks%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computational+Linguistics%3A+Techni)
6. The COLING 2016 Organizing Committee. [find](https://scholar.google.com/scholar?q=The+COLING+2016+Organizing+Committee%2E)
7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
8. Rich Caruana. 1998. Multitask learning. In Learning to learn, pages 95–133. Springer. [find](https://scholar.google.com/scholar?q=Rich+Caruana%2E+1998%2E+Multitask+learning%2E+In+Learning+to+learn%2C+pages+95%E2%80%93133%2E+Springer%2E)
9. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1657–1668. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
10. Yoeng-Jin Chu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396–1400. [find](https://scholar.google.com/scholar?q=Yoeng%2DJin+Chu%2E+1965%2E+On+the+shortest+arborescence+of+a+directed+graph%2E+Science+Sinica%2C+14%3A1396%E2%80%931400%2E)
11. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+international+conference+on+Mac)
12. Timothy Dozat and Christopher D Manning. 2016. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv:1611.01734. [find](https://scholar.google.com/scholar?q=Timothy+Dozat+and+Christopher+D+Manning%2E+2016%2E+Deep+biaffine+attention+for+neural+dependency+parsing%2E+arXiv+preprint+arXiv%3A1611%2E01734%2E) [our card](https://md.research.elamur.ai/free/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/)
13. Timothy Dozat, Peng Qi, and Christopher D Manning. 2017. Stanford’s graph-based neural dependency parser at the conll 2017 shared task. Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 20–30. [find](https://scholar.google.com/scholar?q=Timothy+Dozat%2C+Peng+Qi%2C+and+Christopher+D+Manning%2E+2017%2E+Stanford%E2%80%99s+graph%2Dbased+neural+dependency+parser+at+the+conll+2017+shared+task%2E+Proceedings+of+the+CoNLL+2017+Shared+Task%3A+Multilingual+Parsing+) [our card](https://md.research.elamur.ai/free/papers/ud-2017-dozat-stanford-s-graph-based-neural/)
14. Jack Edmonds. 1967. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):233–240. [find](https://scholar.google.com/scholar?q=Jack+Edmonds%2E+1967%2E+Optimum+branchings%2E+Journal+of+Research+of+the+national+Bureau+of+Standards+B%2C+71%284%29%3A233%E2%80%93240%2E)
15. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. [find](https://scholar.google.com/scholar?q=Kazuma+Hashimoto%2C+Caiming+Xiong%2C+Yoshimasa+Tsuruoka%2C+and+Richard+Socher%2E+2016%2E+A+joint+many%2Dtask+model%3A+Growing+a+neural+network+for+multiple+nlp+tasks%2E+arXiv+preprint+arXiv%3A1611%2E01587%2E) [our card](https://md.research.elamur.ai/free/papers/hashimoto-2016-joint-many-task-model-growing/)
16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
17. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
18. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. — [arXiv:1605.05101](https://arxiv.org/abs/1605.05101) [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2016%2E+Recurrent+neural+network+for+text+classification+with+multi%2Dtask+learning%2E+arXiv+preprint+arXiv%3A1605%2E05101%2E+%E2%80%94+%5BarXiv%3A1605%2E05101%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
19. Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. 2017. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5334–5343. [find](https://scholar.google.com/scholar?q=Yongxi+Lu%2C+Abhishek+Kumar%2C+Shuangfei+Zhai%2C+Yu+Cheng%2C+Tara+Javidi%2C+and+Rogerio+Feris%2E+2017%2E+Fully%2Dadaptive+feature+sharing+in+multi%2Dtask+networks+with+applications+in+person+attribute+classification%2E+I)
20. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+large+annotated+corpus+of+english%3A+The+penn+treebank%2E+Computational+linguistics%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In LREC, pages 216–223. [find](https://scholar.google.com/scholar?q=Marco+Marelli%2C+Stefano+Menini%2C+Marco+Baroni%2C+Luisa+Bentivogli%2C+Raffaella+Bernardi%2C+Roberto+Zamparelli%2C+et+al%2E+2014%2E+A+sick+cure+for+the+evaluation+of+compositional+distributional+semantic+models%2E+In+L)
22. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computat… [find](https://scholar.google.com/scholar?q=Ryan+McDonald%2C+Joakim+Nivre%2C+Yvonne+Quirmbach%2DBrundage%2C+Yoav+Goldberg%2C+Dipanjan+Das%2C+Kuzman+Ganchev%2C+Keith+Hall%2C+Slav+Petrov%2C+Hao+Zhang%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+et+al%2E+2013%2E+Universal+dependency+annotation)
23. Elliot Meyerson and Risto Miikkulainen. 2017. Beyond shared hierarchies: Deep multitask learning through soft layer ordering. arXiv preprint arXiv:1711.00108. — [arXiv:1711.00108](https://arxiv.org/abs/1711.00108) [find](https://scholar.google.com/scholar?q=Elliot+Meyerson+and+Risto+Miikkulainen%2E+2017%2E+Beyond+shared+hierarchies%3A+Deep+multitask+learning+through+soft+layer+ordering%2E+arXiv+preprint+arXiv%3A1711%2E00108%2E+%E2%80%94+%5BarXiv%3A1711%2E00108%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
24. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994–4003. [find](https://scholar.google.com/scholar?q=Ishan+Misra%2C+Abhinav+Shrivastava%2C+Abhinav+Gupta%2C+and+Martial+Hebert%2E+2016%2E+Cross%2Dstitch+networks+for+multi%2Dtask+learning%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recogniti)
25. Joakim Nivre, Lars Ahrenberg Zeljko Agic, et al. 2017. Universal dependencies 2.0. lindat/clarin digital library at the institute of formal and applied linguistics, charles university, prague. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Lars+Ahrenberg+Zeljko+Agic%2C+et+al%2E+2017%2E+Universal+dependencies+2%2E0%2E+lindat%2Fclarin+digital+library+at+the+institute+of+formal+and+applied+linguistics%2C+charles+university%2C+prague%2E)
26. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+process)
27. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. arXiv preprint arXiv:1604.05529. [find](https://scholar.google.com/scholar?q=Barbara+Plank%2C+Anders+S%C3%B8gaard%2C+and+Yoav+Goldberg%2E+2016%2E+Multilingual+part%2Dof%2Dspeech+tagging+with+bidirectional+long+short%2Dterm+memory+models+and+auxiliary+loss%2E+arXiv+preprint+arXiv%3A1604%2E05529%2E) [our card](https://md.research.elamur.ai/free/papers/plank-2016-multilingual-part-speech-tagging-bidirectional/)
28. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2017. Sluice networks: Learning what to share between loosely related tasks. arXiv preprint arXiv:1705.08142. — [arXiv:1705.08142](https://arxiv.org/abs/1705.08142) [find](https://scholar.google.com/scholar?q=Sebastian+Ruder%2C+Joachim+Bingel%2C+Isabelle+Augenstein%2C+and+Anders+S%C3%B8gaard%2E+2017%2E+Sluice+networks%3A+Learning+what+to+share+between+loosely+related+tasks%2E+arXiv+preprint+arXiv%3A1705%2E08142%2E+%E2%80%94+%5BarXiv%3A1705%2E08)
29. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 231–235. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard+and+Yoav+Goldberg%2E+2016%2E+Deep+multi%2Dtask+learning+with+low+level+tasks+supervised+at+lower+layers%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/papers/sogaard-2016-deep-multi-task-learning-low/)
30. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume… [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Dan+Klein%2C+Christopher+D+Manning%2C+and+Yoram+Singer%2E+2003%2E+Feature%2Drich+part%2Dof%2Dspeech+tagging+with+a+cyclic+dependency+network%2E+In+Proceedings+of+the+2003+Conference+of+the+North+A)