---
title: "Tied Multitask Learning for Neural Speech Translation — Anastasopoulos 2018"
description: "They propose a \"triangle\" model in which the translation decoder attends both to the encoder's frame… 5% CER. 21 references, cited by 79 papers here."
authors: ["Antonios Anastasopoulos", "David Chiang"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)"
source: "https://aclanthology.org/N18-1008/"
doi: "10.18653/v1/n18-1008"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / Tied Multitask Learning for Neural Speech Translation

# Tied Multitask Learning for Neural Speech Translation

**Authors:** [Antonios Anastasopoulos](https://md.research.elamur.ai/free/en/authors/antonios-anastasopoulos/) and [David Chiang](https://md.research.elamur.ai/free/en/authors/david-chiang/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-1008/) · [PDF](https://aclanthology.org/N18-1008.pdf) · [DOI 10.18653/v1/n18-1008](https://doi.org/10.18653/v1/n18-1008) · [OpenAlex](https://openalex.org/W2788575190)

**Citations:** 171 citations (OpenAlex, 2026-10-14) · [79](https://md.research.elamur.ai/free/en/papers/anastasopoulos-2018-tied-multitask-learning-neural-speech/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Endangered-language documentation often yields only small speech collections, each recording paired with a transcription and a translation into a better-resourced language. Common multitask sequence-to-sequence setups (e.g., Weiss et al., 2017) let the transcription and translation decoders share only the speech encoder. Anastasopoulos and Chiang argue that translation should draw on a more abstract representation. They propose a "triangle" model in which the translation decoder attends both to the encoder's frame states and to the hidden states of the transcription decoder, with the two context vectors concatenated. They also add attention regularizers that push the attention matrices toward transitivity and invertibility, and they decode with a two-phase beam search. On Spanish-English, Mboshi-French and Ainu-English speech, the triangle model outperforms single-task, ASR→MT pivot, standard multitask and cascade baselines in almost all cases. It improves transcription by up to 5% CER and translation by up to 2.8 character-level BLEU. Attention-based word discovery also gains about 3 F-score. Similar gains did not appear in high-resource text translation (English, French, German).

## Key points
- **Speech data:**
  - CALLHOME Spanish (LDC2014T23) with English translations: about 20 h, more than 240 speakers.
  - Mboshi-French corpus (Godard et al., 2017): about 4.4 h, three speakers.
  - Glossed Audio Corpus of Ainu Folklore: about 2.5 h, one speaker, evaluated with cross-validation.
- **Example results (vs. single-task):**
  - Mboshi CER fell from 42.3 to 31.8.
  - Spanish→English BLEU rose from 21.6 to 28.8 (triangle model, 1-best search).
- **Training objective:** an equal-weight (λ = 0.5) log-linear mix of the two decoders' likelihoods. The transitivity and invertibility regularizers each have weight 0.2.
- **Implementation:** built in DyNet with dropout 0.2, Adam (learning rate 0.0002), beam size 4 per decoder and length normalization 0.8. Code is available on Bitbucket (antonis/dynet-multitask-models).
- **Model size:** the model is much smaller than Weiss et al.'s 8-layer-encoder system, which was trained on about 163 h. This suits extremely low-resource settings.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/)

## Institutions

- [University of Notre Dame](https://md.research.elamur.ai/free/en/universities/university-of-notre-dame/) — University, Notre Dame, United States · 19 works · 35,879 citations

## Cite

Cite as: Antonios Anastasopoulos and David Chiang. 2018. Tied Multitask Learning for Neural Speech Translation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{anastasopoulos2018tied,
  title = {{Tied Multitask Learning for Neural Speech Translation}},
  author = {Antonios Anastasopoulos and David Chiang},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1008},
  url = {https://doi.org/10.18653/v1/n18-1008}
}
```

- [← previous Conversations Gone Awry: Detecting Early Signs of Conversational Failure](https://md.research.elamur.ai/free/en/papers/zhang-2018-conversations-gone-awry-detecting-early/)
- [next → Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://md.research.elamur.ai/free/en/papers/arora-2018-linear-algebraic-structure-word-senses/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gilles Adda, Sebastian Stüker, Martine Adda-Decker, Odette Ambouroue, Laurent Besacier, David Blachon, Hélène Bonneau-Maynard, Pierre Godard, Fatima Hamlaoui, Dmitry Idiatov, et al. 2016. Breaking the unwritten language barrier: The BULB project. Procedia Computer Science, 81:8–14. [find](https://scholar.google.com/scholar?q=Gilles+Adda%2C+Sebastian+Stu%CC%88ker%2C+Martine+Adda%2DDecker%2C+Odette+Ambouroue%2C+Laurent+Besacier%2C+David+Blachon%2C+He%CC%81le%CC%80ne+Bonneau%2DMaynard%2C+Pierre+Godard%2C+Fatima+Hamlaoui%2C+Dmitry+Idiatov%2C+et+al%2E+2016%2E+Breaking+)
2. Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, and Trevor Cohn. 2016. An attentional model for speech translation without transcription. In Proc. NAACL HLT. [find](https://scholar.google.com/scholar?q=Long+Duong%2C+Antonios+Anastasopoulos%2C+David+Chiang%2C+Steven+Bird%2C+and+Trevor+Cohn%2E+2016%2E+An+attentional+model+for+speech+translation+without+transcription%2E+In+Proc%2E+NAACL+HLT%2E)
3. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. In Proc. ACL. [our card](https://md.research.elamur.ai/free/en/papers/eriguchi-2017-learning-parse-translate-improves-neural/)
4. Antonios Anastasopoulos, David Chiang, and Long Duong. 2016. An unsupervised probability model Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In Proc. AAAI. [find](https://scholar.google.com/scholar?q=Antonios+Anastasopoulos%2C+David+Chiang%2C+and+Long+Duong%2E+2016%2E+An+unsupervised+probability+model+Zhaopeng+Tu%2C+Yang+Liu%2C+Lifeng+Shang%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2017%2E+Neural+machine+translation+with+reco)
5. P. Godard, G. Adda, M. Adda-Decker, J. Benjumea, L. Besacier, J. Cooper-Leavitt, G-N. Kouarata, L. Lamel, H. Maynard, M. Mueller, et al. 2017. A very low resource language speech corpus for computational language documentation experiments. arXiv:1710.03501. [find](https://scholar.google.com/scholar?q=P%2E+Godard%2C+G%2E+Adda%2C+M%2E+Adda%2DDecker%2C+J%2E+Benjumea%2C+L%2E+Besacier%2C+J%2E+Cooper%2DLeavitt%2C+G%2DN%2E+Kouarata%2C+L%2E+Lamel%2C+H%2E+Maynard%2C+M%2E+Mueller%2C+et+al%2E+2017%2E+A+very+low+resource+language+speech+corpus+for+computatio)
6. Haifeng Wang, Hua Wu, and Zhanyi Liu. 2006. Word alignment for languages with scarce resources using bilingual corpora of other language pairs. In Proc. COLING/ACL, pages 874–881. [find](https://scholar.google.com/scholar?q=Haifeng+Wang%2C+Hua+Wu%2C+and+Zhanyi+Liu%2E+2006%2E+Word+alignment+for+languages+with+scarce+resources+using+bilingual+corpora+of+other+language+pairs%2E+In+Proc%2E+COLING%2FACL%2C+pages+874%E2%80%93881%2E)
7. Hynek Hermansky. 1990. Perceptual linear predictive (PLP) analysis of speech. J. Acoustical Society of America, 87(4):1738–1752. [find](https://scholar.google.com/scholar?q=Hynek+Hermansky%2E+1990%2E+Perceptual+linear+predictive+%28PLP%29+analysis+of+speech%2E+J%2E+Acoustical+Society+of+America%2C+87%284%29%3A1738%E2%80%931752%2E)
8. Ron J. Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly transcribe foreign speech. In Proc. INTERSPEECH. [find](https://scholar.google.com/scholar?q=Ron+J%2E+Weiss%2C+Jan+Chorowski%2C+Navdeep+Jaitly%2C+Yonghui+Wu%2C+and+Zhifeng+Chen%2E+2017%2E+Sequence%2Dto%2Dsequence+models+can+directly+transcribe+foreign+speech%2E+In+Proc%2E+INTERSPEECH%2E)
9. Aren Jansen, Kenneth Church, and Hynek Hermansky. 2010. Towards spoken term discovery at scale with zero resources. In Proc. INTERSPEECH. [find](https://scholar.google.com/scholar?q=Aren+Jansen%2C+Kenneth+Church%2C+and+Hynek+Hermansky%2E+2010%2E+Towards+spoken+term+discovery+at+scale+with+zero+resources%2E+In+Proc%2E+INTERSPEECH%2E)
10. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Brid)
11. Suyoun Kim, Takaaki Hori, and Shinji Watanabe. 2017. Joint CTC-attention based end-to-end speech recognition using multi-task learning. In Proc. ICASSP. [find](https://scholar.google.com/scholar?q=Suyoun+Kim%2C+Takaaki+Hori%2C+and+Shinji+Watanabe%2E+2017%2E+Joint+CTC%2Dattention+based+end%2Dto%2Dend+speech+recognition+using+multi%2Dtask+learning%2E+In+Proc%2E+ICASSP%2E)
12. Tomer Levinboim and David Chiang. 2015. Multi-task word alignment triangulation for low-resource languages. In Proc. NAACL HLT. [find](https://scholar.google.com/scholar?q=Tomer+Levinboim+and+David+Chiang%2E+2015%2E+Multi%2Dtask+word+alignment+triangulation+for+low%2Dresource+languages%2E+In+Proc%2E+NAACL+HLT%2E)
13. Tomer Levinboim, Ashish Vaswani, and David Chiang. 2015. Model invertibility regularization: Sequence alignment with or without parallel data. In Proc. NAACL HLT. [find](https://scholar.google.com/scholar?q=Tomer+Levinboim%2C+Ashish+Vaswani%2C+and+David+Chiang%2E+2015%2E+Model+invertibility+regularization%3A+Sequence+alignment+with+or+without+parallel+data%2E+In+Proc%2E+NAACL+HLT%2E)
14. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In Proc. ICLR. [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
15. Evgeny Matusov, Stephan Kanthak, and Hermann Ney. 2005. On the integration of speech recognition and statistical machine translation. In Ninth European Conference on Speech Communication and Technology. [find](https://scholar.google.com/scholar?q=Evgeny+Matusov%2C+Stephan+Kanthak%2C+and+Hermann+Ney%2E+2005%2E+On+the+integration+of+speech+recognition+and+statistical+machine+translation%2E+In+Ninth+European+Conference+on+Speech+Communication+and+Technolog)
16. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. DyNet: The dynamic neural network toolkit. arXiv:1701.03980. [our card](https://md.research.elamur.ai/free/en/papers/neubig-2017-dynet-dynamic-neural-network-toolkit/)
17. Hermann Ney. 1999. Speech translation: Coupling of recognition and translation. In Proc. ICASSP, volume 1. [find](https://scholar.google.com/scholar?q=Hermann+Ney%2E+1999%2E+Speech+translation%3A+Coupling+of+recognition+and+translation%2E+In+Proc%2E+ICASSP%2C+volume+1%2E)
18. Toan Q. Nguyen and David Chiang. 2017. Transfer learning across low-resource related languages for neural machine translation. In Proc. IJCNLP. [our card](https://md.research.elamur.ai/free/en/papers/nguyen-2017-transfer-learning-across-low-resource/)
19. Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. 2013. Improved speech-to-text translation with the Fisher and Callhome Spanish-English speech translation corpus. In Proc. IWSLT. [find](https://scholar.google.com/scholar?q=Matt+Post%2C+Gaurav+Kumar%2C+Adam+Lopez%2C+Damianos+Karakos%2C+Chris+Callison%2DBurch%2C+and+Sanjeev+Khudanpur%2E+2013%2E+Improved+speech%2Dto%2Dtext+translation+with+the+Fisher+and+Callhome+Spanish%2DEnglish+speech+transl)
20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. ACL. [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/)
21. Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu. 2017. Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition. In Proc. Interspeech. [find](https://scholar.google.com/scholar?q=Shubham+Toshniwal%2C+Hao+Tang%2C+Liang+Lu%2C+and+Karen+Livescu%2E+2017%2E+Multitask+learning+with+low%2Dlevel+auxiliary+tasks+for+encoder%2Ddecoder+based+speech+recognition%2E+In+Proc%2E+Interspeech%2E)