---
title: "Exact Hard Monotonic Attention for Character-Level Transduction"
description: "Wu and Cotterell argue that earlier monotonic models fell short because they were pipelines… 29 references, cited by 49 papers here."
authors: ["Shijie Wu", "Ryan Cotterell"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1148/"
doi: "10.18653/v1/p19-1148"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wu-2019-exact-hard-monotonic-attention-character/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wu-2019-exact-hard-monotonic-attention-character/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Exact Hard Monotonic Attention for Character-Level Transduction

# Exact Hard Monotonic Attention for Character-Level Transduction

**Authors:** [Shijie Wu](https://md.research.elamur.ai/free/en/authors/shijie-wu/) and [Ryan Cotterell](https://md.research.elamur.ai/free/en/authors/ryan-cotterell/) · **Year:** 2019 · **Venue:** [Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2019](https://md.research.elamur.ai/free/en/venues/acl/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1148/) · [PDF](https://aclanthology.org/P19-1148.pdf) · [DOI 10.18653/v1/p19-1148](https://doi.org/10.18653/v1/p19-1148) · [OpenAlex](https://openalex.org/W2950825792)

**Citations:** 54 citations (OpenAlex, 2026-10-14) · [49](https://md.research.elamur.ai/free/en/papers/wu-2019-exact-hard-monotonic-attention-character/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Character-level transduction tasks such as grapheme-to-phoneme conversion (G2P), named-entity transliteration and morphological inflection map source characters to target characters in an almost entirely left-to-right order. Despite this, non-monotonic soft-attention seq2seq models had been beating monotonic systems. Wu and Cotterell argue that earlier monotonic models fell short because they were pipelines: alignments came from an external aligner rather than being learned together with the transducer.

They extend the hard-attention model of Wu et al. (2018) into a neuralized HMM that treats alignments as latent variables. Monotonicity is enforced by giving zero probability to any backward move in the alignment. Dynamic programming (the forward algorithm) sums exactly over all monotonic alignments in cubic time, and decoding uses a greedy approximation. The authors compare four models of roughly equal size in a controlled setting.

Monotonic hard attention gives the best single-model result on the CoNLL-SIGMORPHON 2017 high-resource inflection data and helps on all three tasks. The results suggest that monotonicity is a useful inductive bias, provided the alignments are trained jointly with the model.

## Key points
- **Models compared** (all about 8.6M parameters):
  - SOFT: soft attention without input-feeding
  - 0-HARD: 0th-order hard attention
  - 0-MONO: 0th-order monotonic hard attention, with no extra parameters
  - 1-MONO: 1st-order monotonic hard attention, using an offset-based transition with window w
- **Cost of the monotonic constraint:** likelihood computation is O(|x|²·|y|·|Σy|), an extra factor of |x| over the non-monotonic 0th-order model's O(|x|·|y|·|Σy|).
- **Morphological inflection, test set:** accuracy is 92.9 for SOFT, 93.8 for 0-HARD and 94.4 for both 0-MONO and 1-MONO. On the dev set, both monotonic models reach 94.8, ahead of the 94.6 scored by Bergmanis et al. (2017) and Makarov and Clematide (2018).
- **Other tasks:**
  - Transliteration accuracy rises from 40.4 (SOFT) to 41.2 (0-MONO).
  - G2P WER is lowest for 1-MONO (28.2 vs 29.3 for SOFT). G2P is the only task where the richer transition clearly helps.
- **Takeaway and code:** the hard monotonic constraint matters more than additional transition parameters. Code is released at github.com/shijie-wu/neural-transducer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations
- [University of Cambridge](https://md.research.elamur.ai/free/en/universities/university-of-cambridge/) — University, Cambridge, United Kingdom · 155 works · 115,562 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 277 works

## Cite

Cite as: Shijie Wu and Ryan Cotterell. 2019. Exact Hard Monotonic Attention for Character-Level Transduction. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wu2019exact,
  title = {{Exact Hard Monotonic Attention for Character-Level Transduction}},
  author = {Shijie Wu and Ryan Cotterell},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1148},
  url = {https://doi.org/10.18653/v1/p19-1148}
}
```

- [← previous Understanding Learning Dynamics Of Language Models with SVCCA](https://md.research.elamur.ai/free/en/papers/saphra-2019-understanding-learning-dynamics-language-models/)
- [next → Exploiting Monolingual Data at Scale for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/wu-2019-exploiting-monolingual-data-scale-neural/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

29 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni and Yoav Goldberg. 2017. [Morphological inflection generation with hard monotonic attention](https://md.research.elamur.ai/free/en/papers/aharoni-2017-morphological-inflection-generation-hard-monotonic/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2004–2015, Vancouver, Canada. Association for Computational Linguistics.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In International Conference on Learning Representations (ICLR), volume abs/1409.0473.
3. Toms Bergmanis, Katharina Kann, Hinrich Schütze, and Sharon Goldwater. 2017. Training data augmentation for low-resource morphological inflection. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 31–39, Vancouver. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Toms+Bergmanis%2C+Katharina+Kann%2C+Hinrich+Sch%C3%BCtze%2C+and+Sharon+Goldwater%2E+2017%2E+Training+data+augmentation+for+low%2Dresource+morphological+inflection%2E+In+Proceedings+of+the+CoNLL+SIGMORPHON+2017+Shared+Ta)
4. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+F%2E+Brown%2C+Vincent+J%2E+Della+Pietra%2C+Stephen+A%2E+Della+Pietra%2C+and+Robert+L%2E+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29)
5. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Christo+Kirov%2C+John+Sylak%2DGlassman%2C+G%C3%A9raldine+Walther%2C+Ekaterina+Vylomova%2C+Patrick+Xia%2C+Manaal+Faruqui%2C+Sandra+K%C3%BCbler%2C+David+Yarowsky%2C+Jason+Eisner%2C+and+Mans+Hulden%2E+2017%2E)
6. The CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+CoNLL%2DSIGMORPHON+2017+shared+task%3A+Universal+morphological+reinflection+in+52+languages%2E+In+Proceedings+of+the+CoNLL%2DSIGMORPHON+2017+Shared+Task%3A+Universal+Morphological+Reinflection%2C+Vancouver%2C+C)
7. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. [find](https://scholar.google.com/scholar?q=Ryan+Cotterell%2C+Christo+Kirov%2C+John+Sylak%2DGlassman%2C+David+Yarowsky%2C+Jason+Eisner%2C+and+Mans+Hulden%2E+2016%2E)
8. The SIGMORPHON 2016 shared task— morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=The+SIGMORPHON+2016+shared+task%E2%80%94+morphological+reinflection%2E+In+Proceedings+of+the+14th+SIGMORPHON+Workshop+on+Computational+Research+in+Phonetics%2C+Phonology%2C+and+Morphology%2C+pages+10%E2%80%9322%2E+Association+)
9. Alex Graves. 2012. [Sequence transduction with recurrent neural networks](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/). arXiv preprint arXiv:1211.3711.
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+J%C3%BCrgen+Schmidhuber%2E+1997%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
11. Diederick P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Diederick+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+International+Conference+on+Learning+Representations+%28ICLR%29%2E)
12. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
13. Peter Makarov and Simon Clematide. 2018. Imitation learning for neural morphological string transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2877–2882. [find](https://scholar.google.com/scholar?q=Peter+Makarov+and+Simon+Clematide%2E+2018%2E+Imitation+learning+for+neural+morphological+string+transduction%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pag)
14. Peter Makarov, Tatiana Ruzsics, and Simon Clematide. 2017. Align and copy: UZH at SIGMORPHON 2017 shared task for morphological reinflection. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 49–57. [find](https://scholar.google.com/scholar?q=Peter+Makarov%2C+Tatiana+Ruzsics%2C+and+Simon+Clematide%2E+2017%2E+Align+and+copy%3A+UZH+at+SIGMORPHON+2017+shared+task+for+morphological+reinflection%2E+Proceedings+of+the+CoNLL+SIGMORPHON+2017+Shared+Task%3A+Univ)
15. Lawrence R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257– 286. [find](https://scholar.google.com/scholar?q=Lawrence+R%2E+Rabiner%2E+1989%2E+A+tutorial+on+hidden+Markov+models+and+selected+applications+in+speech+recognition%2E+Proceedings+of+the+IEEE%2C+77%282%29%3A257%E2%80%93+286%2E)
16. Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, and Douglas Eck. 2017. Online and linear-time attention by enforcing monotonic alignments. In International Conference on Machine Learning (ICML), pages 2837–2846. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Minh%2DThang+Luong%2C+Peter+J%2E+Liu%2C+Ron+J%2E+Weiss%2C+and+Douglas+Eck%2E+2017%2E+Online+and+linear%2Dtime+attention+by+enforcing+monotonic+alignments%2E+In+International+Conference+on+Machine+Learning+%28)
17. Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. 2016. Weighting finite-state transductions with neural context. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 623–633, San Diego, Californi… [find](https://scholar.google.com/scholar?q=Pushpendre+Rastogi%2C+Ryan+Cotterell%2C+and+Jason+Eisner%2E+2016%2E+Weighting+finite%2Dstate+transductions+with+neural+context%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Associa)
18. Mihaela Rosca and Thomas Breuel. 2016. Sequence-to-sequence neural network models for transliteration. arXiv preprint arXiv:1610.09565. — [arXiv:1610.09565](https://arxiv.org/abs/1610.09565) [find](https://scholar.google.com/scholar?q=Mihaela+Rosca+and+Thomas+Breuel%2E+2016%2E+Sequence%2Dto%2Dsequence+neural+network+models+for+transliteration%2E+arXiv+preprint+arXiv%3A1610%2E09565%2E+%E2%80%94+%5BarXiv%3A1610%2E09565%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1610%2E09565%29)
19. Terrence J. Sejnowski and Charles R. Rosenberg. 1987. Parallel networks that learn to pronounce english text. Complex Systems, 1. [find](https://scholar.google.com/scholar?q=Terrence+J%2E+Sejnowski+and+Charles+R%2E+Rosenberg%2E+1987%2E+Parallel+networks+that+learn+to+pronounce+english+text%2E+Complex+Systems%2C+1%2E)
20. Miikka Silfverberg, Adam Wiemerslage, Ling Liu, and Lingshuang Jack Mao. 2017. [Data augmentation for morphological reinflection](https://md.research.elamur.ai/free/en/papers/silfverberg-2017-data-augmentation-morphological-reinflection/). In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 90–99, Vancouver. Association for Computational Linguistics.
21. Katsuhito Sudoh, Shinsuke Mori, and Masaaki Nagata. 2013. Noise-aware character alignment for bootstrapping statistical machine transliteration from bilingual corpora. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 204–209. [find](https://scholar.google.com/scholar?q=Katsuhito+Sudoh%2C+Shinsuke+Mori%2C+and+Masaaki+Nagata%2E+2013%2E+Noise%2Daware+character+alignment+for+bootstrapping+statistical+machine+transliteration+from+bilingual+corpora%2E+In+Proceedings+of+the+2013+Confe)
22. Weiyue Wang, Derui Zhu, Tamer Alkhouli, Zixuan Gan, and Hermann Ney. 2018. Neural hidden Markov model for machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 377–382. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Weiyue+Wang%2C+Derui+Zhu%2C+Tamer+Alkhouli%2C+Zixuan+Gan%2C+and+Hermann+Ney%2E+2018%2E+Neural+hidden+Markov+model+for+machine+translation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computat)
23. R.L. Weide. 1998. The Carnegie Mellon pronouncing dictionary. [find](https://scholar.google.com/scholar?q=R%2EL%2E+Weide%2E+1998%2E+The+Carnegie+Mellon+pronouncing+dictionary%2E)
24. Shijie Wu, Pamela Shapiro, and Ryan Cotterell. 2018. Hard non-monotonic attention for character-level transduction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4425–4438. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shijie+Wu%2C+Pamela+Shapiro%2C+and+Ryan+Cotterell%2E+2018%2E+Hard+non%2Dmonotonic+attention+for+character%2Dlevel+transduction%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
25. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). In Proceedings of the 32nd International Conference on Machine Learning, ICML, pages 2048…
26. Kaisheng Yao and Geoffrey Zweig. 2015. Sequence-to-sequence neural net models for grapheme-to-phoneme conversion. In INTERSPEECH 2015, pages 3330–3334, Dresden, Germany. [find](https://scholar.google.com/scholar?q=Kaisheng+Yao+and+Geoffrey+Zweig%2E+2015%2E+Sequence%2Dto%2Dsequence+neural+net+models+for+grapheme%2Dto%2Dphoneme+conversion%2E+In+INTERSPEECH+2015%2C+pages+3330%E2%80%933334%2C+Dresden%2C+Germany%2E)
27. Lei Yu, Jan Buys, and Phil Blunsom. 2016. Online segment to segment neural transduction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1307–1316. [find](https://scholar.google.com/scholar?q=Lei+Yu%2C+Jan+Buys%2C+and+Phil+Blunsom%2E+2016%2E+Online+segment+to+segment+neural+transduction%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1307%E2%80%931316%2E)
28. Min Zhang, Haizhou Li, Rafael E. Banchs, and A. Kumaran. 2015. Whitepaper of news 2015 shared task on machine transliteration. In NEWS@ACL. [find](https://scholar.google.com/scholar?q=Min+Zhang%2C+Haizhou+Li%2C+Rafael+E%2E+Banchs%2C+and+A%2E+Kumaran%2E+2015%2E+Whitepaper+of+news+2015+shared+task+on+machine+transliteration%2E+In+NEWS%40ACL%2E)
29. Chunting Zhou and Graham Neubig. 2017. Multi-space variational encoder-decoders for semi-supervised labeled sequence transduction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 310–320, Vancouver, Canada. Association for Com… [find](https://scholar.google.com/scholar?q=Chunting+Zhou+and+Graham+Neubig%2E+2017%2E+Multi%2Dspace+variational+encoder%2Ddecoders+for+semi%2Dsupervised+labeled+sequence+transduction%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Comp)

*Updated 24 October 2026*