---
title: "Utilizing Subword — paper by Zalmout and Habash (2020)"
description: "The paper addresses lemmatization of Arabic, where undiacritized words must be mapped to fully… 30 references, cited by 2 papers here."
authors: ["Nasser Zalmout", "Nizar Habash"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.412/"
doi: "10.18653/v1/2020.coling-main.412"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zalmout-2020-utilizing-subword-entities-character-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zalmout-2020-utilizing-subword-entities-character-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/) / Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models

# Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models

**Authors:** [Nasser Zalmout](https://md.research.elamur.ai/free/en/authors/nasser-zalmout/) and [Nizar Habash](https://md.research.elamur.ai/free/en/authors/nizar-habash/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.412/) · [PDF](https://aclanthology.org/2020.coling-main.412.pdf) · [DOI 10.18653/v1/2020.coling-main.412](https://doi.org/10.18653/v1/2020.coling-main.412) · [OpenAlex](https://openalex.org/W3113532658)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/zalmout-2020-utilizing-subword-entities-character-level/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses lemmatization of Arabic, where undiacritized words must be mapped to fully diacritized lemmas despite clitics and templatic morphology that make readings ambiguous. The authors start from a Lematus-style character-level seq2seq model with attention. They change it to use a context of whole neighbouring words instead of a fixed character window. They then add several kinds of cheap subword information:
- FastText n-gram embeddings
- greedy stems, plus orthographic roots and patterns derived from them
- Morfessor base words

These features are fed in two ways: concatenated onto each character embedding at the encoder, or predicted from gold annotations as auxiliary decoder tasks. Experiments use MSA (PATB) and Egyptian Arabic (ARZ). Pretrained FastText embeddings at the encoder match or beat every linguistically motivated subword, and they come close to an oracle that uses gold stems. Relative error reduction over the baseline reaches up to 22% for MSA and 13% for EGY. The systems also handle unseen words and lemmas better. The main lesson is that inexpensive, unsupervised subword signals can compete with costly morphological resources.

## Key points
- **Data:**
  - MSA: Penn Arabic Treebank parts 1–3, about 503K training tokens.
  - EGY: ARZ corpus parts 1–5, about 133K training tokens.
  - Pretraining corpora: Gigaword for MSA and BOLT Arabic Forum Discussions for EGY.
- **Accuracy (baseline → best):**
  - The word-context seq2seq baseline scores 94.1 on MSA and 81.8 on EGY.
  - Adding FastText raises these to 95.4 and 83.3.
  - The gold-stem oracle reaches only 95.6 and 83.6.
- **Weak alternatives:**
  - Multitask learning with gold stems, roots or patterns gave little gain and slightly hurt on EGY.
  - Combining FastText with morphological subwords added nothing significant.
- **Quality of the cheap subwords:** greedy stems agree with gold stems about 65% of the time, and Morfessor base words about 68% in MSA.
- **Unseen words (MSA):** accuracy rose from 67.4 to 73.1.
- **MLE back-off:** using a lookup table for seen words, with the model as back-off for unseen ones, raised EGY to 84.1 but lowered MSA to 95.0.

## Interesting ideas
- Strip the longest known affix sequences greedily from both ends of a word; the resulting "stem" is often wrong but behaves consistently, and that consistency is what makes it a useful feature
- Define the context window as the characters of n whole neighbouring words rather than a fixed character span, which kept word boundaries and gave better results
- Run an oracle with gold features fed through exactly the same pathway to measure the ceiling a feature type could reach before investing in it
- Use a frequency lookup for seen word forms and a generative model only for unseen ones; this helps when training data is small but can hurt when the model is already strong
- Sort errors into valid lemmas that are wrong in context versus invalid, non-existent lemmas, since the two failure types point to different fixes

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): the en lemmatizer could attach a greedy suffix-stripped stem as an extra feature for out-of-vocabulary words, with the affix list harvested from treebank lemma pairs.
- **I2** (tokens): context features for lemma and tag disambiguation should be built from whole neighbouring tokens, not arbitrary character slices.
- **I3** (learn): before building expensive morphology or annotation for a component, plug in gold values once to see whether the possible gain justifies the effort.
- **I4** (tokens): a dictionary-first lemmatizer with a learned fallback, keeping the routing decision explainable and tuned on held-out data per domain.
- **I5** (learn): add this split to lemma evaluation, and check outputs against a lexicon such as WordNet to flag invalid lemmas automatically as a negative control.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University Abu Dhabi](https://md.research.elamur.ai/free/en/universities/new-york-university-abu-dhabi/) — University, Abu Dhabi, United Arab Emirates · 122 works · 4,602 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

💡 Concepts (1): [💡 Lemmatization](https://md.research.elamur.ai/free/en/concepts/lemmatization/) · 20 works

## Cite

Cite as: Nasser Zalmout and Nizar Habash. 2020. Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zalmout2020utilizing,
  title = {{Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models}},
  author = {Nasser Zalmout and Nizar Habash},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.412},
  url = {https://doi.org/10.18653/v1/2020.coling-main.412}
}
```

- [← previous The QMUL/HRBDT contribution to the NADI Arabic Dialect Identification Shared Task](https://md.research.elamur.ai/free/en/papers/aloraini-2020-qmul-hrbdt-contribution-nadi-arabic/)
- [next → #GCDH at WNUT-2020 Task 2: BERT-Based Models for the Detection of Informativeness in English COVID-19 Related Tweets](https://md.research.elamur.ai/free/en/papers/varachkina-2020-gcdh-wnut-2020-task-2/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/9/)

## References

30 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 1171–1179. MIT Press. [↗](https://doi.org/10.48550/arxiv.1506.03099)
2. Toms Bergmanis and Sharon Goldwater. 2018. [Context sensitive neural lemmatization with Lematus](https://md.research.elamur.ai/free/en/papers/bergmanis-2018-context-sensitive-neural-lemmatization-lematus/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1391–1400.
3. Anders Björkelund, Bernd Bohnet, Love Hafdell, and Pierre Nugues. 2010. A high-performance syntactic and semantic dependency parser. In Proceedings of the 23rd International Conference on Computational Linguistics: Demonstrations, pages 33–36. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Anders+Bj%C3%B6rkelund%2C+Bernd+Bohnet%2C+Love+Hafdell%2C+and+Pierre+Nugues%2E+2010%2E+A+high%2Dperformance+syntactic+and+semantic+dependency+parser%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computationa)
4. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146. [↗](https://doi.org/10.1162/tacl_a_00051)
5. Grzegorz Chrupala, Georgiana Dinu, and Josef van Genabith. 2008. [Learning morphology with Morfette](https://md.research.elamur.ai/free/en/papers/chrupala-2008-learning-morphology-morfette/). LREC 2008, pages 2362–2367.
6. Mathias Creutz and Krista Lagus, 2005. Unsupervised morpheme segmentation and morphology induction from text corpora using Morfessor 1.0, chapter A. Publications in computer and information science. Report A. Helsinki University of Technology, Finland. [find](https://scholar.google.com/scholar?q=Mathias+Creutz+and+Krista+Lagus%2C+2005%2E+Unsupervised+morpheme+segmentation+and+morphology+induction+from+text+corpora+using+Morfessor+1%2E0%2C+chapter+A%2E+Publications+in+computer+and+information+science%2E+R)
7. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. [LDC Arabic treebanks and associated corpora: Data divisions manual](https://md.research.elamur.ai/free/en/papers/diab-2013-ldc-arabic-treebanks-associated-corpora/). arXiv preprint arXiv:1309.5652. [↗](https://arxiv.org/abs/1309.5652)
8. Timothy Dozat and Christopher D Manning. 2018. [Simpler but more accurate semantic dependency parsing](https://md.research.elamur.ai/free/en/papers/dozat-2018-simpler-but-more-accurate-semantic/). In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 484–490. [↗](https://doi.org/10.48550/arxiv.1807.01396)
9. Ramy Eskander, Nizar Habash, and Owen Rambow. 2013. Automatic Extraction of Morphological Lexicons from Morphologically Annotated Corpora. In Proceedings of tenth Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Ramy+Eskander%2C+Nizar+Habash%2C+and+Owen+Rambow%2E+2013%2E+Automatic+Extraction+of+Morphological+Lexicons+from+Morphologically+Annotated+Corpora%2E+In+Proceedings+of+tenth+Conference+on+Empirical+Methods+in+Na)
10. Nerea Ezeiza, Iñaki Alegria, José María Arriola, Rubén Urizar, and Itziar Aduriz. 1998. Combining stochastic and rule-based methods for disambiguation in agglutinative languages. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 380–384. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nerea+Ezeiza%2C+I%C3%B1aki+Alegria%2C+Jos%C3%A9+Mar%C3%ADa+Arriola%2C+Rub%C3%A9n+Urizar%2C+and+Itziar+Aduriz%2E+1998%2E+Combining+stochastic+and+rule%2Dbased+methods+for+disambiguation+in+agglutinative+languages%2E+In+Proceedings+of+the)
11. Alexander Fraser, Marion Weller, Aoife Cahill, and Fabienne Cap. 2012. Modeling inflection and word-formation in SMT. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 664–674. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+Fraser%2C+Marion+Weller%2C+Aoife+Cahill%2C+and+Fabienne+Cap%2E+2012%2E+Modeling+inflection+and+word%2Dformation+in+SMT%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+f)
12. Jenna Kanerva, Filip Ginter, and Tapio Salakoski. 2019. Universal lemmatizer: A sequence to sequence model for lemmatizing universal dependencies treebanks. arXiv preprint arXiv:1902.00972. [↗](https://arxiv.org/abs/1902.00972)
13. Diederik P. Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). CoRR, abs/1412.6980. [↗](https://arxiv.org/abs/1412.6980)
14. Daniel Kondratyuk, Tomáš Gavenčiak, Milan Straka, and Jan Hajič. 2018. Lemmatag: Jointly tagging and lemmatizing for morphologically rich languages with BRNNs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4921–4928. [find](https://scholar.google.com/scholar?q=Daniel+Kondratyuk%2C+Tom%C3%A1%C5%A1+Gavenc%CC%8Ciak%2C+Milan+Straka%2C+and+Jan+Hajic%CC%8C%2E+2018%2E+Lemmatag%3A+Jointly+tagging+and+lemmatizing+for+morphologically+rich+languages+with+BRNNs%2E+In+Proceedings+of+the+2018+Conference+)
15. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). arXiv preprint arXiv:1508.04025. [↗](https://arxiv.org/abs/1508.04025)
16. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102–109, Cairo, Egypt. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Ann+Bies%2C+Tim+Buckwalter%2C+and+Wigdan+Mekki%2E+2004%2E+The+Penn+Arabic+Treebank%3A+Building+a+Large%2DScale+Annotated+Arabic+Corpus%2E+In+NEMLAR+Conference+on+Arabic+Language+Resources+and+Tool)
17. Mohamed Maamouri, Sondos Krouna, Dalila Tabessi, Nadia Hamrouni, and Nizar Habash. 2012. Egyptian Arabic Morphological Annotation Guidelines. [find](https://scholar.google.com/scholar?q=Mohamed+Maamouri%2C+Sondos+Krouna%2C+Dalila+Tabessi%2C+Nadia+Hamrouni%2C+and+Nizar+Habash%2E+2012%2E+Egyptian+Arabic+Morphological+Annotation+Guidelines%2E)
18. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1310.4546)
19. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of English. Natural Language Engineering, 7(3):207–223. [↗](https://doi.org/10.1017/s1351324901002728)
20. Thomas Müller, Ryan Cotterell, Alexander Fraser, and Hinrich Schütze. 2015. [Joint lemmatization and morphological tagging with Lemming](https://md.research.elamur.ai/free/en/papers/muller-2015-joint-lemmatization-morphological-tagging-lemming/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2268–2274.
21. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic Gigaword Fifth Edition. LDC catalog number No. LDC2011T11, ISBN 1-58563-595-2. [↗](https://doi.org/10.35111/p02g-rw14)
22. Arfath Pasha, Mohamed Al-Badrashiny, Ahmed El Kholy, Ramy Eskander, Mona Diab, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. [MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic](https://md.research.elamur.ai/free/en/papers/pasha-2014-madamira-fast-comprehensive-tool-morphological/). In In Proceedings of LREC, Reykjavik, Iceland.
23. Tobias Pütz, Daniël De Kok, Sebastian Pütz, and Erhard Hinrichs. 2018. Seq2seq or perceptrons for robust lemmatization. an empirical examination. In Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), December 13–14, 2018, Oslo University, Norway, pages 193–207. Linköping University Electronic Press. [find](https://scholar.google.com/scholar?q=Tobias+P%C3%BCtz%2C+Dani%C3%ABl+De+Kok%2C+Sebastian+P%C3%BCtz%2C+and+Erhard+Hinrichs%2E+2018%2E+Seq2seq+or+perceptrons+for+robust+lemmatization%2E+an+empirical+examination%2E+In+Proceedings+of+the+17th+International+Workshop+on+T)
24. Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. [Arabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking](https://md.research.elamur.ai/free/en/papers/roth-2008-arabic-morphological-tagging-diacritization-lemmatization/). In ACL 2008: The Conference of the Association for Computational Linguistics; Companion Volume, Short Papers, Columbus, Ohio, June. Association for Computational Linguistics.
25. Helmut Schmid, Arne Fitschen, and Ulrich Heid. 2004. [SMOR: A German computational morphology covering derivation, composition and inflection](https://md.research.elamur.ai/free/en/papers/schmid-2004-smor-german-computational-morphology-covering/). In LREC, pages 1–263. Lisbon.
26. Djamé Seddah, Grzegorz Chrupała, Özlem Çetinoğlu, Josef Van Genabith, and Marie Candito. 2010. Lemmatization and lexicalized statistical parsing of morphologically rich languages: the case of French. In Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 85–93. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Djam%C3%A9+Seddah%2C+Grzegorz+Chrupa%C5%82a%2C+%C3%96zlem+%C3%87etinog%CC%86lu%2C+Josef+Van+Genabith%2C+and+Marie+Candito%2E+2010%2E+Lemmatization+and+lexicalized+statistical+parsing+of+morphologically+rich+languages%3A+the+case+of+French%2E)
27. Rico Sennrich and Barry Haddow. 2016. [Linguistic input features improve neural machine translation](https://md.research.elamur.ai/free/en/papers/sennrich-2016-linguistic-input-features-improve-neural/). In Proceedings of the First Conference on Machine Translation: Volume 1, Research Papers, volume 1, pages 83–91.
28. Jennifer Tracey, Haejoong Lee, Stephanie Strassel, and Safa Ismael. 2018. BOLT Arabic Discussion Forum Source Data. LDC catalog number LDC2018T10. [find](https://scholar.google.com/scholar?q=Jennifer+Tracey%2C+Haejoong+Lee%2C+Stephanie+Strassel%2C+and+Safa+Ismael%2E+2018%2E+BOLT+Arabic+Discussion+Forum+Source+Data%2E+LDC+catalog+number+LDC2018T10%2E)
29. Nasser Zalmout and Nizar Habash. 2017. [Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for Arabic](https://md.research.elamur.ai/free/en/papers/zalmout-2017-don-t-throw-those-morphological/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704–713, Copenhagen, Denmark, September. Association for Computational Linguistics.
30. Nasser Zalmout and Nizar Habash. 2019. [Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling](https://md.research.elamur.ai/free/en/papers/zalmout-2019-adversarial-multitask-learning-joint-multi/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1775–1786.

*Updated 1 November 2026*