---
title: "Handling Unknown Words in Statistical Latent-Variable Parsing"
description: "The paper examines how a generative constituency parser should tag words it never saw, or saw only… 25 references, cited by 24 papers here."
authors: ["Mohammed Attia", "Jennifer Foster", "Deirdre Hogan", "Joseph Le Roux", "Lamia Tounsi", "Josef van Genabith"]
year: 2010
venue: "Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages"
source: "https://aclanthology.org/W10-1408/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/attia-2010-handling-unknown-words-statistical-latent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/attia-2010-handling-unknown-words-statistical-latent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French

# Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French

**Authors:** [Mohammed Attia](https://md.research.elamur.ai/free/en/authors/mohammed-attia/), [Jennifer Foster](https://md.research.elamur.ai/free/en/authors/jennifer-foster/), [Deirdre Hogan](https://md.research.elamur.ai/free/en/authors/deirdre-hogan/), [Joseph Le Roux](https://md.research.elamur.ai/free/en/authors/joseph-le-roux/), [Lamia Tounsi](https://md.research.elamur.ai/free/en/authors/lamia-tounsi/) and [Josef van Genabith](https://md.research.elamur.ai/free/en/authors/josef-van-genabith/) · **Year:** 2010 · **Venue:** [Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-statistical-parsing-of-morphologically-rich-languages/) ([NAACL HLT Workshop on Statistical Parsing of Morphologically-Rich Languages 2010](https://md.research.elamur.ai/free/en/venues/naacl-hlt-workshop-on-statistical-parsing-of-morphologically-rich-languages/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W10-1408/) · [PDF](https://aclanthology.org/W10-1408.pdf) · [OpenAlex](https://openalex.org/W2132890663)

**Citations:** 47 citations (OpenAlex, 2026-10-14) · [24](https://md.research.elamur.ai/free/en/papers/attia-2010-handling-unknown-words-statistical-latent/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines how a generative constituency parser should tag words it never saw, or saw only rarely, in training. This matters most for morphologically rich languages and small treebanks. The authors train a Berkeley-style latent-variable PCFG (PCFG-LA) on the Penn Arabic Treebank (ATB), the WSJ part of the Penn-II Treebank and the French Treebank. They compare three ways of handling rare words:

- **Generic symbol:** a language-neutral baseline that maps all rare words to one UNKNOWN terminal.
- **Hand-crafted signatures:** rare words are mapped to morphological classes. English uses the Berkeley parser's classes, French uses prior work, and Arabic uses new heuristics based on affixes and morphotactics.
- **Learnt signatures:** affix classes are chosen automatically by ranking candidate affixes with information gain.

Morphological signatures help most for Arabic and least for English. The Arabic gain of over 3 F-points produces state-of-the-art results for that treebank. Affix signatures learnt automatically perform about as well as hand-written ones. This means the one language-specific part of such a parser could in principle be induced from annotated data alone.

## Key points
- **Unknown-word rates:** on the development sets, the share of words absent from training is 5.40% for Arabic, 2.83% for English and 5.98% for French. When training sizes are equalised, Arabic has the worst problem and English the mildest.
- **Simple UNKNOWN model:** treating words seen at most once as unknown works best (dev F: Arabic 79.53, English 89.47, French 83.88). Raising the cutoff to 10 hurts every language.
- **Learnt signature lists:** the information-gain method selected 24 English suffixes, 48 French suffixes and 38 Arabic prefixes and suffixes.
- **Test F-scores** (no signature / hand-crafted / information gain):
  - Arabic: 78.34 / 81.59 / 81.33
  - English: 89.48 / 89.65 / 89.77
  - French: 83.74 / 85.77 / 85.55
- **Comparison and setup:** on the same Arabic data, Bikel's parser scores 76.50 and the Berkeley parser without signatures scores 75.28. All experiments use gold tokenisation and sentences of up to 40 words.

## Interesting ideas
- Rather than pooling every unseen word into one bucket, back off each one to a class named by its visible affixes (e.g. UNK-ed), so the tag guess depends on the word's form.
- The rare-word cutoff is a real trade-off: reserving more probability for unknowns damages statistics for known-but-rare words, so the cutoff should be tuned rather than set high.
- Ranking candidate affixes by information gain for POS prediction gives signature lists nearly as good as expert-written ones, with no linguistic knowledge required.
- Comparing against gold-tag parsing reveals how much of the parsing error comes from tagging mistakes; for French the gap was striking.
- The authors measure unknown-word rates on training sets cut to equal size, which separates true language difficulty from mere treebank size.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The en tagger could map out-of-vocabulary words to affix-signature classes, keeping the guess explainable ("tagged VBD because of -ed").
- **I2** (learn): When training the perceptron parser or tagger, sweep the rare-word threshold on development data instead of hard-coding it.
- **I3** (tokens): Unknown-word classes for the en tagger could be induced and refreshed automatically from treebank data, with the ranked list kept as an inspectable artifact.
- **I4** (tree): Running gparse and the perceptron parser with gold tags as an upper-bound control would show whether tagger fixes or parser fixes are the better investment.
- **I5** (learn): Evaluations across treebanks or UD dialects should control for training size before conclusions are drawn about domain or dialect difficulty.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Dublin City University](https://md.research.elamur.ai/free/en/universities/dublin-city-university/) — University, Dublin, Ireland · 70 works · 9,447 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [French](https://md.research.elamur.ai/free/en/topics/french/) · 70 works

## Cite

Cite as: Mohammed Attia, Jennifer Foster, Deirdre Hogan, Joseph Le Roux, Lamia Tounsi and Josef van Genabith. 2010. Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French. Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{attia2010handling,
  title = {{Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French}},
  author = {Mohammed Attia, Jennifer Foster, Deirdre Hogan, Joseph {Le Roux}, Lamia Tounsi and Josef {van Genabith}},
  booktitle = {Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages},
  year = {2010},
  url = {https://aclanthology.org/W10-1408/}
}
```

- [← previous Generating Entailment Rules from FrameNet](https://md.research.elamur.ai/free/en/papers/aharon-2010-generating-entailment-rules-framenet/)
- [next → SemEval-2010 Task 10: Linking Events and Their Participants in Discourse](https://md.research.elamur.ai/free/en/papers/ruppenhofer-2010-semeval-2010-task-10-linking/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anne Abeillé, Lionel Clément, and François Toussenel, 2003. Treebanks: Building and Using Parsed Corpora, chapter Building a Treebank for French. Kluwer, Dordrecht. [find](https://scholar.google.com/scholar?q=Anne+Abeille%CC%81%2C+Lionel+Cle%CC%81ment%2C+and+Franc%CC%A7ois+Toussenel%2C+2003%2E+Treebanks%3A+Building+and+Using+Parsed+Corpora%2C+chapter+Building+a+Treebank+for+French%2E+Kluwer%2C+Dordrecht%2E)
2. Abhishek Arun and Frank Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: The case of French. In ACL. The Association for Computer Linguistics. [find](https://scholar.google.com/scholar?q=Abhishek+Arun+and+Frank+Keller%2E+2005%2E+Lexicalization+in+crosslinguistic+probabilistic+parsing%3A+The+case+of+French%2E+In+ACL%2E+The+Association+for+Computer+Linguistics%2E)
3. Kenneth R. Beesley and Lauri Karttunen. 2003. Finite State Morphology. CSLI studies in computational linguistics. [find](https://scholar.google.com/scholar?q=Kenneth+R%2E+Beesley+and+Lauri+Karttunen%2E+2003%2E+Finite+State+Morphology%2E+CSLI+studies+in+computational+linguistics%2E)
4. Kenneth R. Beesley. 1998. [Arabic morphology using only finite-state operations](https://md.research.elamur.ai/free/en/papers/beesley-1998-arabic-morphology-only-finite-state/). In The Workshop on Computational Approaches to Semitic Languages.
5. Ann Bies and Mohammed Maamouri. 2003. Penn Arabic Treebank guidelines. Technical Report TB-1-28-03. [find](https://scholar.google.com/scholar?q=Ann+Bies+and+Mohammed+Maamouri%2E+2003%2E+Penn+Arabic+Treebank+guidelines%2E+Technical+Report+TB%2D1%2D28%2D03%2E)
6. Dan Bikel. 2004. On the Parameter Space of Generative Lexicalized Parsing Models. Ph.D. thesis, University of Pennslyvania. [find](https://scholar.google.com/scholar?q=Dan+Bikel%2E+2004%2E+On+the+Parameter+Space+of+Generative+Lexicalized+Parsing+Models%2E+Ph%2ED%2E+thesis%2C+University+of+Pennslyvania%2E)
7. Marie Candito and Benoit Crabbé. 2009. Improving generative statistical parsing with semi-supervised word clustering. In Proceedings of IWPT’09. [find](https://scholar.google.com/scholar?q=Marie+Candito+and+Benoit+Crabbe%CC%81%2E+2009%2E+Improving+generative+statistical+parsing+with+semi%2Dsupervised+word+clustering%2E+In+Proceedings+of+IWPT%E2%80%9909%2E)
8. Marie Candito, Benoı̂t Crabbé, and Djamé Seddah. 2009. On statistical parsing of French with supervised and semi-supervised strategies. In Proceedings of the EACL 2009 Workshop on Computational Linguistic Aspects of Grammatical Inference, pages 49–57, Athens, Greece, March. [find](https://scholar.google.com/scholar?q=Marie+Candito%2C+Beno%C4%B1%CC%82t+Crabbe%CC%81%2C+and+Djame%CC%81+Seddah%2E+2009%2E+On+statistical+parsing+of+French+with+supervised+and+semi%2Dsupervised+strategies%2E+In+Proceedings+of+the+EACL+2009+Workshop+on+Computational+Ling)
9. Eugene Charniak. 2000. [A maximum-entropy-inspired parser](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/). In Proceedings of the Annual Meeting of the North American Association for Computational Linguistics (NAACL-00), pages 132–139, Seattle, Washington.
10. Benoı̂t Crabbé and Marie Candito. 2008. Expériences d’analyse syntaxique statistique du français. In Actes de TALN. [find](https://scholar.google.com/scholar?q=Beno%C4%B1%CC%82t+Crabbe%CC%81+and+Marie+Candito%2E+2008%2E+Expe%CC%81riences+d%E2%80%99analyse+syntaxique+statistique+du+franc%CC%A7ais%2E+In+Actes+de+TALN%2E)
11. Yoav Goldberg, Reut Tsarfaty, Meni Adler, and Michael Elhadad. 2009. Enhancing unlexicalized parsing performance using a wide coverage lexicon, fuzzy tag-set mapping, and EM-HMM-based lexical probabilities. In EACL, pages 327–335. The Association for Computer Linguistics. [find](https://scholar.google.com/scholar?q=Yoav+Goldberg%2C+Reut+Tsarfaty%2C+Meni+Adler%2C+and+Michael+Elhadad%2E+2009%2E+Enhancing+unlexicalized+parsing+performance+using+a+wide+coverage+lexicon%2C+fuzzy+tag%2Dset+mapping%2C+and+EM%2DHMM%2Dbased+lexical+probabil)
12. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. Mada+tokan: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR). [find](https://scholar.google.com/scholar?q=Nizar+Habash%2C+Owen+Rambow%2C+and+Ryan+Roth%2E+2009%2E+Mada%2Btokan%3A+A+toolkit+for+Arabic+tokenization%2C+diacritization%2C+morphological+disambiguation%2C+pos+tagging%2C+stemming+and+lemmatization%2E+In+Proceedings+of+)
13. Nizar Habash. 2008. Four techniques for online handling of out-of-vocabulary words in arabic-english statistical machine translation. In Proceedings of Association for Computational Linguistics, pages 57–60. [find](https://scholar.google.com/scholar?q=Nizar+Habash%2E+2008%2E+Four+techniques+for+online+handling+of+out%2Dof%2Dvocabulary+words+in+arabic%2Denglish+statistical+machine+translation%2E+In+Proceedings+of+Association+for+Computational+Linguistics%2C+pages)
14. Zhongqiang Huang and Mary Harper. 2009. Self-training pcfg grammars with latent annotations across languages. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Singapore, August. [find](https://scholar.google.com/scholar?q=Zhongqiang+Huang+and+Mary+Harper%2E+2009%2E+Self%2Dtraining+pcfg+grammars+with+latent+annotations+across+languages%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
15. Mark Johnson. 1998. [PCFG models of linguistic tree representations](https://md.research.elamur.ai/free/en/papers/johnson-1998-pcfg-models-linguistic-tree-representations/). Computational Linguistics, 24(4):613–632.
16. Dan Klein and Chris Manning. 2003. Accurate unlexicalised parsing. In Proceedings of the 41st Annual Meeting of the ACL. [find](https://scholar.google.com/scholar?q=Dan+Klein+and+Chris+Manning%2E+2003%2E+Accurate+unlexicalised+parsing%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+ACL%2E)
17. Mohammed Maamouri and Ann Bies. 2004. [Developing an Arabic Treebank: Methods, guidelines, procedures, and tools](https://md.research.elamur.ai/free/en/papers/maamouri-2004-developing-arabic-treebank-methods-guidelines/). In Workshop on Computational Approaches to Arabic Script-based Languages, COLING.
18. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. [The Penn Treebank: Annotating predicate argument structure](https://md.research.elamur.ai/free/en/papers/marcus-1994-penn-treebank-annotating-predicate-argument/). In Proceedings of the 1994 ARPA Speech and Natural Language Workshop, pages 114–119, Princeton, New Jersey.
19. Takuya Matsuzaki, Yusuke Miyao, and Jun’ichi Tsujii. 2005. [Probabilistic CFG with latent annotations](https://md.research.elamur.ai/free/en/papers/matsuzaki-2005-probabilistic-cfg-latent-annotations/). In Proceedings of the 43rd Annual Meeting of the ACL, pages 75–82, Ann Arbor, June.
20. Rani Nelken and Stuart M. Shieber. 2005. [Arabic diacritization using weighted finite-state transducers](https://md.research.elamur.ai/free/en/papers/nelken-2005-arabic-diacritization-weighted-finite-state/). In ACL-05 Workshop on Computational Approaches to Semitic Languages.
21. Slav Petrov and Dan Klein. 2007. [Improved inference for unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/petrov-2007-improved-inference-unlexicalized-parsing/). In Proceedings of HLT-NAACL, Rochester, NY, April.
22. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. [Learning accurate, compact and interpretable tree annotation](https://md.research.elamur.ai/free/en/papers/petrov-2006-learning-accurate-compact-interpretable-tree/). In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the ACL, Sydney, Australia, July.
23. Slav Petrov. 2009. Coarse-to-Fine Natural Language Processing. Ph.D. thesis, University of California at Berkeley, Berkeley, CA, USA. [find](https://scholar.google.com/scholar?q=Slav+Petrov%2E+2009%2E+Coarse%2Dto%2DFine+Natural+Language+Processing%2E+Ph%2ED%2E+thesis%2C+University+of+California+at+Berkeley%2C+Berkeley%2C+CA%2C+USA%2E)
24. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Language Processing (NeMLaP-1), pages 44–49. [find](https://scholar.google.com/scholar?q=Helmut+Schmid%2E+1994%2E+Probabilistic+part%2Dof%2Dspeech+tagging+using+decision+trees%2E+In+Proceedings+of+the+International+Conference+on+New+Methods+in+Language+Processing+%28NeMLaP%2D1%29%2C+pages+44%E2%80%9349%2E)
25. Huihsin Tseng, Daniel Jurafsky, and Christopher Manning. 2005. Morphological features help POS tagging of unknown words across language varieties. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing. [find](https://scholar.google.com/scholar?q=Huihsin+Tseng%2C+Daniel+Jurafsky%2C+and+Christopher+Manning%2E+2005%2E+Morphological+features+help+POS+tagging+of+unknown+words+across+language+varieties%2E+In+Proceedings+of+the+Fourth+SIGHAN+Workshop+on+Chine)

*Updated 30 October 2026*