---
title: "Crowd-sourcing evaluation of automatically acquired,"
description: "Borg and Gatt address the problem of grouping morphologically related words in Maltese without… 25 references, cited by 3 papers here."
authors: ["Claudia Borg", "Albert Gatt"]
year: 2014
venue: "Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)"
source: "https://aclanthology.org/L14-1617/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/borg-2014-crowd-sourcing-evaluation-automatically-acquired/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/borg-2014-crowd-sourcing-evaluation-automatically-acquired/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/) / Crowd-sourcing evaluation of automatically acquired, morphologically related word groupings

# Crowd-sourcing evaluation of automatically acquired, morphologically related word groupings

**Authors:** [Claudia Borg](https://md.research.elamur.ai/free/en/authors/claudia-borg/) and [Albert Gatt](https://md.research.elamur.ai/free/en/authors/albert-gatt/) · **Year:** 2014 · **Venue:** [Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2014](https://md.research.elamur.ai/free/en/venues/lrec/2014/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L14-1617/) · [PDF](https://aclanthology.org/L14-1617.pdf) · [OpenAlex](https://openalex.org/W2251744981)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/borg-2014-crowd-sourcing-evaluation-automatically-acquired/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Borg and Gatt address the problem of grouping morphologically related words in Maltese without supervision. Maltese is an under-resourced language whose morphology mixes Semitic root-and-pattern processes with Romance/English stem-and-affix ones, and it has no gold-standard lexicon to evaluate against. Starting from a filtered wordlist from the MLRS corpus, they find likely prefixes and suffixes with trie-based transitional probabilities. They keep every candidate segmentation and build one cluster per candidate stem. Next, LSA similarity is used to compute a "semantic cohesiveness" score, which guides discarding bad clusters and merging good ones. Because no gold standard exists, the output was judged by linguists and by native-speaker non-experts recruited through crowd-sourcing. Judges removed words that did not belong and rated each cluster's quality. Once outlier judgements were dropped, agreement was high in both groups. Most clusters were rated good or very good. The method worked better on concatenative morphology but was still usable on non-concatenative forms. The judged clusters are also a first step toward a Maltese gold standard and a morphological analyser.

## Key points
- Data: the MLRS corpus (125M tokens) was reduced to 67,434 word types. Function words, numbers, proper nouns, foreign words and types seen fewer than 10 times were removed.
- Segmentation takes the top 10% of ranked affixes (200 prefixes, 400 suffixes). Keeping all candidate segmentations yields 21,381 initial, overlapping clusters.
- Agreement was measured with Krippendorff's alpha (R irr library). For crowd judges it rose from 0.598 to 0.897 once 210 outlier evaluations (11%) were removed; experts reached 0.948.
- Clusters rated good or very good: 57% by experts and 74% by the crowd. The Spearman correlation between rating and share of words removed was 0.737 for experts and 0.788 for the crowd (p \< .001).
- Expert subset: 61% of the 80 concatenative clusters were left intact, against 45% of the 73 non-concatenative ones. Clusters losing over a third of their words occurred at similar rates (19% vs 18%).

## Interesting ideas
- Affixes are found by looking for points in a character trie where the path before a boundary is nearly deterministic and branching opens up just after it; this is cheap, needs no labels, and yields a ranked affix inventory.
- Instead of committing to one segmentation early, every candidate stem seeds its own cluster, and a later semantic filter decides which groupings survive.
- Cluster quality is scored as the spread of distributional similarity between the cluster head and its members; a wide spread flags groupings that look alike on the surface but are unrelated in meaning.
- The evaluation task asks judges only to remove words that do not belong, then checks that the share removed correlates with an overall quality rating; this makes the task easy for non-experts and gives a built-in consistency check.
- Experts gave lower ratings than the crowd, plausibly because they also penalised clusters missing related words, so precision and completeness need separate judgement; results were also split by morphological process type.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): Unknown or rare English words in the en crate could be split into a candidate stem plus a suffix from the ranked list, giving the lemmatizer and tagger an explainable fallback guess.
- **I2** (tokens): Ambiguous lemma candidates (homographs, irregular forms) could be kept as competing hypotheses until context-based evidence picks one, rather than being fixed by a single early rule.
- **I3** (world): When compiling verb classes or ontology groupings, this dispersion score could flag suspicious groups for review, with the reason being which members drift away from the head.
- **I4** (learn): LLM teacher tiers or human checkers could audit silver groupings (frames, verb-class memberships) by striking out wrong members, with the removed share serving as a measurable quality signal.
- **I5** (learn): Evaluations should report purity and coverage separately and stratify by phenomenon (e.g., regular vs irregular morphology), so a high crowd-style score cannot hide gaps in completeness.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Malta](https://md.research.elamur.ai/free/en/universities/university-of-malta/) — University, Msida, Malta · 29 works · 894 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works

## Cite

Cite as: Claudia Borg and Albert Gatt. 2014. Crowd-sourcing evaluation of automatically acquired, morphologically related word groupings. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{borg2014crowd,
  title = {{Crowd-sourcing evaluation of automatically acquired, morphologically related word groupings}},
  author = {Claudia Borg and Albert Gatt},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/2hnjmbvpoir4},
  url = {https://doi.org/10.63317/2hnjmbvpoir4}
}
```

- [← previous A Lemma Based Evaluator for Semitic Language Text Summarization Systems](https://md.research.elamur.ai/free/en/papers/el-shishtawy-2014-lemma-based-evaluator-semitic-language/)
- [next → LAF-Fabric: a data analysis tool for Linguistic Annotation Framework with an application to the Hebrew Bible](https://md.research.elamur.ai/free/en/papers/roorda-2014-laf-fabric-data-analysis-tool/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/4/)

## References

25 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Artstein, R. and Poesio, M. (2008). [Inter-Coder Agreement for Computational Linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555—596. [↗](https://doi.org/10.1162/coli.07-034-r2)
2. Baroni, M., Matiasek, J., and Trost, H. (2002). [Unsupervised discovery of morphologically related words based on orthographic and semantic similarity](https://md.research.elamur.ai/free/en/papers/baroni-2002-unsupervised-discovery-morphologically-related-words/). In Proceedings of the ACL-02 workshop on Morphological and phonological learning - Volume 6, MPL ’02, pages 48–57. Association for Computational Linguistics.
3. Brincat, J. M. (2011). Maltese and other Languages. Midsea Books, Malta. [find](https://scholar.google.com/scholar?q=Brincat%2C+J%2E+M%2E+%282011%29%2E+Maltese+and+other+Languages%2E+Midsea+Books%2C+Malta%2E)
4. Camilleri, J. J. (2013). A Computational Grammar and Lexicon for Maltese. Master’s thesis, University of Gothenburg, September. [find](https://scholar.google.com/scholar?q=Camilleri%2C+J%2E+J%2E+%282013%29%2E+A+Computational+Grammar+and+Lexicon+for+Maltese%2E+Master%E2%80%99s+thesis%2C+University+of+Gothenburg%2C+September%2E)
5. Can, B. and Manandhar, S. (2012). Probabilistic hierarchical clustering of morphological paradigms. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 654–663, Avignon, France, April. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Can%2C+B%2E+and+Manandhar%2C+S%2E+%282012%29%2E+Probabilistic+hierarchical+clustering+of+morphological+paradigms%2E+In+Proceedings+of+the+13th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+L)
6. Creutz, M. and Lagus, K. (2002). [Unsupervised discovery of morphemes](https://md.research.elamur.ai/free/en/papers/creutz-2002-unsupervised-discovery-morphemes/). In Proceedings of the ACL-02 workshop on Morphological and phonological learning - Volume 6, MPL ’02, pages 21–30. Association for Computational Linguistics.
7. Creutz, M. and Lagus, K. (2004). Induction of a simple morphology for highly-inflecting languages. In Proceedings of the Seventh Meeting of the ACL Special Interest Group in Computational Phonology, pages 43–51. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Creutz%2C+M%2E+and+Lagus%2C+K%2E+%282004%29%2E+Induction+of+a+simple+morphology+for+highly%2Dinflecting+languages%2E+In+Proceedings+of+the+Seventh+Meeting+of+the+ACL+Special+Interest+Group+in+Computational+Phonology%2C+p)
8. Creutz, M. and Lagus, K. (2005). Inducing the morphological lexicon of a natural language from unannotated text. In Proceedings of AKRR’05, International and Interdisciplinary Conference on Adaptive Knowledge Representation and Reasoning, pages 106–113. [find](https://scholar.google.com/scholar?q=Creutz%2C+M%2E+and+Lagus%2C+K%2E+%282005%29%2E+Inducing+the+morphological+lexicon+of+a+natural+language+from+unannotated+text%2E+In+Proceedings+of+AKRR%E2%80%9905%2C+International+and+Interdisciplinary+Conference+on+Adaptive+K)
9. Dasgupta, S. and Ng, V. (2007). High-performance, language-independent morphological segmentation. In NAACL HLT 2007: Proceedings of the Main Conference, pages 155–163. [find](https://scholar.google.com/scholar?q=Dasgupta%2C+S%2E+and+Ng%2C+V%2E+%282007%29%2E+High%2Dperformance%2C+language%2Dindependent+morphological+segmentation%2E+In+NAACL+HLT+2007%3A+Proceedings+of+the+Main+Conference%2C+pages+155%E2%80%93163%2E)
10. Fabri, R. (2009). Stem allomorphy in the Maltese verb. In Ilsienna - Our Language, volume 1, pages 1–20, Germany. Brockmeyer Verlag. [find](https://scholar.google.com/scholar?q=Fabri%2C+R%2E+%282009%29%2E+Stem+allomorphy+in+the+Maltese+verb%2E+In+Ilsienna+%2D+Our+Language%2C+volume+1%2C+pages+1%E2%80%9320%2C+Germany%2E+Brockmeyer+Verlag%2E)
11. Farrugia, A. (2008). A computational analysis of the Maltese broken plural. Bachelor’s Thesis, University of Malta. [find](https://scholar.google.com/scholar?q=Farrugia%2C+A%2E+%282008%29%2E+A+computational+analysis+of+the+Maltese+broken+plural%2E+Bachelor%E2%80%99s+Thesis%2C+University+of+Malta%2E)
12. Gatt, A. and Čéplö, S. (2013). Digital Corpora and Other Electronic Resources for Maltese. In Proceedings of the Corpus Linguistic Conference, Lancaster, UK. [find](https://scholar.google.com/scholar?q=Gatt%2C+A%2E+and+C%CC%8Ce%CC%81plo%CC%88%2C+S%2E+%282013%29%2E+Digital+Corpora+and+Other+Electronic+Resources+for+Maltese%2E+In+Proceedings+of+the+Corpus+Linguistic+Conference%2C+Lancaster%2C+UK%2E)
13. Goldsmith, J. (2001). Unsupervised learning of the morphology of a natural language. Computational Linguistics, 27:153–198, June. [↗](https://doi.org/10.1162/089120101750300490)
14. Habash, N. and Rambow, O. (2005). [Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop](https://md.research.elamur.ai/free/en/papers/habash-2005-arabic-tokenization-part-speech-tagging/). In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 573–580, Stroudsburg, PA, USA. Association for Computational Linguistics.
15. Jurgens, D. and Stevens, K. (2010). The S-Space Package: An Open Source Package for Word Space Models. In System Papers of the Association of Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jurgens%2C+D%2E+and+Stevens%2C+K%2E+%282010%29%2E+The+S%2DSpace+Package%3A+An+Open+Source+Package+for+Word+Space+Models%2E+In+System+Papers+of+the+Association+of+Computational+Linguistics%2E)
16. Keshava, S. and Pitler, E. (2006). A simpler, intuitive approach to morpheme induction. In PASCAL Challenge Workshop on Unsupervised Segmentation of Words into Morphemes, pages 31–35. [find](https://scholar.google.com/scholar?q=Keshava%2C+S%2E+and+Pitler%2C+E%2E+%282006%29%2E+A+simpler%2C+intuitive+approach+to+morpheme+induction%2E+In+PASCAL+Challenge+Workshop+on+Unsupervised+Segmentation+of+Words+into+Morphemes%2C+pages+31%E2%80%9335%2E)
17. Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. [find](https://scholar.google.com/scholar?q=Krippendorff%2C+K%2E+%282011%29%2E+Computing+Krippendorff%E2%80%99s+Alpha%2DReliability%2E)
18. Kurimo, M., Virpioja, S., Turunen, V., and Lagus, K. (2010). Morpho challenge competition 2005–2010: evaluations and results. In Proceedings of the 11th Meeting of the ACL Special Interest Group on Computational Morphology and Phonology, SIGMORPHON ’10, pages 87–95. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kurimo%2C+M%2E%2C+Virpioja%2C+S%2E%2C+Turunen%2C+V%2E%2C+and+Lagus%2C+K%2E+%282010%29%2E+Morpho+challenge+competition+2005%E2%80%932010%3A+evaluations+and+results%2E+In+Proceedings+of+the+11th+Meeting+of+the+ACL+Special+Interest+Group+on+Co)
19. Roark, B. and Sproat, R. (2007). Computational Approach to Morphology and Syntax. Oxford University Press. [find](https://scholar.google.com/scholar?q=Roark%2C+B%2E+and+Sproat%2C+R%2E+%282007%29%2E+Computational+Approach+to+Morphology+and+Syntax%2E+Oxford+University+Press%2E)
20. Schembri, T. (2006). The Broken Plural in Maltese: An Analysis. Bachelor’s Thesis, University of Malta. [find](https://scholar.google.com/scholar?q=Schembri%2C+T%2E+%282006%29%2E+The+Broken+Plural+in+Maltese%3A+An+Analysis%2E+Bachelor%E2%80%99s+Thesis%2C+University+of+Malta%2E)
21. Schone, P. and Jurafsky, D. (2000). Knowledge-free induction of morphology using latent semantic analysis. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning - Volume 7, ConLL ’00, pages 67–72. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Schone%2C+P%2E+and+Jurafsky%2C+D%2E+%282000%29%2E+Knowledge%2Dfree+induction+of+morphology+using+latent+semantic+analysis%2E+In+Proceedings+of+the+2nd+workshop+on+Learning+language+in+logic+and+the+4th+conference+on+Co)
22. Schone, P. and Jurafsky, D. (2001). [Knowledge-free induction of inflectional morphologies](https://md.research.elamur.ai/free/en/papers/schone-2001-knowledge-free-induction-inflectional-morphologies/). In Proceedings of the second meeting of the North American Chapter of the Association for Computational Linguistics on Language technologies, NAACL ’01, pages 1–9. Association for Computational Linguistics.
23. Snyder, B. and Barzilay, R. (2008). [Unsupervised multilingual learning for morphological segmentation](https://md.research.elamur.ai/free/en/papers/snyder-2008-unsupervised-multilingual-learning-morphological-segmentation/). In Proceedings of ACL-08: HLT, pages 737–745, Columbus, Ohio, June. Association for Computational Linguistics.
24. Spagnol, M. (2011). A tale of two morphologies. Verb structure and argument alternations in Maltese. Ph.D. thesis, University of Konstanz. [find](https://scholar.google.com/scholar?q=Spagnol%2C+M%2E+%282011%29%2E+A+tale+of+two+morphologies%2E+Verb+structure+and+argument+alternations+in+Maltese%2E+Ph%2ED%2E+thesis%2C+University+of+Konstanz%2E)
25. Yarowsky, D. and Wicentowski, R. (2000). Minimally supervised morphological analysis by multimodal alignment. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL ’00, pages 207–216, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yarowsky%2C+D%2E+and+Wicentowski%2C+R%2E+%282000%29%2E+Minimally+supervised+morphological+analysis+by+multimodal+alignment%2E+In+Proceedings+of+the+38th+Annual+Meeting+on+Association+for+Computational+Linguistics%2C+AC)

*Updated 1 November 2026*