---
title: "How Specialized are Specialized Corpora? Behavioral Evaluation of Corpus — Francom 2010"
description: "The work matters because it offers a cheap, non-corpus validity test for resources that cannot simply be enlarged. 29 references, cited by 1 paper here."
authors: ["Jerid Francom", "Amy LaCross", "Adam Ussishkin"]
year: 2010
venue: "Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)"
source: "https://aclanthology.org/L10-1454/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/francom-2010-how-specialized-are-specialized-corpora/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/francom-2010-how-specialized-are-specialized-corpora/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/) / How Specialized are Specialized Corpora? Behavioral Evaluation of Corpus Representativeness for Maltese.

# How Specialized are Specialized Corpora? Behavioral Evaluation of Corpus Representativeness for Maltese.

**Authors:** Jerid Francom, Amy LaCross and Adam Ussishkin · **Year:** 2010 · **Venue:** Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/L10-1454/) · [PDF](https://aclanthology.org/L10-1454.pdf) · [OpenAlex](https://openalex.org/W189094258)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 140,884 papers whose bibliographies we read

## Summary
Corpora for low-density languages are often small and narrowly sourced, so they count as "specialized." Their builders have no corpus-internal way to tell whether the word distributions are still representative. The authors flip a well-known psycholinguistic result: since frequency predicts behavior, behavior can serve as an external check on corpus frequencies. They test this on the PsyCoL Maltese Lexical Corpus (PMLC), a web-crawled collection seeded from newspapers. In an online experiment, native speakers rated how familiar they found every Semitic-origin verb, and these ratings were compared with the verbs' log frequencies in the corpus. The comparison used three views: individual tokens, frequency bins and verbal binyan classes. Higher-frequency verbs tended to get higher familiarity ratings. The two-bin and three-bin groupings showed significant contrasts, but several expected contrasts did not appear. The authors mainly blame ceiling effects and skew in the ratings. The work matters because it offers a cheap, non-corpus validity test for resources that cannot simply be enlarged.

## Key points
- PMLC has 3,323,325 tokens and roughly 53,000 unique types. It merges two web crawls: PsyCoL (59.8%) and Albert Gatt (40.2%). All seed sources were Maltese online newspapers such as Illum and L-orizzont.
- The rating study covered all 1536 Semitic-origin verbs listed in the Aquilina (2000) Maltese-English dictionary. 107 native speakers rated them on a slider running from "not familiar" to "very familiar."
- In the corpus, binyan classes differed significantly in log frequency, for example Binyan 7 vs Binyan 2 (β = .54, t = 6.0) and Binyan 7 vs Binyan 5. The familiarity ratings showed no significant differences between binyanim, even though both measures ranked the classes in the same order.
- The ratings bunched near the top of the scale and were categorical in shape, which weakened the statistics. The authors propose a 7-point scale for future runs.
- As a corpus-internal sanity check, word length falls as frequency rises, the Zipf-like pattern expected across languages.

## Interesting ideas
- Validate a resource with an independent external signal that is known to track the property you care about, instead of trusting corpus-internal statistics alone.
- Compare grouped bins (high vs low, or three tiers) as well as item-by-item correlations, since a coarse grouping can reveal a trend that noisy per-item data hides.
- Test category-level agreement, such as morphological verb classes, in addition to individual words. Matching class rankings is evidence even when pairwise tests fail.
- Treat a ceiling effect in the measuring instrument as a possible cause of null results before blaming the resource. Change the response scale if needed.
- Use a universal distributional law, such as word length falling with frequency, as a cheap plausibility check on a corpus.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Silver treebanks could be checked against a separate signal, such as LLM-teacher plausibility judgments or human spot ratings. Skewed agreement would flag domain gaps in the silver data.
- **I2** (learn): Evaluation reports could bucket test items by lexical frequency of the head verb. This would show whether parser or frame accuracy collapses on rare verbs.
- **I3** (world): Compiled verb-class knowledge from VerbNet could be checked by comparing class-level corpus frequencies with expectations. This would expose classes that are under-represented in the training texts.
- **I4** (learn): When a negative control or witness-agreement metric saturates, first ask whether the metric can discriminate at all, then conclude the model is fine or broken.
- **I5** (tokens): Running a Zipf-style length/frequency check on tokenizer output for new text sources could reveal tokenization or encoding glitches early.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/)

## Institutions

- [University of Arizona](https://md.research.elamur.ai/free/en/universities/university-of-arizona/) — University, Tucson, United States · 15 works · 9,997 citations
- [Wake Forest University](https://md.research.elamur.ai/free/en/universities/wake-forest-university/) — University, Winston-Salem, United States · 2 works · 789 citations

## Cite

Cite as: Jerid Francom, Amy LaCross and Adam Ussishkin. 2010. How Specialized are Specialized Corpora? Behavioral Evaluation of Corpus Representativeness for Maltese. Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{francom2010how,
  title = {{How Specialized are Specialized Corpora? Behavioral Evaluation of Corpus Representativeness for Maltese.}},
  author = {Jerid Francom, Amy LaCross and Adam Ussishkin},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/4awanv6bvpd6},
  url = {https://doi.org/10.63317/4awanv6bvpd6}
}
```

- [← previous Towards multi-lingual summarization: A comparative analysis of sentence extraction methods on English and Hebrew corpora](https://md.research.elamur.ai/free/en/papers/litvak-2010-multi-lingual-summarization-comparative-analysis/)
- [next → Towards a discourse-pragmatic description of left-dislocation in biblical Hebrew](https://md.research.elamur.ai/free/en/papers/westbury-2010-towards-discourse-pragmatic-description-left/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/page/2/)

## References

29 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J.R. Alameda and F. Cuetos. 1995. Diccionario de frecuencias de las unidades lingüı́sticas del castellano. [find](https://scholar.google.com/scholar?q=J%2ER%2E+Alameda+and+F%2E+Cuetos%2E+1995%2E+Diccionario+de+frecuencias+de+las+unidades+lingu%CC%88%C4%B1%CC%81sticas+del+castellano%2E)
2. C.J. Alvarez, M. Carreiras, and M. Taft. 2001. Syllables and morphemes: Contrasting frequency effects in Spanish. Journal of Experimental Psychology Learning Memory and Cognition, 27(2):545–555. [find](https://scholar.google.com/scholar?q=C%2EJ%2E+Alvarez%2C+M%2E+Carreiras%2C+and+M%2E+Taft%2E+2001%2E+Syllables+and+morphemes%3A+Contrasting+frequency+effects+in+Spanish%2E+Journal+of+Experimental+Psychology+Learning+Memory+and+Cognition%2C+27%282%29%3A545%E2%80%93555%2E)
3. J. Aquilina. 2000. Maltese-English Dictionary. Midsea Books, Santa Venera, Malta. [find](https://scholar.google.com/scholar?q=J%2E+Aquilina%2E+2000%2E+Maltese%2DEnglish+Dictionary%2E+Midsea+Books%2C+Santa+Venera%2C+Malta%2E)
4. R.H. Baayen, R. Piepenbrock, and H. van Rijn. 1993. The CELEX lexical database (CD-ROM). Linguistic Data Consortium, University of Pennsylvania, Philadelphia, PA. [find](https://scholar.google.com/scholar?q=R%2EH%2E+Baayen%2C+R%2E+Piepenbrock%2C+and+H%2E+van+Rijn%2E+1993%2E+The+CELEX+lexical+database+%28CD%2DROM%29%2E+Linguistic+Data+Consortium%2C+University+of+Pennsylvania%2C+Philadelphia%2C+PA%2E)
5. D.A. Balota, M. Pilotti, and M.J. Cortese. 2001. Subjective frequency estimates for 2,938 monosyllabic words. Memory and Cognition, 29(4):639–647. [find](https://scholar.google.com/scholar?q=D%2EA%2E+Balota%2C+M%2E+Pilotti%2C+and+M%2EJ%2E+Cortese%2E+2001%2E+Subjective+frequency+estimates+for+2%2C938+monosyllabic+words%2E+Memory+and+Cognition%2C+29%284%29%3A639%E2%80%93647%2E)
6. E. Bates, S. D Amico, T. Jacobsen, A. Szekely, E. Andonova, A. Devescovi, D. Herron, C.C. Lu, T. Pechmann, C. Pléh, et al. 2003. Timed picture naming in seven languages. Psychonomic Bulletin and Review, 10(2):344– 380. [find](https://scholar.google.com/scholar?q=E%2E+Bates%2C+S%2E+D+Amico%2C+T%2E+Jacobsen%2C+A%2E+Szekely%2C+E%2E+Andonova%2C+A%2E+Devescovi%2C+D%2E+Herron%2C+C%2EC%2E+Lu%2C+T%2E+Pechmann%2C+C%2E+Ple%CC%81h%2C+et+al%2E+2003%2E+Timed+picture+naming+in+seven+languages%2E+Psychonomic+Bulletin+and+Revi)
7. D. Biber and S. Conrad. 2001. Quantitative corpus-based research: Much more than bean counting. TESOL Quarterly, pages 331–336. [find](https://scholar.google.com/scholar?q=D%2E+Biber+and+S%2E+Conrad%2E+2001%2E+Quantitative+corpus%2Dbased+research%3A+Much+more+than+bean+counting%2E+TESOL+Quarterly%2C+pages+331%E2%80%93336%2E)
8. D. Biber. 1993. Representativeness in corpus design. Literary and linguistic computing, 8(4):243–257. [find](https://scholar.google.com/scholar?q=D%2E+Biber%2E+1993%2E+Representativeness+in+corpus+design%2E+Literary+and+linguistic+computing%2C+8%284%29%3A243%E2%80%93257%2E)
9. A.J. Borg and M. Azzopardi-Alexander. 1997. Maltese. Routledge. [find](https://scholar.google.com/scholar?q=A%2EJ%2E+Borg+and+M%2E+Azzopardi%2DAlexander%2E+1997%2E+Maltese%2E+Routledge%2E)
10. L. Borin. 2009. Linguistic diversity in the information society. Proceedings of the SALTMIL 2009 workshop on Information Retrieval and Information Extraction for Less Resourced Languages., pages 1–7. [find](https://scholar.google.com/scholar?q=L%2E+Borin%2E+2009%2E+Linguistic+diversity+in+the+information+society%2E+Proceedings+of+the+SALTMIL+2009+workshop+on+Information+Retrieval+and+Information+Extraction+for+Less+Resourced+Languages%2E%2C+pages+1%E2%80%937%2E)
11. C.M. Connine, J. Mullennix, E. Shernoff, and J. Yelen. 1990. Word familiarity and frequency in visual and auditory word recognition. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(6):1084– 1096. [find](https://scholar.google.com/scholar?q=C%2EM%2E+Connine%2C+J%2E+Mullennix%2C+E%2E+Shernoff%2C+and+J%2E+Yelen%2E+1990%2E+Word+familiarity+and+frequency+in+visual+and+auditory+word+recognition%2E+Journal+of+Experimental+Psychology%3A+Learning%2C+Memory%2C+and+Cognition)
12. W.N. Francis, H. Kučera, and A.W. Mackie. 1982. Frequency analysis of English usage: Lexicon and grammar. Houghton Mifflin Harcourt (HMH). [find](https://scholar.google.com/scholar?q=W%2EN%2E+Francis%2C+H%2E+Kuc%CC%8Cera%2C+and+A%2EW%2E+Mackie%2E+1982%2E+Frequency+analysis+of+English+usage%3A+Lexicon+and+grammar%2E+Houghton+Mifflin+Harcourt+%28HMH%29%2E)
13. J. Francom, A. Ussishkin, and D. Woudstra. 2009. Creating a Web-based Lexical Corpus and Information-extraction Tools for the Semitic Language Maltese. Proceedings of the SALTMIL 2009 workshop on Information Retrieval and Information Extraction for Less Resourced Languages., pages 9–16. [find](https://scholar.google.com/scholar?q=J%2E+Francom%2C+A%2E+Ussishkin%2C+and+D%2E+Woudstra%2E+2009%2E+Creating+a+Web%2Dbased+Lexical+Corpus+and+Information%2Dextraction+Tools+for+the+Semitic+Language+Maltese%2E+Proceedings+of+the+SALTMIL+2009+workshop+on+Info)
14. M.A. Gernsbacher. 1984. Resolving 20 years of inconsistent interactions between lexical familiarity and orthography, concreteness, and polysemy. Journal of Experimental Psychology: General, 113(2):256–281. [find](https://scholar.google.com/scholar?q=M%2EA%2E+Gernsbacher%2E+1984%2E+Resolving+20+years+of+inconsistent+interactions+between+lexical+familiarity+and+orthography%2C+concreteness%2C+and+polysemy%2E+Journal+of+Experimental+Psychology%3A+General%2C+113%282%29%3A256)
15. G. Gilquin and S.T. Gries. 2009. Corpora and experimental methods: A state-of-the-art review. Corpus Linguistics and Linguistic Theory, 5(1):1–26. [find](https://scholar.google.com/scholar?q=G%2E+Gilquin+and+S%2ET%2E+Gries%2E+2009%2E+Corpora+and+experimental+methods%3A+A+state%2Dof%2Dthe%2Dart+review%2E+Corpus+Linguistics+and+Linguistic+Theory%2C+5%281%29%3A1%E2%80%9326%2E)
16. J. Grainger. 1990. Word frequency and neighborhood frequency effects in lexical decision and naming. Journal of Memory and Language, 29(2):228–244. [find](https://scholar.google.com/scholar?q=J%2E+Grainger%2E+1990%2E+Word+frequency+and+neighborhood+frequency+effects+in+lexical+decision+and+naming%2E+Journal+of+Memory+and+Language%2C+29%282%29%3A228%E2%80%93244%2E)
17. S.T. Gries. 2008. Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics, 13(4):403–437. [find](https://scholar.google.com/scholar?q=S%2ET%2E+Gries%2E+2008%2E+Dispersions+and+adjusted+frequencies+in+corpora%2E+International+Journal+of+Corpus+Linguistics%2C+13%284%29%3A403%E2%80%93437%2E)
18. S.T. Gries. 2009. Dispersions and adjusted frequencies in corpora: further explorations. [find](https://scholar.google.com/scholar?q=S%2ET%2E+Gries%2E+2009%2E+Dispersions+and+adjusted+frequencies+in+corpora%3A+further+explorations%2E)
19. W. Li. 1992. Random texts exhibit Zipf’s-law-like word frequency distribution. IEEE Transactions on Information Theory, 38(6):1842–1845. [find](https://scholar.google.com/scholar?q=W%2E+Li%2E+1992%2E+Random+texts+exhibit+Zipf%E2%80%99s%2Dlaw%2Dlike+word+frequency+distribution%2E+IEEE+Transactions+on+Information+Theory%2C+38%286%29%3A1842%E2%80%931845%2E)
20. A. McEnery and R. Xiao. 2007. Parallel and comparable corpora: What are they up to? [find](https://scholar.google.com/scholar?q=A%2E+McEnery+and+R%2E+Xiao%2E+2007%2E+Parallel+and+comparable+corpora%3A+What+are+they+up+to%3F)
21. T. McEnery, R. Xiao, and Y. Tono. 2006. Corpus-based language studies: an advanced resource book. Routledge. [find](https://scholar.google.com/scholar?q=T%2E+McEnery%2C+R%2E+Xiao%2C+and+Y%2E+Tono%2E+2006%2E+Corpus%2Dbased+language+studies%3A+an+advanced+resource+book%2E+Routledge%2E)
22. M. Mifsud. 1995. Loan verbs in Maltese: a descriptive and comparative study. Brill. [find](https://scholar.google.com/scholar?q=M%2E+Mifsud%2E+1995%2E+Loan+verbs+in+Maltese%3A+a+descriptive+and+comparative+study%2E+Brill%2E)
23. F. Moscoso del Prado Martı́n, A. Deutsch, R. Frost, R. Schreuder, N.H. De Jong, and R.H. Baayen. 2005. Changing places: A cross-language perspective on frequency and family size in Dutch and Hebrew. Journal of Memory and Language, 53(4):496–512. [find](https://scholar.google.com/scholar?q=F%2E+Moscoso+del+Prado+Mart%C4%B1%CC%81n%2C+A%2E+Deutsch%2C+R%2E+Frost%2C+R%2E+Schreuder%2C+N%2EH%2E+De+Jong%2C+and+R%2EH%2E+Baayen%2E+2005%2E+Changing+places%3A+A+cross%2Dlanguage+perspective+on+frequency+and+family+size+in+Dutch+and+Hebrew%2E+J)
24. H.C. Nusbaum, D.B. Pisoni, and C.K. Davis. 1984. Sizing up the Hoosier mental lexicon: Measuring the familiarity of 20,000 words. Research on Speech Perception Progress Report, 10:357–376. [find](https://scholar.google.com/scholar?q=H%2EC%2E+Nusbaum%2C+D%2EB%2E+Pisoni%2C+and+C%2EK%2E+Davis%2E+1984%2E+Sizing+up+the+Hoosier+mental+lexicon%3A+Measuring+the+familiarity+of+20%2C000+words%2E+Research+on+Speech+Perception+Progress+Report%2C+10%3A357%E2%80%93376%2E)
25. M. Penke and M. Krause. 2002. German noun plurals: A challenge to the dual-mechanism model. Brain and language, 81(1-3):303–311. [find](https://scholar.google.com/scholar?q=M%2E+Penke+and+M%2E+Krause%2E+2002%2E+German+noun+plurals%3A+A+challenge+to+the+dual%2Dmechanism+model%2E+Brain+and+language%2C+81%281%2D3%29%3A303%E2%80%93311%2E)
26. K. Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, incorporating Cleaneval, page 5. [find](https://scholar.google.com/scholar?q=K%2E+Scannell%2E+2007%2E+The+Cru%CC%81bada%CC%81n+Project%3A+Corpus+building+for+under%2Dresourced+languages%2E+Building+and+Exploring+Web+Corpora%3A+Proceedings+of+the+3rd+Web+as+Corpus+Workshop%2C+incorporating+Cleaneval%2C+pa)
27. J.M. Sinclair. 1987. Looking up: an account of the COBUILD project in lexical computing and the development of the Collins COBUILD English language dictionary. Collins ELT. [find](https://scholar.google.com/scholar?q=J%2EM%2E+Sinclair%2E+1987%2E+Looking+up%3A+an+account+of+the+COBUILD+project+in+lexical+computing+and+the+development+of+the+Collins+COBUILD+English+language+dictionary%2E+Collins+ELT%2E)
28. J. Sprouse. 2009. Revisiting Satiation: Evidence for an equalization response strategy. Linguistic Inquiry, 40(1). A. Ussishkin, J. Berry, A. LaCross, H. Velan, and A. Twist. in progress. Family size in Hebrew auditory word recognition. Ms., University of Arizona and Hebrew University of Jerusalem. [find](https://scholar.google.com/scholar?q=J%2E+Sprouse%2E+2009%2E+Revisiting+Satiation%3A+Evidence+for+an+equalization+response+strategy%2E+Linguistic+Inquiry%2C+40%281%29%2E+A%2E+Ussishkin%2C+J%2E+Berry%2C+A%2E+LaCross%2C+H%2E+Velan%2C+and+A%2E+Twist%2E+in+progress%2E+Family+size+)
29. G.K. Zipf. 1949. Human Behavior and the Principle of Least Effort. [find](https://scholar.google.com/scholar?q=G%2EK%2E+Zipf%2E+1949%2E+Human+Behavior+and+the+Principle+of+Least+Effort%2E)