--- title: "Lexicon Development for Varieties of Spoken Colloquial Arabic — Graff 2006" description: "This makes it hard to build speech and lexical resources for DA. The LDC team first tried to annotate Levantine… 5 references, cited by 2 papers here." authors: ["David Graff", "Tim Buckwalter", "Mohamed Maamouri", "Hubert Jin"] year: 2006 venue: "Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)" source: "https://aclanthology.org/L06-1327/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/graff-2006-lexicon-development-varieties-spoken-colloquial/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/graff-2006-lexicon-development-varieties-spoken-colloquial/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / Lexicon Development for Varieties of Spoken Colloquial Arabic # Lexicon Development for Varieties of Spoken Colloquial Arabic **Authors:** [David Graff](https://md.research.elamur.ai/free/en/authors/david-graff/), [Tim Buckwalter](https://md.research.elamur.ai/free/en/authors/tim-buckwalter/), [Mohamed Maamouri](https://md.research.elamur.ai/free/en/authors/mohamed-maamouri/) and Hubert Jin · **Year:** 2006 · **Venue:** Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L06-1327/) · [PDF](https://aclanthology.org/L06-1327.pdf) · [OpenAlex](https://openalex.org/W2758826174) **Citations:** 14 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/graff-2006-lexicon-development-varieties-spoken-colloquial/cited-by/) in our library, of the 140,884 papers whose bibliographies we read ## Summary Arabic-speaking communities are diglossic. Modern Standard Arabic (MSA) is the written and formal language, while the dialects (DA) people actually speak have no agreed spelling or grammar. This makes it hard to build speech and lexical resources for DA. The LDC team first tried to annotate Levantine Arabic telephone transcripts by correcting candidate analyses from the Buckwalter Morphological Analyzer. Because Levantine differs from MSA more than expected, most words needed manual fixes. Working through the text in order also meant frequent forms were annotated many times, and inconsistently. For Iraqi Arabic they redesigned the workflow around four parts: - a two-layer transcription (a normative consonant-skeleton spelling plus a fully vowelled pronunciation spelling), - annotation of morphology, part of speech and English gloss (MPG) done once per distinct word type rather than per occurrence, - custom annotation tools, - a relational database that links tokens, lexicon entries and morphemes. The result is a pilot Iraqi lexicon covering a transcript corpus of nearly 120,000 tokens. Its stored morpheme data can seed a dialect-specific analyzer, which matters for low-resource varieties that lack a written standard. ## Key points - The Buckwalter Morphological Analyzer parses over 99% of MSA words correctly. On Levantine Arabic transcripts (LDC2006T07), however, most forms needed manual repair of segmentation and POS. - The Iraqi Arabic data come from telephone conversations recorded in 2004 by Appen Pty. Ltd. About 13,000 distinct word forms received manual MPG analysis, covering nearly 120,000 word tokens. - Transcription has two layers: - a skeletal (undiacritized) form that converges on consensus spellings, - a diacritized form that records the actual vowels, gemination and consonant changes. - The database has four main tables: "tokes", lex, morph and "segs". Importing MPG analyses remaps every token to new analyzed lex entries, so the old unanalyzed entries become "unattested forms". - The planned dialect analyzer follows the Buckwalter design. It uses lookup tables of prefixes, suffixes and stems, plus string matching that proposes allowed segmentations for unseen words. ## Interesting ideas - Annotate each distinct word type once and project that decision onto every token, which removes repeated work and the drift in how frequent forms get treated. - Keep two parallel spellings for each word: a normalized identity form and a faithful surface form. Each layer then serves its own purpose without one corrupting the other. - Store the corpus relationally, so that spelling merges, typo fixes and revised analyses are single updates that immediately and consistently change the regenerated corpus. - Build a morphological analyzer directly from accumulated manual segmentations (affix tables, stem lists and allowed combinations) and refine it as annotation grows. - Do not assume a closely related resource transfers cleanly: measure how much of a borrowed tool's output needs fixing before relying on it. ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): Silver-data repairs could be stored per type or per pattern in a table and pushed to every matching token, so frequent errors get fixed consistently in one place. - **I2** (tokens): The en crate could keep both the raw surface token and its normalized form (spelling variants, contractions), so lemmas stay stable while the original text remains recoverable for explanations. - **I3** (tree): Treebank conversions and dialect fixes could be kept as database edits over token-to-analysis links, rather than as edited copies of CoNLL-U files, which keeps every change traceable. - **I4** (tokens): The morphology module could induce affix and stem tables from verified analyses and propose explainable segmentations for unseen words, instead of relying on an opaque model. - **I5** (learn): Before adopting an external parser or lexicon as a witness, measure its correction rate on a small sample from the new domain, as a cheap negative check. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) Topics: [🐪 Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) ## Institutions - [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 143 works · 56,645 citations ## Cite Cite as: David Graff, Tim Buckwalter, Mohamed Maamouri and Hubert Jin. 2006. Lexicon Development for Varieties of Spoken Colloquial Arabic. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{graff2006lexicon, title = {{Lexicon Development for Varieties of Spoken Colloquial Arabic}}, author = {David Graff, Tim Buckwalter, Mohamed Maamouri and Hubert Jin}, booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)}, year = {2006}, doi = {10.63317/4ym94r88vhgv}, url = {https://doi.org/10.63317/4ym94r88vhgv} } ``` - [← previous BESTCUT: A Graph Algorithm for Coreference Resolution](https://md.research.elamur.ai/free/en/papers/nicolae-2006-bestcut-graph-algorithm-coreference-resolution/) - [next → Experimental detection of vowel pronunciation variants in Amharic](https://md.research.elamur.ai/free/en/papers/pellegrini-2006-experimental-detection-vowel-pronunciation-variants/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/) ## References 5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Buckwalter, T. (2004): ‘Buckwalter Arabic Morphological Analyzer Version 2.0’, LDC Corpus Catalog No.LDC2004L02. [find](https://scholar.google.com/scholar?q=Buckwalter%2C+T%2E+%282004%29%3A+%E2%80%98Buckwalter+Arabic+Morphological+Analyzer+Version+2%2E0%E2%80%99%2C+LDC+Corpus+Catalog+No%2ELDC2004L02%2E) 2. Ferguson, Charles (1959b): ‘Diglossia’, Word, 15, pp. 325-340. [find](https://scholar.google.com/scholar?q=Ferguson%2C+Charles+%281959b%29%3A+%E2%80%98Diglossia%E2%80%99%2C+Word%2C+15%2C+pp%2E+325%2D340%2E) 3. Maamouri, M., T. Buckwalter, C. Cieri (2004a): “Dialectal Arabic Telephone Speech Corpus: Principles, Tool Design, and Transcription Conventions”, NEMLAR International Conference on Arabic Language Resources and Tools, Cairo; [http://papers.ldc.upenn.edu/NEMLAR2004/Dialectal-Arabic-telephone-speech-corpus.pdf](http://papers.ldc.upenn.edu/NEMLAR2004/Dialectal-Arabic-telephone-speech-corpus.pdf) [↗](http://papers.ldc.upenn.edu/NEMLAR2004/Dialectal-Arabic-telephone-speech-corpus.pdf) 4. Maamouri, M., D. Graff, H. Jin, C. Cieri, T. Buckwalter (2004b): ‘Dialectal Arabic Orthography-based Transcriptions and CTS Levantine Arabic Collection’, EARS RT-04 Workshop; [http://www.ldc.upenn.edu/Projects/EARS/Arabic/EARSMtgFINAL.12142004.doc](http://www.ldc.upenn.edu/Projects/EARS/Arabic/EARSMtgFINAL.12142004.doc) [↗](http://www.ldc.upenn.edu/Projects/EARS/Arabic/EARSMtgFINAL.12142004.doc) 5. Maamouri, Mohamed, Tim Buckwater, Hubert Jin, David Graff (2006): Levantine Arabic Transcripts, LDC Corpus Catalog No.LDC2006T07. [find](https://scholar.google.com/scholar?q=Maamouri%2C+Mohamed%2C+Tim+Buckwater%2C+Hubert+Jin%2C+David+Graff+%282006%29%3A+Levantine+Arabic+Transcripts%2C+LDC+Corpus+Catalog+No%2ELDC2006T07%2E)