---
title: "T-code compression for Arabic computational morphology (2003)"
description: "Arabic builds words by fitting consonantal roots into templates and then adding inflectional affixes. 11 references, cited by 1 paper here."
authors: ["Jim Yaghi", "Mark R. Titchener", "Sane Yagi"]
year: 2003
venue: "Proceedings of the Australasian Language Technology Workshop 2003"
source: "https://aclanthology.org/U03-1010/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yaghi-2003-t-code-compression-arabic-computational/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yaghi-2003-t-code-compression-arabic-computational/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / T-code compression for Arabic computational morphology

# T-code compression for Arabic computational morphology

**Authors:** Jim Yaghi, Mark R. Titchener and [Sane Yagi](https://md.research.elamur.ai/free/en/authors/sane-yagi/) · **Year:** 2003 · **Venue:** [Proceedings of the Australasian Language Technology Workshop 2003](https://md.research.elamur.ai/free/en/venues/alta/) ([ALTA 2003](https://md.research.elamur.ai/free/en/venues/alta/2003/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/U03-1010/) · [PDF](https://aclanthology.org/U03-1010.pdf) · [OpenAlex](https://openalex.org/W2251031308)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/yaghi-2003-t-code-compression-arabic-computational/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Arabic builds words by fitting consonantal roots into templates and then adding inflectional affixes. Because of this, exact-match tools such as spell checkers, search engines and indexers work badly, and a full list of word forms becomes very large. The authors take a generate-then-look-up approach. A rule-driven generator, written as Perl regular expressions, produces every inflected verb form together with its root, template and affix identifiers, about 2 million words in total. The forms are stored in a dictionary indexed by their undiacritised spelling. The dictionary is compressed with T-Codes, a self-synchronising subclass of Huffman-style variable-length codes, and gets a compact per-entry layout that allows fast scanning. The result is an 8 MB dictionary that includes its search information. That is more than 90% smaller than an uncompressed disk B-Tree, and lookup speed stays comparable. A small web CGI uses the dictionary to extract roots and parse words. The authors also propose sending words between devices as coded ID triples. This matters because analysis becomes a cheap, exact lookup into an enumerated and explainable form space.

## Key points
- The Word Data Creator Environment (WCE) is a GUI for editing the MainDictionary Table (root, root class, pattern, variant) and the Templates Table (tense/voice template, transformation rules, affix list).
- The Word Generation Engine (WGE) runs three stages in sequence:
  - Stem Transformer applies ordered regex rules one pass at a time.
  - Affixer builds 17 morphosyntactic forms per stem.
  - Slotter places the radicals into the F/M/L/R position markers.
- Coverage is verbs only, about 2 million generated forms. Each entry stores numeric root, template and affix IDs so the fully diacritised word can be regenerated.
- Each entry starts with a 12-bit field giving the skip distance to the next entry with a different prefix, plus a 4-bit field counting letters shared with the next word. The final dictionary is 8 MB.
- Worked example: yaktubwna = {12,884, 460, 30} takes 18 bits as T-Codes versus 48 bits as raw 16-bit integers. A T-Code finite-state decoder that reads whole bytes is planned to make decoding faster.

## Interesting ideas
- Instead of analysing words by reversing rules, enumerate every surface form from lexicon plus rules, tag each with its generating IDs, and make analysis a lookup that returns all matching parses.
- Represent a word as a small tuple (root, template, affix) and regenerate the surface form on demand, with variable-length codes assigned by corpus frequency of each ID.
- Index entries under a normalised key (here the word with diacritics removed), then filter candidates by whatever extra detail the query actually supplies.
- Apply transformation rules strictly one per pass in a fixed order, feeding each output into the next rule, with a tool to preview the result after any edit.
- Strip candidate affix combinations from longest to shortest, push every stripped variant into a queue, and look all of them up, keeping every successful parse.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tokens): The en crate could precompile a lemma × paradigm form table in which every form carries its lemma and rule ID, so each analysis is an exact, explainable lookup rather than a guess.
- **I2** (back): The text → graph → text compressor could encode tokens as (lemma, inflection-feature) IDs with frequency-based codes and rely on the generator to restore the exact spelling.
- **I3** (tokens): Lexicon lookups could key on case-folded, apostrophe- and hyphen-normalised forms, then rerank by the original spelling, so noisy input still hits the right entries.
- **I4** (learn): Self-induced repair rules could run as an ordered pipeline that logs each intermediate state, which makes it easy to see which rule broke a tree.
- **I5** (tokens): Clitic and contraction splitting ('s, n't, -ly derivations) could generate several segmentation candidates checked against the form table, passing ambiguity on to the parser instead of committing early.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) · 2,908 works · [🧩 Morphology](https://md.research.elamur.ai/free/en/topics/morphology/) · 405 works · [🪶 Efficient models & compression](https://md.research.elamur.ai/free/en/topics/efficient-models/) · 242 works

## Cite

Cite as: Jim Yaghi, Mark R. Titchener and Sane Yagi. 2003. T-code compression for Arabic computational morphology. Proceedings of the Australasian Language Technology Workshop 2003.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yaghi2003t,
  title = {{T-code compression for Arabic computational morphology}},
  author = {Jim Yaghi, Mark R. Titchener and Sane Yagi},
  booktitle = {Proceedings of the Australasian Language Technology Workshop 2003},
  year = {2003},
  url = {https://aclanthology.org/U03-1010/}
}
```

- [← previous Discourse Chunking: A Tool in Dialogue Act Tagging](https://md.research.elamur.ai/free/en/papers/midgley-2003-discourse-chunking-tool-dialogue-act/)
- [next → Developing a Typology of Dialogue Acts: Some Boundary Problems](https://md.research.elamur.ai/free/en/papers/hennoste-2003-developing-typology-dialogue-acts-some/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/)

## References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Evan L Antworth. 1990. PC-KIMMO: a two-level processor for morphological analysis. Occasional Publications in Academic Computing, 16. [find](https://scholar.google.com/scholar?q=Evan+L+Antworth%2E+1990%2E+PC%2DKIMMO%3A+a+two%2Dlevel+processor+for+morphological+analysis%2E+Occasional+Publications+in+Academic+Computing%2C+16%2E)
2. Kenneth R Beesley. 2001. Finite-state morphological analysis and generation of arabic at xerox research: Status and plans in 2001. In ARABIC Language Processing: Status and Prospects. Arabic NLP Workshop at ACL/EACL 2001, July. [find](https://scholar.google.com/scholar?q=Kenneth+R+Beesley%2E+2001%2E+Finite%2Dstate+morphological+analysis+and+generation+of+arabic+at+xerox+research%3A+Status+and+plans+in+2001%2E+In+ARABIC+Language+Processing%3A+Status+and+Prospects%2E+Arabic+NLP+Works)
3. The Unicode Consortium, 2003. The Unicode Standard, Version 4.0, chapter 2, page 29. Addison-Wesley, Reading, MA. ISBN 0-321-18578-1. [find](https://scholar.google.com/scholar?q=The+Unicode+Consortium%2C+2003%2E+The+Unicode+Standard%2C+Version+4%2E0%2C+chapter+2%2C+page+29%2E+Addison%2DWesley%2C+Reading%2C+MA%2E+ISBN+0%2D321%2D18578%2D1%2E)
4. Jeffery E. F. Friedl. 2002. Mastering Regular Expressions. O’Reilly, 2nd edition, July. [find](https://scholar.google.com/scholar?q=Jeffery+E%2E+F%2E+Friedl%2E+2002%2E+Mastering+Regular+Expressions%2E+O%E2%80%99Reilly%2C+2nd+edition%2C+July%2E)
5. Ulrich Gunther. 1998. Robust Source Coding With Generalised T-Codes. Ph.D. thesis, University of Auckland. [find](https://scholar.google.com/scholar?q=Ulrich+Gunther%2E+1998%2E+Robust+Source+Coding+With+Generalised+T%2DCodes%2E+Ph%2ED%2E+thesis%2C+University+of+Auckland%2E)
6. Lauri Karttunen and Kenneth R Beesley. 1992. Two-level rule compiler. Technical Report ISTL-92-2, Xerox, Xerox Palo Alto Research Center, Palo Alto, California. L. Karttunen, J-P. [find](https://scholar.google.com/scholar?q=Lauri+Karttunen+and+Kenneth+R+Beesley%2E+1992%2E+Two%2Dlevel+rule+compiler%2E+Technical+Report+ISTL%2D92%2D2%2C+Xerox%2C+Xerox+Palo+Alto+Research+Center%2C+Palo+Alto%2C+California%2E+L%2E+Karttunen%2C+J%2DP%2E)
7. Chanod, G. Grefenstette, and A. Schiller. 1997. Regular expressions for language engineering. In Natural Language Engineering, pages 238–305. February 5. [find](https://scholar.google.com/scholar?q=Chanod%2C+G%2E+Grefenstette%2C+and+A%2E+Schiller%2E+1997%2E+Regular+expressions+for+language+engineering%2E+In+Natural+Language+Engineering%2C+pages+238%E2%80%93305%2E+February+5%2E)
8. Lauri Karttunen. 1993. Finite-state lexicon compiler. Technical Report ISTL-NLTT-1993-04-02, Xerox, Xerox Palo Alto Research Center, Palo Alto, California, April. [find](https://scholar.google.com/scholar?q=Lauri+Karttunen%2E+1993%2E+Finite%2Dstate+lexicon+compiler%2E+Technical+Report+ISTL%2DNLTT%2D1993%2D04%2D02%2C+Xerox%2C+Xerox+Palo+Alto+Research+Center%2C+Palo+Alto%2C+California%2C+April%2E)
9. Kirubalaratnam Nithyaganesh. 1998. The Talk-Net Project Real-Time Speech Communication Using T-Codes. MSc thesis, University of Auckland. [find](https://scholar.google.com/scholar?q=Kirubalaratnam+Nithyaganesh%2E+1998%2E+The+Talk%2DNet+Project+Real%2DTime+Speech+Communication+Using+T%2DCodes%2E+MSc+thesis%2C+University+of+Auckland%2E)
10. Mark R Titchener. 1984. Digital encoding by means of new t-codes to provide improved data synchronization and message integrity. In Technical Note, IEE Proceedings, volume 131 of 4, pages 51–53, July. [find](https://scholar.google.com/scholar?q=Mark+R+Titchener%2E+1984%2E+Digital+encoding+by+means+of+new+t%2Dcodes+to+provide+improved+data+synchronization+and+message+integrity%2E+In+Technical+Note%2C+IEE+Proceedings%2C+volume+131+of+4%2C+pages+51%E2%80%9353%2C+July%2E)
11. Mark R Titchener. 1997. The synchronization of variable-length codes. IEEE Transactions on Information Theory, 43:683–691, March. [↗](https://doi.org/10.1109/18.556123)

*Updated 1 November 2026*