---
title: "EMILLE, A 67-Million Word Corpus of Indic — Baker 2002"
description: "The paper reports on the interim state of the EMILLE (Enabling Minority Language Engineering) project of the Universities of Lancaster and Sheffield."
authors: ["Paul Baker", "Andrew Hardie", "Tony McEnery", "Hamish Cunningham", "Rob Gaizauskas"]
year: 2002
venue: "Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02)"
source: "https://aclanthology.org/L02-1319/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/baker-2002-emille-67-million-word-corpus/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/baker-2002-emille-67-million-word-corpus/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2002](https://md.research.elamur.ai/free/papers/2002/) / EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation

# EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation

**Authors:** Paul Baker, Andrew Hardie, Tony McEnery, Hamish Cunningham, Rob Gaizauskas · **Year:** 2002 · **Venue:** Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/L02-1319/) · [PDF](https://aclanthology.org/L02-1319.pdf) · [OpenAlex](https://openalex.org/W173060823)

**Citations:** 52 citations (openalex-snapshot · 2026-10-14) · [13](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper reports on the interim state of the EMILLE (Enabling Minority Language Engineering) project of the Universities of Lancaster and Sheffield. The project aims to build a large corpus of South Asian languages and an environment for processing them automatically. The authors first planned written, spoken and parallel corpora for seven languages. Electronic texts in Indic scripts were scarce, and OCR for these scripts was still immature. The data were therefore collected by hand from news websites that used a single downloadable 8-bit ttf font, with permission obtained from the publishers. These fonts use glyph encodings that are incompatible with one another, so a separate conversion table to Unicode had to be written for each source. Markup followed the CES standard. A collaboration with the Indian CIIL extended coverage to 14 languages and added more genres. In parallel, the authors developed the GATE architecture, above all its Unicode support, and designed an EAGLES-based part-of-speech tagset for Urdu. For low-resource languages, this work is a useful example of how to build a corpus when texts are spread across incompatible encodings.

## Key points
- Initial plan: at least 9 million words of written text for each of Bangla, Gujarati, Hindi, Panjabi, Singhalese, Tamil and Urdu. For the five languages with large communities in the UK, they also planned 500 thousand words of spoken language and 200 thousand words of parallel translations from English for each.
- As of April 2002, the EMILLE/CIIL corpus contained about 60.9 million words in 14 languages. The largest parts were Tamil (10.1 million), Hindi (8.8 million) and Gujarati (7.8 million).
- The authors found no Unicode texts in Indian languages on the web. Websites published texts as images, as PDF, through dedicated programs (e.g., Urdu 98), in 8-bit ttf fonts, or in embedded fonts. Only the ttf-font option proved suitable for collection. Font developers effectively ignored the ISCII standard.
- The Urdu tagset was built within the EAGLES framework. Gender, number and case fit this scheme almost unchanged (the oblique case was labelled as dative). Difficulties arose with verbal categories and with language-specific phenomena, such as the "zimmah" problem in borrowed vocabulary.
- The authors later plan to release a POS tagger for Urdu, font conversion programs, and Unicode sentence alignment tools.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/topics/part-of-speech-tagging/)

## Cited by

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 12 not yet.

- [Universal Dependencies for Punjabi](https://md.research.elamur.ai/free/papers/arora-2022-universal-dependencies-punjabi/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{baker2002emille,
  title = {{EMILLE, A 67-Million Word Corpus of Indic Languages: Data Collection, Mark-up and Harmonisation}},
  author = {Baker, Paul and Hardie, Andrew and McEnery, Tony and Cunningham, Hamish and Gaizauskas, Rob},
  booktitle = {Proceedings of the Third International Conference on Language Resources and Evaluation (LREC'02)},
  year = {2002},
  doi = {10.63317/24zr5nmbm3t3},
  url = {https://doi.org/10.63317/24zr5nmbm3t3}
}
```

- [← previous Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms](https://md.research.elamur.ai/free/papers/collins-2002-discriminative-training-methods-hidden-markov/)
- [next → Floresta Sintá(c)tica: A treebank for Portuguese](https://md.research.elamur.ai/free/papers/afonso-2002-floresta-sinta-c-tica-treebank/)
- [↑ back to papers of 2002](https://md.research.elamur.ai/free/papers/2002/)

## References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baker, J. P., Burnard, L., McEnery, A. M. & Wilson, A. (1998), Techniques for the Evaluation of Language Corpora: a report from the front. Proceedings of the First International Conference on Language Resources and Evaluation: Granada. [find](https://scholar.google.com/scholar?q=Baker%2C+J%2E+P%2E%2C+Burnard%2C+L%2E%2C+McEnery%2C+A%2E+M%2E+%26+Wilson%2C+A%2E+%281998%29%2C+Techniques+for+the+Evaluation+of+Language+Corpora%3A+a+report+from+the+front%2E+Proceedings+of+the+First+International+Conference+on+Language)
2. Baker, J.P., Lie. M., McEnery, A.M. and Sebba, M. 2000. Building a Corpus of Spoken Sylheti. Literary and Linguistic Computing, 15:419-431. [find](https://scholar.google.com/scholar?q=Baker%2C+J%2EP%2E%2C+Lie%2E+M%2E%2C+McEnery%2C+A%2EM%2E+and+Sebba%2C+M%2E+2000%2E+Building+a+Corpus+of+Spoken+Sylheti%2E+Literary+and+Linguistic+Computing%2C+15%3A419%2D431%2E)
3. Bureau of Indian Standards, Indian Standard Code for Information Interchange, IS13194, 1991. [find](https://scholar.google.com/scholar?q=Bureau+of+Indian+Standards%2C+Indian+Standard+Code+for+Information+Interchange%2C+IS13194%2C+1991%2E)
4. Crowdy, S. 1995. The BNC spoken corpus. In G. Leech, G. Myers and J. Thomas (eds.), Spoken English on computer: transcription, mark-up and application. Longman:London. [find](https://scholar.google.com/scholar?q=Crowdy%2C+S%2E+1995%2E+The+BNC+spoken+corpus%2E+In+G%2E+Leech%2C+G%2E+Myers+and+J%2E+Thomas+%28eds%2E%29%2C+Spoken+English+on+computer%3A+transcription%2C+mark%2Dup+and+application%2E+Longman%3ALondon%2E)
5. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V. and Wilks, Y. 2000. Experience of using GATE for NLP R&D. In Proceedings of the Workshop on Using Toolsets and Architectures To Build NLP Systems at COLING-2000:Luxembourg. [find](https://scholar.google.com/scholar?q=Cunningham%2C+H%2E%2C+Maynard%2C+D%2E%2C+Bontcheva%2C+K%2E%2C+Tablan%2C+V%2E+and+Wilks%2C+Y%2E+2000%2E+Experience+of+using+GATE+for+NLP+R%26D%2E+In+Proceedings+of+the+Workshop+on+Using+Toolsets+and+Architectures+To+Build+NLP+Systems)
6. Leech, G. and Wilson, A. (1999) Standards for tagsets. In van Halteren, H. (ed.), Syntactic Wordclass Tagging. Kluwer:Dordrecht. [find](https://scholar.google.com/scholar?q=Leech%2C+G%2E+and+Wilson%2C+A%2E+%281999%29+Standards+for+tagsets%2E+In+van+Halteren%2C+H%2E+%28ed%2E%29%2C+Syntactic+Wordclass+Tagging%2E+Kluwer%3ADordrecht%2E)
7. McEnery, A., Baker, J.P. and Burnard, L. 2000. ‘Corpus Resources and Minority Language Engineering’, in M. Gavrilidou, G. Carayannis, S. Markantontou, S. Piperidis and G. Stainhauoer (eds) Proceedings of the Second International Conference on Language Resources and Evaluation:Athens. [find](https://scholar.google.com/scholar?q=McEnery%2C+A%2E%2C+Baker%2C+J%2EP%2E+and+Burnard%2C+L%2E+2000%2E+%E2%80%98Corpus+Resources+and+Minority+Language+Engineering%E2%80%99%2C+in+M%2E+Gavrilidou%2C+G%2E+Carayannis%2C+S%2E+Markantontou%2C+S%2E+Piperidis+and+G%2E+Stainhauoer+%28eds%29+Proceedings)
8. McEnery, A.M. & Ostler, N. 2000. ‘A New Agenda for Corpus Linguistics – Working With All of the World’s Languages’, Literary and Linguistic Computing, 15:401-418. [find](https://scholar.google.com/scholar?q=McEnery%2C+A%2EM%2E+%26+Ostler%2C+N%2E+2000%2E+%E2%80%98A+New+Agenda+for+Corpus+Linguistics+%E2%80%93+Working+With+All+of+the+World%E2%80%99s+Languages%E2%80%99%2C+Literary+and+Linguistic+Computing%2C+15%3A401%2D418%2E)
9. Oksefjell, S. 1999. A Description of the English-Norwegian Parallel Corpus: Compilation and Further Developments. International Journal of Corpus Linguistics, 4:197-219. [find](https://scholar.google.com/scholar?q=Oksefjell%2C+S%2E+1999%2E+A+Description+of+the+English%2DNorwegian+Parallel+Corpus%3A+Compilation+and+Further+Developments%2E+International+Journal+of+Corpus+Linguistics%2C+4%3A197%2D219%2E)
10. Pal, U. and Chaudhuri, B.B. 1995. Computer recognition of printed Bangla script. International Journal of System Science, 26:2107-2123. [find](https://scholar.google.com/scholar?q=Pal%2C+U%2E+and+Chaudhuri%2C+B%2EB%2E+1995%2E+Computer+recognition+of+printed+Bangla+script%2E+International+Journal+of+System+Science%2C+26%3A2107%2D2123%2E)
11. Schmidt, R. L. 1999. Urdu: an essential grammar. Routledge:London. [find](https://scholar.google.com/scholar?q=Schmidt%2C+R%2E+L%2E+1999%2E+Urdu%3A+an+essential+grammar%2E+Routledge%3ALondon%2E)
12. Singh, S., McEnery, A. and Baker, J.P. 2000. Building a Parallel Corpus of English/Panjabi. In J. Veronis (ed.), Parallel Text Processing, Kluwer:Dordrecht. [find](https://scholar.google.com/scholar?q=Singh%2C+S%2E%2C+McEnery%2C+A%2E+and+Baker%2C+J%2EP%2E+2000%2E+Building+a+Parallel+Corpus+of+English%2FPanjabi%2E+In+J%2E+Veronis+%28ed%2E%29%2C+Parallel+Text+Processing%2C+Kluwer%3ADordrecht%2E)