--- title: "Building Large Monolingual Dictionaries at — Goldhahn 2012" description: "The paper reports on how the Leipzig Corpora Collection (LCC) is scaling its corpus-based monolingual dictionaries to many more languages, with special…" authors: ["Dirk Goldhahn", "Thomas Eckart", "Uwe Quasthoff"] year: 2012 venue: "Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)" source: "https://aclanthology.org/L12-1154/" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/goldhahn-2012-building-large-monolingual-dictionaries-leipzig/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2012](https://md.research.elamur.ai/free/papers/2012/) / Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages # Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages **Authors:** Dirk Goldhahn, Thomas Eckart, Uwe Quasthoff · **Year:** 2012 · **Venue:** Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/L12-1154/) · [PDF](https://aclanthology.org/L12-1154.pdf) · [OpenAlex](https://openalex.org/W2250600805) **Citations:** 345 citations (openalex-snapshot · 2026-10-14) · [188](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper reports on how the Leipzig Corpora Collection (LCC) is scaling its corpus-based monolingual dictionaries to many more languages, with special attention to "low density" languages that have little text online. Each LCC dictionary lists word-form frequencies (no lemmatization), example sentences, significant co-occurrences and a semantic map of the strongest co-occurrences. At the time of writing, 136 languages were available online. The authors gather text from four kinds of sources: - online newspapers listed by ABYZ News Links, - generic Web crawling with the distributed FindLinks client and an HTTrack-based crawler, - search-engine bootstrapping seeded with frequent words from UDHR and Watchtower texts, - Wikipedia dumps. A largely language-independent pipeline then processes the text: HTML-stripping, language identification with LangSepa, sentence segmentation, cleaning, deduplication, sentence scrambling for copyright reasons, database construction and statistics. Because hundreds of corpora cannot be checked by hand, quality is monitored automatically with statistical features. The resulting resources are free to query and download, which supports typological and comparative research across many languages. ## Key points - About 32,000 ABYZ News Links sources in over 120 languages were crawled. This covered roughly 100 languages and produced more than 250 GB of stripped text, from which dictionaries for 70 languages were built. - The FindLinks volunteer client had downloaded around 1 TB of raw text, with about 10 users fetching over 30 million pages per day. None of this data had yet been turned into corpora. - Wikipedia dumps for more than 200 languages yielded 34 GB of text and about 70 dictionaries. Small Wikipedias were excluded because their stub articles create near-duplicate sentences. - LangSepa identifies languages from stop-words and character uni- and trigram distributions. Using profiles built from LCC, UDHR and Watchtower texts, it covers about 450 languages. - For comparison, Crúbadán covers 1023 languages, but for nearly 60% of them it holds 5 or fewer documents. Web As Corpus queries Bing directly for 34 languages without any cleaning or statistics. ## Interesting ideas - **Parallel texts as seeds for rare languages:** Widely translated texts like the UDHR can supply seed vocabularies and language-ID profiles where nothing else exists. Combining 3–5 frequent words into search queries then retrieves fresh Web pages, a cheap way to start a corpus for an under-resourced language. - **Statistical quality checks instead of manual review:** Corpora can be checked by measuring sentence-length distributions, character and n-gram profiles, rare characters and conformity to Zipf's law. In the paper's example, an odd two-peaked sentence-length curve in the Sundanese Wikipedia exposed boilerplate, so such checks scale to hundreds of corpora. - **Sentence-level scrambling for licensing:** The pipeline breaks up document structure and keeps only individual sentences, relying on German law, which does not protect single sentences. This lets the project redistribute Web-derived data, and it could be a model for sharing corpora when source licenses are restrictive. - **Biasing crawls toward smaller countries:** The crawler deliberately limits URLs from large TLDs like .com or .de, which surfaces sites from smaller countries and possibly rare languages. Using language-speaker data such as Ethnologue to steer crawling toward languages that still lack coverage could help, for example, find more Ukrainian or minority-language content. - **Standardized corpus sizes:** Publishing corpora in fixed sizes makes statistics comparable across languages and sources. This is a useful design choice for any multilingual benchmark or frequency resource. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 188 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 176 not yet. - [Detección de terminologización en sustantivos](https://md.research.elamur.ai/free/papers/ahumada-2025-detecting-terminologisation-nouns/) - [The UD-NewsCrawl Treebank: Reflections and Challenges from a Large-scale Tagalog Syntactic Annotation Project](https://md.research.elamur.ai/free/papers/aquino-2025-ud-newscrawl-treebank-reflections-challenges/) - [An Open Dataset and Model for Language Identification](https://md.research.elamur.ai/free/papers/burchell-2023-open-dataset-model-language-identification/) - [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://md.research.elamur.ai/free/papers/caswell-2020-language-id-wild-unexpected/) - [Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs](https://md.research.elamur.ai/free/papers/dunn-2021-finding-variants-construction-based-dialectometry/) - [GlotLID: Language Identification for Low-Resource Languages](https://md.research.elamur.ai/free/papers/kargaran-2023-glotlid-language-identification-low-resource/) - [Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet](https://md.research.elamur.ai/free/papers/kocmi-2024-findings-wmt24-general-machine-translation/) - [Findings of the WMT25 General Machine Translation Shared Task: Time to Stop Evaluating on Easy Test Sets](https://md.research.elamur.ai/free/papers/kocmi-2025-findings-wmt25-general-machine-translation/) - [Are idioms surprising?](https://md.research.elamur.ai/free/papers/philipp-2023-are-idioms-surprising/) - [PARSEME 2.0 Multilingual Corpus of Multiword Expressions](https://md.research.elamur.ai/free/papers/savary-2026-parseme-2-0-multilingual-corpus/) - [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/papers/sengupta-2023-jais-jais-chat-arabic-centric/) - [Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks](https://md.research.elamur.ai/free/papers/sun-2021-cross-cultural-similarity-features-cross/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{goldhahn2012building, title = {{Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages}}, author = {Goldhahn, Dirk and Eckart, Thomas and Quasthoff, Uwe}, booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)}, year = {2012}, doi = {10.63317/4e5i253k8v5a}, url = {https://doi.org/10.63317/4e5i253k8v5a} } ``` - [← previous Building Large Corpora from the Web Using a New Efficient Tool Chain](https://md.research.elamur.ai/free/papers/schafer-2012-building-large-corpora-web-new/) - [next → Classifying Dialogue Acts in Multi-party Live Chats](https://md.research.elamur.ai/free/papers/kim-2012-classifying-dialogue-acts-multi-party/) - [↑ back to papers of 2012](https://md.research.elamur.ai/free/papers/2012/) ## References 12 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Baroni, M.; Bernardini, S. (2004). BootCaT: Bootstrapping corpora and terms from the web. Proceedings of LREC 2004. [find](https://scholar.google.com/scholar?q=Baroni%2C+M%2E%3B+Bernardini%2C+S%2E+%282004%29%2E+BootCaT%3A+Bootstrapping+corpora+and+terms+from+the+web%2E+Proceedings+of+LREC+2004%2E) [our card](https://md.research.elamur.ai/free/papers/baroni-2004-bootcat-bootstrapping-corpora-terms-web/) 2. Biemann, C.; Heyer, G.; Quasthoff, U.; Richter, M. (2007). The Leipzig Corpora Collection - Monolingual corpora of standard size. Proceedings of Corpus Linguistic 2007, Birmingham, UK. [find](https://scholar.google.com/scholar?q=Biemann%2C+C%2E%3B+Heyer%2C+G%2E%3B+Quasthoff%2C+U%2E%3B+Richter%2C+M%2E+%282007%29%2E+The+Leipzig+Corpora+Collection+%2D+Monolingual+corpora+of+standard+size%2E+Proceedings+of+Corpus+Linguistic+2007%2C+Birmingham%2C+UK%2E) 3. Eckart, T.; Quasthoff, U.; Goldhahn, D. (2012). Language Statistics-Based Quality Assurance for Large Corpora. Proceedings of Asia Pacific Corpus Linguistics Conference 2012, Auckland, New Zealand. [find](https://scholar.google.com/scholar?q=Eckart%2C+T%2E%3B+Quasthoff%2C+U%2E%3B+Goldhahn%2C+D%2E+%282012%29%2E+Language+Statistics%2DBased+Quality+Assurance+for+Large+Corpora%2E+Proceedings+of+Asia+Pacific+Corpus+Linguistics+Conference+2012%2C+Auckland%2C+New+Zealand%2E) 4. Heyer, G.; Quasthoff, U. (2004). Calculating Communities by Link Analysis of URLs. Proceedings of IICS-04, Guadalajara, Mexico and Springer LNCS 3473. [find](https://scholar.google.com/scholar?q=Heyer%2C+G%2E%3B+Quasthoff%2C+U%2E+%282004%29%2E+Calculating+Communities+by+Link+Analysis+of+URLs%2E+Proceedings+of+IICS%2D04%2C+Guadalajara%2C+Mexico+and+Springer+LNCS+3473%2E) 5. Kilgarriff, A.; Grefenstette, G. (2001). Web as Corpus. In Proceedings of Corpus Linguistics 2001 Conference, Lancaster. [find](https://scholar.google.com/scholar?q=Kilgarriff%2C+A%2E%3B+Grefenstette%2C+G%2E+%282001%29%2E+Web+as+Corpus%2E+In+Proceedings+of+Corpus+Linguistics+2001+Conference%2C+Lancaster%2E) 6. Pollmächer, J. (2011). Separierung mit FindLinks gecrawlter Texte nach Sprachen. Bachelor Thesis, University of Leipzig. [find](https://scholar.google.com/scholar?q=Pollm%C3%A4cher%2C+J%2E+%282011%29%2E+Separierung+mit+FindLinks+gecrawlter+Texte+nach+Sprachen%2E+Bachelor+Thesis%2C+University+of+Leipzig%2E) 7. Quasthoff, U. (1998). Projekt der deutsche Wortschatz. Heyer, G., Wolff, Ch. (eds.), Linguistik und neue Medien, Wiesbaden, pp. 93-99. [find](https://scholar.google.com/scholar?q=Quasthoff%2C+U%2E+%281998%29%2E+Projekt+der+deutsche+Wortschatz%2E+Heyer%2C+G%2E%2C+Wolff%2C+Ch%2E+%28eds%2E%29%2C+Linguistik+und+neue+Medien%2C+Wiesbaden%2C+pp%2E+93%2D99%2E) 8. Quasthoff, U.; Richter, M.; Biemann, C. (2006a). Corpus Portal for Search in Monolingual Corpora. Proceedings of LREC-06. [http://www.ethnologue.com](http://www.ethnologue.com) [↗](http://www.ethnologue.com) 9. Quasthoff, U.; Biemann, C. (2006b). Measuring Monolinguality. Proceedings of LREC-06 workshop on Quality assurance and quality measurement for language and speech resources. [find](https://scholar.google.com/scholar?q=Quasthoff%2C+U%2E%3B+Biemann%2C+C%2E+%282006b%29%2E+Measuring+Monolinguality%2E+Proceedings+of+LREC%2D06+workshop+on+Quality+assurance+and+quality+measurement+for+language+and+speech+resources%2E) 10. Richter, M.; Quasthoff, U.; Hallsteinsdóttir, E.; Biemann, C. (2006). Exploiting the Leipzig Corpora Collection. Proceedings of the Information Society Language Technologies Conference (IS-LTC), Ljubljana, Slovenia. [find](https://scholar.google.com/scholar?q=Richter%2C+M%2E%3B+Quasthoff%2C+U%2E%3B+Hallsteinsd%C3%B3ttir%2C+E%2E%3B+Biemann%2C+C%2E+%282006%29%2E+Exploiting+the+Leipzig+Corpora+Collection%2E+Proceedings+of+the+Information+Society+Language+Technologies+Conference+%28IS%2DLTC%29%2C+Ljubl) 11. Scannell, K. (2007). The Crúbadán Project: Corpus building for under-resourced languages. In Proc. WAC-3: Building and Exploring Web Corpora, Louvain-la-Neuve, Belgium. [find](https://scholar.google.com/scholar?q=Scannell%2C+K%2E+%282007%29%2E+The+Cr%C3%BAbad%C3%A1n+Project%3A+Corpus+building+for+under%2Dresourced+languages%2E+In+Proc%2E+WAC%2D3%3A+Building+and+Exploring+Web+Corpora%2C+Louvain%2Dla%2DNeuve%2C+Belgium%2E) 12. Sharoff, S. (2006). Creating general-purpose corpora using automated search engine queries. In M. Baroni and S. Bernardini, editors, WaCky! Working papers on the Web as Corpus. Gedit, Bologna. [find](https://scholar.google.com/scholar?q=Sharoff%2C+S%2E+%282006%29%2E+Creating+general%2Dpurpose+corpora+using+automated+search+engine+queries%2E+In+M%2E+Baroni+and+S%2E+Bernardini%2C+editors%2C+WaCky%21+Working+papers+on+the+Web+as+Corpus%2E+Gedit%2C+Bologna%2E)