---
title: "A New Massive Multilingual Dataset for High-Performance"
description: "Large language models and machine translation systems need huge amounts of text. 31 references, cited by 51 papers here."
authors: ["Ona de Gibert", "Graeme Nail", "Nikolay Arefyev", "Marta Bañón", "Jelmer van der Linde", "Shaoxiong Ji", "Jaume Zaragoza-Bernabeu", "Mikko Aulamo", "Gema Ramírez-Sánchez", "Andrey Kutuzov", "Sampo Pyysalo", "Stephan Oepen", "Jörg Tiedemann"]
year: 2024
venue: "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)"
source: "https://aclanthology.org/2024.lrec-main.100/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / A New Massive Multilingual Dataset for High-Performance Language Technologies

# A New Massive Multilingual Dataset for High-Performance Language Technologies

**Authors:** [Ona de Gibert](https://md.research.elamur.ai/free/en/authors/ona-de-gibert/), Graeme Nail, [Nikolay Arefyev](https://md.research.elamur.ai/free/en/authors/nikolay-arefyev/), [Marta Bañón](https://md.research.elamur.ai/free/en/authors/marta-banon/), Jelmer van der Linde, [Shaoxiong Ji](https://md.research.elamur.ai/free/en/authors/shaoxiong-ji/), [Jaume Zaragoza-Bernabeu](https://md.research.elamur.ai/free/en/authors/jaume-zaragoza-bernabeu/), [Mikko Aulamo](https://md.research.elamur.ai/free/en/authors/mikko-aulamo/), [Gema Ramírez-Sánchez](https://md.research.elamur.ai/free/en/authors/gema-ramirez-sanchez/), [Andrey Kutuzov](https://md.research.elamur.ai/free/en/authors/andrey-kutuzov/), [Sampo Pyysalo](https://md.research.elamur.ai/free/en/authors/sampo-pyysalo/), [Stephan Oepen](https://md.research.elamur.ai/free/en/authors/stephan-oepen/) and [Jörg Tiedemann](https://md.research.elamur.ai/free/en/authors/jorg-tiedemann/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2024](https://md.research.elamur.ai/free/en/venues/lrec/2024/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.lrec-main.100/) · [PDF](https://aclanthology.org/2024.lrec-main.100.pdf) · [OpenAlex](https://openalex.org/W4393117029)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [51](https://md.research.elamur.ai/free/en/papers/gibert-2024-new-massive-multilingual-dataset-high/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models and machine translation systems need huge amounts of text. Most existing corpora reprocess the same web snapshots, especially CommonCrawl. The HPLT project addresses this by building the HPLT language resources. They draw on CommonCrawl and on Internet Archive crawls that had not previously been used at this scale for multilingual corpora. About 1.85 PB of WARC data was processed on HPC infrastructure (Sigma2, CESNET storage, and the EuroHPC cluster LUMI) with open-source pipelines:

- warc2text for text extraction, with CLD2 for document-level language identification
- FastSpell for finer paragraph-level language identification
- Monotextor for monolingual cleaning and deduplication
- Bitextor and Bicleaner AI for document and sentence alignment and pair scoring

The release has three parts. The monolingual part covers 75 languages and roughly 5.6 trillion words after document-level deduplication, with an emphasis on low- and medium-resource languages. An English-centric parallel corpus is derived from it, and English-pivoted synthetic data adds further pairs. The data is CC0-licensed and distributed via the project site and OPUS, together with the software. It is among the largest open text collections available for pre-training and MT.

## Key points
- **Sources:** the crawls are Internet Archive WIDE15, WIDE16 and WIDE17 plus CommonCrawl CC-MAIN-2022-40. Together they yielded 87.4 TB compressed (230.7 TB uncompressed) text, with WIDE16 the largest contributor.
- **monoHPLT:** 75 languages (Ukrainian included), about 5.25 billion documents and 5.64 trillion words. Paragraph-level metadata is attached.
- **biHPLT:** 18 English-centric language pairs, over 96 million sentence pairs, about 1.4 billion English tokens.
- **multiHPLT:** 171 language pairs and 157 million synthetic sentence pairs, obtained by pivoting through English.
- **Released models and tools:**
  - 22 Bitextor MT models covering 9 languages
  - 9 new Bicleaner AI models
  - download and processing scripts on GitHub

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) — University, Edinburgh, United Kingdom · 359 works · 77,352 citations
- [University of Helsinki](https://md.research.elamur.ai/free/en/universities/university-of-helsinki/) — University, Helsinki, Finland · 50 works · 5,318 citations
- [University of Oslo](https://md.research.elamur.ai/free/en/universities/university-of-oslo/) — University, Oslo, Norway · 51 works · 4,155 citations
- [University of Turku](https://md.research.elamur.ai/free/en/universities/university-of-turku/) — University, Turku, Finland · 17 works · 4,890 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 828 works

## Cite

Cite as: Ona de Gibert, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer van der Linde, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen and Jörg Tiedemann. 2024. A New Massive Multilingual Dataset for High-Performance Language Technologies. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gibert2024new,
  title = {{A New Massive Multilingual Dataset for High-Performance Language Technologies}},
  author = {Ona {de Gibert}, Graeme Nail, Nikolay Arefyev, Marta Bañón, Jelmer {van der Linde}, Shaoxiong Ji, Jaume Zaragoza-Bernabeu, Mikko Aulamo, Gema Ramírez-Sánchez, Andrey Kutuzov, Sampo Pyysalo, Stephan Oepen and Jörg Tiedemann},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.100/}
}
```

- [← previous MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://md.research.elamur.ai/free/en/papers/pi-2024-mllm-protector-ensuring-mllm-s/)
- [next → Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. [Towards a cleaner document-oriented multilingual crawled corpus](https://md.research.elamur.ai/free/en/papers/abadji-2022-towards-cleaner-document-oriented-multilingual/). In Thirteenth Language Resources and Evaluation Conference-LREC 2022.
2. Mikko Aulamo, Nikolay Bogoychev, Shaoxiong Ji, Graeme Nail, Gema Ramírez-Sánchez, Jörg Tiedemann, Jelmer van der Linde, and Jaume Zaragoza. 2023. HPLT: High performance language technologies. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 517–… [find](https://scholar.google.com/scholar?q=Mikko+Aulamo%2C+Nikolay+Bogoychev%2C+Shaoxiong+Ji%2C+Graeme+Nail%2C+Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2C+J%C3%B6rg+Tiedemann%2C+Jelmer+van+der+Linde%2C+and+Jaume+Zaragoza%2E+2023%2E+HPLT%3A+High+performance+language+technologies%2E+In+Proce)
3. Marta Bañón, Jaume Zaragoza-Bernabeu, Gema Ramírez-Sánchez, and Sergio Ortiz-Rojas. 2024. Fastspell: the Langid Magic Spell. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. In press. [find](https://scholar.google.com/scholar?q=Marta+Ba%C3%B1%C3%B3n%2C+Jaume+Zaragoza%2DBernabeu%2C+Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2C+and+Sergio+Ortiz%2DRojas%2E+2024%2E+Fastspell%3A+the+Langid+Magic+Spell%2E+In+Proceedings+of+the+2024+Joint+International+Conference+on+Computational+)
4. Marta Bañón, Pinzhen Chen, Barry Haddow, Kenneth Heafield, Hieu Hoang, Miquel Esplà-Gomis, Mikel L Forcada, Amir Kamran, Faheem Kirefu, Philipp Koehn, et al. 2020. [Paracrawl: Web-scale acquisition of parallel corpora](https://md.research.elamur.ai/free/en/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/). In Proceedings of the 58th Annual Meeting of the Association for Computational Lin…
5. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, et al. 2022. Building machine translation systems for the next thousand languages. [find](https://scholar.google.com/scholar?q=Ankur+Bapna%2C+Isaac+Caswell%2C+Julia+Kreutzer%2C+Orhan+Firat%2C+Daan+van+Esch%2C+Aditya+Siddhant%2C+Mengmeng+Niu%2C+Pallavi+Baljekar%2C+Xavier+Garcia%2C+Wolfgang+Macherey%2C+et+al%2E+2022%2E+Building+machine+translation+sys)
6. Andrei Z Broder. 2000. Identifying and filtering near-duplicate documents. In Annual symposium on combinatorial pattern matching, pages 1–10. Springer. [find](https://scholar.google.com/scholar?q=Andrei+Z+Broder%2E+2000%2E+Identifying+and+filtering+near%2Dduplicate+documents%2E+In+Annual+symposium+on+combinatorial+pattern+matching%2C+pages+1%E2%80%9310%2E+Springer%2E)
7. Miquel Esplà-Gomis, Mikel Forcada, Sergio Ortiz-Rojas, and Jorge Ferrández-Tordera. 2016. Bitextor’s participation in WMT’16: shared task on document alignment. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 685–691, Berlin, Germany. Association fo… [find](https://scholar.google.com/scholar?q=Miquel+Espl%C3%A0%2DGomis%2C+Mikel+Forcada%2C+Sergio+Ortiz%2DRojas%2C+and+Jorge+Ferr%C3%A1ndez%2DTordera%2E+2016%2E+Bitextor%E2%80%99s+participation+in+WMT%E2%80%9916%3A+shared+task+on+document+alignment%2E+In+Proceedings+of+the+First+Conference+)
8. Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, et al. 2018. [Marian: Fast neural machine translation in C++](https://md.research.elamur.ai/free/en/papers/junczys-dowmunt-2018-marian-fast-neural-machine-translation/). In Proceedings of ACL 2018, System Demonstrations, pages 116–121.
9. Adam Kilgarriff and Gregory Grefenstette. 2003. [Introduction to the Special Issue on the Web as Corpus](https://md.research.elamur.ai/free/en/papers/kilgarriff-2003-introduction-special-issue-web-as/). Computational Linguistics, 29(3):333– 347.
10. Yoon Kim and Alexander M. Rush. 2016. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). CoRR, abs/1606.07947.
11. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. [Quality at a glance: An audit of web-crawled multilingual datasets](https://md.research.elamur.ai/free/en/papers/kreutzer-2021-quality-glance-audit-web-crawled/). Transactions of the Association for Computational Ling…
12. Taja Kuzman, Peter Rupnik, and Nikola Ljubešić. 2023. Get to know your parallel data: Performing english variety and genre classification over macocu corpora. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial 2023), pages 91–103. [find](https://scholar.google.com/scholar?q=Taja+Kuzman%2C+Peter+Rupnik%2C+and+Nikola+Ljube%C5%A1i%C4%87%2E+2023%2E+Get+to+know+your+parallel+data%3A+Performing+english+variety+and+genre+classification+over+macocu+corpora%2E+In+Tenth+Workshop+on+NLP+for+Similar+Lang)
13. Veronika Laippala, Anna Salmela, Samuel Rönnqvist, Alham Fikri Aji, Li-Hsin Chang, Asma Dhifallah, Larissa Goulart, Henna Kortelainen, Marc Pàmies, Deise Prina Dutra, et al. 2022. Towards better structured and less noisy web data: Oscar with register annotations. In Proceedings of the Eighth Worksho… [find](https://scholar.google.com/scholar?q=Veronika+Laippala%2C+Anna+Salmela%2C+Samuel+R%C3%B6nnqvist%2C+Alham+Fikri+Aji%2C+Li%2DHsin+Chang%2C+Asma+Dhifallah%2C+Larissa+Goulart%2C+Henna+Kortelainen%2C+Marc+P%C3%A0mies%2C+Deise+Prina+Dutra%2C+et+al%2E+2022%2E+Towards+better+struc)
14. Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón, and Sergio Ortiz-Rojas. 2020. [Bifixer and bicleaner: two open-source tools to clean your parallel data](https://md.research.elamur.ai/free/en/papers/sanchez-2020-bifixer-bicleaner-two-open-source/). In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 291– 298, Lisboa, Portugal. Euro…
15. Rico Sennrich and Martin Volk. 2010. Mt-based sentence alignment for ocr-generated parallel texts. In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Research Papers. [find](https://scholar.google.com/scholar?q=Rico+Sennrich+and+Martin+Volk%2E+2010%2E+Mt%2Dbased+sentence+alignment+for+ocr%2Dgenerated+parallel+texts%2E+In+Proceedings+of+the+9th+Conference+of+the+Association+for+Machine+Translation+in+the+Americas%3A+Rese)
16. Ole Tange. 2023. Gnu parallel 20230122 (’bolsonar-istas’). GNU Parallel is a general parallelizer to run multiple serial command line programs in parallel without changing them. [find](https://scholar.google.com/scholar?q=Ole+Tange%2E+2023%2E+Gnu+parallel+20230122+%28%E2%80%99bolsonar%2Distas%E2%80%99%29%2E+GNU+Parallel+is+a+general+parallelizer+to+run+multiple+serial+command+line+programs+in+parallel+without+changing+them%2E)
17. Jaume Zaragoza-Bernabeu, Gema Ramírez-Sánchez, Marta Bañón, and Sergio Ortiz Rojas. 2022. Bicleaner AI: Bicleaner goes neural. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 824– 831, Marseille, France. European Language Resources Association. 10. [find](https://scholar.google.com/scholar?q=Jaume+Zaragoza%2DBernabeu%2C+Gema+Ram%C3%ADrez%2DS%C3%A1nchez%2C+Marta+Ba%C3%B1%C3%B3n%2C+and+Sergio+Ortiz+Rojas%2E+2022%2E+Bicleaner+AI%3A+Bicleaner+goes+neural%2E+In+Proceedings+of+the+Thirteenth+Language+Resources+and+Evaluation+Confer)
18. Language Resource References Adebara, Ife and Elmadany, AbdelRahim and Abdul-Mageed, Muhammad and Alcoba Inciarte, Alcides. 2023. SERENGETI: Massively Multilingual Language Models for Africa. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Language+Resource+References+Adebara%2C+Ife+and+Elmadany%2C+AbdelRahim+and+Abdul%2DMageed%2C+Muhammad+and+Alcoba+Inciarte%2C+Alcides%2E+2023%2E+SERENGETI%3A+Massively+Multilingual+Language+Models+for+Africa%2E+Associat)
19. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
20. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758. [find](https://scholar.google.com/scholar?q=Jesse+Dodge%2C+Maarten+Sap%2C+Ana+Marasovi%C4%87%2C+William+Agnew%2C+Gabriel+Ilharco%2C+Dirk+Groeneveld%2C+Margaret+Mitchell%2C+and+Matt+Gardner%2E+2021%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+cle)
21. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+et+al%2E+2020%2E+The+pile%3A+An+800gb+dataset+of+diverse+text+for+lan)
22. ImaniGooghari, Ayyoob and Lin, Peiqin and Kargaran, Amir Hossein and Severini, Silvia and Jalili Sabet, Masoud and Kassner, Nora and Ma, Chunlan and Schmid, Helmut and Martins, André and Yvon, François and Schütze, Hinrich. 2023. Glot500: Scaling Multilingual Corpora and Language Models to 500 Langu… [find](https://scholar.google.com/scholar?q=ImaniGooghari%2C+Ayyoob+and+Lin%2C+Peiqin+and+Kargaran%2C+Amir+Hossein+and+Severini%2C+Silvia+and+Jalili+Sabet%2C+Masoud+and+Kassner%2C+Nora+and+Ma%2C+Chunlan+and+Schmid%2C+Helmut+and+Martins%2C+Andr%C3%A9+and+Yvon%2C+Fran%C3%A7oi)
23. Christopher A and Lee, Katherine and Xin, Derrick and Kusupati, Aditya and Stella, Romi and Bapna, Ankur and others. 2023. [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/). Google.
24. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Proc… [find](https://scholar.google.com/scholar?q=Hugo+Lauren%C3%A7on%2C+Lucile+Saulnier%2C+Thomas+Wang%2C+Christopher+Akiki%2C+Albert+Villanova+del+Moral%2C+Teven+Le+Scao%2C+Leandro+Von+Werra%2C+Chenghao+Mou%2C+Eduardo+Gonz%C3%A1lez+Ponferrada%2C+Huu+Nguyen%2C+et+al%2E+2022%2E+The+b)
25. Nguyen, Thuat and Van Nguyen, Chien and Lai, Viet Dac and Man, Hieu and Ngo, Nghia Trung and Dernoncourt, Franck and Rossi, Ryan A and Nguyen, Thien Huu. 2023. [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://md.research.elamur.ai/free/en/papers/nguyen-2023-culturax-cleaned-enormous-multilingual-dataset/). The University of Oregon NLP Group.
26. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Édouard Grave, Armand Joulin, and Angela Fan. 2021. [Ccmatrix: Mining billions of high-quality parallel sentences on the web](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Co…
27. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. [find](https://scholar.google.com/scholar?q=Pedro+Javier+Ortiz+Su%C3%A1rez%2C+Beno%C3%AEt+Sagot%2C+and+Laurent+Romary%2E+2019%2E+Asynchronous+pipeline+for+processing+huge+corpora+on+medium+to+low+resource+infrastructures%2E+In+7th+Workshop+on+the+Challenges+in+the)
28. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 2214–2218. [find](https://scholar.google.com/scholar?q=J%C3%B6rg+Tiedemann%2E+2012%2E+Parallel+data%2C+tools+and+interfaces+in+OPUS%2E+In+Proceedings+of+the+Eighth+International+Conference+on+Language+Resources+and+Evaluation+%28LREC%E2%80%9912%29%2C+pages+2214%E2%80%932218%2E)
29. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/).
30. Sha Yuan and Hanyu Zhao and Zhengxiao Du and Ming Ding and Xiao Liu and Yukuo Cen and Xu Zou and Zhilin Yang and Jie Tang. 2021. WuDao-Corpora: A super large-scale Chinese corpora for pre-training language models. Elsevier. [find](https://scholar.google.com/scholar?q=Sha+Yuan+and+Hanyu+Zhao+and+Zhengxiao+Du+and+Ming+Ding+and+Xiao+Liu+and+Yukuo+Cen+and+Xu+Zou+and+Zhilin+Yang+and+Jie+Tang%2E+2021%2E+WuDao%2DCorpora%3A+A+super+large%2Dscale+Chinese+corpora+for+pre%2Dtraining+lan)
31. Kudugunta, Sneha and Caswell, Isaac and Zhang, Biao and Garcia, Xavier and Choquette-Choo, [find](https://scholar.google.com/scholar?q=Kudugunta%2C+Sneha+and+Caswell%2C+Isaac+and+Zhang%2C+Biao+and+Garcia%2C+Xavier+and+Choquette%2DChoo%2C)

*Updated 29 October 2026*