--- title: "ParaCrawl: Web-scale parallel corpora for the — Esplà 2019" description: "Machine translation for many European languages is held back by a lack of parallel training data." authors: ["Miquel Esplà", "Mikel Forcada", "Gema Ramírez-Sánchez", "Hieu Hoang"] year: 2019 venue: "Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks" source: "https://aclanthology.org/W19-6721/" licence: "CC BY-ND 4.0 (stated in the PDF)" licence_url: "https://creativecommons.org/licenses/by-nd/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/espla-2019-paracrawl-web-scale-parallel-corpora/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/espla-2019-paracrawl-web-scale-parallel-corpora/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2019](https://md.research.elamur.ai/free/papers/2019/page/3/) / ParaCrawl: Web-scale parallel corpora for the languages of the EU # ParaCrawl: Web-scale parallel corpora for the languages of the EU **Authors:** Miquel Esplà, Mikel Forcada, [Gema Ramírez-Sánchez](https://md.research.elamur.ai/free/authors/gema-ramirez-sanchez/), [Hieu Hoang](https://md.research.elamur.ai/free/authors/hieu-hoang/) · **Year:** 2019 · **Venue:** Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks · **Licence:** [CC BY-ND 4.0 (stated in the PDF)](https://creativecommons.org/licenses/by-nd/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W19-6721/) · [PDF](https://aclanthology.org/W19-6721.pdf) · [OpenAlex](https://openalex.org/W2977458338) **Citations:** 69 citations (openalex-snapshot · 2026-10-14) · [61](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Machine translation for many European languages is held back by a lack of parallel training data. This extended abstract describes ParaCrawl and its successor Paracrawl-2, two Connecting Europe Facility actions that mine the web for translated text in EU languages. The completed first action paired English with each of the other 23 official EU languages. The ongoing second action adds new combinations, including Spanish with Catalan, Basque and Galician, and the two Norwegian written standards. A consortium of five university and industry partners releases the corpora under the CC0 license. It also maintains the open-source Bitextor pipeline, which handles every step from downloading HTML pages to producing a cleaned, deduplicated bitext, and includes the Bicleaner noise filter. Experts checked samples of version 3, and the problems they found were fixed before version 4 was released. Adding version 4 to WMT 2018 baselines raised BLEU for almost every pair tested. The project matters because it supplies both large free datasets and scalable, reusable tools for building more of them. ## Key points - **ParaCrawl version 4** covers English paired with 23 EU languages. Sizes range from about 195K segment pairs (Maltese) to about 31.4M (French), with Spanish at about 22.0M and German at about 16.3M. The data is available on paracrawl.eu and ELRC-share. - **Quality checks:** language experts reviewed 2,000-sentence random samples per pair in version 3. Adding version 4 data to WMT 2018 baselines improved BLEU in 11 of 12 language pairs. - **Bitextor 7** is a five-stage pipeline: 1. crawling 2. preprocessing and normalization 3. document alignment 4. segment alignment 5. filtering, deduplication and formatting It is modular, scales to clusters (SLURM, PBS Pro) and cloud platforms (Azure), and its components can also be used on their own. - **Bicleaner**, the integrated filtering tool, was among the top systems in the WMT 2018 parallel corpus cleaning shared task. - **Planned work** includes: - adding Icelandic, Norwegian, Basque, Catalan/Valencian and Galician - handling non-HTML formats (PDF, DOCX, ODT) - domain identification - mining the Internet Archive - repairing noisy segments rather than discarding them A further action, Paracrawl-3 (2018-EU-IA-0063), has already been funded. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 61 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 54 not yet. - [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://md.research.elamur.ai/free/papers/alves-2024-tower-open-multilingual-large-language/) - [An Open Dataset and Model for Language Identification](https://md.research.elamur.ai/free/papers/burchell-2023-open-dataset-model-language-identification/) - [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://md.research.elamur.ai/free/papers/fomicheva-2022-mlqe-pe-multilingual-quality-estimation/) - [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://md.research.elamur.ai/free/papers/kishky-2020-ccaligned-massive-collection-cross-lingual/) - [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://md.research.elamur.ai/free/papers/kreutzer-2021-quality-glance-audit-web-crawled/) - [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://md.research.elamur.ai/free/papers/kudugunta-2023-madlad-400-multilingual-document-level/) - [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://md.research.elamur.ai/free/papers/sellam-2020-learning-evaluate-translation-beyond-english/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{espla2019paracrawl, title = {{ParaCrawl: Web-scale parallel corpora for the languages of the EU}}, author = {Esplà, Miquel and Forcada, Mikel and Ramírez-Sánchez, Gema and Hoang, Hieu}, booktitle = {Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks}, year = {2019}, url = {https://aclanthology.org/W19-6721/} } ``` - [← previous Opuscula Ugaritico-Accadico-Hebraica: Relative particles, paʿam, and Amraphel](https://md.research.elamur.ai/free/papers/hbo-2019-wikander-opuscula-ugaritico-accadico-hebraica-relative/) - [next → Passage Re-ranking with BERT](https://md.research.elamur.ai/free/papers/nogueira-2019-passage-re-ranking-bert/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/papers/2019/page/3/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Sánchez-Cartagena, Vı́ctor M., Marta Bañón, Sergio Ortiz-Rojas, and Gema Ramı́rez-Sánchez. 2018. Prompsit’s submission to the WMT 2018 parallel corpus filtering shared task. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 968–975, Brussels, Belg… [find](https://scholar.google.com/scholar?q=Sa%CC%81nchez%2DCartagena%2C+V%C4%B1%CC%81ctor+M%2E%2C+Marta+Ban%CC%83o%CC%81n%2C+Sergio+Ortiz%2DRojas%2C+and+Gema+Ram%C4%B1%CC%81rez%2DSa%CC%81nchez%2E+2018%2E+Prompsit%E2%80%99s+submission+to+the+WMT+2018+parallel+corpus+filtering+shared+task%2E+In+Proceedings+of+the+)