---
title: "Filtering and Mining Parallel Data in a Joint Multilingual Space"
description: "Schwenk proposes one method for both cleaning existing bitexts and mining new ones. 32 references, cited by 71 papers here."
authors: ["Holger Schwenk"]
year: 2018
venue: "Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/P18-2037/"
doi: "10.18653/v1/p18-2037"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schwenk-2018-filtering-mining-parallel-data-joint/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schwenk-2018-filtering-mining-parallel-data-joint/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/) / Filtering and Mining Parallel Data in a Joint Multilingual Space

# Filtering and Mining Parallel Data in a Joint Multilingual Space

**Authors:** [Holger Schwenk](https://md.research.elamur.ai/free/en/authors/holger-schwenk/) · **Year:** 2018 · **Venue:** [Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2018](https://md.research.elamur.ai/free/en/venues/acl/2018/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P18-2037/) · [PDF](https://aclanthology.org/P18-2037.pdf) · [DOI 10.18653/v1/p18-2037](https://doi.org/10.18653/v1/p18-2037) · [OpenAlex](https://openalex.org/W2798389157)

**Citations:** 96 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/schwenk-2018-filtering-mining-parallel-data-joint/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Neural MT systems are sensitive to noisy training bitexts, and much useful parallel text sits unaligned in comparable corpora such as news. Schwenk proposes one method for both cleaning existing bitexts and mining new ones. A single shared encoder (a 3-layer BLSTM) is trained on nine Europarl languages without any language indicator. It maps sentences from any of these languages into a common 1024-dimensional space. The cosine distance between a sentence pair in that space is then thresholded: per pair to filter a corpus, or over all source-target combinations to mine one. No extra classifiers or hand-crafted features are needed.

On the BUCC shared task, the approach reaches F1 scores in the low-to-mid 70s across four language pairs. It trails only the heavily tuned VIC system. Filtering Common Crawl improves a competitive WMT'14 En→De fairseq-py baseline by 0.3 BLEU, and mined news text helps a system trained only on human-translated data.

The approach matters because it is generic: it works for any pair within the joint space, needs only a threshold tuned per task, and does not depend on the MT architecture.

## Key points
- **Encoder:** a shared BLSTM trained on about 2M Europarl sentences per language for nine languages, with a 20k joint BPE vocabulary and max-pooled sentence vectors. The ru/zh experiments used earlier embeddings trained on the UN corpus.
- **BUCC test F1:**
  - en-fr 75.8
  - en-de 76.9
  - en-ru 73.8
  - en-zh 71.6

  These clearly beat the RALI, NLP2CT and H2 systems (except roughly matching H2 on en-fr) but stay below VIC.
- **Filtering Common Crawl:**
  - Pre-processing removed about 19% of the corpus: sentences with more than 3 commas, sentences of 50+ words, and pairs failing language ID (about 6% had the wrong language).
  - Distance filtering at threshold 1.0 then raised BLEU from 24.75 to 25.06, a gain of 0.3 BLEU while discarding 25% of the training data.
- **Mining WMT news:** the search compared 144M English and 187M German sentences, retrieving k=20 nearest neighbours per source sentence with FAISS.
  - Mined data alone gave 4.18–7.77 BLEU.
  - Adding it to Europarl+News Commentary gave +0.45 BLEU (21.87→22.32).
  - Adding it to the filtered-Common-Crawl system gave no gain.
- **Sentence length:** mined sentences average about 8 words, compared with 24 in the WMT'14 training data. Erroneous alignments tend to occur on sentences dominated by named entities or numbers.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 467 works

## Cite

Cite as: Holger Schwenk. 2018. Filtering and Mining Parallel Data in a Joint Multilingual Space. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schwenk2018filtering,
  title = {{Filtering and Mining Parallel Data in a Joint Multilingual Space}},
  author = {Holger Schwenk},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2037},
  url = {https://doi.org/10.18653/v1/p18-2037}
}
```

- [← previous Epitran: Precision G2P for Many Languages](https://md.research.elamur.ai/free/en/papers/mortensen-2018-epitran-precision-g2p-many-languages/)
- [next → DeepProbLog: Neural Probabilistic Logic Programming](https://md.research.elamur.ai/free/en/papers/manhaeve-2018-deepproblog-neural-probabilistic-logic/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/3/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. [Billion-scale similarity search with gpus](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). arXiv preprint arXiv:1702.08734.
2. Ashish Vaswani et al. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In NIPS.
3. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. [Domain adaptation via pseudo in-domain data selection](https://md.research.elamur.ai/free/en/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/). In EMNLP, pages 355–362.
4. Melvin Johnson et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. In [https://arxiv.org/](https://arxiv.org/) abs/1611.04558. [↗](https://arxiv.org/)
5. Andoni Azpeitia, Thierry Etchegoyhen, and Eva Martı́nez Garcia. 2018. Extracting parallel sentences from comparable corpora with STACC variants. In BUCC. [find](https://scholar.google.com/scholar?q=Andoni+Azpeitia%2C+Thierry+Etchegoyhen%2C+and+Eva+Mart%C4%B1%CC%81nez+Garcia%2E+2018%2E+Extracting+parallel+sentences+from+comparable+corpora+with+STACC+variants%2E+In+BUCC%2E)
6. Chongman Leong, Derek F. Wong, and Lidia S. Chao. 2018. Um-paligner: Neural network-based parallel sentence identification model. In BUCC. [find](https://scholar.google.com/scholar?q=Chongman+Leong%2C+Derek+F%2E+Wong%2C+and+Lidia+S%2E+Chao%2E+2018%2E+Um%2Dpaligner%3A+Neural+network%2Dbased+parallel+sentence+identification+model%2E+In+BUCC%2E)
7. Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and artificial noise both break neural machine translation. In https://arxiv.org/abs/1711. 02173. [↗](https://arxiv.org/abs/1711)
8. Minh-Thang Luong, Hieu Pham, and Christophe D. Manning. 2015. [Effective approaches to attention-based neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-effective-approaches-attention-based-neural/). In EMNLP, pages 1412–1421.
9. Houda Bouamor and Hassan Sajjad. 2018. H2@bucc18: Parallel sentence extraction from comparable coprora using multlingual sentence embeddings. In BUCC. [find](https://scholar.google.com/scholar?q=Houda+Bouamor+and+Hassan+Sajjad%2E+2018%2E+H2%40bucc18%3A+Parallel+sentence+extraction+from+comparable+coprora+using+multlingual+sentence+embeddings%2E+In+BUCC%2E)
10. Aditua Mogadala and Achim Rettinger. 2016. Bilingual word embeddings from parallel and non-parallel corpora for cross-language classification. In NAACL, pages 692–702. [find](https://scholar.google.com/scholar?q=Aditua+Mogadala+and+Achim+Rettinger%2E+2016%2E+Bilingual+word+embeddings+from+parallel+and+non%2Dparallel+corpora+for+cross%2Dlanguage+classification%2E+In+NAACL%2C+pages+692%E2%80%93702%2E)
11. Sarath Chandar, Mitesh M. Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2013. Multilingual deep learning. In NIPS DL wshop. [find](https://scholar.google.com/scholar?q=Sarath+Chandar%2C+Mitesh+M%2E+Khapra%2C+Balaraman+Ravindran%2C+Vikas+Raykar%2C+and+Amrita+Saha%2E+2013%2E+Multilingual+deep+learning%2E+In+NIPS+DL+wshop%2E)
12. Dragos Stefan Munteanu and Daniel Marcu. 2005. [Improving machine translation performance by exploiting non-parallel corpora](https://md.research.elamur.ai/free/en/papers/munteanu-2005-improving-machine-translation-performance-exploiting/). Computational Linguistics, 31(4):477–504.
13. Junyoung Chunga, Kyunghyun Choa, and Yoshua Bengio. 2016. [A character-level decoder without explicit segmentation for neural machine translation](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/). In https://arxiv.org/abs/1603.06147. [↗](https://arxiv.org/abs/1603.06147)
14. Hieu Pham, Minh-Thang Luong, and Christopher D. Manning. 2015. Learning distributed representations for multilingual text sequences. In Workshop on Vector Space Modeling for NLP. [find](https://scholar.google.com/scholar?q=Hieu+Pham%2C+Minh%2DThang+Luong%2C+and+Christopher+D%2E+Manning%2E+2015%2E+Learning+distributed+representations+for+multilingual+text+sequences%2E+In+Workshop+on+Vector+Space+Modeling+for+NLP%2E)
15. Cristina España-Bonet, Ádám Csaba Varga, Alberto Barrón-Cedeño, and Josef van Genabith. 2017. An empirical analysis of nmt-derived interlingual embeddings and their use in parallel sentence identification. IEEE Journal of Selected Topics in Sigmal Processing, pages 1340–1348. [find](https://scholar.google.com/scholar?q=Cristina+Espan%CC%83a%2DBonet%2C+A%CC%81da%CC%81m+Csaba+Varga%2C+Alberto+Barro%CC%81n%2DCeden%CC%83o%2C+and+Josef+van+Genabith%2E+2017%2E+An+empirical+analysis+of+nmt%2Dderived+interlingual+embeddings+and+their+use+in+parallel+sentence+ident)
16. Sadaf Abdul Rauf and Holger Schwenk. 2009. On the use of comparable corpora to improve SMT performance. In EACL, pages 16–23. [find](https://scholar.google.com/scholar?q=Sadaf+Abdul+Rauf+and+Holger+Schwenk%2E+2009%2E+On+the+use+of+comparable+corpora+to+improve+SMT+performance%2E+In+EACL%2C+pages+16%E2%80%9323%2E)
17. Philip Resnik and Noah A. Smith. 2003. [The web as a parallel corpus](https://md.research.elamur.ai/free/en/papers/resnik-2003-web-as-parallel-corpus/). Computational Linguistics, 29:349– 380.
18. Thierry Etchegoyhen and Andoni Azpeitia. 2016. Set-theoretic alignment of comparable corpora. In ACL. [find](https://scholar.google.com/scholar?q=Thierry+Etchegoyhen+and+Andoni+Azpeitia%2E+2016%2E+Set%2Dtheoretic+alignment+of+comparable+corpora%2E+In+ACL%2E)
19. Lucı́a Santamarı́a and Amittai Axelrod. 2017. Data selection with cluster-based language difference models and cynical selection. In IWSLT, pages 137–145. [find](https://scholar.google.com/scholar?q=Luc%C4%B1%CC%81a+Santamar%C4%B1%CC%81a+and+Amittai+Axelrod%2E+2017%2E+Data+selection+with+cluster%2Dbased+language+difference+models+and+cynical+selection%2E+In+IWSLT%2C+pages+137%E2%80%93145%2E)
20. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. [Convolutional Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/). In ICML.
21. Holger Schwenk and Matthijs Douze. 2017. [Learning joint multilingual sentence representations with neural machine translation](https://md.research.elamur.ai/free/en/papers/schwenk-2017-learning-joint-multilingual-sentence-representations/). In ACL workshop on Representation Learning for NLP.
22. Francis Grégoire and Philippe Langlais. 2017. [find](https://scholar.google.com/scholar?q=Francis+Gre%CC%81goire+and+Philippe+Langlais%2E+2017%2E)
23. Bucc 2017 shared task: a first attempt toward a deep learning framework for identifying parallel sentences in comparable corpora. In BUCC, pages 46–50. [find](https://scholar.google.com/scholar?q=Bucc+2017+shared+task%3A+a+first+attempt+toward+a+deep+learning+framework+for+identifying+parallel+sentences+in+comparable+corpora%2E+In+BUCC%2C+pages+46%E2%80%9350%2E)
24. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In ACL, pages 86–96. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2016a%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+In+ACL%2C+pages+86%E2%80%9396%2E)
25. [Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh…](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
26. Masao Utiyama and Hitoshi Isahara. 2003. Reliable measures for aligning Japanese-English news articles and sentences. In ACL, pages 72–79. [find](https://scholar.google.com/scholar?q=Masao+Utiyama+and+Hitoshi+Isahara%2E+2003%2E+Reliable+measures+for+aligning+Japanese%2DEnglish+news+articles+and+sentences%2E+In+ACL%2C+pages+72%E2%80%9379%2E)
27. Karl Moritz Hermann and Phil Blunsom. 2014. [Multilingual models for compositional distributed semantics](https://md.research.elamur.ai/free/en/papers/hermann-2014-multilingual-models-compositional-distributed-semantics/). In ACL, pages 58–68.
28. Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. [Dynamic data selection for neural machine translation](https://md.research.elamur.ai/free/en/papers/wees-2017-dynamic-data-selection-neural-machine/). In EMNLP, pages 1400–1410.
29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In ACL, pages 1715–1725.
30. Yonghui Wu et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. In https://arxiv.org/abs/1610.05011. [↗](https://arxiv.org/abs/1610.05011)
31. Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016. Cross-lingual sentiment classification with bilingual document representation learning. In ACL. [find](https://scholar.google.com/scholar?q=Xinjie+Zhou%2C+Xiaojun+Wan%2C+and+Jianguo+Xiao%2E+2016%2E+Cross%2Dlingual+sentiment+classification+with+bilingual+document+representation+learning%2E+In+ACL%2E)
32. Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spottign parallel sentences in comparable corpora. In BUCC, pages 60–67. [find](https://scholar.google.com/scholar?q=Pierre+Zweigenbaum%2C+Serge+Sharoff%2C+and+Reinhard+Rapp%2E+2018%2E+Overview+of+the+third+bucc+shared+task%3A+Spottign+parallel+sentences+in+comparable+corpora%2E+In+BUCC%2C+pages+60%E2%80%9367%2E)

*Updated 24 October 2026*