--- title: "Effective Parallel Corpus Mining using Bilingual — Guo 2018" description: "Neural machine translation needs large amounts of clean parallel text." authors: ["Mandy Guo", "Qinlan Shen", "Yinfei Yang", "Heming Ge", "Daniel Cer", "Gustavo Hernandez Abrego", "Keith Stevens", "Noah Constant", "Yun-Hsuan Sung", "Brian Strope", "Ray Kurzweil"] year: 2018 venue: "Proceedings of the Third Conference on Machine Translation: Research Papers" source: "https://aclanthology.org/W18-6317/" doi: "10.18653/v1/w18-6317" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/guo-2018-effective-parallel-corpus-mining-bilingual/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/guo-2018-effective-parallel-corpus-mining-bilingual/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2018](https://md.research.elamur.ai/free/papers/2018/page/2/) / Effective Parallel Corpus Mining using Bilingual Sentence Embeddings # Effective Parallel Corpus Mining using Bilingual Sentence Embeddings **Authors:** [Mandy Guo](https://md.research.elamur.ai/free/authors/mandy-guo/), Qinlan Shen, [Yinfei Yang](https://md.research.elamur.ai/free/authors/yinfei-yang/), [Heming Ge](https://md.research.elamur.ai/free/authors/heming-ge/), [Daniel Cer](https://md.research.elamur.ai/free/authors/daniel-cer/), [Gustavo Hernandez Abrego](https://md.research.elamur.ai/free/authors/gustavo-hernandez-abrego/), [Keith Stevens](https://md.research.elamur.ai/free/authors/keith-stevens/), [Noah Constant](https://md.research.elamur.ai/free/authors/noah-constant/), [Yun-Hsuan Sung](https://md.research.elamur.ai/free/authors/yun-hsuan-sung/), [Brian Strope](https://md.research.elamur.ai/free/authors/brian-strope/), [Ray Kurzweil](https://md.research.elamur.ai/free/authors/ray-kurzweil/) · **Year:** 2018 · **Venue:** Proceedings of the Third Conference on Machine Translation: Research Papers · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W18-6317/) · [PDF](https://aclanthology.org/W18-6317.pdf) · [DOI 10.18653/v1/w18-6317](https://doi.org/10.18653/v1/w18-6317) · [OpenAlex](https://openalex.org/W2886198413) **Citations:** 104 citations (openalex-snapshot · 2026-10-14) · [62](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Neural machine translation needs large amounts of clean parallel text. Earlier mining pipelines were heavy, distributed and hand-engineered. The authors, from Google AI, propose a lightweight alternative based on bilingual sentence embeddings. Each language has its own Deep Averaging Network (DAN) encoder, which averages learned word and bigram embeddings and passes them through a feedforward network with residual connections. A candidate pair is scored by the dot product of its two vectors, so large-scale nearest-neighbour search becomes practical. Training treats the other pairs in a mini-batch as negatives. The key addition is "hard" negatives: sentences that a baseline model ranks highly but that are not real translations. These force the model to separate true translations from sentences that are only on the same topic. Experiments on en-fr and en-es show that hard negatives consistently beat random ones. The method reconstructs much of the United Nations Parallel Corpus, and when adapted to documents it matches a far costlier system. NMT models trained on the mined data come close to models trained on the gold data. This makes cheap, scalable bitext mining and filtering a practical option. ## Key points - Sentence-level reconstruction of the UN Parallel Corpus reaches P@1 of 48.9% for en-fr and 54.9% for en-es. Hard negatives were computed for only 20% of the training data to save compute. - At the document level, matching accuracy is comparable to the much heavier approach of Uszkoreit et al. (2010). - NMT systems trained on the reconstructed pairs score within 1–2 BLEU of systems trained on the original corpus. - On ParaCrawl, the model's scores correlate only modestly with Zipporah (Pearson's r ≈ 0.4). Adding the model-filtered ParaCrawl data to WMT data gave the best BLEU, about 0.5 above alternatives. - Translation professionals rated more of the model's selected pairs as GOOD than Zipporah's selections: 76.0% vs 72.0% for en-fr and 74.5% vs 74.0% for en-es. Common errors were partial translations and pairs with identical text on both sides. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/shelves/machine-translation/) Topics: [Machine translation](https://md.research.elamur.ai/free/topics/machine-translation/) ## Cited by Cited by 62 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 55 not yet. - [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://md.research.elamur.ai/free/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/) - [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://md.research.elamur.ai/free/papers/artetxe-2019-margin-based-parallel-corpus-mining/) - [ParaCrawl: Web-Scale Acquisition of Parallel Corpora](https://md.research.elamur.ai/free/papers/banon-2020-paracrawl-web-scale-acquisition-parallel/) - [Language-agnostic BERT Sentence Embedding](https://md.research.elamur.ai/free/papers/feng-2020-language-agnostic-bert-sentence-embedding/) - [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://md.research.elamur.ai/free/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/) - [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://md.research.elamur.ai/free/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/) - [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://md.research.elamur.ai/free/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{guo2018effective, title = {{Effective Parallel Corpus Mining using Bilingual Sentence Embeddings}}, author = {Guo, Mandy and Shen, Qinlan and Yang, Yinfei and Ge, Heming and Cer, Daniel and Abrego, Gustavo Hernandez and Stevens, Keith and Constant, Noah and Sung, Yun-Hsuan and Strope, Brian and Kurzweil, Ray}, booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers}, year = {2018}, doi = {10.18653/v1/w18-6317}, url = {https://doi.org/10.18653/v1/w18-6317} } ``` - [← previous Edition 1.1 of the PARSEME Shared Task on Automatic Identification of Verbal Multiword Expressions](https://md.research.elamur.ai/free/papers/ramisch-2018-edition-1-1-parseme-shared/) - [next → Enhanced Word Representations for Bridging Anaphora Resolution](https://md.research.elamur.ai/free/papers/hou-2018-enhanced-word-representations-bridging-anaphora/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/papers/2018/page/2/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceed… [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Daniel+Cer%2C+Mona+Diab%2C+and+Aitor+Gonzalez%2DAgirre%2E+2012%2E+Semeval%2D2012+task+6%3A+A+pilot+on+semantic+textual+similarity%2E+In+%2ASEM+2012%3A+The+First+Joint+Conference+on+Lexical+and+Computational) [our card](https://md.research.elamur.ai/free/papers/agirre-2012-semeval-2012-task-6-pilot/) 2. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136–144. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexandra+Antonova+and+Alexey+Misyurev%2E+2011%2E+Building+a+web%2Dbased+parallel+corpus+and+filtering+out+machine%2Dtranslated+text%2E+In+Proceedings+of+the+4th+Workshop+on+Building+and+Using+Comparable+Corpor) 3. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Christian+Buck%2C+Chris+Callison%2DBurch%2C+Christian+Federmann%2C+Barry+Haddow%2C+Philipp+Koehn%2C+Christof+Monz%2C+Matt+Post%2C+Radu+Soricut%2C+and+Lucia+Specia%2E+2013%2E+Findings+of+the+2013+Workshop+on+) [our card](https://md.research.elamur.ai/free/papers/bojar-2013-findings-2013-workshop-statistical-machine/)