---
title: "Large Scale Parallel Document Mining for Machine Translation"
description: "Statistical machine translation needs parallel text, which is scarce compared with… 0.6 BLEU. 20 references, cited by 75 papers here."
authors: ["Jakob Uszkoreit", "Jay Ponte", "Ashok Popat", "Moshe Dubiner"]
year: 2010
venue: "Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)"
source: "https://aclanthology.org/C10-1124/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Large Scale Parallel Document Mining for Machine Translation

# Large Scale Parallel Document Mining for Machine Translation

**Authors:** [Jakob Uszkoreit](https://md.research.elamur.ai/free/en/authors/jakob-uszkoreit/), [Jay Ponte](https://md.research.elamur.ai/free/en/authors/jay-ponte/), [Ashok Popat](https://md.research.elamur.ai/free/en/authors/ashok-popat/) and Moshe Dubiner · **Year:** 2010 · **Venue:** [Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2010](https://md.research.elamur.ai/free/en/venues/coling/2010/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/C10-1124/) · [PDF](https://aclanthology.org/C10-1124.pdf) · [OpenAlex](https://openalex.org/W2101096097)

**Citations:** 142 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/uszkoreit-2010-large-scale-parallel-document-mining/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Statistical machine translation needs parallel text, which is scarce compared with monolingual web data. Earlier mining methods depended on metadata such as publication dates or document structure, and that information is unreliable on the open Web. The authors treat parallel document discovery as cross-language near-duplicate detection. They first translate every document into English with a baseline phrase-based SMT system, even if its quality is modest. Rare word n-gram "shingles" then generate candidate pairs, and very frequent shingles are dropped so that runtime grows linearly with corpus size. Each candidate pair is scored by idf-weighted cosine similarity over lower-order n-grams. Pairs are kept only if each document is in the other's n-best list, and the output is cleaned further at the sentence level. The whole pipeline runs on MapReduce. It was applied to 2.5 billion web pages and 1.5 million OCR-digitized books. Training on the mined data consistently raised BLEU for several European language pairs. For Arabic and Chinese, it matched or slightly beat the date-dependent Munteanu and Marcu corpora.

## Key points
- **Input data:**
  - The "web" set has 2.5B pages covering six languages: Czech, English, French, German, Hungarian and Spanish.
  - The "books" set has 1.5M public-domain OCR books, mostly English and French, with fewer Spanish volumes.
  - The web reference alignments come from the cross-language hyperlinks of america.gov.
- **Baseline translation:**
  - The baseline is a phrase-based log-linear SMT system trained on Europarl, the DGT Multilingual Translation Memory and the UN ODS corpus.
  - Its weights are tuned with MERT on WMT dev sets, and it uses a 4-gram language model.
- **Complexity:** Capping the posting-list size of matching n-grams at c gives a time cost of O(|D|·m·c·s), which is linear in both corpus size and document length.
- **Comparison with Munteanu and Marcu (2005):** Both datasets were added to a UN ODS baseline.
  - Chinese→English on NIST 2008 rose from 19.79 to 22.02, versus 21.69 with the Munteanu and Marcu data.
  - For Arabic, the two approaches gave similar results.
- **Robustness to baseline quality:**
  - The authors weakened the Czech→English baseline by up to roughly 9 BLEU.
  - The systems trained on the resulting mined data lost less than 0.6 BLEU.

## Interesting ideas
- **Translate first, then deduplicate.** Running a cheap, imperfect MT pass turns a cross-lingual matching problem into a monolingual one. Standard near-duplicate detection tools then apply directly, and the experiments show this holds up even with a weak translator.
- **Two separate feature sets.** Long, rare n-grams are used only to propose candidate pairs, and short, common n-grams are used only to score them. This separates recall-oriented scoring from the cost of candidate generation, a pattern that suits any large-scale pairwise matching job.
- **Dropping overly frequent keys as a scaling tool.** Discarding posting lists above a frequency threshold removes the quadratic blow-up. Because the distribution is long-tailed, only a small fraction of features is lost.
- **Prefer the simpler scoring function.** Binary idf-weighted cosine outperformed tf-idf and BM25 here, which the authors attribute to queries being as long as the documents. This is a useful reminder to re-check IR defaults when queries are whole documents.
- **Cheap consistency filters.** - Mutual n-best agreement removes one-sided matches. - A normalized permutation edit distance over shared n-gram positions removes spurious matches between long documents such as books. - Both checks are inexpensive and could be reused in other alignment pipelines.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 353 works · 364,806 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (1): [🔁 Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) · 958 works

## Cite

Cite as: Jakob Uszkoreit, Jay Ponte, Ashok Popat and Moshe Dubiner. 2010. Large Scale Parallel Document Mining for Machine Translation. Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{uszkoreit2010large,
  title = {{Large Scale Parallel Document Mining for Machine Translation}},
  author = {Jakob Uszkoreit, Jay Ponte, Ashok Popat and Moshe Dubiner},
  booktitle = {Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)},
  year = {2010},
  url = {https://aclanthology.org/C10-1124/}
}
```

- [← previous Automatic Domain Adaptation for Parsing](https://md.research.elamur.ai/free/en/papers/mcclosky-2010-automatic-domain-adaptation-parsing/)
- [next → Sentence and Expression Level Annotation of Opinions in User-Generated Discourse](https://md.research.elamur.ai/free/en/papers/toprak-2010-sentence-expression-level-annotation-opinions/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdul-Rauf, Sadaf and Holger Schwenk. 2009. On the use of comparable corpora to improve SMT performance. In EACL, pages 16–23. [find](https://scholar.google.com/scholar?q=Abdul%2DRauf%2C+Sadaf+and+Holger+Schwenk%2E+2009%2E+On+the+use+of+comparable+corpora+to+improve+SMT+performance%2E+In+EACL%2C+pages+16%E2%80%9323%2E)
2. Broder, Andrei Z. 2000. Identifying and filtering near-duplicate documents. In COM ’00: Proceedings of the 11th Annual Symposium on Combinatorial Pattern Matching, pages 1–10, London, UK. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Broder%2C+Andrei+Z%2E+2000%2E+Identifying+and+filtering+near%2Dduplicate+documents%2E+In+COM+%E2%80%9900%3A+Proceedings+of+the+11th+Annual+Symposium+on+Combinatorial+Pattern+Matching%2C+pages+1%E2%80%9310%2C+London%2C+UK%2E+Springer%2DVer)
3. Chen, Jiang and Jian-Yun Nie. 2000. Parallel web text mining for cross-language IR. In In In Proc. of RIAO, pages 62–77. [find](https://scholar.google.com/scholar?q=Chen%2C+Jiang+and+Jian%2DYun+Nie%2E+2000%2E+Parallel+web+text+mining+for+cross%2Dlanguage+IR%2E+In+In+In+Proc%2E+of+RIAO%2C+pages+62%E2%80%9377%2E)
4. Cormode, Graham, S. Muthukrishnan, and Süleyman Cenk Sahinalp. 2001. Permutation editing and matching via embeddings. In ICALP ’01: Proceedings of the 28th International Colloquium on Automata, Languages and Programming,, pages 481–492, London, UK. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Cormode%2C+Graham%2C+S%2E+Muthukrishnan%2C+and+Su%CC%88leyman+Cenk+Sahinalp%2E+2001%2E+Permutation+editing+and+matching+via+embeddings%2E+In+ICALP+%E2%80%9901%3A+Proceedings+of+the+28th+International+Colloquium+on+Automata%2C+Langu)
5. Dean, Jeffrey and Sanjay Ghemawat. 2004. MapReduce: Simplified data processing on large clusters. In Proceedings of the Sixth Symposium on Operating System Design and Implementation (OSDI-04), San Francisco, CA, USA. [find](https://scholar.google.com/scholar?q=Dean%2C+Jeffrey+and+Sanjay+Ghemawat%2E+2004%2E+MapReduce%3A+Simplified+data+processing+on+large+clusters%2E+In+Proceedings+of+the+Sixth+Symposium+on+Operating+System+Design+and+Implementation+%28OSDI%2D04%29%2C+San+Fra)
6. Do, Thi-Ngoc-Diep, Viet-Bac Le, Brigitte Bigi, Laurent Besacier Eric, and Castelli. 2009. Mining a comparable text corpus for a Vietnamese - French statistical machine translation system. In Proceedings of the 4th EACL Workshop on Statistical Machine Translation, pages 165–172, Athens, Greece, March… [find](https://scholar.google.com/scholar?q=Do%2C+Thi%2DNgoc%2DDiep%2C+Viet%2DBac+Le%2C+Brigitte+Bigi%2C+Laurent+Besacier+Eric%2C+and+Castelli%2E+2009%2E+Mining+a+comparable+text+corpus+for+a+Vietnamese+%2D+French+statistical+machine+translation+system%2E+In+Proceedin)
7. European Commission Directorate-General for Translation. 2007. DGT-TM parallel corpus. [http://langtech.jrc.it/DGT-TM.html](http://langtech.jrc.it/DGT-TM.html). [↗](http://langtech.jrc.it/DGT-TM.html)
8. Harding, Stephen M., W. Bruce Croft, and C. Weir. 1997. Probabilistic retrieval of OCR degraded text using n-grams. In ECDL ’97: Proceedings of the First European Conference on Research and Advanced Technology for Digital Libraries, pages 345–359, London, UK. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Harding%2C+Stephen+M%2E%2C+W%2E+Bruce+Croft%2C+and+C%2E+Weir%2E+1997%2E+Probabilistic+retrieval+of+OCR+degraded+text+using+n%2Dgrams%2E+In+ECDL+%E2%80%9997%3A+Proceedings+of+the+First+European+Conference+on+Research+and+Advanced+T)
9. Henzinger, Monika. 2006. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In SIGIR ’06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 284– 291, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Henzinger%2C+Monika%2E+2006%2E+Finding+near%2Dduplicate+web+pages%3A+a+large%2Dscale+evaluation+of+algorithms%2E+In+SIGIR+%E2%80%9906%3A+Proceedings+of+the+29th+annual+international+ACM+SIGIR+conference+on+Research+and+devel)
10. Koehn, Philipp. 2002. Europarl: A multilingual corpus for evaluation of machine translation. Draft. [find](https://scholar.google.com/scholar?q=Koehn%2C+Philipp%2E+2002%2E+Europarl%3A+A+multilingual+corpus+for+evaluation+of+machine+translation%2E+Draft%2E)
11. Macherey, Wolfgang, Franz Och, Ignacio Thayer, and Jakob Uszkoreit. 2008. [Lattice-based minimum error rate training for statistical machine translation](https://md.research.elamur.ai/free/en/papers/macherey-2008-lattice-based-minimum-error-rate/). In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 725–734, Honolulu, Hi, October. Association for Co…
12. Manber, Udi. 1994. Finding similar files in a large file system. In Proceedings of the USENIX Winter 1994 Technical Conferenc. [find](https://scholar.google.com/scholar?q=Manber%2C+Udi%2E+1994%2E+Finding+similar+files+in+a+large+file+system%2E+In+Proceedings+of+the+USENIX+Winter+1994+Technical+Conferenc%2E)
13. Munteanu, Dragos Stefan and Daniel Marcu. 2005. [Improving machine translation performance by exploiting non-parallel corpora](https://md.research.elamur.ai/free/en/papers/munteanu-2005-improving-machine-translation-performance-exploiting/). Comput. Linguist., 31(4):477–504.
14. Munteanu, Dragos Stefan and Daniel Marcu. 2006. [Extracting parallel sub-sentential fragments from non-parallel corpora](https://md.research.elamur.ai/free/en/papers/munteanu-2006-extracting-parallel-sub-sentential-fragments/). In ACL.
15. Och, Franz Josef and Hermann Ney. 2002. [Discriminative training and maximum entropy models for statistical machine translation](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 295–302, Philadelphia, PA, USA.
16. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, USA.
17. Resnik, Philip and Noah A. Smith. 2003. [The web as a parallel corpus](https://md.research.elamur.ai/free/en/papers/resnik-2003-web-as-parallel-corpus/). Computational Linguistics, 29:349–380.
18. Robertson, S E, S Walker, S Jones, M M Hancock-Beaulieu, and M Gatford. 1995. [Okapi at TREC–3](https://md.research.elamur.ai/free/en/papers/robertson-1994-okapi-trec-3/). In Proceedings of the Third Text REtrieval Conference (TREC-3).
19. Udupa, Raghavendra, K. Saravanan, A. Kumaran, and Jagadeesh Jagarlamudi. 2009. Mint: A method for effective and scalable mining of named entity transliterations from large comparable corpora. In EACL, pages 799–807. [find](https://scholar.google.com/scholar?q=Udupa%2C+Raghavendra%2C+K%2E+Saravanan%2C+A%2E+Kumaran%2C+and+Jagadeesh+Jagarlamudi%2E+2009%2E+Mint%3A+A+method+for+effective+and+scalable+mining+of+named+entity+transliterations+from+large+comparable+corpora%2E+In+EACL%2C)
20. United Nations. 2006. ODS UN parallel corpus. [http://ods.un.org/](http://ods.un.org/). [↗](http://ods.un.org/)

*Updated 24 October 2026*