---
title: "Extracting Parallel Sentences from Comparable — Smith 2010"
description: "Statistical MT systems depend heavily on parallel training data, which is scarce for most language pairs and domains."
authors: ["Jason R. Smith", "Chris Quirk", "Kristina Toutanova"]
year: 2010
venue: "Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N10-1063/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/smith-2010-extracting-parallel-sentences-comparable-corpora/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/smith-2010-extracting-parallel-sentences-comparable-corpora/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment

# Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment

**Authors:** [Jason R. Smith](https://md.research.elamur.ai/free/en/authors/jason-r-smith/), [Chris Quirk](https://md.research.elamur.ai/free/en/authors/chris-quirk/), [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) · **Year:** 2010 · **Venue:** Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N10-1063/) · [PDF](https://aclanthology.org/N10-1063.pdf) · [OpenAlex](https://openalex.org/W2105673178)

**Citations:** 192 citations (openalex-snapshot · 2026-10-14) · [110](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Statistical MT systems depend heavily on parallel training data, which is scarce for most language pairs and domains. The authors treat Wikipedia as a comparable corpus: articles on the same topic are already paired across languages by user-made "interwiki" links, so the task reduces to finding parallel sentences inside each linked article pair. Instead of the usual binary parallel/non-parallel classifier, they propose a Maximum Entropy ranking model. For each source sentence, it picks a target sentence or "null". They also propose a first-order linear-chain CRF that aligns the whole document at once. The CRF exploits the fact that parallel sentences tend to appear in clusters. Features come from IBM Model 1 and HMM word alignments, sentence length and position, distortion, and Wikipedia markup (links, shared image captions, lists). A lexicon induced from comparable text adds further features. Experiments on Spanish-, German- and Bulgarian-English show that the extracted sentences raise BLEU, most strongly on Wikipedia-domain text. This makes Wikipedia a practical parallel-data source for language pairs with few resources.

## Key points
- Wikipedia already provides document alignment through interwiki links. Table 1 gives the counts of English-paired articles, for example French 496K, German 488K, Spanish 311K and Czech 87K.
- The extracted data were large: roughly 1.69M German-English, 1.91M Spanish-English and 146K Bulgarian-English sentence pairs were added to training.
- Adding the extracted data to medium-sized systems improved BLEU in every language pair. On the new Wikitest set, gains were +6.1 (es-en), +5.2 (de-en) and +10.1 (bg-en).
- On Wikitest, a medium system plus the Wikipedia extracts matched or beat the large system alone. Adding the extracts to the large system still gave +2.2 to +3.5 BLEU on Wikitest, while changes on Test A were small.
- The authors released manually aligned document pairs and the hand-cleaned "Wikitest" test set, with about 500 sentences per language pair.

## Interesting ideas
- **Rank instead of classify.** Choosing the best match (or "null") for each source sentence avoids the heavy class imbalance of labelling all sentence pairs as positive or negative. The same framing could help any one-to-at-most-one matching task, such as entity linking or record deduplication.
- **Global, non-monotonic document alignment.** A CRF with distortion features rewards alignments that follow the previous aligned sentence. This captures the tendency of parallel content to come in clusters, and the idea could transfer to aligning translated web pages or subtitles.
- **Use structural metadata as weak signals.** Matching hyperlinks are weighted by their inverse frequency, and shared image captions or list membership act as indicators. The image and list indicators take a negative value when only one side matches. Such cheap markup cues could be reused when mining bilingual sites or Ukrainian–English Wikipedia pairs.
- **Induce a lexicon from non-parallel article pairs.** A word-level model trained on linked articles adds translations that the seed parallel data lacks. This raises recall without the drawbacks of self-training, which can only learn words that already occur in extracted parallel sentences.
- **Build an in-domain, hand-cleaned test set.** Testing only on Europarl-like data would have hidden most of the benefit. A small, manually edited Wikipedia test set showed where the extracted data actually helps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

110 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 102 not yet.

- [Neural CRF Model for Sentence Alignment in Text Simplification](https://md.research.elamur.ai/free/en/papers/jiang-2020-neural-crf-model-sentence-alignment/)
- [Learning How to Simplify From Explicit Labeling of Complex-Simplified Text Pairs](https://md.research.elamur.ai/free/en/papers/manchego-2017-learning-how-simplify-explicit-labeling/)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://md.research.elamur.ai/free/en/papers/schwenk-2019-ccmatrix-mining-billions-high-quality/)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://md.research.elamur.ai/free/en/papers/schwenk-2019-wikimatrix-mining-135m-parallel-sentences/)
- [Dirt Cheap Web-Scale Parallel Text from the Common Crawl](https://md.research.elamur.ai/free/en/papers/smith-2013-dirt-cheap-web-scale-parallel/)
- [Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/wein-2023-measuring-fine-grained-semantic-equivalence/)
- [Assessing the Cross-linguistic Utility of Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/wein-2024-assessing-cross-linguistic-utility-abstract/)
- [Crowdsourcing Translation: Professional Quality from Non-Professionals](https://md.research.elamur.ai/free/en/papers/zaidan-2011-crowdsourcing-translation-professional-quality-non/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{smith2010extracting,
  title = {{Extracting Parallel Sentences from Comparable Corpora using Document Level Alignment}},
  author = {Smith, Jason R. and Quirk, Chris and Toutanova, Kristina},
  booktitle = {Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/N10-1063/}
}
```

- [← previous Extracting Opinion Targets in a Single and Cross-Domain Setting with Conditional Random Fields](https://md.research.elamur.ai/free/en/papers/jakob-2010-extracting-opinion-targets-single-cross/)
- [next → Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation](https://md.research.elamur.ai/free/en/papers/burch-2010-findings-2010-joint-workshop-statistical/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. F Adafre and M. de Rijke. 2006. Finding similar sentences across multiple languages in wikipedia. In Proceedings of EACL, pages 62–69. [find](https://scholar.google.com/scholar?q=S%2E+F+Adafre+and+M%2E+de+Rijke%2E+2006%2E+Finding+similar+sentences+across+multiple+languages+in+wikipedia%2E+In+Proceedings+of+EACL%2C+pages+62%E2%80%9369%2E)
2. Phil Blunsom and Trevor Cohn. 2006. Discriminative word alignment with conditional random fields. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Phil+Blunsom+and+Trevor+Cohn%2E+2006%2E+Discriminative+word+alignment+with+conditional+random+fields%2E+In+Proceedings+of+ACL%2E)
3. P. F Brown, V. J Della Pietra, S. A Della Pietra, and R. L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=P%2E+F+Brown%2C+V%2E+J+Della+Pietra%2C+S%2E+A+Della+Pietra%2C+and+R%2E+L+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+linguistics%2C+19%282%29%3A263%E2%80%93311%2E)
4. P. Fung and P. Cheung. 2004. Multi-level bootstrapping for extracting parallel sentences from a quasi-comparable corpus. In Proceedings of the 20th international conference on Computational Linguistics, page 1051. [find](https://scholar.google.com/scholar?q=P%2E+Fung+and+P%2E+Cheung%2E+2004%2E+Multi%2Dlevel+bootstrapping+for+extracting+parallel+sentences+from+a+quasi%2Dcomparable+corpus%2E+In+Proceedings+of+the+20th+international+conference+on+Computational+Linguistic)
5. W. A Gale and K. W Church. 1991. Identifying word correspondences in parallel texts. In Proceedings of the workshop on Speech and Natural Language, pages 152–157. [find](https://scholar.google.com/scholar?q=W%2E+A+Gale+and+K%2E+W+Church%2E+1991%2E+Identifying+word+correspondences+in+parallel+texts%2E+In+Proceedings+of+the+workshop+on+Speech+and+Natural+Language%2C+pages+152%E2%80%93157%2E)
6. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL, pages 771–779. [find](https://scholar.google.com/scholar?q=Aria+Haghighi%2C+Percy+Liang%2C+Taylor+Berg%2DKirkpatrick%2C+and+Dan+Klein%2E+2008%2E+Learning+bilingual+lexicons+from+monolingual+corpora%2E+In+Proceedings+of+ACL%2C+pages+771%E2%80%93779%2E) [our card](https://md.research.elamur.ai/free/en/papers/haghighi-2008-learning-bilingual-lexicons-monolingual-corpora/)
7. Roy Bar-Haim Ido, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entailment challenge. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim+Ido%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+2006%2E+The+second+pascal+recognising+textual+entailment+challenge%2E)
8. P. Koehn and K. Knight. 2002. Learning a translation lexicon from monolingual corpora. In Proceedings of the ACL Workshop on Unsupervised Lexical Acquisition. [find](https://scholar.google.com/scholar?q=P%2E+Koehn+and+K%2E+Knight%2E+2002%2E+Learning+a+translation+lexicon+from+monolingual+corpora%2E+In+Proceedings+of+the+ACL+Workshop+on+Unsupervised+Lexical+Acquisition%2E) [our card](https://md.research.elamur.ai/free/en/papers/koehn-2002-learning-translation-lexicon-monolingual-corpora/)
9. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of HLT/NAACL, pages 127–133, Edmonton, Canada, May. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+Josef+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+HLT%2FNAACL%2C+pages+127%E2%80%93133%2C+Edmonton%2C+Canada%2C+May%2E)
10. P. Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+MT+summit%2C+volume+5%2E)
11. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning, pages 282–289. [find](https://scholar.google.com/scholar?q=J%2E+Lafferty%2C+A%2E+McCallum%2C+and+F%2E+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proceedings+of+the+18th+International+Conference+on+Machin)
12. R. C Moore. 2002. Fast and accurate sentence alignment of bilingual corpora. Lecture Notes in Computer Science, 2499:135–144. [find](https://scholar.google.com/scholar?q=R%2E+C+Moore%2E+2002%2E+Fast+and+accurate+sentence+alignment+of+bilingual+corpora%2E+Lecture+Notes+in+Computer+Science%2C+2499%3A135%E2%80%93144%2E)
13. D. S Munteanu and D. Marcu. 2005. Improving machine translation performance by exploiting non-parallel corpora. Computational Linguistics, 31(4):477–504. [find](https://scholar.google.com/scholar?q=D%2E+S+Munteanu+and+D%2E+Marcu%2E+2005%2E+Improving+machine+translation+performance+by+exploiting+non%2Dparallel+corpora%2E+Computational+Linguistics%2C+31%284%29%3A477%E2%80%93504%2E) [our card](https://md.research.elamur.ai/free/en/papers/munteanu-2005-improving-machine-translation-performance-exploiting/)
14. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+160%E2%80%93167%2C+Sapporo%2C+Ja)
15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelpha, Pennsylvania, USA. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+Annual+Meeting+of+the+Association+for+Computati) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
16. R. Rapp. 1999. Automatic identification of word translations from unrelated English and German corpora. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=R%2E+Rapp%2E+1999%2E+Automatic+identification+of+word+translations+from+unrelated+English+and+German+corpora%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/rapp-1999-automatic-identification-word-translations-unrelated/)
17. P. Resnik and N. A Smith. 2003. The web as a parallel corpus. Computational Linguistics, 29(3):349–380. [find](https://scholar.google.com/scholar?q=P%2E+Resnik+and+N%2E+A+Smith%2E+2003%2E+The+web+as+a+parallel+corpus%2E+Computational+Linguistics%2C+29%283%29%3A349%E2%80%93380%2E) [our card](https://md.research.elamur.ai/free/en/papers/resnik-2003-web-as-parallel-corpus/)
18. C. Tillmann and J. Xu. 2009. A simple sentence-level extraction algorithm for comparable data. In Proceedings of HLT/NAACL, pages 93–96. [find](https://scholar.google.com/scholar?q=C%2E+Tillmann+and+J%2E+Xu%2E+2009%2E+A+simple+sentence%2Dlevel+extraction+algorithm+for+comparable+data%2E+In+Proceedings+of+HLT%2FNAACL%2C+pages+93%E2%80%9396%2E)
19. C. Tillmann. 2009. A Beam-Search extraction algorithm for comparable data. In Proceedings of ACL, pages 225–228. [find](https://scholar.google.com/scholar?q=C%2E+Tillmann%2E+2009%2E+A+Beam%2DSearch+extraction+algorithm+for+comparable+data%2E+In+Proceedings+of+ACL%2C+pages+225%E2%80%93228%2E)
20. S. Vogel, H. Ney, and C. Tillmann. 1996. HMM-based word alignment in statistical translation. In Proceedings of the 16th conference on Computational linguistics-Volume 2, pages 836–841. [find](https://scholar.google.com/scholar?q=S%2E+Vogel%2C+H%2E+Ney%2C+and+C%2E+Tillmann%2E+1996%2E+HMM%2Dbased+word+alignment+in+statistical+translation%2E+In+Proceedings+of+the+16th+conference+on+Computational+linguistics%2DVolume+2%2C+pages+836%E2%80%93841%2E)
21. Wikipedia. 2004. Wikipedia, the free encyclopedia. [Online; accessed 20-November-2009]. [find](https://scholar.google.com/scholar?q=Wikipedia%2E+2004%2E+Wikipedia%2C+the+free+encyclopedia%2E+%5BOnline%3B+accessed+20%2DNovember%2D2009%5D%2E)
22. B. Zhao and S. Vogel. 2002. Adaptive parallel sentences mining from web bilingual news collection. In Proceedings of the 2002 IEEE International Conference on Data Mining, page 745. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=B%2E+Zhao+and+S%2E+Vogel%2E+2002%2E+Adaptive+parallel+sentences+mining+from+web+bilingual+news+collection%2E+In+Proceedings+of+the+2002+IEEE+International+Conference+on+Data+Mining%2C+page+745%2E+IEEE+Computer+Soci)