---
title: "Tuning as Ranking — Hopkins 2011"
description: "Hopkins and May address a practical bottleneck in statistical MT: MERT, the standard way to set feature weights, becomes unreliable once a model has more…"
authors: ["Mark Hopkins", "Jonathan May"]
year: 2011
venue: "Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D11-1125/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Tuning as Ranking

# Tuning as Ranking

**Authors:** [Mark Hopkins](https://md.research.elamur.ai/free/en/authors/mark-hopkins/), [Jonathan May](https://md.research.elamur.ai/free/en/authors/jonathan-may/) · **Year:** 2011 · **Venue:** Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D11-1125/) · [PDF](https://aclanthology.org/D11-1125.pdf) · [OpenAlex](https://openalex.org/W2180952760)

**Citations:** 232 citations (OpenAlex, 2026-10-14) · [159](https://md.research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Hopkins and May address a practical bottleneck in statistical MT: MERT, the standard way to set feature weights, becomes unreliable once a model has more than a few dozen features. MIRA-based tuning can handle thousands of features, but it is complex and rarely adopted. The authors propose PRO (pairwise ranking optimization), which keeps MERT's loop of k-best candidate generation and changes only the optimization step. Instead of maximizing the gold score of the single top-ranked output, PRO treats tuning as learning to order candidate translations correctly. Pairs of candidates for the same source sentence become difference vectors labeled by which member scores better under sentence-level BLEU+1. An ordinary linear binary classifier then learns the weights. A synthetic test shows MERT failing to recover known weights as dimensionality grows. Large-scale experiments on phrase-based and syntax-based systems, several language pairs, and baseline and extended feature sets show PRO matching or beating MERT and MIRA. PRO is also more stable across runs. Because it can be added to an existing MERT setup in hours, it lowers the cost of research on rich feature sets.

## Key points
- In a synthetic experiment (500 "sentences", 100 candidates each, dimensionality from 10 to 1000), MERT's line optimization was run with 20 random starts. Its ability to recover a known gold weight vector, measured by cosine similarity, dropped rapidly as dimensionality increased.
- PRO uses the MegaM classifier and BLEU+1 as the local gold score. The authors report similar results with the WEKA SVM module and the Stanford classifier.
- In all 12 configurations (system type × language pair × feature set), PRO's test-set results equaled or exceeded those of both MIRA and MERT. The extended features include word-pair features numbering 6,723 in principle.
- Over five repeated Urdu-English PBMT baseline runs, the standard deviation of final test BLEU was 0.13 for MERT and only 0.05 for PRO.
- Adoption gap motivating the work: in the ACL 2010 proceedings, 15 MT papers reported tuning with MERT and only one with MIRA.

## Interesting ideas
- **Reduce ranking to binary classification on difference vectors.** If a scorer is linear, "A should outrank B" becomes a classification problem over the vector x(A) − x(B). This turns any listwise or structured preference task into something a stock classifier can solve. It could be applied to reranking parsers, search results, or candidate normalizations.
- **Change one module and keep the surrounding architecture.** The authors left MERT's candidate generation untouched and replaced only the optimizer. This minimal-divergence design greatly eases adoption in existing pipelines, which is a useful principle for introducing any new method into established tooling.
- **Weight smoothing across iterations.** Each iteration's classifier is trained from scratch, so the new weights are blended with the previous ones (Ψ = 0.1). This cheap interpolation trick adds stability to any iterative retrain-from-scratch loop.
- **Validate the method on synthetic data with a known answer.** Generating data from a hidden gold weight vector and checking whether the optimizer recovers it gives a clean diagnostic of scalability. This kind of test is worth reusing before running costly real experiments.
- **Report stability across repeated runs.** Rerunning the same configuration several times and comparing the spread of final scores reveals optimizer variance that a single-run comparison would hide.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{hopkins2011tuning,
  title = {{Tuning as Ranking}},
  author = {Hopkins, Mark and May, Jonathan},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1125/}
}
```

- [← previous Literal and Metaphorical Sense Identification through Concrete and Abstract Context](https://md.research.elamur.ai/free/en/papers/turney-2011-literal-metaphorical-sense-identification-through/)
- [next → Crowdsourcing Translation: Professional Quality from Non-Professionals](https://md.research.elamur.ai/free/en/papers/zaidan-2011-crowdsourcing-translation-professional-quality-non/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/)

## References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, pages 89–96, Bonn, Germany. ACM. [find](https://scholar.google.com/scholar?q=Chris+Burges%2C+Tal+Shaked%2C+Erin+Renshaw%2C+Ari+Lazier%2C+Matt+Deeds%2C+Nicole+Hamilton%2C+and+Greg+Hullender%2E+2005%2E+Learning+to+rank+using+gradient+descent%2E+In+Proceedings+of+the+22nd+International+Conference+) [our card](https://md.research.elamur.ai/free/en/papers/burges-2005-learning-rank-using-gradient/)
2. Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129–136, Corvalis, OR. [find](https://scholar.google.com/scholar?q=Zhe+Cao%2C+Tao+Qin%2C+Tie%2DYan+Liu%2C+Ming%2DFeng+Tsai%2C+and+Hang+Li%2E+2007%2E+Learning+to+rank%3A+From+pairwise+approach+to+listwise+approach%2E+In+Proceedings+of+the+24th+International+Conference+on+Machine+Learning)
3. Samidh Chatterjee and Nicola Cancedda. 2010. Minimum error rate training by sampling the translation lattice. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 606–615, Cambridge, MA, October. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samidh+Chatterjee+and+Nicola+Cancedda%2E+2010%2E+Minimum+error+rate+training+by+sampling+the+translation+lattice%2E+In+Proceedings+of+the+2010+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
4. Wei Chen, Tie-Yan Liu, Yanyan Lan, Zhi-Ming Ma, and Hang Li. 2009. Ranking measures and loss functions in learning to rank. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 315–323. [find](https://scholar.google.com/scholar?q=Wei+Chen%2C+Tie%2DYan+Liu%2C+Yanyan+Lan%2C+Zhi%2DMing+Ma%2C+and+Hang+Li%2E+2009%2E+Ranking+measures+and+loss+functions+in+learning+to+rank%2E+In+Y%2E+Bengio%2C+D%2E+Schuurmans%2C+J%2E+Lafferty%2C+C%2E+K%2E+I%2E+Williams%2C+and+A%2E+Culotta%2C)
5. David Chiang, Steve DeNeefe, Yee Seng Chan, and Hwee Tou Ng. 2008a. Decomposability of translation metrics for improved evaluation and efficient algorithms. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 610–619, Honolulu, HI, October. Association fo… [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Steve+DeNeefe%2C+Yee+Seng+Chan%2C+and+Hwee+Tou+Ng%2E+2008a%2E+Decomposability+of+translation+metrics+for+improved+evaluation+and+efficient+algorithms%2E+In+Proceedings+of+the+2008+Conference+on+Em)
6. David Chiang, Yuval Marton, and Philip Resnik. 2008b. Online large-margin training of syntactic and structural translation features. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 224–233, Honolulu, HI, October. Association for Computational Linguist… [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Yuval+Marton%2C+and+Philip+Resnik%2E+2008b%2E+Online+large%2Dmargin+training+of+syntactic+and+structural+translation+features%2E+In+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natur)
7. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 218–226, Boulder, CO, June. Associa… [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Kevin+Knight%2C+and+Wei+Wang%2E+2009%2E+11%2C001+new+features+for+statistical+machine+translation%2E+In+Proceedings+of+Human+Language+Technologies%3A+The+2009+Annual+Conference+of+the+North+American)
8. David Chiang. 2010. Learning to translate with source and target syntax. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1443–1452, Uppsala, Sweden, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=David+Chiang%2E+2010%2E+Learning+to+translate+with+source+and+target+syntax%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+1443%E2%80%931452%2C+Uppsala%2C+Sweden%2C+J) [our card](https://md.research.elamur.ai/free/en/papers/chiang-2010-learning-translate-source-target-syntax/)
9. Brooke Cowan, Ivona Kuc̆erová, and Michael Collins. 2006. A discriminative model for tree-to-tree translation. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 232–241, Sydney, Australia, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Brooke+Cowan%2C+Ivona+Kuc%CC%86erova%CC%81%2C+and+Michael+Collins%2E+2006%2E+A+discriminative+model+for+tree%2Dto%2Dtree+translation%2E+In+Proceedings+of+the+2006+Conference+on+Empirical+Methods+in+Natural+Language+Processin)
10. Koby Crammer and Yoram Singer. 2003. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951–991. [find](https://scholar.google.com/scholar?q=Koby+Crammer+and+Yoram+Singer%2E+2003%2E+Ultraconservative+online+algorithms+for+multiclass+problems%2E+Journal+of+Machine+Learning+Research%2C+3%3A951%E2%80%93991%2E)
11. Hal Daumé III. 2004. Notes on CG and LM-BFGS optimization of logistic regression. Paper available at [http://pub.hal3.name\#daume04cg-bfgs](http://pub.hal3.name\#daume04cg-bfgs), implementation available at http://hal3.name/megam/, August. [↗](http://pub.hal3.name\#daume04cg-bfgs)
12. Yoav Freund, Raj Iyer, Robert E. Schapire, and Yoram Singer. 2003. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933–969. [find](https://scholar.google.com/scholar?q=Yoav+Freund%2C+Raj+Iyer%2C+Robert+E%2E+Schapire%2C+and+Yoram+Singer%2E+2003%2E+An+efficient+boosting+algorithm+for+combining+preferences%2E+Journal+of+Machine+Learning+Research%2C+4%3A933%E2%80%93969%2E)
13. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In HLT-NAACL 2004: Main Proceedings, pages 273–280, Boston, MA, May. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michel+Galley%2C+Mark+Hopkins%2C+Kevin+Knight%2C+and+Daniel+Marcu%2E+2004%2E+What%E2%80%99s+in+a+translation+rule%3F+In+HLT%2DNAACL+2004%3A+Main+Proceedings%2C+pages+273%E2%80%93280%2C+Boston%2C+MA%2C+May%2E+Association+for+Computational+Ling) [our card](https://md.research.elamur.ai/free/en/papers/galley-2004-what-s-translation-rule/)
14. Gloria Goodale. 2008. Language Weaver: fast in translation. The Christian Science Monitor, October 1. [http://www.csmonitor.com/Innovation/Tech-Culture/2008/1001/language-weaver-fast-in-translation](http://www.csmonitor.com/Innovation/Tech-Culture/2008/1001/language-weaver-fast-in-translation). [↗](http://www.csmonitor.com/Innovation/Tech-Culture/2008/1001/language-weaver-fast-in-translation)
15. Jan Hajič, Sandra Carberry, Stephen Clark, and Joakim Nivre, editors. 2010. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, July. [find](https://scholar.google.com/scholar?q=Jan+Hajic%CC%8C%2C+Sandra+Carberry%2C+Stephen+Clark%2C+and+Joakim+Nivre%2C+editors%2E+2010%2E+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E+Association+for+Computational+Ling)
16. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. SIGKDD Explorations, 11(1). [find](https://scholar.google.com/scholar?q=Mark+Hall%2C+Eibe+Frank%2C+Geoffrey+Holmes%2C+Bernhard+Pfahringer%2C+Peter+Reutemann%2C+and+Ian+H%2E+Witten%2E+2009%2E+The+WEKA+data+mining+software%3A+An+update%2E+SIGKDD+Explorations%2C+11%281%29%2E)
17. Ralf Herbrich, Thore Graepel, and Klaus Obermayer. 1999. Support vector learning for ordinal regression. In Proceedings of the 1999 International Conference on Artificial Neural Networks, pages 97–102. [find](https://scholar.google.com/scholar?q=Ralf+Herbrich%2C+Thore+Graepel%2C+and+Klaus+Obermayer%2E+1999%2E+Support+vector+learning+for+ordinal+regression%2E+In+Proceedings+of+the+1999+International+Conference+on+Artificial+Neural+Networks%2C+pages+97%E2%80%93102)
18. Abraham Ittycheriah and Salim Roukos. 2005. A maximum entropy word aligner for Arabic-English machine translation. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 89–96, Vancouver, Canada, October. Association for Compu… [find](https://scholar.google.com/scholar?q=Abraham+Ittycheriah+and+Salim+Roukos%2E+2005%2E+A+maximum+entropy+word+aligner+for+Arabic%2DEnglish+machine+translation%2E+In+Proceedings+of+Human+Language+Technology+Conference+and+Conference+on+Empirical+Me)
19. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
20. Shankar Kumar, Wolfgang Macherey, Chris Dyer, and Franz Och. 2009. Efficient minimum error rate training and minimum bayes-risk decoding for translation hypergraphs and lattices. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference o… [find](https://scholar.google.com/scholar?q=Shankar+Kumar%2C+Wolfgang+Macherey%2C+Chris+Dyer%2C+and+Franz+Och%2E+2009%2E+Efficient+minimum+error+rate+training+and+minimum+bayes%2Drisk+decoding+for+translation+hypergraphs+and+lattices%2E+In+Proceedings+of+the)
21. Alon Lavie and Michael J. Denkowski. 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2–3):105–115, September. [find](https://scholar.google.com/scholar?q=Alon+Lavie+and+Michael+J%2E+Denkowski%2E+2009%2E+The+METEOR+metric+for+automatic+evaluation+of+machine+translation%2E+Machine+Translation%2C+23%282%E2%80%933%29%3A105%E2%80%93115%2C+September%2E)
22. Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 761–… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Alexandre+Bouchard%2DCo%CC%82te%CC%81%2C+Dan+Klein%2C+and+Ben+Taskar%2E+2006%2E+An+end%2Dto%2Dend+discriminative+approach+to+machine+translation%2E+In+Proceedings+of+the+21st+International+Conference+on+Computatio)
23. Wolfgang Macherey, Franz Josef Och, Ignacio Thayer, and Jakob Uszkoreit. 2008. Lattice-based minimum error rate training for statistical machine translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 725– 734, Honolulu, HI, October. Association… [find](https://scholar.google.com/scholar?q=Wolfgang+Macherey%2C+Franz+Josef+Och%2C+Ignacio+Thayer%2C+and+Jakob+Uszkoreit%2E+2008%2E+Lattice%2Dbased+minimum+error+rate+training+for+statistical+machine+translation%2E+In+Proceedings+of+the+2008+Conference+on+E) [our card](https://md.research.elamur.ai/free/en/papers/macherey-2008-lattice-based-minimum-error-rate/)
24. Christopher Manning and Dan Klein. 2003. Optimization, maxent models, and conditional estimation without magic. Tutorial at HLT-NAACL 2003 and ACL 2003. [find](https://scholar.google.com/scholar?q=Christopher+Manning+and+Dan+Klein%2E+2003%2E+Optimization%2C+maxent+models%2C+and+conditional+estimation+without+magic%2E+Tutorial+at+HLT%2DNAACL+2003+and+ACL+2003%2E)
25. I. Dan Melamed, Ryan Green, and Joseph P. Turian. 2003. Precision and recall of machine translation. In Companion Volume of the Proceedings of HLT-NAACL 2003 - Short Papers, pages 61–63, Edmonton, Canada, May–June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=I%2E+Dan+Melamed%2C+Ryan+Green%2C+and+Joseph+P%2E+Turian%2E+2003%2E+Precision+and+recall+of+machine+translation%2E+In+Companion+Volume+of+the+Proceedings+of+HLT%2DNAACL+2003+%2D+Short+Papers%2C+pages+61%E2%80%9363%2C+Edmonton%2C+Can) [our card](https://md.research.elamur.ai/free/en/papers/melamed-2003-precision-recall-machine-translation/)
26. Franz Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 440–447, Hong Kong, October. [find](https://scholar.google.com/scholar?q=Franz+Och+and+Hermann+Ney%2E+2000%2E+Improved+statistical+alignment+models%2E+In+Proceedings+of+the+38th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+440%E2%80%93447%2C+Hong+Kong%2C+October%2E) [our card](https://md.research.elamur.ai/free/en/papers/och-2000-improved-statistical-alignment-models/)
27. Franz Josef Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 295–302, Philadelphia, PA, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Franz+Josef+Och+and+Hermann+Ney%2E+2002%2E+Discriminative+training+and+maximum+entropy+models+for+statistical+machine+translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Computationa) [our card](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/)
28. Franz Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4):417–449. [find](https://scholar.google.com/scholar?q=Franz+Och+and+Hermann+Ney%2E+2004%2E+The+alignment+template+approach+to+statistical+machine+translation%2E+Computational+Linguistics%2C+30%284%29%3A417%E2%80%93449%2E)
29. Franz Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan, July. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Franz+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+Proceedings+of+the+41st+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+160%E2%80%93167%2C+Sapporo%2C+Jap)
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. B LEU : a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, July. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+B+LEU+%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+40th+Annual+Meeting+of+the+Association+for+Comput)
31. Benjamin Roth, Andrew McCallum, Marc Dymetman, and Nicola Cancedda. 2010. Machine translation using overlapping alignments and samplerank. In Proceedings of Association for Machine Translation in the Americas, Denver, CO. [find](https://scholar.google.com/scholar?q=Benjamin+Roth%2C+Andrew+McCallum%2C+Marc+Dymetman%2C+and+Nicola+Cancedda%2E+2010%2E+Machine+translation+using+overlapping+alignments+and+samplerank%2E+In+Proceedings+of+Association+for+Machine+Translation+in+the+)
32. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 177–184, Boston, MA, May 2 - May 7. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Libin+Shen%2C+Anoop+Sarkar%2C+and+Franz+Josef+Och%2E+2004%2E+Discriminative+reranking+for+machine+translation%2E+In+Daniel+Marcu+Susan+Dumais+and+Salim+Roukos%2C+editors%2C+HLT%2DNAACL+2004%3A+Main+Proceedings%2C+pages+1)
33. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231. [find](https://scholar.google.com/scholar?q=Matthew+Snover%2C+Bonnie+Dorr%2C+Richard+Schwartz%2C+Linnea+Micciulla%2C+and+John+Makhoul%2E+2006%2E+A+study+of+translation+edit+rate+with+targeted+human+annotation%2E+In+Proceedings+of+Association+for+Machine+Tran) [our card](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/)
34. Christoph Tillmann and Tong Zhang. 2005. A localized prediction model for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the ACL, pages 557–564, Ann Arbor, MI, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Christoph+Tillmann+and+Tong+Zhang%2E+2005%2E+A+localized+prediction+model+for+statistical+machine+translation%2E+In+Proceedings+of+the+43rd+Annual+Meeting+of+the+ACL%2C+pages+557%E2%80%93564%2C+Ann+Arbor%2C+MI%2C+June%2E+Ass)
35. Wei Wang. 2011. Personal communication. [find](https://scholar.google.com/scholar?q=Wei+Wang%2E+2011%2E+Personal+communication%2E)
36. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2006. NTT statistical machine translation for IWSLT 2006. In Proceedings of IWSLT 2006, pages 95–102. [find](https://scholar.google.com/scholar?q=Taro+Watanabe%2C+Jun+Suzuki%2C+Hajime+Tsukada%2C+and+Hideki+Isozaki%2E+2006%2E+NTT+statistical+machine+translation+for+IWSLT+2006%2E+In+Proceedings+of+IWSLT+2006%2C+pages+95%E2%80%93102%2E)
37. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 764–… [find](https://scholar.google.com/scholar?q=Taro+Watanabe%2C+Jun+Suzuki%2C+Hajime+Tsukada%2C+and+Hideki+Isozaki%2E+2007%2E+Online+large%2Dmargin+training+for+statistical+machine+translation%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+)
38. Deyi Xiong, Qun Liu, and Shouxun Lin. 2006. Maximum entropy based phrase reordering model for statistical machine translation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 521–528, Syd… [find](https://scholar.google.com/scholar?q=Deyi+Xiong%2C+Qun+Liu%2C+and+Shouxun+Lin%2E+2006%2E+Maximum+entropy+based+phrase+reordering+model+for+statistical+machine+translation%2E+In+Proceedings+of+the+21st+International+Conference+on+Computational+Ling) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2006-maximum-entropy-based-phrase-reordering/)