---
title: "Batch Tuning Strategies for Statistical — Cherry 2012"
description: "Classic MERT tuning for statistical machine translation breaks down once a model has more than a few dozen features."
authors: ["Colin Cherry", "George Foster"]
year: 2012
venue: "Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N12-1047/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cherry-2012-batch-tuning-strategies-statistical-machine/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cherry-2012-batch-tuning-strategies-statistical-machine/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/) / Batch Tuning Strategies for Statistical Machine Translation

# Batch Tuning Strategies for Statistical Machine Translation

**Authors:** [Colin Cherry](https://md.research.elamur.ai/free/en/authors/colin-cherry/), [George Foster](https://md.research.elamur.ai/free/en/authors/george-foster/) · **Year:** 2012 · **Venue:** Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N12-1047/) · [PDF](https://aclanthology.org/N12-1047.pdf) · [OpenAlex](https://openalex.org/W2159755860)

**Citations:** 328 citations (openalex-snapshot · 2026-10-14) · [220](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Classic MERT tuning for statistical machine translation breaks down once a model has more than a few dozen features. Many replacement tuners exist, but they had not been compared under shared conditions. Cherry and Foster fill this gap in two ways. First, they recast MIRA, PRO and minimum risk training (MR) as different sentence-level surrogate losses inside one common framework. Second, they introduce three new large-margin tuners that run inside MERT's usual outer loop of decoding k-best lists or lattices and then re-optimizing:
- Batch MIRA over k-best lists
- Batch MIRA over lattices
- a Structured SVM

They compare eight tuning strategies on French–English, English–French and Chinese–English. Each pair is tested with Small, Medium and Big feature sets, and stability is measured by tuning on sub-samples of the tuning set. With few features, all methods perform about the same. As the feature count grows, Batch Lattice MIRA is the most reliable choice. It matches or beats online MIRA while being far simpler to parallelize. This matters in practice because rich-feature tuning can reuse existing MERT infrastructure.

## Key points
- Eight tuners were compared: k-best MERT, lattice MERT, online MIRA, k-best Batch MIRA, lattice Batch MIRA, PRO (max-ent version), MR, and the Structured SVM.
- Online MIRA was parallelized with the shard-and-average scheme of McDonald et al. (2010), as in Moses. It ran roughly an order of magnitude slower than the batch variants.
- Batch Lattice MIRA scored best on test in every Medium setting and in two of the three Big settings. On Big Zh-En, the SVM came first (33.04 vs 32.82 BLEU).
- MERT already shows weaknesses with about 18 features, which Lattice MERT partly fixes. Among the tuners with an explicit regularizer, MR was the most consistent and always stayed within 0.2 BLEU of Batch Lattice MIRA.
- In an overfitting stress test with more than 20k features (Big Fr-En), the λ parameter of SVM and MR gave much wider control over tuning scores than MIRA's step-size cap C.

## Interesting ideas
- **Compare methods by their per-example loss.** Writing each tuner as a per-sentence loss (max over hypotheses, sum over pairs, or expected cost) exposes how they really differ. One example is PRO getting credit for every correctly ordered pair, even pairs that hardly affect BLEU. This lens could help compare ranking or structured learners in other tasks.
- **Run online learning on a fixed search approximation.** Online updates are applied to a fixed, cached approximation of the search space rather than to live decoder output. This keeps the benefits of online learning while allowing parallel batch decoding, which suits any setting where inference is expensive.
- **Test stability by sub-sampling the tuning data.** Instead of only changing random seeds, the authors tune on different subsets of the tuning set. This gives a fair variance estimate even for methods that have little internal randomness.
- **Build a feature set designed to overfit.** Deliberately adding features that should generalize badly is a cheap way to probe how well a learner's regularization works.
- **Use "hope" and "fear" hypotheses.** The update pairs an ambitious but reachable target (high model score plus high metric score) with a hypothesis that is both high-scoring and costly. This trick could carry over to other metric-driven structured prediction problems.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

220 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 209 not yet.

- [Farasa: A Fast and Furious Segmenter for Arabic](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-abdelali-farasa-fast-furious-segmenter-arabic/)
- [Prompsit's submission to WMT 2018 Parallel Corpus Filtering shared task](https://md.research.elamur.ai/free/en/papers/cartagena-2018-prompsit-s-submission-wmt-2018/)
- [A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/)
- [Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-log-linear-combinations-monolingual-bilingual/)
- [Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation](https://md.research.elamur.ai/free/en/papers/ferreira-2017-linguistic-realisation-as-machine-translation/)
- [Achieving Human Parity on Automatic Chinese to English News Translation](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/)
- [Scalable Modified Kneser-Ney Language Model Estimation](https://md.research.elamur.ai/free/en/papers/heafield-2013-scalable-modified-kneser-ney-language/)
- [Measuring Machine Translation Errors in New Domains](https://md.research.elamur.ai/free/en/papers/irvine-2013-measuring-machine-translation-errors-new/)
- [A Challenge Set Approach to Evaluating Machine Translation](https://md.research.elamur.ai/free/en/papers/isabelle-2017-challenge-set-approach-evaluating-machine/)
- [The IIT Bombay English-Hindi Parallel Corpus](https://md.research.elamur.ai/free/en/papers/kunchukuttan-2018-iit-bombay-english-hindi-parallel/)
- [The NiuTrans Machine Translation Systems for WMT19](https://md.research.elamur.ai/free/en/papers/li-2019-niutrans-machine-translation-systems-wmt19/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cherry2012batch,
  title = {{Batch Tuning Strategies for Statistical Machine Translation}},
  author = {Cherry, Colin and Foster, George},
  booktitle = {Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2012},
  url = {https://aclanthology.org/N12-1047/}
}
```

- [← previous Baselines and Bigrams: Simple, Good Sentiment and Topic Classification](https://md.research.elamur.ai/free/en/papers/wang-2012-baselines-bigrams-simple-good-sentiment/)
- [next → Behind the Article: Recognizing Dialog Acts in Wikipedia Talk Pages](https://md.research.elamur.ai/free/en/papers/ferschke-2012-behind-article-recognizing-dialog-acts/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/)

## References

31 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abhishek Arun, Barry Haddow, and Philipp Koehn. 2010. A unified approach to minimum risk training and decoding. In Proceedings of the Joint Workshop on Statistical Machine Translation and MetricsMATR, pages 365–374. [find](https://scholar.google.com/scholar?q=Abhishek+Arun%2C+Barry+Haddow%2C+and+Philipp+Koehn%2E+2010%2E+A+unified+approach+to+minimum+risk+training+and+decoding%2E+In+Proceedings+of+the+Joint+Workshop+on+Statistical+Machine+Translation+and+MetricsMATR%2C)
2. Leon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In International Conference on Computational Statistics, pages 177–187. [find](https://scholar.google.com/scholar?q=Leon+Bottou%2E+2010%2E+Large%2Dscale+machine+learning+with+stochastic+gradient+descent%2E+In+International+Conference+on+Computational+Statistics%2C+pages+177%E2%80%93187%2E)
3. David Chiang, Yuval Marton, and Philip Resnik. 2008. Online large-margin training of syntactic and structural translation features. In EMNLP, pages 224–233. [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Yuval+Marton%2C+and+Philip+Resnik%2E+2008%2E+Online+large%2Dmargin+training+of+syntactic+and+structural+translation+features%2E+In+EMNLP%2C+pages+224%E2%80%93233%2E)
4. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In HLT-NAACL, pages 218–226. [find](https://scholar.google.com/scholar?q=David+Chiang%2C+Kevin+Knight%2C+and+Wei+Wang%2E+2009%2E+11%2C001+new+features+for+statistical+machine+translation%2E+In+HLT%2DNAACL%2C+pages+218%E2%80%93226%2E)
5. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In ACL, pages 176–181. [find](https://scholar.google.com/scholar?q=Jonathan+H%2E+Clark%2C+Chris+Dyer%2C+Alon+Lavie%2C+and+Noah+A%2E+Smith%2E+2011%2E+Better+hypothesis+testing+for+statistical+machine+translation%3A+Controlling+for+optimizer+instability%2E+In+ACL%2C+pages+176%E2%80%93181%2E)
6. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research. [find](https://scholar.google.com/scholar?q=Koby+Crammer%2C+Ofer+Dekel%2C+Joseph+Keshet%2C+Shai+Shalev%2DShwartz%2C+and+Yoram+Singer%2E+2006%2E+Online+passive%2Daggressive+algorithms%2E+Journal+of+Machine+Learning+Research%2E)
7. Kevin Gimpel and Noah A. Smith. 2012. Structured ramp loss minimization for machine translation. In HLT-NAACL, Montreal, Canada, June. [find](https://scholar.google.com/scholar?q=Kevin+Gimpel+and+Noah+A%2E+Smith%2E+2012%2E+Structured+ramp+loss+minimization+for+machine+translation%2E+In+HLT%2DNAACL%2C+Montreal%2C+Canada%2C+June%2E)
8. Eva Hasler, Barry Haddow, and Philipp Koehn. 2011. Margin infused relaxed algorithm for moses. The Prague Bulletin of Mathematical Linguistics, 96:69– 78. [find](https://scholar.google.com/scholar?q=Eva+Hasler%2C+Barry+Haddow%2C+and+Philipp+Koehn%2E+2011%2E+Margin+infused+relaxed+algorithm+for+moses%2E+The+Prague+Bulletin+of+Mathematical+Linguistics%2C+96%3A69%E2%80%93+78%2E)
9. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In EMNLP, pages 1352–1362. [find](https://scholar.google.com/scholar?q=Mark+Hopkins+and+Jonathan+May%2E+2011%2E+Tuning+as+ranking%2E+In+EMNLP%2C+pages+1352%E2%80%931362%2E) [our card](https://md.research.elamur.ai/free/en/papers/hopkins-2011-tuning-as-ranking/)
10. Cho-Jui Hsieh, Kai-Wei Chang, Chih-Jen Lin, S. Sathiya Keerthi, and S. Sundararajan. 2008. A dual coordinate descent method for large-scale linear svm. In ICML. [find](https://scholar.google.com/scholar?q=Cho%2DJui+Hsieh%2C+Kai%2DWei+Chang%2C+Chih%2DJen+Lin%2C+S%2E+Sathiya+Keerthi%2C+and+S%2E+Sundararajan%2E+2008%2E+A+dual+coordinate+descent+method+for+large%2Dscale+linear+svm%2E+In+ICML%2E)
11. Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In HLT-NAACL, pages 127–133. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Franz+J%2E+Och%2C+and+Daniel+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+HLT%2DNAACL%2C+pages+127%E2%80%93133%2E)
12. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In ACL,… [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2C+Hieu+Hoang%2C+Alexandra+Birch%2C+Chris+Callison%2DBurch%2C+Marcello+Federico%2C+Nicola+Bertoldi%2C+Brooke+Cowan%2C+Wade+Shen%2C+Christine+Moran%2C+Richard+Zens%2C+Chris+Dyer%2C+Ondrej+Bojar%2C+Alexandra+Consta)
13. Zhifei Li and Jason Eisner. 2009. First- and second-order expectation semirings with applications to minimum-risk training on translation forests. In EMNLP, pages 40–51. [find](https://scholar.google.com/scholar?q=Zhifei+Li+and+Jason+Eisner%2E+2009%2E+First%2D+and+second%2Dorder+expectation+semirings+with+applications+to+minimum%2Drisk+training+on+translation+forests%2E+In+EMNLP%2C+pages+40%E2%80%9351%2E)
14. Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006. An end-to-end discriminative approach to machine translation. In COLING-ACL, pages 761–768. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Alexandre+Bouchard%2DCo%CC%82te%CC%81%2C+Dan+Klein%2C+and+Ben+Taskar%2E+2006%2E+An+end%2Dto%2Dend+discriminative+approach+to+machine+translation%2E+In+COLING%2DACL%2C+pages+761%E2%80%93768%2E)
15. Chin-Yew Lin and Franz Josef Och. 2004. Orange: a method for evaluating automatic evaluation metrics for machine translation. In COLING, pages 501–507. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin+and+Franz+Josef+Och%2E+2004%2E+Orange%3A+a+method+for+evaluating+automatic+evaluation+metrics+for+machine+translation%2E+In+COLING%2C+pages+501%E2%80%93507%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-orange-method-evaluating-automatic-evaluation/)
16. Wolfgang Machery, Franz Josef Och, Ignacio Thayer, and Jakob Uszkoreit. 2008. Lattice-based minimum error rate training for statistical machine translation. In EMNLP, pages 725–734. [find](https://scholar.google.com/scholar?q=Wolfgang+Machery%2C+Franz+Josef+Och%2C+Ignacio+Thayer%2C+and+Jakob+Uszkoreit%2E+2008%2E+Lattice%2Dbased+minimum+error+rate+training+for+statistical+machine+translation%2E+In+EMNLP%2C+pages+725%E2%80%93734%2E) [our card](https://md.research.elamur.ai/free/en/papers/macherey-2008-lattice-based-minimum-error-rate/)
17. André F. T. Martins, Kevin Gimpel, Noah A. Smith, Eric P. Xing, Pedro M. Q. Aguiar, and Mário A. T. Figueiredo. 2010. Learning structured classifiers with dual coordinate descent. Technical Report CMU-ML-10-109, Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=Andre%CC%81+F%2E+T%2E+Martins%2C+Kevin+Gimpel%2C+Noah+A%2E+Smith%2C+Eric+P%2E+Xing%2C+Pedro+M%2E+Q%2E+Aguiar%2C+and+Ma%CC%81rio+A%2E+T%2E+Figueiredo%2E+2010%2E+Learning+structured+classifiers+with+dual+coordinate+descent%2E+Technical+Report+C)
18. Ryan McDonald, Keith Hall, and Gideon Mann. 2010. Distributed training strategies for the structured perceptron. In ACL, pages 456–464. [find](https://scholar.google.com/scholar?q=Ryan+McDonald%2C+Keith+Hall%2C+and+Gideon+Mann%2E+2010%2E+Distributed+training+strategies+for+the+structured+perceptron%2E+In+ACL%2C+pages+456%E2%80%93464%2E)
19. Franz Joseph Och and Hermann Ney. 2002. Discriminative training and maximum entropy models for statistical machine translation. In ACL, pages 295–302, Philadelphia, PA, July. [find](https://scholar.google.com/scholar?q=Franz+Joseph+Och+and+Hermann+Ney%2E+2002%2E+Discriminative+training+and+maximum+entropy+models+for+statistical+machine+translation%2E+In+ACL%2C+pages+295%E2%80%93302%2C+Philadelphia%2C+PA%2C+July%2E) [our card](https://md.research.elamur.ai/free/en/papers/och-2002-discriminative-training-maximum-entropy-models/)
20. Franz Joseph Och. 2003. Minimum error rate training for statistical machine translation. In ACL, pages 160– 167. [find](https://scholar.google.com/scholar?q=Franz+Joseph+Och%2E+2003%2E+Minimum+error+rate+training+for+statistical+machine+translation%2E+In+ACL%2C+pages+160%E2%80%93+167%2E)
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL, pages 311–318. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+2002%2E+BLEU%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2C+pages+311%E2%80%93318%2E) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
22. Adam Pauls, John Denero, and Dan Klein. 2009. Consensus training for consensus decoding in machine translation. In EMNLP, pages 1418–1427. [find](https://scholar.google.com/scholar?q=Adam+Pauls%2C+John+Denero%2C+and+Dan+Klein%2E+2009%2E+Consensus+training+for+consensus+decoding+in+machine+translation%2E+In+EMNLP%2C+pages+1418%E2%80%931427%2E)
23. Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, and Richard Schwartz. 2011. Expected bleu training for graphs: BBN system description for WMT11 system combination task. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 159– 165. [find](https://scholar.google.com/scholar?q=Antti%2DVeikko+Rosti%2C+Bing+Zhang%2C+Spyros+Matsoukas%2C+and+Richard+Schwartz%2E+2011%2E+Expected+bleu+training+for+graphs%3A+BBN+system+description+for+WMT11+system+combination+task%2E+In+Proceedings+of+the+Sixth+W)
24. Libin Shen, Anoop Sarkar, and Franz Josef Och. 2004. Discriminative reranking for machine translation. In HLT-NAACL, pages 177–184, Boston, Massachusetts, May 2 - May 7. [find](https://scholar.google.com/scholar?q=Libin+Shen%2C+Anoop+Sarkar%2C+and+Franz+Josef+Och%2E+2004%2E+Discriminative+reranking+for+machine+translation%2E+In+HLT%2DNAACL%2C+pages+177%E2%80%93184%2C+Boston%2C+Massachusetts%2C+May+2+%2D+May+7%2E)
25. David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In COLING-ACL, pages 787–794. [find](https://scholar.google.com/scholar?q=David+A%2E+Smith+and+Jason+Eisner%2E+2006%2E+Minimum+risk+annealing+for+training+log%2Dlinear+models%2E+In+COLING%2DACL%2C+pages+787%E2%80%93794%2E)
26. Ioannis Tsochantaridis, Thomas Hofman, Thorsten Joachims, and Yasemin Altun. 2004. Support vector machine learning for interdependent and structured output spaces. In ICML, pages 823–830. [find](https://scholar.google.com/scholar?q=Ioannis+Tsochantaridis%2C+Thomas+Hofman%2C+Thorsten+Joachims%2C+and+Yasemin+Altun%2E+2004%2E+Support+vector+machine+learning+for+interdependent+and+structured+output+spaces%2E+In+ICML%2C+pages+823%E2%80%93830%2E)
27. Taro Watanabe, Jun Suzuki, Hajime Tsukada, and Hideki Isozaki. 2007. Online large-margin training for statistical machine translation. In EMNLP-CoNLL, pages 764–773. [find](https://scholar.google.com/scholar?q=Taro+Watanabe%2C+Jun+Suzuki%2C+Hajime+Tsukada%2C+and+Hideki+Isozaki%2E+2007%2E+Online+large%2Dmargin+training+for+statistical+machine+translation%2E+In+EMNLP%2DCoNLL%2C+pages+764%E2%80%93773%2E)
28. Joern Wuebker, Arne Mauser, and Hermann Ney. 2010. Training phrase translation models with leaving-one-out. In ACL. [find](https://scholar.google.com/scholar?q=Joern+Wuebker%2C+Arne+Mauser%2C+and+Hermann+Ney%2E+2010%2E+Training+phrase+translation+models+with+leaving%2Done%2Dout%2E+In+ACL%2E)
29. Chun-Nam John Yu and Thorsten Joachims. 2009. Learning structural SVMs with latent variables. In ICML. [find](https://scholar.google.com/scholar?q=Chun%2DNam+John+Yu+and+Thorsten+Joachims%2E+2009%2E+Learning+structural+SVMs+with+latent+variables%2E+In+ICML%2E)
30. Richard Zens and Hermann Ney. 2004. Improvements in phrase-based statistical machine translation. In HLT-NAACL, pages 257–264, Boston, USA, May. [find](https://scholar.google.com/scholar?q=Richard+Zens+and+Hermann+Ney%2E+2004%2E+Improvements+in+phrase%2Dbased+statistical+machine+translation%2E+In+HLT%2DNAACL%2C+pages+257%E2%80%93264%2C+Boston%2C+USA%2C+May%2E) [our card](https://md.research.elamur.ai/free/en/papers/zens-2004-improvements-phrase-based-statistical-machine/)
31. Richard Zens, Sǎsa Hasan, and Hermann Ney. 2007. A systematic comparison of training criteria for statistical machine translation. In EMNLP, pages 524–532. [find](https://scholar.google.com/scholar?q=Richard+Zens%2C+Sa%CC%8Csa+Hasan%2C+and+Hermann+Ney%2E+2007%2E+A+systematic+comparison+of+training+criteria+for+statistical+machine+translation%2E+In+EMNLP%2C+pages+524%E2%80%93532%2E)