---
title: "Minimum Risk Annealing for Training Log-Linear Models (2006)"
description: "The paper tackles the mismatch between how NLP systems are evaluated (task metrics such as… 24 references, cited by 81 papers here."
authors: ["David A. Smith", "Jason Eisner"]
year: 2006
venue: "Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions"
source: "https://aclanthology.org/P06-2101/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/smith-2006-minimum-risk-annealing-training-log/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/smith-2006-minimum-risk-annealing-training-log/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / Minimum Risk Annealing for Training Log-Linear Models

# Minimum Risk Annealing for Training Log-Linear Models

**Authors:** [David A. Smith](https://md.research.elamur.ai/free/en/authors/david-smith/) and [Jason Eisner](https://md.research.elamur.ai/free/en/authors/jason-eisner/) · **Year:** 2006 · **Venue:** [Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2006](https://md.research.elamur.ai/free/en/venues/acl/2006/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P06-2101/) · [PDF](https://aclanthology.org/P06-2101.pdf) · [OpenAlex](https://openalex.org/W2137143056)

**Citations:** 136 citations (OpenAlex, 2026-10-14) · [81](https://md.research.elamur.ai/free/en/papers/smith-2006-minimum-risk-annealing-training-log/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles the mismatch between how NLP systems are evaluated (task metrics such as BLEU or labeled attachment accuracy) and how they are usually trained (maximum likelihood). Directly minimizing 1-best error is difficult because the error surface is piecewise flat and full of local optima. The authors instead minimize risk, the expected loss under the model's distribution over a fixed list of candidate analyses. They wrap this in deterministic annealing: a scale or temperature parameter starts the model at a smooth, high-entropy distribution and gradually sharpens it toward the single best hypothesis. They also show how to approximate the expectation of corpus-level nonlinear metrics such as BLEU or precision using a Taylor expansion. The approach is used to tune weights of log-linear combinations of submodels for machine translation (German-English, Finnish-English) and labeled dependency parsing (Bulgarian, Dutch, Slovenian). On MT, annealed minimum risk significantly beats Och-style minimum error training in BLEU. On parsing, the gains are mixed. Across experiments, it was never significantly worse than either baseline.

## Key points
- The models are log-linear rescorers over K-best lists, where each feature can be the log probability of an entire submodel ("products of experts"). A scale parameter γ moves the model from uniform (γ=0), through the ordinary model (γ=1), to Viterbi-like behavior (γ→∞).
- Annealing optimizes expected loss minus T times entropy and cools T over time. Once T is low, γ is raised quickly ("quenching").
- Corpus-level BLEU is handled by minimizing the expectation of −log of the metric. This decomposes into expectations of log counts, which are approximated with a second-order Taylor expansion around their means.
- Parsing setup:
  - Data and parsers: CoNLL-X data, with parsers built from 10 globally normalized log-linear experts.
  - Tuning: weights were tuned on 200-best lists for a 200-sentence dev set.
  - Testing: each language was evaluated on 200 test sentences.
- Labeled accuracy, reported as max-likelihood / min-error / annealed min-risk:

  | Language | Max-likelihood | Min-error | Annealed min-risk |
  |---|---|---|---|
  | Slovenian | 27.78 | 22.52 | 31.16 |
  | Bulgarian | 47.23 | 54.72 | 54.66 |
  | Dutch | 36.78 | 36.78 | 36.71 |

  Only the Slovenian gain for annealing is significant (paired permutation test, 1000 replications).

## Interesting ideas
- **Optimize the expected metric under the model's own distribution.** This gives a smooth, differentiable surrogate for any evaluation measure, including non-decomposable ones. It applies to any reranking or system-combination task where the true metric is piecewise constant.
- **Anneal from high entropy to a peaked distribution.** This continuation strategy helps the optimizer step past local optima that only appear once the objective becomes sharp. It is a cheap alternative to many random restarts in other non-convex tuning problems.
- **Control entropy rather than the scale parameter alone.** In an unregularized log-linear model, the optimizer can cancel γ by shrinking the weights, so annealing must target entropy directly. This pitfall is worth remembering whenever temperature is used as a training knob.
- **Taylor-approximate expectations of ratio metrics.** Turning a corpus-level score into expectations of log counts makes metrics like BLEU, precision, or F-scores trainable as risks.
- **Explain gains through generalization, not better minimization.** The authors note the method sometimes lost on training data yet won on held-out data. They attribute this to several low-loss hypotheses per sentence jointly steering the weights, which is a useful diagnostic for judging a new training objective.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: David A. Smith and Jason Eisner. 2006. Minimum Risk Annealing for Training Log-Linear Models. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{smith2006minimum,
  title = {{Minimum Risk Annealing for Training Log-Linear Models}},
  author = {David A. Smith and Jason Eisner},
  booktitle = {Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions},
  year = {2006},
  doi = {10.3115/1273073.1273174},
  url = {https://doi.org/10.3115/1273073.1273174}
}
```

- [← previous Measuring Agreement on Set-valued Items (MASI) for Semantic and Pragmatic Annotation](https://md.research.elamur.ai/free/en/papers/passonneau-2006-measuring-agreement-set-valued-items/)
- [next → Meaningful Clustering of Senses Helps Boost Word Sense Disambiguation Performance](https://md.research.elamur.ai/free/en/papers/navigli-2006-meaningful-clustering-senses-helps-boost/)
- [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/)

## References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer. 1988. A new algorithm for the estimation of hidden Markov model parameters. In ICASSP, pages 493–496. [find](https://scholar.google.com/scholar?q=L%2E+R%2E+Bahl%2C+P%2E+F%2E+Brown%2C+P%2E+V%2E+de+Souza%2C+and+R%2E+L%2E+Mercer%2E+1988%2E+A+new+algorithm+for+the+estimation+of+hidden+Markov+model+parameters%2E+In+ICASSP%2C+pages+493%E2%80%93496%2E)
2. E. Charniak and M. Johnson. 2005. [Coarse-to-fine n-best parsing and maxent discriminative reranking](https://md.research.elamur.ai/free/en/papers/charniak-2005-coarse-fine-n-best-parsing/). In ACL, pages 173–180.
3. S. F. Chen and R. Rosenfeld. 1999. A gaussian prior for smoothing maximum entropy models. Technical report, CS Dept., Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=S%2E+F%2E+Chen+and+R%2E+Rosenfeld%2E+1999%2E+A+gaussian+prior+for+smoothing+maximum+entropy+models%2E+Technical+report%2C+CS+Dept%2E%2C+Carnegie+Mellon+University%2E)
4. K. Crammer, R. McDonald, and F. Pereira. 2004. New large margin algorithms for structured prediction. In Learning with Structured Outputs (NIPS). [find](https://scholar.google.com/scholar?q=K%2E+Crammer%2C+R%2E+McDonald%2C+and+F%2E+Pereira%2E+2004%2E+New+large+margin+algorithms+for+structured+prediction%2E+In+Learning+with+Structured+Outputs+%28NIPS%29%2E)
5. M. Dreyer, D. A. Smith, and N. A. Smith. 2006. Vine parsing and minimum risk reranking for speed and precision. In CoNLL. [find](https://scholar.google.com/scholar?q=M%2E+Dreyer%2C+D%2E+A%2E+Smith%2C+and+N%2E+A%2E+Smith%2E+2006%2E+Vine+parsing+and+minimum+risk+reranking+for+speed+and+precision%2E+In+CoNLL%2E)
6. G. Elidan and N. Friedman. 2005. Learning hidden variable networks: The information bottleneck approach. JMLR, 6:81–127. [find](https://scholar.google.com/scholar?q=G%2E+Elidan+and+N%2E+Friedman%2E+2005%2E+Learning+hidden+variable+networks%3A+The+information+bottleneck+approach%2E+JMLR%2C+6%3A81%E2%80%93127%2E)
7. V. Goel and W. J. Byrne. 2000. Minimum Bayes-Risk automatic speech recognition. Computer Speech and Language, 14(2):115–135. [find](https://scholar.google.com/scholar?q=V%2E+Goel+and+W%2E+J%2E+Byrne%2E+2000%2E+Minimum+Bayes%2DRisk+automatic+speech+recognition%2E+Computer+Speech+and+Language%2C+14%282%29%3A115%E2%80%93135%2E)
8. J. T. Goodman. 1996. Parsing algorithms and metrics. In ACL, pages 177–183. [find](https://scholar.google.com/scholar?q=J%2E+T%2E+Goodman%2E+1996%2E+Parsing+algorithms+and+metrics%2E+In+ACL%2C+pages+177%E2%80%93183%2E)
9. G. Hinton. 1999. Products of experts. In Proc. of ICANN, volume 1, pages 1–6. [find](https://scholar.google.com/scholar?q=G%2E+Hinton%2E+1999%2E+Products+of+experts%2E+In+Proc%2E+of+ICANN%2C+volume+1%2C+pages+1%E2%80%936%2E)
10. K.-U. Hoffgen, H.-U. Simon, and K. S. Van Horn. 1995. Robust trainability of single neurons. J. of Computer and System Sciences, 50(1):114–125. [find](https://scholar.google.com/scholar?q=K%2E%2DU%2E+Hoffgen%2C+H%2E%2DU%2E+Simon%2C+and+K%2E+S%2E+Van+Horn%2E+1995%2E+Robust+trainability+of+single+neurons%2E+J%2E+of+Computer+and+System+Sciences%2C+50%281%29%3A114%E2%80%93125%2E)
11. D. S. Johnson and F. P. Preparata. 1978. The densest hemisphere problem. Theoretical Comp. Sci., 6(93–107). [find](https://scholar.google.com/scholar?q=D%2E+S%2E+Johnson+and+F%2E+P%2E+Preparata%2E+1978%2E+The+densest+hemisphere+problem%2E+Theoretical+Comp%2E+Sci%2E%2C+6%2893%E2%80%93107%29%2E)
12. S. Katagiri, B.-H. Juang, and C.-H. Lee. 1998. Pattern recognition using a family of design algorithms based upon the generalized probabilistic descent method. Proc. IEEE, 86(11):2345–2373, November. [find](https://scholar.google.com/scholar?q=S%2E+Katagiri%2C+B%2E%2DH%2E+Juang%2C+and+C%2E%2DH%2E+Lee%2E+1998%2E+Pattern+recognition+using+a+family+of+design+algorithms+based+upon+the+generalized+probabilistic+descent+method%2E+Proc%2E+IEEE%2C+86%2811%29%3A2345%E2%80%932373%2C+November%2E)
13. P. Koehn, F. J. Och, and D. Marcu. 2003. Statistical phrase-based translation. In HLT-NAACL, pages 48–54. [find](https://scholar.google.com/scholar?q=P%2E+Koehn%2C+F%2E+J%2E+Och%2C+and+D%2E+Marcu%2E+2003%2E+Statistical+phrase%2Dbased+translation%2E+In+HLT%2DNAACL%2C+pages+48%E2%80%9354%2E)
14. S. Kumar and W. Byrne. 2004. [Minimum bayes-risk decoding for statistical machine translation](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/). In HLT-NAACL.
15. J. Lafferty, A. McCallum, and F. C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. [find](https://scholar.google.com/scholar?q=J%2E+Lafferty%2C+A%2E+McCallum%2C+and+F%2E+C%2E+N%2E+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+ICML%2E)
16. F. J. Och. 2003. Minimum error rate training in statistical machine translation. In ACL, pages 160–167. [find](https://scholar.google.com/scholar?q=F%2E+J%2E+Och%2E+2003%2E+Minimum+error+rate+training+in+statistical+machine+translation%2E+In+ACL%2C+pages+160%E2%80%93167%2E)
17. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. B LEU : A method for automatic evaluation of machine translation. In ACL, pages 311–318. [find](https://scholar.google.com/scholar?q=K%2E+Papineni%2C+S%2E+Roukos%2C+T%2E+Ward%2C+and+W%2E%2DJ%2E+Zhu%2E+2002%2E+B+LEU+%3A+A+method+for+automatic+evaluation+of+machine+translation%2E+In+ACL%2C+pages+311%E2%80%93318%2E)
18. K. A. Papineni. 1999. Discriminative training via linear programming. In ICASSP. [find](https://scholar.google.com/scholar?q=K%2E+A%2E+Papineni%2E+1999%2E+Discriminative+training+via+linear+programming%2E+In+ICASSP%2E)
19. A. Rao and K. Rose. 2001. Deterministically annealed design of Hidden Markov Model speech recognizers. IEEE Trans. on Speech and Audio Processing, 9(2):111–126. [find](https://scholar.google.com/scholar?q=A%2E+Rao+and+K%2E+Rose%2E+2001%2E+Deterministically+annealed+design+of+Hidden+Markov+Model+speech+recognizers%2E+IEEE+Trans%2E+on+Speech+and+Audio+Processing%2C+9%282%29%3A111%E2%80%93126%2E)
20. K. Rose. 1998. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proc. IEEE, 86(11):2210–2239. [find](https://scholar.google.com/scholar?q=K%2E+Rose%2E+1998%2E+Deterministic+annealing+for+clustering%2C+compression%2C+classification%2C+regression%2C+and+related+optimization+problems%2E+Proc%2E+IEEE%2C+86%2811%29%3A2210%E2%80%932239%2E)
21. N. A. Smith and J. Eisner. 2004. Annealing techniques for unsupervised statistical language learning. In ACL, pages 486–493. [find](https://scholar.google.com/scholar?q=N%2E+A%2E+Smith+and+J%2E+Eisner%2E+2004%2E+Annealing+techniques+for+unsupervised+statistical+language+learning%2E+In+ACL%2C+pages+486%E2%80%93493%2E)
22. A. Smith, T. Cohn, and M. Osborne. 2005. Logarithmic opinion pools for conditional random fields. In ACL, pages 18–25. [find](https://scholar.google.com/scholar?q=A%2E+Smith%2C+T%2E+Cohn%2C+and+M%2E+Osborne%2E+2005%2E+Logarithmic+opinion+pools+for+conditional+random+fields%2E+In+ACL%2C+pages+18%E2%80%9325%2E)
23. B. Taskar, D. Klein, M. Collins, D. Koller, and C. Manning. 2004. Max-margin parsing. In EMNLP, pages 1–8. [find](https://scholar.google.com/scholar?q=B%2E+Taskar%2C+D%2E+Klein%2C+M%2E+Collins%2C+D%2E+Koller%2C+and+C%2E+Manning%2E+2004%2E+Max%2Dmargin+parsing%2E+In+EMNLP%2C+pages+1%E2%80%938%2E)
24. N. Ueda and R. Nakano. 1998. Deterministic annealing EM algorithm. Neural Networks, 11(2):271–282. [find](https://scholar.google.com/scholar?q=N%2E+Ueda+and+R%2E+Nakano%2E+1998%2E+Deterministic+annealing+EM+algorithm%2E+Neural+Networks%2C+11%282%29%3A271%E2%80%93282%2E)

*Updated 24 October 2026*