---
title: "Posterior calibration and exploratory analysis for natural language"
description: "The authors propose a practical way to measure miscalibration based on the calibration term of the… 48 references, cited by 47 papers here."
authors: ["Khanh Nguyen", "Brendan O'Connor"]
year: 2015
venue: "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D15-1182/"
doi: "10.18653/v1/d15-1182"
arxiv: "1508.05154"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nguyen-2015-posterior-calibration-exploratory-analysis-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nguyen-2015-posterior-calibration-exploratory-analysis-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Posterior calibration and exploratory analysis for natural language processing models

# Posterior calibration and exploratory analysis for natural language processing models

**Authors:** Khanh Nguyen and [Brendan O'Connor](https://md.research.elamur.ai/free/en/authors/brendan-o-connor/) · **Year:** 2015 · **Venue:** [Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2015](https://md.research.elamur.ai/free/en/venues/emnlp/2015/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D15-1182/) · [PDF](https://aclanthology.org/D15-1182.pdf) · [DOI 10.18653/v1/d15-1182](https://doi.org/10.18653/v1/d15-1182) · [arXiv 1508.05154](https://arxiv.org/abs/1508.05154) · [OpenAlex](https://openalex.org/W1933798900)

**Citations:** 92 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/nguyen-2015-posterior-calibration-exploratory-analysis-natural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Probabilistic NLP models produce full distributions over outputs, yet they are usually judged only by their single best prediction. Nguyen and O'Connor argue that these posteriors should be checked directly for calibration, meaning that events predicted with, say, 70% confidence should actually occur about 70% of the time. Overconfident models weaken any downstream method that relies on lower-ranked alternatives, such as K-best lists, sampling or minimum Bayes risk decoding.

The authors propose a practical way to measure miscalibration based on the calibration term of the Brier score decomposition. Because NLP confidence scores are heavily skewed toward 0 and 1, they replace fixed-width bins with equal-count adaptive bins and add a simulation-based confidence interval. They use this to compare common generative and discriminative tagging and classification models.

They also introduce a coreference sampling algorithm that carries uncertainty into a political event extraction task on New York Times articles, yielding credible intervals for event trends. The work matters because calibrated uncertainty tells analysts when an NLP-based conclusion can be trusted.

## Key points
- **Calibration metric:** CalibErr is defined as the RMS gap between predicted probability and empirical label frequency. It comes from the calibration–refinement decomposition of the Brier score (DeGroot and Fienberg, 1983).
- **Adaptive binning (Algorithm 1):** Each bin holds an equal number of points (bin size β), which keeps standard errors bounded by 0.5/√β across the probability range.
- **Confidence interval (Algorithm 2):** The 95% interval for calibration error is obtained by sampling per-bin label frequencies from a binomial-normal approximation.
- **Exploratory analysis:** Coreference posterior samples produce 95% credible intervals for counts of "attack" events per 3-month period. For example, the Yugoslavia and NATO counts in mid-1999 overlap heavily, so a 1-best system would suggest a difference the model cannot actually support.
- **Error analysis:** Event predictions near 50% probability expose difficult coreference cases. One example is "Mr. Hussein" being linked to Iraq in only about half of the samples.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 83 works · 23,784 citations
- [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 67 works · 64,591 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nguyen2015posterior,
  title = {{Posterior calibration and exploratory analysis for natural language processing models}},
  author = {Khanh Nguyen and Brendan O'Connor},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1182},
  eprint = {1508.05154},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1182}
}
```

- [← previous Investigating the distribution of some (but not all) implicatures using corpora and web-based methods](https://md.research.elamur.ai/free/en/papers/degen-2015-investigating-distribution-some-but-not/)
- [next → Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2015-generating-high-quality-proposition-banks/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

48 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Bamman, Brendan O’Connor, and Noah A. Smith. [Learning latent personas of film characters](https://md.research.elamur.ai/free/en/papers/bamman-2013-learning-latent-personas-film-characters/). In Proceedings of ACL, 2013.
2. Paul N. Bennett. Assessing the calibration of naive Bayes’ posterior estimates. Technical report, Carnegie Mellon University, 2000. [find](https://scholar.google.com/scholar?q=Paul+N%2E+Bennett%2E+Assessing+the+calibration+of+naive+Bayes%E2%80%99+posterior+estimates%2E+Technical+report%2C+Carnegie+Mellon+University%2C+2000%2E)
3. David M. Blei and Peter I. Frazier. Distance dependent Chinese restaurant processes. The Journal of Machine Learning Research, 12: 2461–2488, 2011. [find](https://scholar.google.com/scholar?q=David+M%2E+Blei+and+Peter+I%2E+Frazier%2E+Distance+dependent+Chinese+restaurant+processes%2E+The+Journal+of+Machine+Learning+Research%2C+12%3A+2461%E2%80%932488%2C+2011%2E)
4. Elizabeth Boschee, Premkumar Natarajan, and Ralph Weischedel. Automatic extraction of events from open source text for predictive forecasting. Handbook of Computational Approaches to Counterterrorism, page 51, 2013. [find](https://scholar.google.com/scholar?q=Elizabeth+Boschee%2C+Premkumar+Natarajan%2C+and+Ralph+Weischedel%2E+Automatic+extraction+of+events+from+open+source+text+for+predictive+forecasting%2E+Handbook+of+Computational+Approaches+to+Counterterrorism%2C)
5. Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. [find](https://scholar.google.com/scholar?q=Glenn+W%2E+Brier%2E+Verification+of+forecasts+expressed+in+terms+of+probability%2E+Monthly+weather+review%2C+78%281%29%3A1%E2%80%933%2C+1950%2E)
6. Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643):1512–1519, 2009. [find](https://scholar.google.com/scholar?q=Jochen+Bro%CC%88cker%2E+Reliability%2C+sufficiency%2C+and+the+decomposition+of+proper+scores%2E+Quarterly+Journal+of+the+Royal+Meteorological+Society%2C+135%28643%29%3A1512%E2%80%931519%2C+2009%2E)
7. Niko Brümmer and George Doddington. Likelihood-ratio calibration using prior-weighted proper scoring rules. arXiv preprint arXiv:1307.7981, 2013. — [arXiv:1307.7981](https://arxiv.org/abs/1307.7981) [find](https://scholar.google.com/scholar?q=Niko+Bru%CC%88mmer+and+George+Doddington%2E+Likelihood%2Dratio+calibration+using+prior%2Dweighted+proper+scoring+rules%2E+arXiv+preprint+arXiv%3A1307%2E7981%2C+2013%2E+%E2%80%94+%5BarXiv%3A1307%2E7981%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1307%2E7981%29)
8. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. [Universal Stanford dependencies: A cross-linguistic typology](https://md.research.elamur.ai/free/en/papers/marneffe-2014-universal-stanford-dependencies-cross-linguistic/). In Proceedings of LREC, 2014.
9. Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. The statistician, pages 12–22, 1983. [find](https://scholar.google.com/scholar?q=Morris+H%2E+DeGroot+and+Stephen+E%2E+Fienberg%2E+The+comparison+and+evaluation+of+forecasters%2E+The+statistician%2C+pages+12%E2%80%9322%2C+1983%2E)
10. Pedro Domingos and Michael Pazzani. On the optimality of the simple Bayesian classifier under zero-one loss. Machine learning, 29(2-3):103– 130, 1997. [find](https://scholar.google.com/scholar?q=Pedro+Domingos+and+Michael+Pazzani%2E+On+the+optimality+of+the+simple+Bayesian+classifier+under+zero%2Done+loss%2E+Machine+learning%2C+29%282%2D3%29%3A103%E2%80%93+130%2C+1997%2E)
11. Greg Durrett and Dan Klein. [Easy victories and uphill battles in coreference resolution](https://md.research.elamur.ai/free/en/papers/durrett-2013-easy-victories-uphill-battles-coreference/). In EMNLP, pages 1971–1982, 2013.
12. Greg Durrett and Dan Klein. [A joint model for entity analysis: Coreference, typing, and linking](https://md.research.elamur.ai/free/en/papers/durrett-2014-joint-model-entity-analysis-coreference/). Transactions of the Association for Computational Linguistics, 2:477–490, 2014.
13. Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 618– 626. Association for Computa… [find](https://scholar.google.com/scholar?q=Jenny+Rose+Finkel%2C+Christopher+D%2E+Manning%2C+and+Andrew+Y%2E+Ng%2E+Solving+the+problem+of+cascading+errors%3A+Approximate+Bayesian+inference+for+linguistic+annotation+pipelines%2E+In+Proceedings+of+the+2006+Con)
14. Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian data analysis. Chapman and Hall/CRC, 3rd edition, 2013. [find](https://scholar.google.com/scholar?q=Andrew+Gelman%2C+John+B%2E+Carlin%2C+Hal+S%2E+Stern%2C+David+B%2E+Dunson%2C+Aki+Vehtari%2C+and+Donald+B%2E+Rubin%2E+Bayesian+data+analysis%2E+Chapman+and+Hall%2FCRC%2C+3rd+edition%2C+2013%2E)
15. Kevin Gimpel and Noah A. Smith. Rich source-side context for statistical machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 9–17, 2008. [find](https://scholar.google.com/scholar?q=Kevin+Gimpel+and+Noah+A%2E+Smith%2E+Rich+source%2Dside+context+for+statistical+machine+translation%2E+In+Proceedings+of+the+Third+Workshop+on+Statistical+Machine+Translation%2C+pages+9%E2%80%9317%2C+2008%2E)
16. Kevin Gimpel and Noah A. Smith. Softmax-margin CRFs: Training log-linear models with cost functions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 733–736. Association for Computational Linguistics, 20… [find](https://scholar.google.com/scholar?q=Kevin+Gimpel+and+Noah+A%2E+Smith%2E+Softmax%2Dmargin+CRFs%3A+Training+log%2Dlinear+models+with+cost+functions%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+Chapter+of+the+Ass)
17. Kevin Gimpel, Dhruv Batra, Chris Dyer, and Gregory Shakhnarovich. A systematic exploration of diversity in machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1100–1111, Seattle, Washington, USA, October 2013. Association for Computat… [find](https://scholar.google.com/scholar?q=Kevin+Gimpel%2C+Dhruv+Batra%2C+Chris+Dyer%2C+and+Gregory+Shakhnarovich%2E+A+systematic+exploration+of+diversity+in+machine+translation%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+La)
18. Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. [find](https://scholar.google.com/scholar?q=Tilmann+Gneiting+and+Adrian+E%2E+Raftery%2E+Strictly+proper+scoring+rules%2C+prediction%2C+and+estimation%2E+Journal+of+the+American+Statistical+Association%2C+102%28477%29%3A359%E2%80%93378%2C+2007%2E)
19. Joshua Goodman. Parsing algorithms and metrics. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 177–183, Santa Cruz, California, USA, June 1996. Association for Computational Linguistics. doi: 10.3115/981863.981887. URL [http://www.aclweb.org/antholog…](http://www.aclweb.org/antholog…) [↗](http://www.aclweb.org/antholog…)
20. Aria Haghighi and Dan Klein. [Unsupervised coreference resolution in a nonparametric Bayesian model](https://md.research.elamur.ai/free/en/papers/haghighi-2007-unsupervised-coreference-resolution-nonparametric-bayesian/). In Annual Meeting, Association for Computational Linguistics, volume 45, page 848, 2007.
21. Robert E. Kass, Bradley P. Carlin, Andrew Gelman, and Radford M. Neal. Markov chain Monte Carlo in practice: a roundtable discussion. The American Statistician, 52(2):93–100, 1998. [find](https://scholar.google.com/scholar?q=Robert+E%2E+Kass%2C+Bradley+P%2E+Carlin%2C+Andrew+Gelman%2C+and+Radford+M%2E+Neal%2E+Markov+chain+Monte+Carlo+in+practice%3A+a+roundtable+discussion%2E+The+American+Statistician%2C+52%282%29%3A93%E2%80%93100%2C+1998%2E)
22. Shankar Kumar and William Byrne. [Minimum Bayes-risk decoding for statistical machine translation](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/). In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 169–176, Boston, Massachusetts, USA, May 2 - May 7 2004. Association for Computational Linguistics.
23. Kalev Leetaru and Philip A. Schrodt. GDELT: Global data on events, location, and tone, 1979– 2012. In ISA Annual Convention, volume 2, page 4, 2013. [find](https://scholar.google.com/scholar?q=Kalev+Leetaru+and+Philip+A%2E+Schrodt%2E+GDELT%3A+Global+data+on+events%2C+location%2C+and+tone%2C+1979%E2%80%93+2012%2E+In+ISA+Annual+Convention%2C+volume+2%2C+page+4%2C+2013%2E)
24. Jimmy Lin and Alek Kolcz. Large-scale machine learning at Twitter. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pages 793–804. [find](https://scholar.google.com/scholar?q=Jimmy+Lin+and+Alek+Kolcz%2E+Large%2Dscale+machine+learning+at+Twitter%2E+In+Proceedings+of+the+2012+ACM+SIGMOD+International+Conference+on+Management+of+Data%2C+pages+793%E2%80%93804%2E)
25. Michael C. McCord, J. William Murdock, and Branimir K. Boguraev. Deep parsing in Watson. IBM Journal of Research and Development, 56 (3.4):3–1, 2012. [find](https://scholar.google.com/scholar?q=Michael+C%2E+McCord%2C+J%2E+William+Murdock%2C+and+Branimir+K%2E+Boguraev%2E+Deep+parsing+in+Watson%2E+IBM+Journal+of+Research+and+Development%2C+56+%283%2E4%29%3A3%E2%80%931%2C+2012%2E)
26. David Mimno and David Blei. Bayesian checking for topic models. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 227–237, Edinburgh, Scotland, UK., July 2011. Association for Computational Linguistics. URL [http://www.aclweb.org/anthology/D11-1021](http://www.aclweb.org/anthology/D11-1021). [↗](http://www.aclweb.org/anthology/D11-1021)
27. Makoto Miwa, Sampo Pyysalo, Tadayoshi Hara, and Jun’ichi Tsujii. Evaluating dependency representations for event extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 779–787, Beijing, China, August 2010. [find](https://scholar.google.com/scholar?q=Makoto+Miwa%2C+Sampo+Pyysalo%2C+Tadayoshi+Hara%2C+and+Jun%E2%80%99ichi+Tsujii%2E+Evaluating+dependency+representations+for+event+extraction%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Lingui)
28. Coling 2010 Organizing Committee. URL [http://www.aclweb.org/anthology/C10-1088](http://www.aclweb.org/anthology/C10-1088). [↗](http://www.aclweb.org/anthology/C10-1088)
29. Allan H. Murphy and Robert L. Winkler. A general framework for forecast verification. Monthly Weather Review, 115(7):1330–1338, 1987. [find](https://scholar.google.com/scholar?q=Allan+H%2E+Murphy+and+Robert+L%2E+Winkler%2E+A+general+framework+for+forecast+verification%2E+Monthly+Weather+Review%2C+115%287%29%3A1330%E2%80%931338%2C+1987%2E)
30. Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes. Advances in neural information processing systems, 14: 841, 2002. [find](https://scholar.google.com/scholar?q=Andrew+Ng+and+Michael+Jordan%2E+On+discriminative+vs%2E+generative+classifiers%3A+A+comparison+of+logistic+regression+and+naive+Bayes%2E+Advances+in+neural+information+processing+systems%2C+14%3A+841%2C+2002%2E)
31. Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning, pages 625–632, 2005. [find](https://scholar.google.com/scholar?q=Alexandru+Niculescu%2DMizil+and+Rich+Caruana%2E+Predicting+good+probabilities+with+supervised+learning%2E+In+Proceedings+of+the+22nd+International+Conference+on+Machine+Learning%2C+pages+625%E2%80%93632%2C+2005%2E)
32. Brendan O’Connor, Brandon Stewart, and Noah A. Smith. Learning to extract international relations from political context. In Proceedings of ACL, 2013. [find](https://scholar.google.com/scholar?q=Brendan+O%E2%80%99Connor%2C+Brandon+Stewart%2C+and+Noah+A%2E+Smith%2E+Learning+to+extract+international+relations+from+political+context%2E+In+Proceedings+of+ACL%2C+2013%2E)
33. Naoaki Okazaki. Crfsuite: a fast implementation of conditional random fields (CRFs), 2007. URL [http://www.chokkan.org/software/crfsuite/](http://www.chokkan.org/software/crfsuite/). [↗](http://www.chokkan.org/software/crfsuite/)
34. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
35. John Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in large margin classifiers. MIT Press (2000), 1999. URL [http://research.microsoft.com/pubs/69187/svmprob.ps.gz](http://research.microsoft.com/pubs/69187/svmprob.ps.gz). [↗](http://research.microsoft.com/pubs/69187/svmprob.ps.gz)
36. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. [CoNLL-2011 shared task: Modeling unrestricted coreference in Ontonotes](https://md.research.elamur.ai/free/en/papers/pradhan-2011-conll-2011-shared-task-modeling/). In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27. Association for Co…
37. Jonathon Read. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In Proceedings of the ACL Student Research Workshop, pages 43–48. Association for Computational Linguistics, 2005. [find](https://scholar.google.com/scholar?q=Jonathon+Read%2E+Using+emoticons+to+reduce+dependency+in+machine+learning+techniques+for+sentiment+classification%2E+In+Proceedings+of+the+ACL+Student+Research+Workshop%2C+pages+43%E2%80%9348%2E+Association+for+Compu)
38. Evan Sandhaus. The New York Times Annotated Corpus. Linguistic Data Consortium, LDC2008T19, 2008. [find](https://scholar.google.com/scholar?q=Evan+Sandhaus%2E+The+New+York+Times+Annotated+Corpus%2E+Linguistic+Data+Consortium%2C+LDC2008T19%2C+2008%2E)
39. Philip A. Schrodt. Precedents, progress, and prospects in political event data. International Interactions, 38(4):546–569, 2012. [find](https://scholar.google.com/scholar?q=Philip+A%2E+Schrodt%2E+Precedents%2C+progress%2C+and+prospects+in+political+event+data%2E+International+Interactions%2C+38%284%29%3A546%E2%80%93569%2C+2012%2E)
40. Philip A. Schrodt, Shannon G. Davis, and Judith L. Weddle. KEDS – a program for the machine coding of event data. Social Science Computer Review, 12(4):561 –587, December 1994. doi: 10.1177/089443939401200408. URL [http://ssc.sagepub.com/content/12/4/561.abstract](http://ssc.sagepub.com/content/12/4/561.abstract). — [doi:10.1177/089443939401200408](https://doi.org/10.1177/089443939401200408) [↗](http://ssc.sagepub.com/content/12/4/561.abstract)
41. Sameer Singh, Sebastian Riedel, Brian Martin, Jiaping Zheng, and Andrew McCallum. Joint inference of entities, relations, and coreference. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, pages 1– 6. [find](https://scholar.google.com/scholar?q=Sameer+Singh%2C+Sebastian+Riedel%2C+Brian+Martin%2C+Jiaping+Zheng%2C+and+Andrew+McCallum%2E+Joint+inference+of+entities%2C+relations%2C+and+coreference%2E+In+Proceedings+of+the+2013+Workshop+on+Automated+Knowledge+Ba)
42. David A. Smith and Jason Eisner. [Minimum risk annealing for training log-linear models](https://md.research.elamur.ai/free/en/papers/smith-2006-minimum-risk-annealing-training-log/). In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787–794, Sydney, Australia, July 2006. Association for Computational Linguistics. URL [http://www.aclweb.org/anthology/P06-2101](http://www.aclweb.org/anthology/P06-2101). [↗](http://www.aclweb.org/anthology/P06-2101)
43. Veselin Stoyanov, Alexander Ropson, and Jason Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In International Conference on Artificial Intelligence and Statistics, pages 725–733, 2011. [find](https://scholar.google.com/scholar?q=Veselin+Stoyanov%2C+Alexander+Ropson%2C+and+Jason+Eisner%2E+Empirical+risk+minimization+of+graphical+model+parameters+given+approximate+inference%2C+decoding%2C+and+model+structure%2E+In+International+Conference+)
44. Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. [A global joint model for semantic role labeling](https://md.research.elamur.ai/free/en/papers/toutanova-2008-global-joint-model-semantic-role/). Computational Linguistics, 34(2):161–191, 2008.
45. John W. Tukey. Curves as parameters, and touch estimation. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, pages 681–694, Berkeley, Calif., 1961. University of California Press. URL [http://projecteuclid.…](http://projecteuclid.…) [↗](http://projecteuclid.…)
46. Ashish Venugopal, Andreas Zollmann, Noah A. Smith, and Stephan Vogel. Wider pipelines: Nbest alignments and parses in MT training. In Proceedings of AMTA, 2008. [find](https://scholar.google.com/scholar?q=Ashish+Venugopal%2C+Andreas+Zollmann%2C+Noah+A%2E+Smith%2C+and+Stephan+Vogel%2E+Wider+pipelines%3A+Nbest+alignments+and+parses+in+MT+training%2E+In+Proceedings+of+AMTA%2C+2008%2E)
47. Larry Wasserman. All of nonparametric statistics. Springer Science & Business Media, 2006. [find](https://scholar.google.com/scholar?q=Larry+Wasserman%2E+All+of+nonparametric+statistics%2E+Springer+Science+%26+Business+Media%2C+2006%2E)
48. Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of KDD, pages 694–699. ACM, 2002. [find](https://scholar.google.com/scholar?q=Bianca+Zadrozny+and+Charles+Elkan%2E+Transforming+classifier+scores+into+accurate+multiclass+probability+estimates%2E+In+Proceedings+of+KDD%2C+pages+694%E2%80%93699%2E+ACM%2C+2002%2E)

*Updated 24 October 2026*