---
title: "Automatic Evaluation of Summaries Using N-gram — Lin 2003"
description: "Lin and Hovy ask whether summaries can be scored automatically, the way the machine translation community had begun using BLEU/NIST, instead of relying…"
authors: ["Chin-Yew Lin", "Eduard Hovy"]
year: 2003
venue: "Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics"
source: "https://aclanthology.org/N03-1020/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

# Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics

**Authors:** [Chin-Yew Lin](https://md.research.elamur.ai/free/en/authors/chin-yew-lin/), [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/) · **Year:** 2003 · **Venue:** Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/N03-1020/) · [PDF](https://aclanthology.org/N03-1020.pdf) · [OpenAlex](https://openalex.org/W2150824314)

**Citations:** 1578 citations (openalex-snapshot · 2026-10-14) · [389](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Lin and Hovy ask whether summaries can be scored automatically, the way the machine translation community had begun using BLEU/NIST, instead of relying on expensive and inconsistent human judges. They use DUC 2001 data, covering single-document summaries of 100 words and multi-document summaries of several lengths, with human coverage scores collected in the SEE tool as ground truth. Because DUC judging asks how much of a model summary a peer recovers, they turn BLEU's clipped n-gram precision into a recall measure computed against model units. This gives a family of scores, Ngram(i, j). They test these scores against two requirements: system rankings should match human rankings, and significance decisions should match those based on human scores. Running BLEU without changes correlates positively with human judgments, but not reliably. Simple unigram recall, Ngram(1,1), shows the strongest and most stable agreement. This made cheap, repeatable summary evaluation look feasible and laid the groundwork for later recall-based n-gram metrics.

## Key points
- Human ground truth is a coverage score C. It counts the model units an assessor marks as matched, each multiplied by a completeness factor E that equals 1 when the unit is fully covered, 3/4 when most of it is, 1/2 when some of it is, 1/4 when hardly any of it is, and 0 when none of it is.
- Unmodified IBM BLEU on DUC 2001 reached a Spearman ρ of 0.66 with one reference and 0.82 with three references on the single-document task. On the multi-document task it reached 0.67 and 0.70. All four values were significant at α = 0.01.
- The rankings cover 14 instances in the single-document task (11 systems, 2 humans, 1 baseline) and 16 in the multi-document task (12 systems, 2 humans, 2 baselines).
- The BLEU-like Ngram(1,4) gave Spearman values between about 0.53 and 0.88 depending on the task and summary length. Results were computed both with and without stopwords.
- In recall/precision plots of significance agreement (z-tests at several α levels), Ngram(1,1) and Ngram(2,2) fall in the best region, and Ngram(1,1) is the best overall.

## Interesting ideas
- **Meta-evaluate a metric by its significance decisions, not just correlation.** The method counts system pairs that the metric calls significantly different and checks how often human scores agree, giving a recall and a precision. This tells you whether a cheap metric can be trusted to declare one system better than another, and the check applies to any automatic metric.
- **Match the metric's orientation to the task.** BLEU rewards precision because a translation should not add wrong material. Summary judging asks how much of the reference content was captured, so the authors flip the denominator to the reference side. Before adopting a metric from a neighbouring field, ask whether the human protocol is precision-like or recall-like.
- **Longer n-grams may measure fluency rather than content.** The authors argue that extractive summaries copy grammatical sentences, so 3- and 4-gram matches add little information about content. When outputs are mostly copied text, plain unigram overlap may be enough, and adding more complex components should be justified with evidence.
- **Partial-credit human judgments.** Graded completeness labels (all, most, some, hardly any) turn subjective matching into a numeric score. This is a lightweight annotation design that could be reused for other content-coverage tasks.
- **A recurring "metric track" built on existing human judgments.** Each year's manual results would be used to rank newly proposed automatic metrics, and the winning metric would serve as the in-house evaluator for the following year. This gets more value from costly annotation; NIST spent about 3,000 hours on each DUC.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/)

## Cited by

Cited by

389 papers of our library cite it (of the 140884 whose bibliographies we read); 19 of them have a page here, 370 not yet.

- [Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised](https://md.research.elamur.ai/free/en/papers/angelidis-2018-summarizing-opinions-aspect-extraction-meets/)
- [An Actor-Critic Algorithm for Sequence Prediction](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/)
- [Sentence Fusion for Multidocument News Summarization](https://md.research.elamur.ai/free/en/papers/barzilay-2005-sentence-fusion-multidocument-news-summarization/)
- [Modeling Local Coherence: An Entity-Based Approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/)
- [Neural Summarization by Extracting Sentences and Words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/)
- [Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints](https://md.research.elamur.ai/free/en/papers/durrett-2016-learning-based-single-document-summarization/)
- [Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/)
- [Automatic Evaluation of Translation Quality for Distant Language Pairs](https://md.research.elamur.ai/free/en/papers/isozaki-2010-automatic-evaluation-translation-quality-distant/)
- [Computational Generation of Referring Expressions: A Survey](https://md.research.elamur.ai/free/en/papers/krahmer-2012-computational-generation-referring-expressions-survey/)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
- [Automatically Assessing Machine Summary Content Without a Gold Standard](https://md.research.elamur.ai/free/en/papers/louis-2013-automatically-assessing-machine-summary-content/)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://md.research.elamur.ai/free/en/papers/lu-2022-neurologic-esque-decoding-constrained-text/)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
- [Planning with Learned Entity Prompts for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/narayan-2021-planning-learned-entity-prompts-abstractive/)
- [Sequence Level Training with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/)
- [Heterogeneous Graph Neural Networks for Extractive Document Summarization](https://md.research.elamur.ai/free/en/papers/wang-2020-heterogeneous-graph-neural-networks-extractive/)
- [BARTScore: Evaluating Generated Text as Text Generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/)
- [Extractive Summarization as Text Matching](https://md.research.elamur.ai/free/en/papers/zhong-2020-extractive-summarization-as-text-matching/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lin2003automatic,
  title = {{Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics}},
  author = {Lin, Chin-Yew and Hovy, Eduard},
  booktitle = {Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
  doi = {10.3115/1073445.1073465},
  url = {https://doi.org/10.3115/1073445.1073465}
}
```

- [← previous Arabic Syntactic Trees: from Constituency to Dependency](https://md.research.elamur.ai/free/en/papers/map-2003-zabokrtsky-arabic-syntactic-trees-constituency-dependency/)
- [next → Automatic Pronominal Anaphora Resolution in English Texts](https://md.research.elamur.ai/free/en/papers/liang-2003-automatic-pronominal-anaphora-resolution-english/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/)

## References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Donaway, R.L., Drummey, K.W., and Mather, L.A. 2000. A Comparison of Rankings Produced by Summarization Evaluation Measures. In Proceeding of the Workshop on Automatic Summarization, post-conference workshop of ANLP-NAACL-2000, pp. 69-78, Seattle, WA, 2000. [find](https://scholar.google.com/scholar?q=Donaway%2C+R%2EL%2E%2C+Drummey%2C+K%2EW%2E%2C+and+Mather%2C+L%2EA%2E+2000%2E+A+Comparison+of+Rankings+Produced+by+Summarization+Evaluation+Measures%2E+In+Proceeding+of+the+Workshop+on+Automatic+Summarization%2C+post%2Dconference+w)
2. DUC. 2002. The Document Understanding Conference. [http://duc.nist.gov](http://duc.nist.gov). [↗](http://duc.nist.gov)
3. Fukusima, T. and Okumura, M. 2001. Text Summarization Challenge: Text Summarization Evaluation at NTCIR Workshop2. In Proceedings of the Second NTCIR Workshop on Research in Chinese & Japanese Text Retrieval and Text Summarization, NII, Tokyo, Japan, 2001. [find](https://scholar.google.com/scholar?q=Fukusima%2C+T%2E+and+Okumura%2C+M%2E+2001%2E+Text+Summarization+Challenge%3A+Text+Summarization+Evaluation+at+NTCIR+Workshop2%2E+In+Proceedings+of+the+Second+NTCIR+Workshop+on+Research+in+Chinese+%26+Japanese+Text+Re)
4. Lin, C.-Y. 2001. Summary Evaluation Environment. [http://www.isi.edu/~cyl/SEE](http://www.isi.edu/~cyl/SEE). [↗](http://www.isi.edu/~cyl/SEE)
5. Lin, C.-Y. and E. Hovy. 2002. Manual and Automatic Evaluations of Summaries. In Proceedings of the Workshop on Automatic Summarization, post-conference workshop of ACL-2002, pp. 45-51, Philadelphia, PA, 2002. [find](https://scholar.google.com/scholar?q=Lin%2C+C%2E%2DY%2E+and+E%2E+Hovy%2E+2002%2E+Manual+and+Automatic+Evaluations+of+Summaries%2E+In+Proceedings+of+the+Workshop+on+Automatic+Summarization%2C+post%2Dconference+workshop+of+ACL%2D2002%2C+pp%2E+45%2D51%2C+Philadelphia%2C+P)
6. McKeown, K., R. Barzilay, D. Evans, V. Hatzivassiloglou, J. L. Klavans, A. Nenkova, C. Sable, B. Schiffman, S. Sigelman. Tracking and Summarizing News on a Daily Basis with Columbia’s Newsblaster. In Proceedings of Human Language Technology Conference 2002 (HLT 2002). [find](https://scholar.google.com/scholar?q=McKeown%2C+K%2E%2C+R%2E+Barzilay%2C+D%2E+Evans%2C+V%2E+Hatzivassiloglou%2C+J%2E+L%2E+Klavans%2C+A%2E+Nenkova%2C+C%2E+Sable%2C+B%2E+Schiffman%2C+S%2E+Sigelman%2E+Tracking+and+Summarizing+News+on+a+Daily+Basis+with+Columbia%E2%80%99s+Newsblaster%2E+In+)
7. Mani, I., D. House, G. Klein, L. Hirschman, L. Obrst, T. Firmin, M. Chrzanowski, and B. Sundheim. 1998. The TIPSTER SUMMAC Text Summarization Evaluation: Final Report. MITRE Corp. Tech. Report. [find](https://scholar.google.com/scholar?q=Mani%2C+I%2E%2C+D%2E+House%2C+G%2E+Klein%2C+L%2E+Hirschman%2C+L%2E+Obrst%2C+T%2E+Firmin%2C+M%2E+Chrzanowski%2C+and+B%2E+Sundheim%2E+1998%2E+The+TIPSTER+SUMMAC+Text+Summarization+Evaluation%3A+Final+Report%2E+MITRE+Corp%2E+Tech%2E+Report%2E)
8. NIST. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-Occurrence Statistics. [find](https://scholar.google.com/scholar?q=NIST%2E+2002%2E+Automatic+Evaluation+of+Machine+Translation+Quality+using+N%2Dgram+Co%2DOccurrence+Statistics%2E)
9. Over, P. 2003. Personal Communication. [find](https://scholar.google.com/scholar?q=Over%2C+P%2E+2003%2E+Personal+Communication%2E)
10. Papineni, K., S. Roukos, T. Ward, W.-J. Zhu. 2001. B LEU : a Method for Automatic Evaluation of Machine Translation. IBM Research Report RC22176 (W0109-022). [find](https://scholar.google.com/scholar?q=Papineni%2C+K%2E%2C+S%2E+Roukos%2C+T%2E+Ward%2C+W%2E%2DJ%2E+Zhu%2E+2001%2E+B+LEU+%3A+a+Method+for+Automatic+Evaluation+of+Machine+Translation%2E+IBM+Research+Report+RC22176+%28W0109%2D022%29%2E)
11. Porter, M. F. 1980. An Algorithm for Suffix Stripping. Program, 14, pp. 130-137. [find](https://scholar.google.com/scholar?q=Porter%2C+M%2E+F%2E+1980%2E+An+Algorithm+for+Suffix+Stripping%2E+Program%2C+14%2C+pp%2E+130%2D137%2E)
12. Radev, D. R., S. Blair-Goldensohn, Z. Zhang, and R. S. Raghavan. Newsinessence: A System for Domain-Independent, Real-Time News Clustering and Multi-Document Summarization. In Proceedings of human Language Technology Conference (HLT 2001), San Diego, CA, 2001. [find](https://scholar.google.com/scholar?q=Radev%2C+D%2E+R%2E%2C+S%2E+Blair%2DGoldensohn%2C+Z%2E+Zhang%2C+and+R%2E+S%2E+Raghavan%2E+Newsinessence%3A+A+System+for+Domain%2DIndependent%2C+Real%2DTime+News+Clustering+and+Multi%2DDocument+Summarization%2E+In+Proceedings+of+human+Lan)
13. Spärck Jones, K. and J. R. Galliers. 1996. Evaluating Natural Language Processing Systems: An Analysis and Review. New York: Springer. [find](https://scholar.google.com/scholar?q=Sp%C3%A4rck+Jones%2C+K%2E+and+J%2E+R%2E+Galliers%2E+1996%2E+Evaluating+Natural+Language+Processing+Systems%3A+An+Analysis+and+Review%2E+New+York%3A+Springer%2E)
14. Rath, G.J., Resnick, A., and Savage, T.R. 1961. The Formation of Abstracts by the Selection of Sentences. American Documentation, 12(2), pp. 139-143. Reprinted in Mani, I., and Maybury, M., eds, Advances in Automatic Text Summarization, MIT Press, pp. 287-292. [find](https://scholar.google.com/scholar?q=Rath%2C+G%2EJ%2E%2C+Resnick%2C+A%2E%2C+and+Savage%2C+T%2ER%2E+1961%2E+The+Formation+of+Abstracts+by+the+Selection+of+Sentences%2E+American+Documentation%2C+12%282%29%2C+pp%2E+139%2D143%2E+Reprinted+in+Mani%2C+I%2E%2C+and+Maybury%2C+M%2E%2C+eds%2C+Advan)
15. WAS. 2000. Workshop on Automatic Summarization, post-conference workshop of ANLP-NAACL-2000, Seattle, WA, 2000. [find](https://scholar.google.com/scholar?q=WAS%2E+2000%2E+Workshop+on+Automatic+Summarization%2C+post%2Dconference+workshop+of+ANLP%2DNAACL%2D2000%2C+Seattle%2C+WA%2C+2000%2E)
16. WAS. 2001. Workshop on Automatic Summarization, pre-conference workshop of NAACL-2001, Pittsburgh, PA, 2001. [find](https://scholar.google.com/scholar?q=WAS%2E+2001%2E+Workshop+on+Automatic+Summarization%2C+pre%2Dconference+workshop+of+NAACL%2D2001%2C+Pittsburgh%2C+PA%2C+2001%2E)
17. WAS. 2002. Workshop on Automatic Summarization, post-conference workshop of ACL-2002, Philadelphia, PA, 2002. [find](https://scholar.google.com/scholar?q=WAS%2E+2002%2E+Workshop+on+Automatic+Summarization%2C+post%2Dconference+workshop+of+ACL%2D2002%2C+Philadelphia%2C+PA%2C+2002%2E)