---
title: "Evaluation & benchmarks — papers — Elamur Research"
description: "Evaluation & benchmarks: 684 papers of the Elamur Research library, the most cited first."
canonical: "https://research.elamur.ai/free/en/topics/evaluation-benchmarks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/topics/evaluation-benchmarks/)

[home](https://md.research.elamur.ai/free/en/) / [topics](https://md.research.elamur.ai/free/en/topics/) / Evaluation & benchmarks

# 📏 Evaluation & benchmarks

684 papers · page 1 of 8 · 1–25 · the most cited first · [all topics](https://md.research.elamur.ai/free/en/topics/)

How systems are measured: benchmarks, metrics, human judgements and what the numbers mean.

💡 Explained: [BLEU score](https://md.research.elamur.ai/free/en/concepts/bleu/)

A survey · about 18 minutes to read · every paper named here has its own page in this library

**In this article**

1. [What it is](#a-what-it-is)
2. [History and milestones](#a-history-and-milestones)
3. [Main problems and tasks](#a-main-problems-and-tasks)
4. [Methods and schools](#a-methods-and-schools)
5. [Datasets, resources and benchmarks](#a-datasets-resources-and-benchmarks)
6. [Landmark papers in this library](#a-landmark-papers-in-this-library)
7. [People](#a-people)
8. [Institutions and places](#a-institutions-and-places)
9. [Open questions](#a-open-questions)
10. [Neighbouring topics](#a-neighbouring-topics)
11. [Key works in this library](#a-key-works-in-this-library)

Evaluation is the part of language technology that decides what counts as progress. Every system — a translator, a parser, a question-answering model, a chat assistant — is eventually reduced to numbers: a score on a test set, a correlation with human ratings, a rank on a leaderboard. This topic gathers the papers in this library that build those numbers and examine them: automatic metrics such as BLEU, METEOR, chrF and COMET; benchmark suites such as GLUE, Natural Questions and MMLU-Pro; shared-task campaigns that organise whole communities around common data; and the methodological studies that ask whether any of it measures what it claims to.

The topic is unusual because it is about the instruments rather than the phenomena. A paper on machine translation asks how to translate better; a paper here asks how anyone would know. That makes the field reflexive and often sceptical. Many of its most influential works are critiques — of a metric that rewards the wrong thing, of a benchmark that can be solved without understanding, of a human study too small to support its conclusions.

In this library the subject is dominated by machine translation, where automatic evaluation was invented and refined over two decades, but it reaches far beyond it: reading comprehension, summarisation, coreference, word embeddings, multimodal models, code generation, and the broad and fast-moving evaluation of large language models.

## What it is

Evaluation in natural language processing answers three related questions. **What is being measured?** — a capability such as translation adequacy, factual accuracy, or reasoning. **How is it measured?** — by human judgement, by an automatic metric, or by a test whose answers can be checked mechanically. **How much should the result be trusted?** — which depends on test-set size, annotator agreement, statistical significance, and whether the system may already have seen the test data.

Several distinctions organise the field:

- **Intrinsic versus extrinsic evaluation.** An intrinsic evaluation scores the output directly (is this summary good?); an extrinsic one measures its usefulness in a downstream task (does it help a reader answer questions?).
- **Reference-based versus reference-free metrics.** Classic metrics compare system output with one or more human references; newer ones judge output against the source or an image, or use a learned model that needs no reference at all.
- **Metrics versus benchmarks.** A metric is a scoring function; a benchmark is a fixed collection of inputs, expected outputs and rules for comparison, often with a public leaderboard and a hidden test set.
- **Evaluation versus meta-evaluation.** Meta-evaluation asks how well a metric agrees with human judgement, usually by correlation at the level of whole systems or individual segments. A great deal of the work here is meta-evaluation.

The topic borders closely on [machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/), which supplies most of its metrics and campaigns, and on [language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/), whose rapid growth created a demand for ever harder and broader benchmarks. It differs from both in that its object is the measurement itself.

## History and milestones

### Early evaluation programmes

Systematic evaluation of language technology began in large government-funded programmes in the United States in the late 1980s and 1990s. The Message Understanding Conferences defined information-extraction tasks with fixed scoring, and the ARPA machine translation programme experimented with ways of rating translations by fluency, adequacy and informativeness; [The ARPA MT Evaluation Methodologies](https://md.research.elamur.ai/free/en/papers/white-1994-arpa-mt-evaluation-methodologies-evolution/) records what that programme learned. In coreference, [Algorithms for Scoring Coreference Chains](https://md.research.elamur.ai/free/en/papers/bagga-1998-algorithms-scoring-coreference-chains/) introduced the B-cubed measure as an alternative to the MUC link-based score. Human judgement was the gold standard, and it was slow and expensive.

### The automatic-metric era

The turning point came in 2002 with the IBM paper introducing [BLEU](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/), which showed that a cheap n-gram overlap score could rank systems roughly as human judges did. It is by far the most cited work in this topic, and it set the template for a decade. Summarisation followed with recall-oriented n-gram scores ([Lin and Hovy, 2003](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/)) that grew into ROUGE, and translation gained alternatives built on longest common subsequences ([Lin and Och, 2004](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/)) and on explicit word alignment with stemming and synonyms ([METEOR, 2005](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)). Large shared datasets became benchmarks in their own right: [RCV1](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) for text categorisation and the [ACE programme](https://md.research.elamur.ai/free/en/papers/doddington-2004-automatic-content-extraction-ace-program/) for entity, relation and event extraction.

Landmark papers on this topic: the 12 most cited, in time The 12 most cited papers on this topic span 2002–2023; the dot grows with the citations.

**The numbers**

| Paper | Citations |
| --- | --- |
| [2002 — Bleu: a Method for Automatic Evaluation of Machine Translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) | 32,134 |
| [2004 — RCV1: A New Benchmark Collection for Text Categorization Research](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/) | 2,752 |
| [2004 — Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) | 2,444 |
| [2005 — METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) | 5,504 |
| [2014 — Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) | 2,399 |
| [2015 — chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) | 2,107 |
| [2018 — GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) | 7,169 |
| [2018 — A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) | 4,135 |
| [2019 — Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) | 4,035 |
| [2019 — SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) | 2,190 |
| [2021 — Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) | 2,548 |
| [2023 — A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) | 3,641 |

### Shared tasks and meta-evaluation

From 2006 the annual Workshop on Statistical Machine Translation — from 2016 the Conference on Machine Translation, WMT — combined a translation competition with large-scale human evaluation and a metrics task that ranked the metrics themselves. Its yearly findings papers, from [Koehn and Monz (2006)](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/) through [2014](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/), [2016](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/) and [2019](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/) to [2024](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/), form a continuous record of how human evaluation practice changed: from relative ranking of a few outputs, to direct assessment on a 0–100 scale, to source-based and document-level judgement, and finally to expert error annotation. Similar campaigns ran for spoken translation ([IWSLT](https://md.research.elamur.ai/free/en/papers/cettolo-2014-report-11th-iwslt-evaluation-campaign/)), parsing of morphologically rich languages ([SPMRL 2013](https://md.research.elamur.ai/free/en/papers/seddah-2013-overview-spmrl-2013-shared-task/)) and dialect identification ([VarDial](https://md.research.elamur.ai/free/en/papers/zampieri-2017-findings-vardial-evaluation-campaign/)).

### Benchmarks for general understanding

With pretrained neural models came a new style of benchmark: a suite of many tasks with a single aggregate score. [GLUE](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) (2018) was overtaken by models within about a year, prompting the harder [SuperGLUE](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). The same pattern produced multilingual suites ([XTREME](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/), [XGLUE](https://md.research.elamur.ai/free/en/papers/liang-2020-xglue-new-benchmark-dataset-cross/)), language-specific ones ([IndicNLPSuite](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/), [CLUE](https://md.research.elamur.ai/free/en/papers/xu-2020-clue-chinese-language-understanding/), [IndoNLU](https://md.research.elamur.ai/free/en/papers/wilie-2020-indonlu-benchmark-resources-evaluating-indonesian/)) and domain ones ([LexGLUE](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/), [TweetEval](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/)). At the same time, adversarial and diagnostic work — [Jia and Liang (2017)](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/), [contrast sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/) — showed how far high scores could rest on shallow cues.

### Learned metrics and the LLM era

Around 2020 machine translation evaluation moved from string overlap to learned metrics such as [COMET](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/), and WMT's metrics task concluded in 2022 that neural metrics should replace BLEU ([Freitag et al., 2022](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/)). Large language models then changed the field twice over: they became the systems to be evaluated, on benchmarks such as [MMLU-Pro](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/), [GPQA](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/) and [MMMU](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/), and they became evaluators themselves, as in [GEMBA-MQM](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/) and [Prometheus](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/). Most papers on the topic in this library date from this most recent period.

Papers on this topic by year 684 dated papers from 1990 to 2026; the busiest year is 2026 with 145, and 79% of them appeared in 2018 or later.

**The numbers**

| Year | Papers |
| --- | --- |
| 2026 | 145 |
| 2025 | 36 |
| 2024 | 67 |
| 2023 | 76 |
| 2022 | 41 |
| 2021 | 50 |
| 2020 | 62 |
| 2019 | 33 |
| 2018 | 29 |
| 2017 | 21 |
| 2016 | 14 |
| 2015 | 8 |
| 2014 | 11 |
| 2013 | 11 |
| 2012 | 7 |
| 2011 | 9 |
| 2010 | 8 |
| 2009 | 6 |
| 2008 | 7 |
| 2007 | 4 |
| 2006 | 7 |
| 2005 | 7 |
| 2004 | 6 |
| 2003 | 4 |
| 2002 | 4 |
| 2001 | 2 |
| 2000 | 1 |
| 1999 | 1 |
| 1998 | 2 |
| 1997 | 1 |
| 1995 | 1 |
| 1994 | 2 |
| 1990 | 1 |

## Main problems and tasks

The work collected here falls into several recurring problems.

**Designing automatic metrics.** The basic task is a function that maps a system output (and possibly a source and references) to a quality score. Families include n-gram precision (BLEU), recall-weighted alignment (METEOR), character n-gram F-scores ([chrF](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)), edit distances such as TER and [CDER](https://md.research.elamur.ai/free/en/papers/leusch-2006-cder-efficient-mt-evaluation-block/), semantic and syntactic matching ([YiSi](https://md.research.elamur.ai/free/en/papers/lo-2019-yisi-unified-semantic-mt-quality/), [syntactic features](https://md.research.elamur.ai/free/en/papers/liu-2005-syntactic-features-evaluation-machine-translation/)), and trained regressors over pretrained encoders.

**Meta-evaluating metrics.** Before a metric is trusted, its agreement with human judgement must be checked. This raises its own questions: whether to correlate at system or segment level ([Graham et al., 2015](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/)), how outliers distort correlations ([Tangled up in BLEU](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/)), and whether a metric validated for one task transfers to another — it often does not, as studies of [text simplification](https://md.research.elamur.ai/free/en/papers/sulem-2018-bleu-is-not-suitable-evaluation/) and [summarisation](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/) found.

**Collecting reliable human judgements.** Human evaluation is the reference point, yet it is noisy. Work here covers crowdsourcing quality ([Snow et al., 2008](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/)), scales and protocols, the role of document context ([Läubli et al., 2018](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/)), expert error annotation ([Freitag et al., 2021](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/)), statistical power ([Graham et al., 2020](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/)) and annotator disagreement as signal rather than noise ([Basile et al., 2021](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/)).

**Building benchmarks.** A benchmark needs representative inputs, correct labels, a fair split, and a way to prevent overfitting to the test set. Questions written from real information needs ([Natural Questions](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/), [TyDi QA](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/)) behave differently from questions written while looking at the answer.

**Targeted and diagnostic testing.** Instead of a single average, challenge sets probe specific phenomena: [discourse in translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/), [contrastive grammatical pairs](https://md.research.elamur.ai/free/en/papers/sennrich-2017-how-grammatical-is-character-level/), [gender bias](https://md.research.elamur.ai/free/en/papers/stanovsky-2019-evaluating-gender-bias-machine-translation/), [object hallucination](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/), or [exaggerated safety refusals](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/).

**Quality estimation.** Predicting the quality of a translation without any reference — useful when output must be checked at the moment of use — has its own WMT task series ([2019](https://md.research.elamur.ai/free/en/papers/fonseca-2019-findings-wmt-2019-shared-tasks/), [2020](https://md.research.elamur.ai/free/en/papers/specia-2020-findings-wmt-2020-shared-task-2020-wmt-1-79/), [2023](https://md.research.elamur.ai/free/en/papers/blain-2023-findings-wmt-2023-shared-task/)).

**Contamination and reproducibility.** When models are trained on web-scale data, test sets leak into training data, and published numbers become hard to compare. [Sainz et al. (2023)](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/) and [Balloccu et al. (2024)](https://md.research.elamur.ai/free/en/papers/balloccu-2024-leak-cheat-repeat-data-contamination/) treat this as a central threat; [Post (2018)](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) addressed the older problem of incomparable metric configurations.

## Methods and schools

### Surface-overlap metrics

The first school treats quality as similarity to a human reference, measured on surface strings. BLEU uses clipped n-gram precision with a length penalty; ROUGE flips the emphasis to recall for summaries; METEOR aligns words in stages and penalises fragmented matches; chrF works on characters and so needs no tokeniser, which suits morphologically rich languages. These metrics are transparent, fast and language-agnostic, and their weaknesses are well documented: they reward lexical match rather than meaning, penalise valid paraphrases, and are unreliable for single sentences. Proposals to soften them with paraphrases include [Kauchak and Barzilay (2006)](https://md.research.elamur.ai/free/en/papers/kauchak-2006-paraphrasing-automatic-evaluation/), [Meteor Universal](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) and [smoothing for sentence-level BLEU](https://md.research.elamur.ai/free/en/papers/chen-2014-systematic-comparison-smoothing-techniques-sentence/).

### Learned and embedding-based metrics

A second school learns the metric from human judgements. COMET feeds source, hypothesis and reference through a multilingual encoder and regresses a score; its successors [COMET-22](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/), [xCOMET](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/) and Google's [MetricX-23](https://md.research.elamur.ai/free/en/papers/juraska-2023-metricx-23-google-submission-wmt/) and [MetricX-24](https://md.research.elamur.ai/free/en/papers/juraska-2024-metricx-24-google-submission-wmt/) add error spans and larger backbones. Related ideas appear in [paraphrase-based scoring](https://md.research.elamur.ai/free/en/papers/thompson-2020-automatic-machine-translation-evaluation-many/), in [CLIPScore](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/) for image captions, and in unsupervised dialogue metrics such as [USR](https://md.research.elamur.ai/free/en/papers/mehri-2020-usr-unsupervised-reference-free-evaluation/). The debate over these metrics is about opacity and robustness: a learned scorer may correlate well on average yet be fooled by inputs unlike its training data, and WMT asked in 2024 whether LLM-generated translations were [breaking MT metrics](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/).

### Language models as judges

The newest school prompts a large language model to grade outputs, either holistically or by listing errors. [GEMBA-MQM](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/) and [Fernandes et al. (2023)](https://md.research.elamur.ai/free/en/papers/fernandes-2023-devil-is-errors-leveraging-large/) ask models to mark translation errors in the MQM scheme; [Prometheus](https://md.research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/) trains an open evaluator model on rubrics; [PandaLM](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/) and [Arena-Hard](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/) build judge-based benchmarks for instruction-tuned models. The open questions concern judge bias, self-preference and reproducibility when the judge is a closed commercial model.

### Execution and verifiable checks

Where outputs can be run or checked mechanically, evaluation can avoid both references and judges. [HumanEval](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) scored generated code by unit tests and introduced pass@k; [EvalPlus](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/) showed that thin test suites inflate such scores. [IFEval](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/) applies the same idea to instruction following by using only constraints a short program can verify, and text-to-SQL work examines [how to compare queries fairly](https://md.research.elamur.ai/free/en/papers/dollak-2018-improving-text-sql-evaluation-methodology/).

### Human evaluation protocols

Human evaluation has its own schools. Relative ranking asks judges to order several outputs; direct assessment, championed by Yvette Graham and adopted by WMT, asks for an absolute score on a continuous scale, standardised per annotator; MQM-style annotation asks trained professionals to mark and classify every error. The shift toward MQM, documented by [Freitag et al. (2021)](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/), reflected the finding that crowd ratings could not separate strong systems. The debate over [human parity](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) in translation — answered by [Läubli et al. (2018)](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/) and [Toral et al. (2018)](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/) — turned largely on these protocol choices.

### Adversarial and behavioural testing

A further tradition attacks benchmarks rather than systems. Adversarial distractors, [contrast sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/), [partial-input baselines](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/) and [model-written evaluations](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/) all ask whether a high score reflects the intended skill. [Linzen (2020)](https://md.research.elamur.ai/free/en/papers/linzen-2020-how-can-we-accelerate-progress/) argued more broadly that leaderboards built on i.i.d. test sets reward the wrong kind of generalisation.

## Datasets, resources and benchmarks

The benchmarks in this library can be grouped by what they test.

| Area | Examples in this library |
| --- | --- |
| General language understanding | [GLUE](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/), [SuperGLUE](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/), [MMLU-Pro](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/), [AGIEval](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/) |
| Question answering and reading | [Natural Questions](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/), [DROP](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/), [2WikiMultiHopQA](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/), [KILT](https://md.research.elamur.ai/free/en/papers/petroni-2020-kilt-benchmark-knowledge-intensive-language/), [Belebele](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/) |
| Multilingual | [XTREME](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/), [TyDi QA](https://md.research.elamur.ai/free/en/papers/clark-2020-tydi-qa-benchmark-information-seeking/), [FLORES](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/), [GLUECoS](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/) |
| Expert knowledge and reasoning | [GPQA](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), [OlympiadBench](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/), [MiniF2F](https://md.research.elamur.ai/free/en/papers/zheng-2021-minif2f-cross-system-benchmark-formal/), [PlanBench](https://md.research.elamur.ai/free/en/papers/valmeekam-2022-planbench-extensible-benchmark-evaluating-large/), [LegalBench](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/) |
| Multimodal | [MMMU](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/), [MME](https://md.research.elamur.ai/free/en/papers/fu-2023-mme-comprehensive-evaluation-benchmark-multimodal/), [ChartQA](https://md.research.elamur.ai/free/en/papers/masry-2022-chartqa-benchmark-question-answering-about/), [A-OKVQA](https://md.research.elamur.ai/free/en/papers/schwenk-2022-okvqa-benchmark-visual-question-answering/), [Video-MME](https://md.research.elamur.ai/free/en/papers/fu-2024-video-mme-first-ever-comprehensive/), [ALFRED](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/) |
| Long context and efficiency | [LongBench](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/), [Long Range Arena](https://md.research.elamur.ai/free/en/papers/tay-2020-long-range-arena-benchmark-efficient/) |
| Representations and retrieval | [MTEB](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/), [BEIR](https://md.research.elamur.ai/free/en/papers/thakur-2021-beir-heterogenous-benchmark-zero/), [One Billion Word](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/), [BLESS](https://md.research.elamur.ai/free/en/papers/baroni-2011-how-we-blessed-distributional-semantic/) |
| Safety, bias and truthfulness | [BBQ](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/), [HaluEval](https://md.research.elamur.ai/free/en/papers/li-2023-halueval-large-scale-hallucination-evaluation/), [JailbreakBench](https://md.research.elamur.ai/free/en/papers/chao-2024-jailbreakbench-open-robustness-benchmark-jailbreaking/), [LIAR](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/) |
| Agents and tools | [GAIA](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/), [API-Bank](https://md.research.elamur.ai/free/en/papers/li-2023-api-bank-comprehensive-benchmark-tool/) |

Beyond named benchmarks, the WMT campaigns are themselves a resource: every year they release test sets, system outputs and human judgements, which later metric papers reuse as training and validation data. The [MQM annotations of Freitag et al.](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/) and the [human parity data from Microsoft](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) play the same role at a finer grain.

## Landmark papers in this library

Citations of the papers on this topic, by when they were published Papers published in 2020–2024 gather 33% of the 176,328 citations of the papers on this topic.

**The numbers**

| Published | Papers and their citations |
| --- | --- |
| 2025–2029 | 181 papers, 1,004 citations |
| 2020–2024 | 296 papers, 58,965 citations |
| 2015–2019 | 105 papers, 46,245 citations |
| 2010–2014 | 46 papers, 11,697 citations |
| 2005–2009 | 31 papers, 13,428 citations |
| 2000–2004 | 17 papers, 43,799 citations |
| 1995–1999 | 5 papers, 861 citations |
| 1990–1994 | 3 papers, 329 citations |

### BLEU (2002)

[Papineni and colleagues at IBM](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) proposed scoring a translation by how many of its word sequences of length one to four also occur in reference translations, with counts capped so that repetition is not rewarded and a penalty for output that is too short. Its value was practical: developers could test a change in minutes rather than wait for human raters. BLEU shaped two decades of machine translation and was borrowed, often uncritically, for other generation tasks — a habit that much later work in this topic set out to correct.

### METEOR (2005)

[Banerjee and Lavie](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) answered BLEU's lack of recall and its poor sentence-level behaviour with a metric based on explicit word alignment, matching exact forms, then stems, then WordNet synonyms, and penalising scattered matches. Later versions ([Meteor 1.3](https://md.research.elamur.ai/free/en/papers/denkowski-2011-meteor-1-3-automatic-metric/), [Meteor Universal](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/)) added paraphrase tables, function-word weighting and a way to support new target languages without hand-built resources.

### ROUGE precursors (2003–2004)

[Lin and Hovy](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/) tested whether BLEU-style scoring could judge summaries and found that plain unigram recall tracked human coverage ratings best. With [Lin and Och](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/) the same line produced subsequence- and skip-bigram measures. Together they underlie ROUGE, still the default summarisation metric despite the doubts raised in [Graham (2015)](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/) and [Bhandari et al. (2020)](https://md.research.elamur.ai/free/en/papers/bhandari-2020-re-evaluating-evaluation-text-summarization/).

### GLUE (2018) and SuperGLUE (2019)

[GLUE](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) bundled nine English sentence-level tasks behind a single score, a public leaderboard and a diagnostic set, aiming to reward general-purpose rather than task-specific systems. When pretrained transformers passed its human estimate within months, the same team released [SuperGLUE](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) with harder tasks and human baselines for each. The rapid saturation of both became the standard example of how short a benchmark's useful life can be.

### A Call for Clarity in Reporting BLEU Scores (2018)

[Matt Post](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) showed that "BLEU" in different papers often meant different computations, because authors tokenised references in their own ways, and that the resulting differences could exceed the gains being claimed. The accompanying SacreBLEU tool, which handles preprocessing itself and prints a signature of its settings, became the standard way to report the metric.

### Natural Questions (2019)

[Kwiatkowski and colleagues](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) built a question-answering benchmark from real search queries paired with Wikipedia pages, annotated with long and short answers. Because the questions were not written by people looking at the text, the dataset avoided the lexical overlap that made earlier sets easy; and by collecting several annotations per test item, it treated legitimate disagreement as part of the target.

### Adversarial Examples for Reading Comprehension (2017)

[Jia and Liang](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) inserted a single misleading sentence into SQuAD paragraphs and watched the accuracy of published models collapse while human readers were barely affected. The paper made adversarial evaluation a standard tool and helped motivate benchmarks such as [DROP](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/), whose questions were collected against a live model.

### Evaluating Large Language Models Trained on Code (2021)

The Codex paper by [Chen and colleagues](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) replaced text similarity with functional correctness: a generated function counts as right only if it passes hidden unit tests. Its HumanEval set and pass@k estimator became the default for code models, and [EvalPlus](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/) later showed how much a richer test suite lowers those scores.

### COMET (2020)

[Rei, Stewart, Farinha and Lavie](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) trained a metric on human quality judgements using a cross-lingual pretrained encoder that sees the source, the hypothesis and the reference. It correlated better with people than any surface metric, and within two years learned metrics of this kind had become the recommended replacement for BLEU in WMT.

### Experts, Errors, and Context (2021)

[Freitag and colleagues](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/) had professional translators annotate errors in top WMT systems using the MQM framework, with full document context. The resulting ranking differed sharply from the crowd-based one and placed human translations clearly ahead of machines. The study shifted WMT toward expert error annotation and became the reference human data for judging metrics.

### Human parity claims (2018)

Microsoft's [claim of human parity for Chinese–English news](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/) used careful source-based direct assessment. Two responses — [Läubli, Sennrich and Volk](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/) and [Toral and colleagues](https://md.research.elamur.ai/free/en/papers/toral-2018-attaining-unattainable-reassessing-claims-human/) — showed that with document context, professional raters and attention to translationese, the gap reappeared. The exchange is a compact lesson in how evaluation design determines conclusions.

### A Survey on Evaluation of Large Language Models (2023)

[Chang and colleagues](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) organised the sudden flood of LLM evaluation work by what is evaluated, where, and how, and catalogued general, domain and multimodal benchmarks. It is the most cited LLM-era overview here, alongside [Guo et al. (2023)](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/), and it called for dynamic tests that resist contamination.

### MMLU-Pro (2024) and GPQA (2023)

[MMLU-Pro](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) responded to the saturation of MMLU by expanding answer options to ten, filtering easy items and cleaning labels, which also made scores less sensitive to prompt wording. [GPQA](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/) took a different route: questions written by doctoral-level specialists that skilled non-experts with web access cannot reliably answer, designed as a testbed for supervising systems that may exceed their overseers.

### CEAF and coreference scoring (2005)

[Luo](https://md.research.elamur.ai/free/en/papers/luo-2005-coreference-resolution-performance-metrics/) proposed aligning gold and predicted entities one-to-one and scoring the best alignment, fixing MUC's tendency to reward over-merging. CEAF was later averaged with MUC and [B-cubed](https://md.research.elamur.ai/free/en/papers/bagga-1998-algorithms-scoring-coreference-chains/) to form the CoNLL coreference score; [Moosavi and Strube (2016)](https://md.research.elamur.ai/free/en/papers/moosavi-2016-which-coreference-evaluation-metric-do/) later reopened the question.

### WindowDiff (2002)

[Pevzner and Hearst](https://md.research.elamur.ai/free/en/papers/pevzner-2002-critique-improvement-evaluation-metric-text/) analysed the Pk measure for text segmentation, showed that it treats misses and false alarms unequally and punishes near misses too harshly, and proposed WindowDiff, which compares boundary counts in a sliding window. It is a model of how a careful critique of a metric can change practice in a subfield.

## People

Authors with the most papers on this topic Ondřej Bojar leads with 23 papers; the 12 authors shown wrote 176 of the papers on this topic between them, counting each paper once per author.

**The numbers**

| Author | Papers here · their citations |
| --- | --- |
| [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) | 23 papers · 7,788 citations |
| [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) | 19 papers · 10,488 citations |
| [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/) | 19 papers · 2,818 citations |
| [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/) | 15 papers · 6,539 citations |
| [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/) | 14 papers · 6,292 citations |
| [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) | 14 papers · 2,211 citations |
| [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/) | 13 papers · 6,159 citations |
| [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/) | 13 papers · 5,161 citations |
| [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/) | 12 papers · 3,654 citations |
| [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) | 12 papers · 2,816 citations |
| [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) | 11 papers · 2,429 citations |
| [Tom Kocmi](https://md.research.elamur.ai/free/en/authors/tom-kocmi/) | 11 papers · 1,716 citations |

- [Ondřej Bojar](https://md.research.elamur.ai/free/en/authors/ondrej-bojar/) (Charles University) is the most prolific author on this topic in the library, having led or co-written the WMT findings papers through the 2010s, including [WMT14](https://md.research.elamur.ai/free/en/papers/bojar-2014-findings-2014-workshop-statistical-machine/), [WMT16](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/), [WMT17](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/) and [WMT18](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/), as well as the [WMT17 metrics results](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/).
- [Philipp Koehn](https://md.research.elamur.ai/free/en/authors/philipp-koehn/) (Johns Hopkins University) co-founded the WMT evaluation tradition with [Christof Monz](https://md.research.elamur.ai/free/en/authors/christof-monz/) (University of Amsterdam), beginning with their [2006 comparison of manual and automatic evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2006-manual-automatic-evaluation-machine-translation/), and both remained organisers of the campaign for many years.
- [Markus Freitag](https://md.research.elamur.ai/free/en/authors/markus-freitag/) led the [MQM study of human evaluation](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/), the argument that [references themselves are a source of error](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/), and the WMT metrics tasks from [2021](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/) to [2024](https://md.research.elamur.ai/free/en/papers/freitag-2024-are-llms-breaking-mt-metrics/).
- [Alon Lavie](https://md.research.elamur.ai/free/en/authors/alon-lavie/) (Carnegie Mellon University) connects two generations of metrics: he co-created [METEOR](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) and its successors, and later co-authored [COMET](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/).
- [Ricardo Rei](https://md.research.elamur.ai/free/en/authors/ricardo-rei/) is the lead author of COMET and [COMET-22](https://md.research.elamur.ai/free/en/papers/rei-2022-comet-22-unbabel-ist-2022/), working with [André F. T. Martins](https://md.research.elamur.ai/free/en/authors/andre-f-t-martins/) (Instituto de Telecomunicações and the University of Lisbon), who also co-authored [xCOMET](https://md.research.elamur.ai/free/en/papers/guerreiro-2024-xcomet-transparent-machine-translation-evaluation/) and quality-estimation shared tasks.
- [Christian Federmann](https://md.research.elamur.ai/free/en/authors/christian-federmann/) built and ran much of the human evaluation infrastructure behind WMT and the [human parity study](https://md.research.elamur.ai/free/en/papers/hassan-2018-achieving-human-parity-automatic-chinese/).
- [Yvette Graham](https://md.research.elamur.ai/free/en/authors/yvette-graham/) (Trinity College Dublin) developed direct assessment and studied the statistics of evaluation, including [segment-level metric testing](https://md.research.elamur.ai/free/en/papers/graham-2015-accurate-evaluation-segment-level-machine/), [summarisation metrics](https://md.research.elamur.ai/free/en/papers/graham-2015-re-evaluating-automatic-summarization-bleu/) and [statistical power and translationese](https://md.research.elamur.ai/free/en/papers/graham-2020-statistical-power-translationese-machine-translation/).
- [Barry Haddow](https://md.research.elamur.ai/free/en/authors/barry-haddow/) (University of Edinburgh) co-organised WMT across the period covered here.
- [Lucia Specia](https://md.research.elamur.ai/free/en/authors/lucia-specia/) (Imperial College London) shaped quality estimation as a task and co-organised [multimodal translation](https://md.research.elamur.ai/free/en/papers/barrault-2018-findings-third-shared-task-multimodal/) and [simplification benchmarking](https://md.research.elamur.ai/free/en/papers/manchego-2020-data-driven-sentence-simplification-survey/).
- [Tom Kocmi](https://md.research.elamur.ai/free/en/authors/tom-kocmi/) brought language models into MT evaluation with [GEMBA-MQM](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/) and led the [WMT24 general task findings](https://md.research.elamur.ai/free/en/papers/kocmi-2024-findings-wmt24-general-machine-translation/).
- [Matt Post](https://md.research.elamur.ai/free/en/authors/matt-post/) (Microsoft) is known here for [SacreBLEU](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) and for co-organising several WMT campaigns.
- [Nitika Mathur](https://md.research.elamur.ai/free/en/authors/nitika-mathur/) showed in [Tangled up in BLEU](https://md.research.elamur.ai/free/en/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/) how outlier systems inflate metric correlations, and led the [WMT20 metrics task](https://md.research.elamur.ai/free/en/papers/mathur-2020-results-wmt20-metrics-shared-task/).
- [Chris Callison-Burch](https://md.research.elamur.ai/free/en/authors/chris-callison-burch/) organised early WMT evaluations, including the [(meta-)evaluation of 2007](https://md.research.elamur.ai/free/en/papers/burch-2007-meta-evaluation-machine-translation/) and the [2012 findings](https://md.research.elamur.ai/free/en/papers/burch-2012-findings-2012-workshop-statistical-machine/).
- [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) co-led [GLUE](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/), [SuperGLUE](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) and [GPQA](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), with [Alex Wang](https://md.research.elamur.ai/free/en/authors/alex-wang/) as first author of both GLUE papers.
- [Reut Tsarfaty](https://md.research.elamur.ai/free/en/authors/reut-tsarfaty/) (Bar-Ilan University) contributed to parsing evaluation, from [robust dependency-parsing evaluation](https://md.research.elamur.ai/free/en/papers/tsarfaty-2011-evaluating-dependency-parsing-robust-heuristics/) to the [SPMRL shared task](https://md.research.elamur.ai/free/en/papers/seddah-2013-overview-spmrl-2013-shared-task/) and the [HeQ Hebrew reading benchmark](https://md.research.elamur.ai/free/en/papers/cohen-2023-heq-large-diverse-hebrew-reading/).
- [Nizar Habash](https://md.research.elamur.ai/free/en/authors/nizar-habash/) (New York University Abu Dhabi) works on Arabic evaluation, including recent studies of [diacritics](https://md.research.elamur.ai/free/en/papers/inoue-2026-do-diacritics-matter-evaluating-impact/) and [medical question answering](https://md.research.elamur.ai/free/en/papers/abu-daoud-2026-medarabench-large-scale-arabic-medical/).

## Institutions and places

The institutions most represented reflect the two halves of the topic. European machine translation groups — Charles University in Prague, the University of Edinburgh, the University of Amsterdam, ETH Zurich, the German Research Centre for Artificial Intelligence and the Lisbon cluster around the Instituto de Telecomunicações — carry the WMT tradition and the learned-metric work. Large North American universities and industrial labs — Carnegie Mellon, Johns Hopkins, Stanford, the University of Washington, New York University, the Allen Institute for AI, Microsoft, Google and Amazon — dominate the benchmark side, especially for question answering and large language models. Chinese institutions, among them the Chinese Academy of Sciences and Zhejiang University, figure prominently in recent LLM and multimodal benchmarks, and the Mohamed bin Zayed University of Artificial Intelligence in multilingual evaluation.

Where the authors on this topic work now Carnegie Mellon University is the current home of authors of 53 papers on this topic; counted are the authors with a page here, by their latest institution.

**The numbers**

| Institution | Papers |
| --- | --- |
| [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) | 53 papers |
| [Google](https://md.research.elamur.ai/free/en/companies/google/) | 51 papers |
| [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) | 48 papers |
| [ETH Zurich](https://md.research.elamur.ai/free/en/universities/eth-zurich/) | 34 papers |
| [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) | 33 papers |
| [University of Edinburgh](https://md.research.elamur.ai/free/en/universities/university-of-edinburgh/) | 33 papers |
| [Charles University](https://md.research.elamur.ai/free/en/universities/charles-university/) | 31 papers |
| [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) | 28 papers |
| [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) | 28 papers |
| [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) | 24 papers |

By country, the United States leads, followed by the United Kingdom and China, then Germany, Czechia and Switzerland — the last two largely because of the WMT organisers.

Countries of the authors on this topic United States comes first: authors working there now wrote 322 papers on this topic.

**The numbers**

| Country | Papers |
| --- | --- |
| [United States](https://md.research.elamur.ai/free/en/countries/united-states/) | 322 papers |
| [China](https://md.research.elamur.ai/free/en/countries/china/) | 132 papers |
| [United Kingdom](https://md.research.elamur.ai/free/en/countries/united-kingdom/) | 113 papers |
| [Germany](https://md.research.elamur.ai/free/en/countries/germany/) | 73 papers |
| [Switzerland](https://md.research.elamur.ai/free/en/countries/switzerland/) | 46 papers |
| [Italy](https://md.research.elamur.ai/free/en/countries/italy/) | 43 papers |
| [Canada](https://md.research.elamur.ai/free/en/countries/canada/) | 41 papers |
| [Netherlands](https://md.research.elamur.ai/free/en/countries/netherlands/) | 39 papers |
| [France](https://md.research.elamur.ai/free/en/countries/france/) | 36 papers |
| Hong Kong | 35 papers |

The proceedings of the Language Resources and Evaluation Conference (LREC), whose very name names the topic, carry more of these papers than any other venue. They are followed by Computational Linguistics, the Transactions of the ACL, the main ACL and EMNLP conferences, and the yearly proceedings of the Conference on Machine Translation.

Where the papers on this topic were published Language Resources and Evaluation Conference (LREC) carries the most, 91 of 684.

**The numbers**

| Venue | Papers |
| --- | --- |
| [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/) | 91 papers |
| [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/) | 68 papers |
| [Conference on Machine Translation (WMT)](https://md.research.elamur.ai/free/en/venues/wmt/) | 62 papers |
| [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) | 56 papers |
| [Findings of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/findings/) | 43 papers |
| [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/) | 22 papers |
| [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/) | 18 papers |
| [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/) | 17 papers |
| [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/) | 15 papers |
| [International Conference on Computational Linguistics (COLING)](https://md.research.elamur.ai/free/en/venues/coling/) | 14 papers |

## Open questions

**Validity.** Does a metric or benchmark measure the property it names? [Reiter's structured review](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/) concluded that BLEU is supported for diagnosing machine translation systems but not for single texts or other tasks. Every newer metric faces the same question.

**Saturation and shelf life.** Benchmarks are overtaken ever faster. Building harder ones — expert-written, adversarial, multimodal — buys time but raises costs, and harder is not the same as more meaningful.

**Contamination.** When training data is undisclosed and drawn from the web, it is difficult to know whether a model has seen a test set. Proposed remedies include held-out canary strings, post-cutoff test material, dynamic benchmarks and contamination audits, none of them complete.

**Who judges the judges.** Learned metrics and LLM judges are trained on, or prompted with, data whose biases they inherit; they may prefer outputs from models like themselves. Their reliability has to be checked against human judgement, which is itself expensive and variable.

**Human disagreement.** For many tasks — offensive language, sentiment, summary quality — there is no single right answer. Treating disagreement as noise to be averaged away may discard exactly the information an evaluation should keep.

**References.** Reference-based metrics are only as good as their references. Work such as [Freitag et al. (2020)](https://md.research.elamur.ai/free/en/papers/freitag-2020-bleu-might-be-guilty-but/) and [Dhingra et al. (2019)](https://md.research.elamur.ai/free/en/papers/dhingra-2019-handling-divergent-reference-texts-when/) shows that poor or divergent references distort scores, which pushes the field toward reference-free and source-based assessment.

**Languages beyond English.** Most benchmarks and metrics were built for English or a few high-resource languages. Efforts such as [FLORES](https://md.research.elamur.ai/free/en/papers/guzman-2019-flores-evaluation-datasets-low-resource/), [Belebele](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/), [BLEnD](https://md.research.elamur.ai/free/en/papers/myung-2024-blend-benchmark-llms-everyday-knowledge/) and [IndicNLPSuite](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/) extend coverage, but translated test sets carry translationese and cultural mismatch.

**What a single number hides.** Aggregate scores conceal trade-offs between accuracy, robustness, fairness and cost. Multi-metric reporting, as in [ERASER](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/), and fine-grained error typing, as in [ERRANT](https://md.research.elamur.ai/free/en/papers/bryant-2017-automatic-annotation-evaluation-error-types/), are partial answers.

## Neighbouring topics

Evaluation touches almost every topic on the site, since each needs its own measures.

- [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) is the closest neighbour, with the largest overlap; the [machine translation shelf](https://md.research.elamur.ai/free/en/topics/machine-translation/) holds the wider literature.
- [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) supplies the systems that most recent benchmarks test, and [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) supplies many of the hardest tasks.
- [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/), [Summarization](https://md.research.elamur.ai/free/en/topics/summarization/), [Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) and [Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) each have metric debates of their own.
- [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) and [Low-resource languages](https://md.research.elamur.ai/free/en/topics/low-resource-languages/) contribute multilingual benchmarks; [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) and [Hebrew & Aramaic](https://md.research.elamur.ai/free/en/topics/hebrew-aramaic/) add Arabic and Hebrew test sets.
- [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) and [Dependency parsing](https://md.research.elamur.ai/free/en/topics/dependency-parsing/) have long histories of scoring disputes.
- [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) and [Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) extend evaluation to images, video and audio.
- [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) and [Sentiment & emotion](https://md.research.elamur.ai/free/en/topics/sentiment-emotion/) raise questions of disagreement and harm; [Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/) brings domain benchmarks; [Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) evaluates explanations.

Neighbouring topics: papers shared Machine translation shares the most: 202 of the 684 papers here (30%) stand on both.

**The numbers**

| Topic | Papers shared |
| --- | --- |
| [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) | 202 shared · 30% |
| [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) | 119 shared · 17% |
| [Semitic languages](https://md.research.elamur.ai/free/en/topics/semitic-languages/) | 78 shared · 11% |
| [English](https://md.research.elamur.ai/free/en/topics/english/) | 63 shared · 9% |
| [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) | 44 shared · 6% |
| [Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) | 38 shared · 6% |
| [Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) | 32 shared · 5% |
| [Coreference resolution](https://md.research.elamur.ai/free/en/topics/coreference-resolution/) | 28 shared · 4% |
| [Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) | 26 shared · 4% |
| [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/) | 24 shared · 4% |

On the site's shelves, the topic stands mostly on [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/), [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) and [Natural language processing](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/), with smaller parts on [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) and [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/).

The shelves this topic stands on Artificial Intelligence: 662 of the 684 papers here.

**The numbers**

| Shelf | Papers |
| --- | --- |
| [Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) | 662 papers |
| [Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) | 639 papers |
| [Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) | 78 papers |
| [Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) | 41 papers |
| [Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) | 19 papers |
| [Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) | 18 papers |
| [Literature and Literary Theory](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/literature-and-literary-theory/) | 14 papers |
| [Developmental and Educational Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/developmental-and-educational-psychology/) | 13 papers |
| [Social Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/social-psychology/) | 13 papers |
| [Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) | 12 papers |
| [Signal Processing](https://md.research.elamur.ai/free/en/shelves/computer-science/signal-processing/) | 12 papers |
| [Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) | 7 papers |

## Key works in this library

- [BLEU (Papineni et al.)](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) — the n-gram precision metric that started automatic MT evaluation.
- [METEOR](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/) — alignment-based metric with stems, synonyms and a recall emphasis.
- [chrF](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/) — character n-gram F-score that needs no tokeniser.
- [Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/) — the recall-based groundwork for ROUGE.
- [A Call for Clarity in Reporting BLEU Scores](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/) — SacreBLEU and standardised metric signatures.
- [A Structured Review of the Validity of BLEU](https://md.research.elamur.ai/free/en/papers/reiter-2018-structured-review-validity-bleu/) — what the evidence does and does not support.
- [COMET](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) — a learned metric over a cross-lingual encoder.
- [Results of WMT22 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/) — the recommendation to move from BLEU to neural metrics.
- [Experts, Errors, and Context](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/) — expert MQM annotation as a new human gold standard.
- [Has Machine Translation Achieved Human Parity?](https://md.research.elamur.ai/free/en/papers/laubli-2018-has-machine-translation-achieved-human/) — the case for document-level human evaluation.
- [Findings of the 2016 Conference on Machine Translation](https://md.research.elamur.ai/free/en/papers/bojar-2016-findings-2016-conference-machine-translation/) — the first WMT as a full conference.
- [GLUE](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) — the multi-task understanding benchmark with a single score.
- [SuperGLUE](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/) — its harder successor after rapid saturation.
- [Natural Questions](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) — question answering from real search queries.
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/) — distractor sentences that exposed shallow matching.
- [DROP](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) — reading comprehension that requires counting and arithmetic.
- [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) — HumanEval and the pass@k measure.
- [XTREME](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/) — zero-shot cross-lingual transfer across forty languages.
- [MTEB](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/) — a unified benchmark for text embeddings.
- [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) — a map of LLM evaluation.
- [MMLU-Pro](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/) — a harder, less prompt-sensitive knowledge benchmark.
- [GPQA](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/) — expert questions for oversight research.
- [MMMU](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/) — college-level multimodal reasoning.
- [Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/) — the POPE probe for invented objects.
- [On Coreference Resolution Performance Metrics](https://md.research.elamur.ai/free/en/papers/luo-2005-coreference-resolution-performance-metrics/) — the CEAF entity-alignment score.

## Papers (684)

**1** [2](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/2/) … [8](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/8/) [next →](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/2/)

1. [**Bleu: a Method for Automatic Evaluation of Machine Translation**Kishore Papineni, Salim Roukos, Todd Ward, et al.*2002 · 32,134 citations*](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/)
2. **[GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) [EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP](https://md.research.elamur.ai/free/en/venues/emnlp-workshop-blackboxnlp-analyzing-and-interpreting-neural-networks-for-nlp/)**Alex Wang, Amanpreet Singh, Julian Michael, et al.*2018 · 7,169 citations*
3. [**METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments**Satanjeev Banerjee, Alon Lavie*2005 · 5,504 citations*](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/)
4. [**A Call for Clarity in Reporting BLEU Scores**Matt Post*2018 · 4,135 citations*](https://md.research.elamur.ai/free/en/papers/post-2018-call-clarity-reporting-bleu-scores/)
5. **[Natural Questions: A Benchmark for Question Answering Research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) [Transactions of the Association for Computational Linguistics (TACL)](https://md.research.elamur.ai/free/en/venues/tacl/)**Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, et al.*2019 · 4,035 citations*
6. **[A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) [ACM Transactions on Intelligent Systems and Technology](https://md.research.elamur.ai/free/en/venues/acm-transactions-on-intelligent-systems-and-technology/)**Yupeng Chang, Xu Wang, Jindong Wang, et al.*2023 · 3,641 citations*
7. [**RCV1: A New Benchmark Collection for Text Categorization Research JMLR 5**David D. Lewis, Yiming Yang, Tony G. Rose, et al.*2004 · 2,752 citations*](https://md.research.elamur.ai/free/en/papers/lewis-2004-rcv1-benchmark-collection-text-categorization/)
8. [**Evaluating Large Language Models Trained on Code arXiv:2107.03374**Mark Chen, Jerry Tworek, Heewoo Jun, et al.*2021 · 2,548 citations*](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
9. **[Statistical Significance Tests for Machine Translation Evaluation](https://md.research.elamur.ai/free/en/papers/koehn-2004-statistical-significance-tests-machine-translation/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Philipp Koehn*2004 · 2,444 citations*
10. **[Meteor Universal: Language Specific Translation Evaluation for Any Target Language](https://md.research.elamur.ai/free/en/papers/denkowski-2014-meteor-universal-language-specific-translation/) [Conference on Machine Translation (WMT)](https://md.research.elamur.ai/free/en/venues/wmt/)**Michael Denkowski, Alon Lavie*2014 · 2,399 citations*
11. [**SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems arXiv:1905.00537**Alex Wang, Yada Pruksachatkun, Nikita Nangia, et al.*2019 · 2,190 citations*](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
12. [**chrF: character n-gram F-score for automatic MT evaluation**Maja Popović*2015 · 2,107 citations*](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/)
13. **[Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/) [Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)](https://md.research.elamur.ai/free/en/venues/naacl/)**Chin-Yew Lin, Eduard Hovy*2003 · 1,971 citations*
14. [**Adversarial Examples for Evaluating Reading Comprehension Systems**Robin Jia, Percy Liang*2017 · 1,942 citations*](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
15. **[“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/) [Annual Meeting of the Association for Computational Linguistics (ACL)](https://md.research.elamur.ai/free/en/venues/acl/)**William Yang Wang*2017 · 1,807 citations*
16. [**COMET: A Neural Framework for MT Evaluation**Ricardo Rei, Craig Stewart, Ana C Farinha, et al.*2020 · 1,784 citations*](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/)
17. **[How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://md.research.elamur.ai/free/en/papers/liu-2016-how-not-evaluate-your-dialogue/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Chia-Wei Liu, Ryan Lowe, Iulian Serban, et al.*2016 · 1,744 citations*
18. **[Evaluating WordNet-based Measures of Lexical Semantic Relatedness](https://md.research.elamur.ai/free/en/papers/budanitsky-2006-evaluating-wordnet-based-measures-lexical/) [Computational Linguistics](https://md.research.elamur.ai/free/en/venues/cl/)**Alexander Budanitsky, Graeme Hirst*2006 · 1,614 citations*
19. **[CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Jack Hessel, Ari Holtzman, Maxwell Forbes, et al.*2021 · 1,499 citations*
20. **[The Automatic Content Extraction (ACE) Program – Tasks, Data, and Evaluation](https://md.research.elamur.ai/free/en/papers/doddington-2004-automatic-content-extraction-ace-program/) [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/)**George Doddington, Alexis Mitchell, Mark Przybocki, et al.*2004 · 1,470 citations*
21. **[A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/) [Language Resources and Evaluation Conference (LREC)](https://md.research.elamur.ai/free/en/venues/lrec/)**Marco Marelli, Stefano Menini, Marco Baroni, et al.*2014 · 1,090 citations*
22. **[MTEB: Massive Text Embedding Benchmark](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-mteb-massive-text-embedding-benchmark/) [Conference of the European Chapter of the Association for Computational Linguistics (EACL)](https://md.research.elamur.ai/free/en/venues/eacl/)**Niklas Muennighoff, Nouamane Tazi, Loïc Magne, et al.*2022 · 1,021 citations*
23. **[Evaluating Object Hallucination in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-evaluating-object-hallucination-large-vision/) [Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/)**Yifan Li, Yifan Du, Kun Zhou, et al.*2023 · 986 citations*
24. [**GPQA: A Graduate-Level Google-Proof Q&A Benchmark arXiv:2311.12022**David Rein, Betty Li Hou, Asa Cooper Stickland, et al.*2023 · 964 citations*](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
25. [**TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification arXiv:2010.12421**Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, et al.*2020 · 952 citations*](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/)

**1** [2](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/2/) … [8](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/8/) [next →](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/page/2/)

Updated 24 October 2026

- [← previous topic English](https://md.research.elamur.ai/free/en/topics/english/)
- [next topic → French](https://md.research.elamur.ai/free/en/topics/french/)
- [↑ back to topics](https://md.research.elamur.ai/free/en/topics/)