--- title: "MENLI — paper by Chen and Eger (2023) — summary and ideas" description: "They propose MENLI, a family of metrics that take pre-trained NLI models off the shelf… 43% accuracy. 1 reference, cited by 36 papers here." authors: ["Yanran Chen", "Steffen Eger"] year: 2023 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2023.tacl-1.47/" doi: "10.1162/tacl_a_00576" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/chen-2023-menli-robust-evaluation-metrics-natural/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-menli-robust-evaluation-metrics-natural/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / MENLI: Robust Evaluation Metrics from Natural Language Inference # MENLI: Robust Evaluation Metrics from Natural Language Inference **Authors:** [Yanran Chen](https://md.research.elamur.ai/free/en/authors/yanran-chen/) and [Steffen Eger](https://md.research.elamur.ai/free/en/authors/steffen-eger/) · **Year:** 2023 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2023](https://md.research.elamur.ai/free/en/venues/tacl/2023/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2023.tacl-1.47/) · [PDF](https://aclanthology.org/2023.tacl-1.47.pdf) · [DOI 10.1162/tacl\_a\_00576](https://doi.org/10.1162/tacl_a_00576) · [OpenAlex](https://openalex.org/W4384211370) **Citations:** 38 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/chen-2023-menli-robust-evaluation-metrics-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Popular BERT-based metrics for text generation, such as BERTScore and MoverScore, score candidates by how semantically close they are to a reference or source. Chen and Eger argue that this is the wrong objective. A translation can be very similar to the reference and still contain wrong numbers, names or facts. They propose MENLI, a family of metrics that take pre-trained NLI models off the shelf and score candidates by entailment (bi-directional for references, source-to-summary for summarization). The work covers MT and summarization, in both reference-based and reference-free settings. To test robustness, the authors build a preference-based adversarial suite that needs no human annotators. Each item asks whether a metric prefers a correct paraphrase over a fluent but factually corrupted candidate. **Main results:** - NLI metrics are far more robust than standard metrics. - On standard benchmarks they beat existing summarization metrics but trail SOTA MT metrics. - Combining NLI metrics with existing ones improves both robustness and correlation with human judgments. This suggests that inference-based modeling is a promising basis for trustworthy evaluation. ## Key points - On the adversarial suite, BLEU, ROUGE, MoverScore and BERTScore perform at or below chance. In MT, NLI metrics beat the best unsupervised standard metric by roughly 30–50%. - Ensembles of standard and NLI metrics gain about 15–30% in adversarial robustness and about 5–30% on standard benchmarks. One exception: cross-lingual (reference-free MT) NLI metrics are weak, so standard metrics make better ensemble partners there. - On WMT20 de, RoMe averages 43% accuracy versus over 80% for the NLI metrics. On BAGEL both reach about 0.23 Spearman's ρ. - Sentence-level NLI variants for summarization scored lower than plain NLI-R on adversarial data (NLI-R 72%): - SummaCZS: 53% - FalseSum-tuned models: about 55–68% - Code and data are available at github.com/cyr19/MENLI. Choosing the pooling strategy on held-out datasets changes results only slightly (about −1.9% to +0.8%). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) — University, Darmstadt, Germany · 109 works · 35,920 citations - [Bielefeld University](https://md.research.elamur.ai/free/en/universities/bielefeld-university/) — University, Bielefeld, Germany · 35 works · 1,196 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works ## Cite Cite as: Yanran Chen and Steffen Eger. 2023. MENLI: Robust Evaluation Metrics from Natural Language Inference. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{chen2023menli, title = {{MENLI: Robust Evaluation Metrics from Natural Language Inference}}, author = {Yanran Chen and Steffen Eger}, journal = {Transactions of the Association for Computational Linguistics}, year = {2023}, doi = {10.1162/tacl\_a\_00576}, url = {https://doi.org/10.1162/tacl_a_00576} } ``` - [← previous AlignBench: Benchmarking Chinese Alignment of Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-alignbench-benchmarking-chinese-alignment/) - [next → ReLoRA: High-Rank Training Through Low-Rank Updates](https://md.research.elamur.ai/free/en/papers/lialin-2023-relora-high-rank-training-through/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. 2019. [On adversarial removal of hypothesis-only bias in natural language inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-adversarial-removal-hypothesis-only-bias/). In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 256–262, Minneapolis, Min… *Updated 31 October 2026*