--- title: "DnA-Eval — paper by Li et al. (2025) — summary and ideas" description: "The authors propose DnA-Eval, a zero-shot protocol modelled on scoring rubrics from pedagogy. 5 references, cited by 10 papers here." authors: ["Minzhi Li", "Zhengyuan Liu", "Shumin Deng", "Shafiq Joty", "Nancy Chen", "Min-Yen Kan"] year: 2025 venue: "Proceedings of the 31st International Conference on Computational Linguistics" source: "https://aclanthology.org/2025.coling-main.156/" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/li-2025-dna-eval-enhancing-large-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2025-dna-eval-enhancing-large-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) / DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation # DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation **Authors:** [Minzhi Li](https://md.research.elamur.ai/free/en/authors/minzhi-li/), [Zhengyuan Liu](https://md.research.elamur.ai/free/en/authors/zhengyuan-liu/), [Shumin Deng](https://md.research.elamur.ai/free/en/authors/shumin-deng/), [Shafiq Joty](https://md.research.elamur.ai/free/en/authors/shafiq-joty/), [Nancy Chen](https://md.research.elamur.ai/free/en/authors/nancy-chen/) and [Min-Yen Kan](https://md.research.elamur.ai/free/en/authors/min-yen-kan/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.156/) · [PDF](https://aclanthology.org/2025.coling-main.156.pdf) · [OpenAlex](https://openalex.org/W4399062248) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/li-2025-dna-eval-enhancing-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Using LLMs as judges of generated text is cheap and scalable, but how reliable they are remains unclear. Standard meta-evaluation asks a model once which of two responses is better and measures agreement with human labels. This hides the reasoning, so a model can match humans by luck. The authors propose DnA-Eval, a zero-shot protocol modelled on scoring rubrics from pedagogy. In the decomposition stage, the LLM uses the evaluation criteria given in the instruction, or invents its own when none are provided. It then scores both candidate responses on each aspect and proposes weights reflecting how much each aspect matters for that instance. In the aggregation stage, an external calculation module computes weighted overall scores and picks the winner. The approach needs no extra data and no fine-tuning, and it works with proprietary and open models. Across several meta-evaluation benchmarks it raised agreement by up to 39.6% compared with direct scoring and CoT. Its intermediate outputs (aspects and weights) also make the judging process easier to inspect. ## Key points - Evaluated with ChatGPT, GPT-4, Llama2-13B and Mistral-7B on benchmarks including MTBench, LLMBar, FairEval and InstruSum; the reported gain reaches 39.6%. - Human raters judged model-proposed aspects on a 1–5 Likert scale. Average scores ranged from 4.64 to 4.95 for relevance, clarity and comprehensiveness. - Weight analysis on MTBench task categories showed that models adapt to the task, for example giving less weight to creativity and more to accuracy in math than in writing. - Rankings of model weights were compared with crowdworker rankings using top-k Kendall's τ distance. Models generally diverge from humans more than two human annotators diverge from each other. ChatGPT came closest to human weightings and Mistral was furthest. - Limitations include higher inference cost and a fixed number of aspects (three). The authors also note that human tie labels can understate the model: GPT-4 sometimes spotted subtle differences that annotators had labelled as ties. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [National University of Singapore](https://md.research.elamur.ai/free/en/universities/national-university-of-singapore/) — University, Singapore, Singapore · 207 works · 39,576 citations - [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 184 works · 27,484 citations - [Institute for Infocomm Research](https://md.research.elamur.ai/free/en/labs/institute-for-infocomm-research/) — Research institute, Singapore, Singapore · 45 works · 14,598 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works ## Cite Cite as: Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen and Min-Yen Kan. 2025. DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation. Proceedings of the 31st International Conference on Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{li2025dna, title = {{DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation}}, author = {Minzhi Li, Zhengyuan Liu, Shumin Deng, Shafiq Joty, Nancy Chen and Min-Yen Kan}, booktitle = {Proceedings of the 31st International Conference on Computational Linguistics}, year = {2025}, url = {https://aclanthology.org/2025.coling-main.156/} } ``` - [← previous Disentangling Language and Culture for Evaluating Multilingual Large Language Models](https://md.research.elamur.ai/free/en/papers/ying-2025-disentangling-language-culture-evaluating-multilingual/) - [next → EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport Alignments](https://md.research.elamur.ai/free/en/papers/truong-2025-emo-embedding-model-distillation-intra/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/9/) ## References 5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. [A multi-task, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/). arXiv preprint arXiv:2302.04023. 2. Susan M Brookhart. 2018. Appropriate criteria: Key to effective rubrics. In Frontiers in Education, volume 3, page 22. Frontiers Media SA. [find](https://scholar.google.com/scholar?q=Susan+M+Brookhart%2E+2018%2E+Appropriate+criteria%3A+Key+to+effective+rubrics%2E+In+Frontiers+in+Education%2C+volume+3%2C+page+22%2E+Frontiers+Media+SA%2E) 3. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. [Chateval: Towards better llm-based evaluators through multi-agent debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/). arXiv preprint arXiv:2308.07201. 4. Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study. arXiv preprint arXiv:2304.00723. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) [find](https://scholar.google.com/scholar?q=Yi+Chen%2C+Rui+Wang%2C+Haiyun+Jiang%2C+Shuming+Shi%2C+and+Ruifeng+Xu%2E+2023%2E+Exploring+the+use+of+large+language+models+for+reference%2Dfree+text+quality+evaluation%3A+A+preliminary+empirical+study%2E+arXiv+preprint) 5. Steffi Chern, Ethan Chern, Graham Neubig, and Pengfei Liu. 2024. Can large language models be trusted for evaluation? scalable meta-evaluation of llms as evaluators via agent debate. arXiv preprint arXiv:2401.16788. [find](https://scholar.google.com/scholar?q=Steffi+Chern%2C+Ethan+Chern%2C+Graham+Neubig%2C+and+Pengfei+Liu%2E+2024%2E+Can+large+language+models+be+trusted+for+evaluation%3F+scalable+meta%2Devaluation+of+llms+as+evaluators+via+agent+debate%2E+arXiv+preprint+ar) *Updated 30 October 2026*