---
title: "CT-FineBench — paper by Yuan et al. (2026)"
description: "Existing metrics for automatically generated CT reports are poor at checking whether diagnostic details are correct. 26 references."
authors: ["Ruifeng Yuan", "Wanxing Chang", "Weiwei Cao", "Bowen Shi", "Zhongyu Wei", "Ling Zhang", "Jianpeng Zhang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2038/"
doi: "10.18653/v1/2026.acl-long.2038"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yuan-2026-ct-finebench-diagnostic-fidelity-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuan-2026-ct-finebench-diagnostic-fidelity-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/) / CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation

# CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation

**Authors:** [Ruifeng Yuan](https://md.research.elamur.ai/free/en/authors/ruifeng-yuan/), Wanxing Chang, Weiwei Cao, [Bowen Shi](https://md.research.elamur.ai/free/en/authors/bowen-shi/), [Zhongyu Wei](https://md.research.elamur.ai/free/en/authors/zhongyu-wei/), Ling Zhang and Jianpeng Zhang · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2038/) · [PDF](https://aclanthology.org/2026.acl-long.2038.pdf) · [DOI 10.18653/v1/2026.acl-long.2038](https://doi.org/10.18653/v1/2026.acl-long.2038) · [OpenAlex](https://openalex.org/W7166864394)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Existing metrics for automatically generated CT reports are poor at checking whether diagnostic details are correct. Lexical scores (BLEU, ROUGE), embedding similarity (BERTScore), entity matching (RadGraph, RaTEScore) and LLM judges (GREEN) either reward surface similarity, use overly coarse entity types, or give opaque verdicts. CT-FineBench addresses this by treating evaluation as fact-checking through question answering.

The benchmark was built from CT-RATE (chest) and Merlin (abdomen) in two steps:
- **Attribute schema:** An LLM mined finding–attribute–content triplets from reference reports. Rare groups were filtered out. Human annotators then removed, split, merged and documented attributes such as location, size, density and margin.
- **QA pairs:** Each positive finding's attributes in the gold report were turned into questions.

At evaluation time, an LLM answers these questions from the generated report and compares its answers with the gold answers. The resulting score agrees with expert ratings better than prior metrics in most settings and catches fine-grained errors more reliably. This gives an interpretable, attribute-level standard for judging clinical trustworthiness.

## Key points
- **Human study:** 100 generated reports per dataset (CT-CHAT outputs for CT-RATE, Merlin outputs for Merlin) were rated by two experts on a 10-point scale.
- **CT-RATE results:** CT-FineBench reached a Pearson correlation of 0.622 with human scores, compared with 0.521 for RaTEScore, the strongest baseline. It also led on Kendall and Spearman.
- **Merlin results:** CT-FineBench had the best Kendall (0.306) and Spearman (0.401) correlations. On Pearson, RadGraph was slightly higher (0.369 vs 0.326).
- **Smaller evaluator models:** The pipeline uses Qwen3-Max by default. Swapping in Qwen3-32b produced scores correlating above 0.9 with Qwen3-Max and nearly identical accuracy, so the benchmark can run locally.
- **Limitations:** The metric is recall-oriented and does not penalize hallucinated findings absent from the reference. Coverage is limited to the findings labeled in the source datasets. A training-set version, CT-FineData, was released but not yet used to train models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 45 works · 6,874 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works · [🩺 Medicine](https://md.research.elamur.ai/free/en/shelves/medicine/) · 275 works · [🩺 Health Informatics](https://md.research.elamur.ai/free/en/shelves/medicine/health-informatics/) · 214 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Ruifeng Yuan, Wanxing Chang, Weiwei Cao, Bowen Shi, Zhongyu Wei, Ling Zhang and Jianpeng Zhang. 2026. CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yuan2026ct,
  title = {{CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation}},
  author = {Ruifeng Yuan, Wanxing Chang, Weiwei Cao, Bowen Shi, Zhongyu Wei, Ling Zhang and Jianpeng Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2038},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2038}
}
```

- [← previous CSULoRA: Closest Safe Update Low-Rank Adaptation](https://md.research.elamur.ai/free/en/papers/marchenko-2026-csulora-closest-safe-update-low/)
- [next → CUET_SYNTHETICA@EEUCA 2026: Gated Cross-Modal Attention with Domain-Adapted Text Encoding for Vaccine-Critical Meme Detection](https://md.research.elamur.ai/free/en/papers/zaman-2026-cuet-synthetica-eeuca-2026-gated/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/28/)

## References

26 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, and 1 others. 2024. Merlin: A vision language foundation model for 3d computed tomography. Research Square, pages rs–3. [find](https://scholar.google.com/scholar?q=Louis+Blankemeier%2C+Joseph+Paul+Cohen%2C+Ashwin+Kumar%2C+Dave+Van+Veen%2C+Syed+Jamal+Safdar+Gardezi%2C+Magdalini+Paschali%2C+Zhihong+Chen%2C+Jean%2DBenoit+Delbrouck%2C+Eduardo+Reis%2C+Cesar+Truyts%2C+and+1+others%2E+2024%2E+M)
2. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
3. Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. [Qafacteval: Improved qa-based factual consistency evaluation for summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/). In Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technolo…
4. Ibrahim Ethem Hamamci, Sezgin Er, and Bjoern Menze. 2024a. Ct2rep: Automated radiology report generation for 3d medical imaging. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 476–486. Springer. [find](https://scholar.google.com/scholar?q=Ibrahim+Ethem+Hamamci%2C+Sezgin+Er%2C+and+Bjoern+Menze%2E+2024a%2E+Ct2rep%3A+Automated+radiology+report+generation+for+3d+medical+imaging%2E+In+International+Conference+on+Medical+Image+Computing+and+Computer%2DAss)
5. Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, and 1 others. 2024b. [find](https://scholar.google.com/scholar?q=Ibrahim+Ethem+Hamamci%2C+Sezgin+Er%2C+Chenyu+Wang%2C+Furkan+Almas%2C+Ayse+Gulnihan+Simsek%2C+Sevval+Nil+Esirgun%2C+Irem+Dogan%2C+Omer+Faruk+Durugol%2C+Benjamin+Hou%2C+Suprosanna+Shit%2C+and+1+others%2E+2024b%2E)
6. Developing generalist foundation models from a multimodal dataset for 3d computed tomography. arXiv preprint arXiv:2403.17834. — [arXiv:2403.17834](https://arxiv.org/abs/2403.17834) [find](https://scholar.google.com/scholar?q=Developing+generalist+foundation+models+from+a+multimodal+dataset+for+3d+computed+tomography%2E+arXiv+preprint+arXiv%3A2403%2E17834%2E+%E2%80%94+%5BarXiv%3A2403%2E17834%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2403%2E17834%29)
7. Philipp Harzig, Yan-Ying Chen, Francine Chen, and Rainer Lienhart. 2019. Addressing data bias problems for chest x-ray image report generation. arXiv preprint arXiv:1908.02123. — [arXiv:1908.02123](https://arxiv.org/abs/1908.02123) [find](https://scholar.google.com/scholar?q=Philipp+Harzig%2C+Yan%2DYing+Chen%2C+Francine+Chen%2C+and+Rainer+Lienhart%2E+2019%2E+Addressing+data+bias+problems+for+chest+x%2Dray+image+report+generation%2E+arXiv+preprint+arXiv%3A1908%2E02123%2E+%E2%80%94+%5BarXiv%3A1908%2E02123%5D%28ht)
8. Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, and 1 others. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106.14463. — [arXiv:2106.14463](https://arxiv.org/abs/2106.14463) [find](https://scholar.google.com/scholar?q=Saahil+Jain%2C+Ashwin+Agrawal%2C+Adriel+Saporta%2C+Steven+QH+Truong%2C+Du+Nguyen+Duong%2C+Tan+Bui%2C+Pierre+Chambon%2C+Yuhao+Zhang%2C+Matthew+P+Lungren%2C+Andrew+Y+Ng%2C+and+1+others%2E+2021%2E+Radgraph%3A+Extracting+clinical+)
9. Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, and 1 others. 2025. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668. [find](https://scholar.google.com/scholar?q=Songtao+Jiang%2C+Yuan+Wang%2C+Sibo+Song%2C+Tianxiang+Hu%2C+Chenyi+Zhou%2C+Bin+Pu%2C+Yan+Zhang%2C+Zhibo+Yang%2C+Yang+Feng%2C+Joey+Tianyi+Zhou%2C+and+1+others%2E+2025%2E+Hulu%2Dmed%3A+A+transparent+generalist+model+towards+holisti)
10. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. [Llava-med: Training a large language-and-vision assistant for biomedicine in one day](https://md.research.elamur.ai/free/en/papers/li-2023-llava-med-training-large-language/). Advances in Neural Information Processing Systems, 36:28541–28564.
11. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
12. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
13. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. [Med-flamingo: a multimodal medical few-shot learner](https://md.research.elamur.ai/free/en/papers/moor-2023-med-flamingo-multimodal-medical-few/). In Machine Learning for Health (ML4H), pages 353–367. PMLR.
14. Nassir Navab, J Hornegger, WM Wells, AF Frangi, and D Hutchison. 2015. Medical image computing and computer-assisted intervention. In Proceedings of the MICCAI 2015 18th International Conference, Munich, Germany, pages 5–9. [find](https://scholar.google.com/scholar?q=Nassir+Navab%2C+J+Hornegger%2C+WM+Wells%2C+AF+Frangi%2C+and+D+Hutchison%2E+2015%2E+Medical+image+computing+and+computer%2Dassisted+intervention%2E+In+Proceedings+of+the+MICCAI+2015+18th+International+Conference%2C+Muni)
15. OpenAI. 2025. Gpt-5 system openai.com/index/gpt-5-system-card. card. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Gpt%2D5+system+openai%2Ecom%2Findex%2Fgpt%2D5%2Dsystem%2Dcard%2E+card%2E)
16. Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson Md, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, and 1 others. 2024. Green: Generative radiology report evaluation and error notation. In Findings of the association for compu… [find](https://scholar.google.com/scholar?q=Sophie+Ostmeier%2C+Justin+Xu%2C+Zhihong+Chen%2C+Maya+Varma%2C+Louis+Blankemeier%2C+Christian+Bluethgen%2C+Arne+Edward+Michalson+Md%2C+Michael+Moseley%2C+Curtis+Langlotz%2C+Akshay+S+Chaudhari%2C+and+1+others%2E+2024%2E+Green%3A)
17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
18. Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren. 2020. Chexbert: [combining automatic labelers and expert annotations for accurate radiology report labeling using bert](https://md.research.elamur.ai/free/en/papers/smit-2020-combining-automatic-labelers-expert-annotations/). arXiv preprint arXiv:2004.09167.
19. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. 2025. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications, 16(1):7866. [find](https://scholar.google.com/scholar?q=Chaoyi+Wu%2C+Xiaoman+Zhang%2C+Ya+Zhang%2C+Hui+Hui%2C+Yanfeng+Wang%2C+and+Weidi+Xie%2E+2025%2E+Towards+generalist+foundation+model+for+radiology+by+leveraging+web%2Dscale+2d%263d+medical+data%2E+Nature+Communications%2C+16%28)
20. Yu Xin, Gorkem Can Ates, Kuang Gong, and Wei Shao. 2025. Med3dvlm: An efficient vision-language model for 3d medical image analysis. arXiv preprint arXiv:2503.20047. — [arXiv:2503.20047](https://arxiv.org/abs/2503.20047) [find](https://scholar.google.com/scholar?q=Yu+Xin%2C+Gorkem+Can+Ates%2C+Kuang+Gong%2C+and+Wei+Shao%2E+2025%2E+Med3dvlm%3A+An+efficient+vision%2Dlanguage+model+for+3d+medical+image+analysis%2E+arXiv+preprint+arXiv%3A2503%2E20047%2E+%E2%80%94+%5BarXiv%3A2503%2E20047%5D%28https%3A%2F%2Farxiv)
21. Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, and 1 others. 2025. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044. — [arXiv:2506.07044](https://arxiv.org/abs/2506.07044) [find](https://scholar.google.com/scholar?q=Weiwen+Xu%2C+Hou+Pong+Chan%2C+Long+Li%2C+Mahani+Aljunied%2C+Ruifeng+Yuan%2C+Jianyu+Wang%2C+Chenghao+Xiao%2C+Guizhen+Chen%2C+Chaoqun+Liu%2C+Zhaodonghui+Li%2C+and+1+others%2E+2025%2E+Lingshu%3A+A+generalist+foundation+model+for+)
22. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
23. Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y Ng, and 1 others. 2023. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9). [find](https://scholar.google.com/scholar?q=Feiyang+Yu%2C+Mark+Endo%2C+Rayan+Krishnan%2C+Ian+Pan%2C+Andy+Tsai%2C+Eduardo+Pontes+Reis%2C+Eduardo+Kaiser+Ururahy+Nunes+Fonseca%2C+Henrique+Min+Ho+Lee%2C+Zahra+Shakeri+Hossein+Abad%2C+Andrew+Y+Ng%2C+and+1+others%2E+2023%2E+)
24. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
25. Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024. Ratescore: A metric for radiology report generation. arXiv preprint arXiv:2406.16845. [find](https://scholar.google.com/scholar?q=Weike+Zhao%2C+Chaoyi+Wu%2C+Xiaoman+Zhang%2C+Ya+Zhang%2C+Yanfeng+Wang%2C+and+Weidi+Xie%2E+2024%2E+Ratescore%3A+A+metric+for+radiology+report+generation%2E+arXiv+preprint+arXiv%3A2406%2E16845%2E)
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in neural information processing systems, 36:46595–46623.

*Updated 2 November 2026*