---
title: "Benchmarking the Fine-Grained Discriminability in Image-Text"
description: "The authors build two benchmarks, MSCOCO-CCD and Flickr30k-CCD, to address this. 26 references."
authors: ["Zhen Wang", "Xi Zhou", "Yating Yang", "Bo Ma", "Lei Wang", "Rui Dong", "Azmat Anwar", "Siru Miao"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1525/"
doi: "10.18653/v1/2026.findings-acl.1525"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-benchmarking-fine-grained-discriminability-image/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-benchmarking-fine-grained-discriminability-image/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/) / Benchmarking the Fine-Grained Discriminability in Image-Text Retrieval via Controlled Contrastive Differences

# Benchmarking the Fine-Grained Discriminability in Image-Text Retrieval via Controlled Contrastive Differences

**Authors:** [Zhen Wang](https://md.research.elamur.ai/free/en/authors/zhen-wang/), Xi Zhou, Yating Yang, Bo Ma, [Lei Wang](https://md.research.elamur.ai/free/en/authors/lei-wang/), Rui Dong, Azmat Anwar and Siru Miao · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1525/) · [PDF](https://aclanthology.org/2026.findings-acl.1525.pdf) · [DOI 10.18653/v1/2026.findings-acl.1525](https://doi.org/10.18653/v1/2026.findings-acl.1525) · [OpenAlex](https://openalex.org/W7166827557)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Image-text retrieval models often return candidates that match a query broadly but get specific details wrong. Standard benchmarks such as MSCOCO and Flickr30k cannot measure this weakness because their captions and candidate pools are coarse-grained. The authors build two benchmarks, MSCOCO-CCD and Flickr30k-CCD, to address this.

Construction follows a two-level taxonomy of image content with four top-level dimensions: entity, scene, event/action, and style & presentation. Using a Human-LLM collaborative pipeline, they create contrastive samples that each differ from an anchor in exactly one labeled aspect. These samples come either from licensed web image search or from controlled image editing. The captions are then rewritten so they are detailed enough to tell similar images apart.

They also propose a pairwise metric, FG-CDA, that tests whether a model ranks the true match above its contrastive twin. Experiments on 25 embedding models show that contrastive samples sharply reduce retrieval quality. Subtle features such as emotions and scene attributes remain especially hard. The resource therefore enables diagnosis of retrieval failures by feature type.

## Key points
- **Pipeline:**
  - Qwen3-VL-32B proposes key details and contrastive strategies, which two human annotators verify.
  - Contrastive images are obtained via the Google Image Search API (restricted to public-domain/CC licenses) or generated with Qwen-Image-Edit.
  - Humans filter low-quality or overly different images and refine the captions.
- **Scale:**
  - MSCOCO-CCD has 19,590 images, compared with 5,000 in the original test set.
  - Flickr30k-CCD has 3,560 images, compared with 1,000.
  - Average caption length rises from roughly 12–14 to about 30 tokens.
  - Top-5 candidate image similarity increases, for example from 0.449 to 0.631 on MSCOCO.
- **Quality and distribution:**
  - Human evaluation reports annotation accuracy above 95%.
  - Entity attribute and scene attribute are the most frequent contrast types.
  - Entity emotion is rare, at 1–4% of samples.
- **Evaluation:**
  - Metrics are Recall@1/5 plus FG-CDA.
  - The models tested are 11 CLIP-series models (CLIP, BLIP, SigLIP, AltCLIP, BGE-VL) and 14 MLLM-based models (VLM2Vec, GME, Ops-MM-embedding, UniME, BGE-VL, RzenEmbed-v2).
- **Findings:**
  - I2T accuracy on entity emotion falls below 70%.
  - Models with strong overall retrieval, such as RzenEmbed-v2-7B, also discriminate fine-grained differences best.
  - Code and data are released at github.com/miaomiao1215/CCD.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations

Also on the byline: [Xinjiang Technical Institute of Physics & Chemistry](https://ror.org/00x44h034).

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Zhen Wang, Xi Zhou, Yating Yang, Bo Ma, Lei Wang, Rui Dong, Azmat Anwar and Siru Miao. 2026. Benchmarking the Fine-Grained Discriminability in Image-Text Retrieval via Controlled Contrastive Differences. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026benchmarking,
  title = {{Benchmarking the Fine-Grained Discriminability in Image-Text Retrieval via Controlled Contrastive Differences}},
  author = {Zhen Wang, Xi Zhou, Yating Yang, Bo Ma, Lei Wang, Rui Dong, Azmat Anwar and Siru Miao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1525},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1525}
}
```

- [← previous Benchmarking Text Embedding Models for South African Languages](https://md.research.elamur.ai/free/en/papers/villiers-2026-benchmarking-text-embedding-models-south/)
- [next → Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces](https://md.research.elamur.ai/free/en/papers/shi-2026-benchmarking-web-agent-safety-under/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/23/)

## References

26 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra,… — [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-3-technical-report-highly/)
2. Ibrahim M. Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. 2023. Getting vit in shape: Scaling laws for compute-optimal model design. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orlean… [find](https://scholar.google.com/scholar?q=Ibrahim+M%2E+Alabdulmohsin%2C+Xiaohua+Zhai%2C+Alexander+Kolesnikov%2C+and+Lucas+Beyer%2E+2023%2E+Getting+vit+in+shape%3A+Scaling+laws+for+compute%2Doptimal+model+design%2E+In+Advances+in+Neural+Information+Processing+S)
3. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. [Qwen3-vl technical report](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen3-vl-technical-report/). Preprint, ar…
4. Weijing Chen, Linli Yao, and Qin Jin. 2023a. Rethinking benchmarks for cross-modal image-text retrieval. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 1241–1251. ACM. [find](https://scholar.google.com/scholar?q=Weijing+Chen%2C+Linli+Yao%2C+and+Qin+Jin%2E+2023a%2E+Rethinking+benchmarks+for+cross%2Dmodal+image%2Dtext+retrieval%2E+In+Proceedings+of+the+46th+International+ACM+SIGIR+Conference+on+Research+and+Development+in+In)
5. Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Qinghong Yang, and Ledell Wu. 2023b. Altclip: Altering the language encoder in CLIP for extended language capabilities. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 8666–8682. Association for C… [find](https://scholar.google.com/scholar?q=Zhongzhi+Chen%2C+Guang+Liu%2C+Bo%2DWen+Zhang%2C+Qinghong+Yang%2C+and+Ledell+Wu%2E+2023b%2E+Altclip%3A+Altering+the+language+encoder+in+CLIP+for+extended+language+capabilities%2E+In+Findings+of+the+Association+for+Compu)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In 9th In…
8. Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. 2025. Breaking the modality barrier: Universal embedding learning with multimodal llms. CoRR, abs/2504.17432. — [arXiv:2504.17432](https://arxiv.org/abs/2504.17432) [find](https://scholar.google.com/scholar?q=Tiancheng+Gu%2C+Kaicheng+Yang%2C+Ziyong+Feng%2C+Xingjun+Wang%2C+Yanzhao+Zhang%2C+Dingkun+Long%2C+Yingda+Chen%2C+Weidong+Cai%2C+and+Jiankang+Deng%2E+2025%2E+Breaking+the+modality+barrier%3A+Universal+embedding+learning+with)
9. Weijian Jian, Yajun Zhang, Dawei Liang, Chunyu Xie, Yixiao He, Dawei Leng, and Yuhui Yin. 2025. Rzenembed: Towards comprehensive multimodal retrieval. CoRR, abs/2510.27350. and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30: Annual Conferen… — [arXiv:2510.27350](https://arxiv.org/abs/2510.27350)
10. Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, and Wenhu Chen. 2025. Vlm2vec: Training vision-language models for massive multimodal embedding tasks. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ziyan+Jiang%2C+Rui+Meng%2C+Xinyi+Yang%2C+Semih+Yavuz%2C+Yingbo+Zhou%2C+and+Wenhu+Chen%2E+2025%2E+Vlm2vec%3A+Training+vision%2Dlanguage+models+for+massive+multimodal+embedding+tasks%2E+In+The+Thirteenth+International+Conf)
11. Edward Vendrow, Omiros Pantazis, Alexander Shepard, Gabriel J. Brostow, Kate E. Jones, Oisin Mac Aodha, Sara Beery, and Grant Van Horn. 2024. INQUIRE: A natural world text-to-image retrieval benchmark. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information P… [find](https://scholar.google.com/scholar?q=Edward+Vendrow%2C+Omiros+Pantazis%2C+Alexander+Shepard%2C+Gabriel+J%2E+Brostow%2C+Kate+E%2E+Jones%2C+Oisin+Mac+Aodha%2C+Sara+Beery%2C+and+Grant+Van+Horn%2E+2024%2E+INQUIRE%3A+A+natural+world+text%2Dto%2Dimage+retrieval+benchmark)
12. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. [BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of…
13. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. [Qwen2-vl: Enhancing vision-language model’s perception of the…](https://md.research.elamur.ai/free/en/papers/wang-2024-qwen2-vl-enhancing-vision/)
14. Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. [BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation](https://md.research.elamur.ai/free/en/papers/li-2022-blip-bootstrapping-language-image-pre/). In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings…
15. Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. 2024b. Cross-modal retrieval: A systematic review of methods and future directions. Proc. IEEE, 112(11):1716–1754. [find](https://scholar.google.com/scholar?q=Tianshi+Wang%2C+Fengling+Li%2C+Lei+Zhu%2C+Jingjing+Li%2C+Zheng+Zhang%2C+and+Heng+Tao+Shen%2E+2024b%2E+Cross%2Dmodal+retrieval%3A+A+systematic+review+of+methods+and+future+directions%2E+Proc%2E+IEEE%2C+112%2811%29%3A1716%E2%80%931754%2E)
16. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. [Microsoft COCO: common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V,…
17. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. [Visual instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-visual-instruction-tuning/). In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
18. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the 38th Internatio…
19. Mingxing Tan and Quoc V. Le. 2019. [Efficientnet: Rethinking model scaling for convolutional neural networks](https://md.research.elamur.ai/free/en/papers/tan-2019-efficientnet-rethinking-model-scaling-convolutional/). In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6…
20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. [A survey on multimodal large language models](https://md.research.elamur.ai/free/en/papers/yin-2023-survey-multimodal-large-language-models/). CoRR, abs/2306.13549.
21. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. Assoc. Comput. Linguistics, 2:67–78. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Trans%2E+Assoc%2E+C)
22. Yan Zeng, Xinsong Zhang, and Hang Li. 2022. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 25994–2600… [find](https://scholar.google.com/scholar?q=Yan+Zeng%2C+Xinsong+Zhang%2C+and+Hang+Li%2E+2022%2E+Multi%2Dgrained+vision+language+pre%2Dtraining%3A+Aligning+texts+with+visual+concepts%2E+In+International+Conference+on+Machine+Learning%2C+ICML+2022%2C+17%2D23+July+2022)
23. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. [Sigmoid loss for language image pre-training](https://md.research.elamur.ai/free/en/papers/zhai-2023-sigmoid-loss-language-image-pre/). In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941–11952. IEEE.
24. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. GME: improving universal multimodal retrieval by multimodal llms. CoRR, abs/2412.16855. — [arXiv:2412.16855](https://arxiv.org/abs/2412.16855) [find](https://scholar.google.com/scholar?q=Xin+Zhang%2C+Yanzhao+Zhang%2C+Wen+Xie%2C+Mingxin+Li%2C+Ziqi+Dai%2C+Dingkun+Long%2C+Pengjun+Xie%2C+Meishan+Zhang%2C+Wenjie+Li%2C+and+Min+Zhang%2E+2024%2E+GME%3A+improving+universal+multimodal+retrieval+by+multimodal+llms%2E+CoR)
25. Junjie Zhou, Yongping Xiong, Zheng Liu, Ze Liu, Shitao Xiao, Yueze Wang, Bo Zhao, Chen Jason Zhang, and Defu Lian. 2025. [Megapairs: Massive data synthesis for universal multimodal retrieval](https://md.research.elamur.ai/free/en/papers/zhou-2025-megapairs-massive-data-synthesis-universal/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa…
26. Tester T2I I2T A B 96.5 95.8 97.8 97.5 *Table 3: The human evaluation accuracies on two tasks.* A T2I R1 R5 CLIP-series embedding models CLIP ViT-B(149M) 46.9 86.4 BGE-VL-base ViT-B(149M) 66.0 97.7 BLIP-coco ViT-B(149M) 58.1 94.4 BLIP-flickr30k ViT-B(149M) 62.1 96.5 ALIGN-base EfficientNet 53.9 95.0… [find](https://scholar.google.com/scholar?q=Tester+T2I+I2T+A+B+96%2E5+95%2E8+97%2E8+97%2E5+%2ATable+3%3A+The+human+evaluation+accuracies+on+two+tasks%2E%2A+A+T2I+R1+R5+CLIP%2Dseries+embedding+models+CLIP+ViT%2DB%28149M%29+46%2E9+86%2E4+BGE%2DVL%2Dbase+ViT%2DB%28149M%29+66%2E0+97%2E7+BL)

*Updated 1 November 2026*