---
title: "Semantic Contrastive Adaptation for Multimodal Figurative"
description: "The paper also releases FIGMENT, a multilingual figurative grounding test in five languages. 24 references."
authors: ["Ayaan Siddiqui"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2026.acl-srw.12/"
doi: "10.18653/v1/2026.acl-srw.12"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/siddiqui-2026-semantic-contrastive-adaptation-multimodal-figurative/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/siddiqui-2026-semantic-contrastive-adaptation-multimodal-figurative/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/) / Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding

# Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding

**Authors:** Ayaan Siddiqui · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-srw.12/) · [PDF](https://aclanthology.org/2026.acl-srw.12.pdf) · [DOI 10.18653/v1/2026.acl-srw.12](https://doi.org/10.18653/v1/2026.acl-srw.12) · [OpenAlex](https://openalex.org/W7166859064)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Vision–language models such as CLIP and BLIP-2 handle literal image–text matching well, but they perform poorly when an image must be matched to the non-literal sense of an idiom, metaphor or simile. They usually pick the image that depicts the words literally. The author argues this comes from a systematic preference for literal matches, not from too little model capacity. The task is therefore recast as contrastive "semantic deviation", in which the model is trained to score the figurative image higher than a literal distractor that looks visually plausible. To do this, CLIP is fine-tuned with LoRA on triples of a phrase, its figurative image and a mined hard literal negative.

On the IRFL benchmark, this beats zero-shot CLIP and BLIP-2, weighted and MLP ensembles of CLIP backbones, and ConceptNet-enriched text. The paper also releases FIGMENT, a multilingual figurative grounding test in five languages. Although trained only on English data, the adapted model transfers well to these languages. This suggests that lightweight, targeted adaptation can noticeably improve non-literal multimodal reasoning.

## Key points
- On the IRFL "mixed" task, LoRA-CLIP improves on zero-shot CLIP for all three categories:
  - idioms: about 72% vs 22%
  - metaphors: 48% vs 30%
  - similes: 62% vs 44%
  - Humans score 99.7%, so a gap remains.
- Common alternatives give little benefit:
  - Ensembles of ViT-B/32, ViT-B/16, RN50 and RN50×64 fail to consistently beat the strongest single backbone (RN50×64).
  - ConceptNet augmentation of the text also adds little.
- FIGMENT covers French, Chinese, Spanish, Hindi and Arabic.
  - The adapted model reaches 68.7–84.6% accuracy.
  - Zero-shot mCLIP reaches only 4.1–28.8%.
  - Literal bias falls to 15–31%, compared with up to 96% for mCLIP.
- The zero-shot models show near-zero or negative similarity margins between figurative and literal images. The adapted model shows clearly positive margins, which line up with its lower literal bias.
- Adapting the higher-level visual attention layers gives the largest gains. Text-side adaptation helps mainly for idioms when definitions are available.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Birla Institute of Technology and Science - Hyderabad Campus](https://md.research.elamur.ai/free/en/universities/birla-institute-of-technology-and-science-hyderabad-campus/) — University, Hyderabad, India · 2 works

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🖼️ Vision & language](https://md.research.elamur.ai/free/en/topics/multimodal/) · 657 works · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 217 works

## Cite

Cite as: Ayaan Siddiqui. 2026. Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{siddiqui2026semantic,
  title = {{Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding}},
  author = {Ayaan Siddiqui},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.12},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.12}
}
```

- [← previous Semantic Clustering of Obfuscated Command-Line Detections for Alert Reduction](https://md.research.elamur.ai/free/en/papers/stepankova-2026-semantic-clustering-obfuscated-command-line/)
- [next → Semantic Distance for Presburger Formula Generation](https://md.research.elamur.ai/free/en/papers/taniguchi-2026-semantic-distance-presburger-formula-generation/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/42/)

## References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, and Marzyeh Ghassemi. 2025. Vision-language models do not understand negation. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). [find](https://scholar.google.com/scholar?q=Kumail+Alhamoud%2C+Shaden+Alshammari%2C+Yonglong+Tian%2C+Guohao+Li%2C+Philip+Torr%2C+Yoon+Kim%2C+and+Marzyeh+Ghassemi%2E+2025%2E+Vision%2Dlanguage+models+do+not+understand+negation%2E+In+Proceedings+of+the+Conference+on+)
2. Julia Birke and Anoop Sarkar. 2006. A clustering approach for nearly unsupervised recognition of nonliteral language. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Julia+Birke+and+Anoop+Sarkar%2E+2006%2E+A+clustering+approach+for+nearly+unsupervised+recognition+of+nonliteral+language%2E+In+11th+Conference+of+the+European+Chapter+of+the+Association+for+Computational+Li)
3. Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. 2023. mclip: Multilingual clip via cross-lingual transfer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. A… [find](https://scholar.google.com/scholar?q=Guanhua+Chen%2C+Lu+Hou%2C+Yun+Chen%2C+Wenliang+Dai%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Qun+Liu%2C+Jia+Pan%2C+and+Wenping+Wang%2E+2023%2E+mclip%3A+Multilingual+clip+via+cross%2Dlingual+transfer%2E+In+Proceedings+of+the+61st+Annual+)
4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In Proceedings of the International Conference on Machine Learning (ICML).
5. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Dietterich%2E+2000%2E+Ensemble+methods+in+machine+learning%2E+In+Multiple+Classifier+Systems%2C+Berlin%2C+Heidelberg%2E+Springer+Berlin+Heidelberg%2E)
6. Dan Fass. 1991. [met*: A method for discriminating metonymy and metaphor by computer](https://md.research.elamur.ai/free/en/papers/fass-1991-met-method-discriminating-metonymy-metaphor/). Computational Linguistics, 17:49–90.
7. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. [Language-agnostic BERT sentence embedding](https://md.research.elamur.ai/free/en/papers/feng-2020-language-agnostic-bert-sentence-embedding/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL).
8. Md Kamrul Hasan, Wasifur Rahman, AmirAli Bagher Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, and Mohammed (Ehsan) Hoque. 2019. UR-FUNNY: A multimodal language dataset for understanding humor. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Proces… [find](https://scholar.google.com/scholar?q=Md+Kamrul+Hasan%2C+Wasifur+Rahman%2C+AmirAli+Bagher+Zadeh%2C+Jianyuan+Zhong%2C+Md+Iftekhar+Tanveer%2C+Louis%2DPhilippe+Morency%2C+and+Mohammed+%28Ehsan%29+Hoque%2E+2019%2E+UR%2DFUNNY%3A+A+multimodal+language+dataset+for+unders)
9. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, et al. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML). [find](https://scholar.google.com/scholar?q=Neil+Houlsby%2C+Andrei+Giurgiu%2C+Stanislaw+Jastrzebski%2C+et+al%2E+2019%2E+Parameter%2Defficient+transfer+learning+for+NLP%2E+In+Proceedings+of+the+36th+International+Conference+on+Machine+Learning+%28ICML%29%2E)
10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations (ICLR).
11. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. [Scaling up visual and vision-language representation learning with noisy text supervision](https://md.research.elamur.ai/free/en/papers/jia-2021-scaling-up-visual-vision-language/).
12. Hui Li, Peng Wang, and Chunhua Shen. 2020. Scene text recognition from two-dimensional perspective. In Proceedings of the AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Hui+Li%2C+Peng+Wang%2C+and+Chunhua+Shen%2E+2020%2E+Scene+text+recognition+from+two%2Ddimensional+perspective%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2E)
13. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. [Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models](https://md.research.elamur.ai/free/en/papers/li-2023-blip-2-bootstrapping-language-image/). ICML’23. JMLR.org.
14. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+ConceptNet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Proceedings+of+the+Thirty%2DFirst+AAAI+Conference+on+Artificial+Intelligence%2E)
15. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. [Representation learning with contrastive predictive coding](https://md.research.elamur.ai/free/en/papers/oord-2018-representation-learning-contrastive-predictive-coding/). CoRR.
16. Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023. IRFL: Image recognition of figurative language. In Findings of the Association for Computational Linguistics: EMNLP 2023. [find](https://scholar.google.com/scholar?q=Ron+Yosef%2C+Yonatan+Bitton%2C+and+Dafna+Shahaf%2E+2023%2E+IRFL%3A+Image+recognition+of+figurative+language%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2E)
17. Siting Li, Pang Wei Koh, and Simon Shaolei Du. 2024. On erroneous agreements of CLIP image embeddings. arXiv preprint arXiv:2411.05195. — [arXiv:2411.05195](https://arxiv.org/abs/2411.05195) [find](https://scholar.google.com/scholar?q=Siting+Li%2C+Pang+Wei+Koh%2C+and+Simon+Shaolei+Du%2E+2024%2E+On+erroneous+agreements+of+CLIP+image+embeddings%2E+arXiv+preprint+arXiv%3A2411%2E05195%2E+%E2%80%94+%5BarXiv%3A2411%2E05195%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E05195%29)
18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28ACL%29%2E)
19. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. [find](https://scholar.google.com/scholar?q=Sourab+Mangrulkar%2C+Sylvain+Gugger%2C+Lysandre+Debut%2C+Younes+Belkada%2C+Sayak+Paul%2C+and+Benjamin+Bossan%2E+2022%2E+PEFT%3A+State%2Dof%2Dthe%2Dart+parameter%2Defficient+fine%2Dtuning+methods%2E)
20. Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan. 2023. Filtering, distillation, and hard negatives for vision–language pre-training. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition… [find](https://scholar.google.com/scholar?q=Filip+Radenovic%2C+Abhimanyu+Dubey%2C+Abhishek+Kadian%2C+Todor+Mihaylov%2C+Simon+Vandenhende%2C+Yash+Patel%2C+Yi+Wen%2C+Vignesh+Ramanathan%2C+and+Dhruv+Mahajan%2E+2023%2E+Filtering%2C+distillation%2C+and+hard+negatives+for+v)
21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. [Learning transferable visual models from natural language supervision](https://md.research.elamur.ai/free/en/papers/radford-2021-learning-transferable-visual-models-natural/). In Proceedings of the International C…
22. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2025. [Understanding figurative meaning through explainable visual entailment](https://md.research.elamur.ai/free/en/papers/saakyan-2025-understanding-figurative-meaning-through-explainable/). In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL).
23. Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. [FaceNet: A unified embedding for face recognition and clustering](https://md.research.elamur.ai/free/en/papers/schroff-2015-facenet-unified-embedding-face-recognition/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
24. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. [Laion-5b: an open larg…](https://md.research.elamur.ai/free/en/papers/schuhmann-2022-laion-5b-open-large-scale/)

*Updated 1 November 2026*