---
title: "It is not a piece of cake for GPT: Explaining Textual Entailment"
description: "The paper investigates how well modern LLMs handle textual entailment between sentence pairs that… 34 references, cited by 5 papers here."
authors: ["Giuseppe Gallipoli", "Luca Cagliero"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.646/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gallipoli-2025-it-is-not-piece-cake/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gallipoli-2025-it-is-not-piece-cake/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/) / It is not a piece of cake for GPT: Explaining Textual Entailment Recognition in the presence of Figurative Language

# It is not a piece of cake for GPT: Explaining Textual Entailment Recognition in the presence of Figurative Language

**Authors:** [Giuseppe Gallipoli](https://md.research.elamur.ai/free/en/authors/giuseppe-gallipoli/) and [Luca Cagliero](https://md.research.elamur.ai/free/en/authors/luca-cagliero/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.646/) · [PDF](https://aclanthology.org/2025.coling-main.646.pdf) · [OpenAlex](https://openalex.org/W7219272743)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/gallipoli-2025-it-is-not-piece-cake/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper investigates how well modern LLMs handle textual entailment between sentence pairs that contain figurative language. The task requires a binary label (entailment or contradiction) plus a natural-language explanation of the non-literal expression. Using the FLUTE benchmark, the authors compare open-source models of up to 9B parameters with GPT-3.5 Turbo and GPT-4o. They test four regimes: zero-shot prompting, few-shot prompting with several example-selection strategies, Chain-of-Thought prompting that injects knowledge from an external scene-elaboration model, and LightWeight Fine-Tuning. Prompts are tuned per model and setting on validation data.

Predicting the label is relatively easy for LLMs, but explaining it well is much harder. The GPT models trail much smaller open models on explanation quality. Injecting external knowledge through CoT does not help. Fine-tuned models, with Gemma2 9B the best, beat the earlier state of the art (DREAM-FLUTE, Cross-Task). This matters because it shows that model size and proprietary status do not guarantee figurative-language understanding, and that explanation quality must be evaluated separately from label accuracy.

## Key points
- Data: the FLUTE benchmark (English only) covers four figurative types: sarcasm, metaphor, simile and idiom. Each premise–hypothesis pair carries an E/C label and a gold explanation; neutral pairs are excluded.
- Prompt variants were tuned on the validation set and chosen by acc@60, a metric reflecting both label and explanation quality. Few-shot prompting with 5–10 examples beat zero-shot, while CoT with external knowledge gave no gain.
- After fine-tuning, Gemma2 9B surpassed DREAM-FLUTE (T5 3B based) and Cross-Task. In a GPT-4o-judged A/B test, its outputs were preferred in roughly 65% of comparisons against both systems.
- GPT-4o was also used as an "expert reviewer" of Gemma2 outputs, and the results got worse.
  - It never changed only the label, edited only the explanation in 121 cases, edited both in 48, and left 1329 untouched.
  - Under a conservative rule, it made 27 wrong edits versus 21 right ones.
- Code and models are released under CC BY-NC-SA (FigurativeTER-E_LLM on GitHub).

## Interesting ideas
- The authors select prompts by a joint score that credits a label only when the accompanying explanation is good enough, so a right answer with a bad justification does not count.
- Both the label and the explanation are tied specifically to the figurative expression rather than to the rest of the sentence, which localizes where understanding must happen.
- An LLM reviewer was audited by counting its correct versus harmful edits, including a gated mode where edits apply only when the reviewer disagrees on the label. The audit showed the reviewer introduced roughly as many errors as fixes.
- Few-shot examples were drawn with deliberate strategies (random vs. balanced across labels and figurative types) rather than ad hoc.
- A previously reported benefit, scene elaboration fed through CoT, disappeared when the setup moved to LLMs. Published gains may therefore not transfer across model families and should be re-tested.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Score readers on school tests with a metric that requires both the correct answer and a valid trace through frames and world, so lucky guesses are not rewarded.
- **I2** (frames): The irony/implicature module in prag could mark the exact non-literal span and attach its literal reading, making the inference step auditable.
- **I3** (learn): Before trusting an LLM teacher tier to repair silver data, measure its edit precision against gold or negative controls and allow edits only under disagreement gating.
- **I4** (learn): When prompting LLM teachers for silver frames or labels, balance demonstrations across construction types so the teacher's output is not skewed toward frequent cases.
- **I5** (world): Since Mova's world-of-text is itself an explicit scene model, check through ablations that its elaborations actually change answers instead of assuming extra context helps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Politecnico di Torino](https://md.research.elamur.ai/free/en/universities/politecnico-di-torino/) — University, Turin, Italy · 11 works · 18 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 217 works

## Cite

Cite as: Giuseppe Gallipoli and Luca Cagliero. 2025. It is not a piece of cake for GPT: Explaining Textual Entailment Recognition in the presence of Figurative Language. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{gallipoli2025it,
  title = {{It is not a piece of cake for GPT: Explaining Textual Entailment Recognition in the presence of Figurative Language}},
  author = {Giuseppe Gallipoli and Luca Cagliero},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.646/}
}
```

- [← previous Irony Detection, Reasoning and Understanding in Zero-shot Learning](https://md.research.elamur.ai/free/en/papers/yi-2025-irony-detection-reasoning-understanding-zero/)
- [next → Large Language Models and Arabic Content: A Review](https://md.research.elamur.ai/free/en/papers/rhel-2025-large-language-models-arabic-content/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/11/)

## References

34 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rodrigo Agerri. 2008. [Metaphor in textual entailment](https://md.research.elamur.ai/free/en/papers/agerri-2008-metaphor-textual-entailment/). In Coling 2008: Companion volume: Posters, pages 3–6, Manchester, UK. Coling 2008 Organizing Committee.
2. Rodrigo Agerri, John Barnden, Mark Lee, and Alan Wallington. 2008. [Textual entailment as an evaluation framework for metaphor resolution: A proposal](https://md.research.elamur.ai/free/en/papers/agerri-2008-textual-entailment-as-evaluation-framework/). In Semantics in Text Processing. STEP 2008 Conference Proceedings, pages 357–363. College Publications. [↗](https://doi.org/10.3115/1626481.1626509)
3. Irina Bigoulaeva, Rachneet Singh Sachdeva, Harish Tayyar Madabushi, Aline Villavicencio, and Iryna Gurevych. 2022. Effective cross-task transfer learning for explainable natural language inference with t5. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 54–60, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Irina+Bigoulaeva%2C+Rachneet+Singh+Sachdeva%2C+Harish+Tayyar+Madabushi%2C+Aline+Villavicencio%2C+and+Iryna+Gurevych%2E+2022%2E+Effective+cross%2Dtask+transfer+learning+for+explainable+natural+language+inference+wit)
4. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. [e-snli: natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 9560–9572, Red Hook, NY, USA. Curran Associates Inc. [↗](https://doi.org/10.48550/arxiv.1812.01193)
5. Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. 2021. [Figurative language in recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/chakrabarty-2021-figurative-language-recognizing-textual-entailment/). In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 3354– 3361. Association for Computational Linguistics.
6. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. [FLUTE: Figurative language understanding through textual explanations](https://md.research.elamur.ai/free/en/papers/chakrabarty-2022-flute-figurative-language-understanding-through/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM Trans. Intell. Syst. Technol., 15(3). [↗](https://doi.org/10.1145/3641289)
8. Yuling Gu, Bhavana Dalvi, and Peter Clark. 2022a. DREAM: Improving situational QA by first elaborating the situation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1127, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuling+Gu%2C+Bhavana+Dalvi%2C+and+Peter+Clark%2E+2022a%2E+DREAM%3A+Improving+situational+QA+by+first+elaborating+the+situation%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Associa)
9. Yuling Gu, Yao Fu, Valentina Pyatkin, Ian Magnusson, Bhavana Dalvi Mishra, and Peter Clark. 2022b. Just-DREAM-about-it: Figurative language understanding with DREAM-FLUTE. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 84–93, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yuling+Gu%2C+Yao+Fu%2C+Valentina+Pyatkin%2C+Ian+Magnusson%2C+Bhavana+Dalvi+Mishra%2C+and+Peter+Clark%2E+2022b%2E+Just%2DDREAM%2Dabout%2Dit%3A+Figurative+language+understanding+with+DREAM%2DFLUTE%2E+In+Proceedings+of+the+3rd+Wo)
10. Randal Holme. 2004. Using Figurative Language, pages 28–58. Palgrave Macmillan UK, London. [find](https://scholar.google.com/scholar?q=Randal+Holme%2E+2004%2E+Using+Figurative+Language%2C+pages+28%E2%80%9358%2E+Palgrave+Macmillan+UK%2C+London%2E)
11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
12. Harsh Jhamtani, Varun Gangal, Eduard H. Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating robustness of dialog models to popular figurative language constructs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7476– 7485. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Harsh+Jhamtani%2C+Varun+Gangal%2C+Eduard+H%2E+Hovy%2C+and+Taylor+Berg%2DKirkpatrick%2E+2021%2E+Investigating+robustness+of+dialog+models+to+popular+figurative+language+constructs%2E+In+Proceedings+of+the+2021+Confere)
13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. [↗](https://arxiv.org/abs/2310.06825)
14. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
16. Gemma Team. 2024a. Gemma: Open Models Based on Gemini Research and Technology. Preprint, arXiv:2403.08295. [↗](https://arxiv.org/abs/2403.08295)
17. Llama Team. 2024. [The Llama 3 Herd of Models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). Preprint, arXiv:2407.21783. [↗](https://arxiv.org/abs/2407.21783)
18. Gemma Team. 2024b. Gemma2: Improving Open Language Models at a Practical Size. Preprint, arXiv:2408.00118. [↗](https://arxiv.org/abs/2408.00118)
19. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft). [↗](https://github.com/huggingface/peft)
20. Maarten Grootendorst. 2020. Minimal keyword extraction with BERT. [find](https://scholar.google.com/scholar?q=Maarten+Grootendorst%2E+2020%2E+Minimal+keyword+extraction+with+BERT%2E)
21. OpenAI. 2024a. OpenAI GPT-3.5. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024a%2E+OpenAI+GPT%2D3%2E5%2E)
22. OpenAI. 2024b. OpenAI GPT-4o. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+OpenAI+GPT%2D4o%2E)
23. Dylan Phelps, Thomas M. R. Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. 2024. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In [Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024](https://md.research.elamur.ai/free/en/papers/anon-2024-proceedings-joint-workshop-multiword-expressions/), pages 178–187, Torino, Italia. ELRA and ICCL.
24. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). J. Mach. Learn. Res., 21(1). [↗](https://doi.org/10.48550/arxiv.1910.10683)
25. Arkadiy Saakyan, Tuhin Chakrabarty, Debanjan Ghosh, and Smaranda Muresan. 2022. A report on the FigLang 2022 shared task on understanding figurative language. In Proceedings of the 3rd Workshop on Figurative Language Processing (FLP), pages 178–183, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Arkadiy+Saakyan%2C+Tuhin+Chakrabarty%2C+Debanjan+Ghosh%2C+and+Smaranda+Muresan%2E+2022%2E+A+report+on+the+FigLang+2022+shared+task+on+understanding+figurative+language%2E+In+Proceedings+of+the+3rd+Workshop+on+Fig)
26. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [↗](https://doi.org/10.48550/arxiv.1904.09675)
27. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Advances in Neural Information Processing Systems, volume 36, pages 46595–46623. Curran Associates, Inc.
28. Arkadiy Saakyan, Shreyas Kulkarni, Tuhin Chakrabarty, and Smaranda Muresan. 2024. V-flute: Visual figurative language understanding with textual explanations. Preprint, arXiv:2405.01474. [↗](https://arxiv.org/abs/2405.01474)
29. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. [BLEURT: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.704)
30. Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022. [IMPLI: Investigating NLI models’ performance on figurative language](https://md.research.elamur.ai/free/en/papers/stowe-2022-impli-investigating-nli-models-performance/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2022.acl-long.369)
31. Hugo Touvron et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). CoRR, abs/2307.09288. [↗](https://arxiv.org/abs/2307.09288)
32. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv:2310.16944. [↗](https://arxiv.org/abs/2310.16944)
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
34. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In

*Updated 1 November 2026*