--- title: "Creation and Validation of a Monolingual Spanish NLI Dataset for Metaphor — Sanchez-Montero 2026" description: "The authors address this gap by building an NLI dataset, entirely in Spanish, that tests how well models interpret metaphors. 23 references." authors: ["Alec Sanchez-Montero", "Gemma Bel-Enguix", "Sergio Luis Ojeda Trueba"] year: 2026 venue: "Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026" source: "https://aclanthology.org/2026.nonliteral-1.7/" doi: "10.63317/3j5ogzih7uuv" licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)" licence_url: "https://creativecommons.org/licenses/by-nc/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/montero-2026-creation-validation-monolingual-spanish-nli/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/montero-2026-creation-validation-monolingual-spanish-nli/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) / Creation and Validation of a Monolingual Spanish NLI Dataset for Metaphor Interpretation via Model-in-the-Loop # Creation and Validation of a Monolingual Spanish NLI Dataset for Metaphor Interpretation via Model-in-the-Loop **Authors:** Alec Sanchez-Montero, [Gemma Bel-Enguix](https://md.research.elamur.ai/free/en/authors/gemma-bel-enguix/) and [Sergio Luis Ojeda Trueba](https://md.research.elamur.ai/free/en/authors/sergio-luis-ojeda-trueba/) · **Year:** 2026 · **Venue:** Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026 · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.nonliteral-1.7/) · [PDF](https://aclanthology.org/2026.nonliteral-1.7.pdf) · [DOI 10.63317/3j5ogzih7uuv](https://doi.org/10.63317/3j5ogzih7uuv) · [OpenAlex](https://openalex.org/W7166529033) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Most NLI benchmarks for figurative language, such as flute, are in English. Spanish has few comparable resources, and building them by hand is slow. The authors address this gap by building an NLI dataset, entirely in Spanish, that tests how well models interpret metaphors. Each metaphorical sentence serves as the Hypothesis and is paired with two literal Premises, one that entails it and one that contradicts it. The metaphors come from Spanish science-popularization tweets. Candidate premises were generated with a Model-in-the-Loop pipeline and then curated by human experts. Before curation, an ablation study compared two prompting setups. In the plain setup, models saw only the sentence. In the scaffolded setup, models also received expert explanations of the metaphor. The automated metrics favoured the plain prompts, since scaffolded outputs scored lower on BERTScore and showed higher surprisal. Human judges reached the opposite verdict and found the scaffolded premises more accurate and more strictly literal. The authors conclude that fluency-oriented metrics penalize the reasoning a metaphor actually requires. The dataset is released on GitHub to support Spanish NLU evaluation that targets understanding rather than surface generation. ## Key points - **Source data:** Spanish tweets on Popular Communication of Science (PCS), previously annotated by Sánchez-Montero et al. (2025). Only the 200 instances with perfect human agreement (soft-label score 1.0) on the metaphor category were kept. - **Task format:** the task follows the flute paradigm. Each metaphorical Hypothesis gets a literal Entailment premise and a literal Contradiction premise. - **Generator models:** gpt-4.1-mini, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3 and gemma-3-12b-it were used to compare a proprietary model with open-weight ones. - **Ablation design:** a baseline prompt was compared with a scaffolded prompt that adds expert natural-language explanations of the metaphor and its type (e.g., Direct, Indirect, Personification). Outputs were scored with BERTScore and Cross-Entropy, then judged by humans. - **Main finding:** automated metrics and human judgment disagreed. Scaffolding lowered fluency scores but produced better literal premises in human evaluation. The authors read this as evidence that LLMs may need explicit symbolic cues to perform structural alignment, a point framed within the Career of Metaphor theory. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) ## Institutions - [Universitat Pompeu Fabra](https://md.research.elamur.ai/free/en/universities/universitat-pompeu-fabra/) — University, Barcelona, Spain · 27 works · 5,604 citations - [Universidad Nacional Autónoma de México](https://md.research.elamur.ai/free/en/universities/universidad-nacional-autonoma-de-mexico/) — University, Mexico City, Mexico · 5 works · 283 citations ## Cite Cite as: Alec Sanchez-Montero, Gemma Bel-Enguix and Sergio Luis Ojeda Trueba. 2026. Creation and Validation of a Monolingual Spanish NLI Dataset for Metaphor Interpretation via Model-in-the-Loop. Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{sanchezmontero2026creation, title = {{Creation and Validation of a Monolingual Spanish NLI Dataset for Metaphor Interpretation via Model-in-the-Loop}}, author = {Alec Sanchez-Montero, Gemma Bel-Enguix and Sergio Luis Ojeda Trueba}, booktitle = {Proceedings of Learning Non-Literal Expressions with Small Data @ LREC 2026}, year = {2026}, doi = {10.63317/3j5ogzih7uuv}, url = {https://doi.org/10.63317/3j5ogzih7uuv} } ``` - [← previous Creating a Hybrid Rule and Neural Network Based Semantic Tagger Using Silver Standard Data: The PyMUSAS Framework for Multilingual Semantic Annotation](https://md.research.elamur.ai/free/en/papers/moore-2026-creating-hybrid-rule-neural-network/) - [next → CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps](https://md.research.elamur.ai/free/en/papers/yun-2026-crit-qa-evaluating-multi-hop/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/5/) ## References 23 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Kathleen Ahrens, Christian Burgers, and Yin Zhong. 2024. Making the unseen seen: The role of signaling and novelty in rating metaphors. Journal of Psycholinguistic Research, 53(3):36. [find](https://scholar.google.com/scholar?q=Kathleen+Ahrens%2C+Christian+Burgers%2C+and+Yin+Zhong%2E+2024%2E+Making+the+unseen+seen%3A+The+role+of+signaling+and+novelty+in+rating+metaphors%2E+Journal+of+Psycholinguistic+Research%2C+53%283%29%3A36%2E) 2. George Lakoff and Mark Leonard Johnson. 1980. Metaphors We Live By. University of Chicago press. [find](https://scholar.google.com/scholar?q=George+Lakoff+and+Mark+Leonard+Johnson%2E+1980%2E+Metaphors+We+Live+By%2E+University+of+Chicago+press%2E) 3. Informe Belmont. 1978. Principios éticos y directrices para la protección de sujetos humanos de investigación. Estados Unidos de Norteamérica: Reporte de la Comisión Nacional para la Protección de Sujetos Humanos de Investigación Biomédica y de Comportamiento. [find](https://scholar.google.com/scholar?q=Informe+Belmont%2E+1978%2E+Principios+%C3%A9ticos+y+directrices+para+la+protecci%C3%B3n+de+sujetos+humanos+de+investigaci%C3%B3n%2E+Estados+Unidos+de+Norteam%C3%A9rica%3A+Reporte+de+la+Comisi%C3%B3n+Nacional+para+la+Protecci%C3%B3n+de+Suj) 4. Brian F. Bowdle and Dedre Gentner. 2005. The career of metaphor. Psychological Review, 112(1):193–216. [find](https://scholar.google.com/scholar?q=Brian+F%2E+Bowdle+and+Dedre+Gentner%2E+2005%2E+The+career+of+metaphor%2E+Psychological+Review%2C+112%281%29%3A193%E2%80%93216%2E) 5. Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. 2022. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437–4452.… [find](https://scholar.google.com/scholar?q=Emmy+Liu%2C+Chenxuan+Cui%2C+Kenneth+Zheng%2C+and+Graham+Neubig%2E+2022%2E+Testing+the+ability+of+language+models+to+interpret+figurative+language%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Cha) 6. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159. Association for Computational Linguistics… [our card](https://md.research.elamur.ai/free/en/papers/chakrabarty-2022-flute-figurative-language-understanding-through/) 7. Simone Mazzoli, Alice Suozzi, and Gianluca Lebani. 2025. Language models and the magic of metaphor: A comparative evaluation with human judgments. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 710–721. CEUR Workshop Proceedings. [find](https://scholar.google.com/scholar?q=Simone+Mazzoli%2C+Alice+Suozzi%2C+and+Gianluca+Lebani%2E+2025%2E+Language+models+and+the+magic+of+metaphor%3A+A+comparative+evaluation+with+human+judgments%2E+In+Proceedings+of+the+Eleventh+Italian+Conference+on+) 8. Zi-Yuan Chen and Yining Mao. 2023. MetaMapper: Interpretable metaphor detection. [find](https://scholar.google.com/scholar?q=Zi%2DYuan+Chen+and+Yining+Mao%2E+2023%2E+MetaMapper%3A+Interpretable+metaphor+detection%2E) 9. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, and Janko Altenschmidt et al. 2024. Gpt-4 technical report. [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/) 10. Minjin Choi, Sunkyung Lee, Eunseong Choi, Heesoo Park, Junhyuk Lee, Dongwon Lee, and Jongwuk Lee. 2021. MelBERT: Metaphor detection via contextualized late interaction using metaphorical identification theories. In Proceedings of the 2021 Conference of the North American Chapter of the Association f… [find](https://scholar.google.com/scholar?q=Minjin+Choi%2C+Sunkyung+Lee%2C+Eunseong+Choi%2C+Heesoo+Park%2C+Junhyuk+Lee%2C+Dongwon+Lee%2C+and+Jongwuk+Lee%2E+2021%2E+MelBERT%3A+Metaphor+detection+via+contextualized+late+interaction+using+metaphorical+identificatio) 11. Pragglejaz. 2007. MIP: A method for identifying metaphorically used words in discourse. Metaphor and Symbol, 22(1):1–39. [find](https://scholar.google.com/scholar?q=Pragglejaz%2E+2007%2E+MIP%3A+A+method+for+identifying+metaphorically+used+words+in+discourse%2E+Metaphor+and+Symbol%2C+22%281%29%3A1%E2%80%9339%2E) 12. Eduardo Puraivan, Irene Renau, and Nicolás Riquelme. 2024. Metaphor identification and interpretation in corpora with ChatGPT. SN Computer Science, 5(8):976. [find](https://scholar.google.com/scholar?q=Eduardo+Puraivan%2C+Irene+Renau%2C+and+Nicol%C3%A1s+Riquelme%2E+2024%2E+Metaphor+identification+and+interpretation+in+corpora+with+ChatGPT%2E+SN+Computer+Science%2C+5%288%29%3A976%2E) 13. Quen: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… [find](https://scholar.google.com/scholar?q=Quen%3A+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yan) 14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Kartikay+Khandelwal%2C+Naman+Goyal%2C+Vishrav+Chaudhary%2C+Guillaume+Wenzek%2C+Francisco+Guzm%C3%A1n%2C+Edouard+Grave%2C+Myle+Ott%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2020%2E+Unsupervised+cross%2Dlingu) 15. Matteo Fuoli, Weihang Huang, Jeannette Littlemore, Sarah Turner, and Ellen Wilding. 2025. Metaphor identification using large language models: A comparison of rag, prompt engineering, and fine-tuning. [find](https://scholar.google.com/scholar?q=Matteo+Fuoli%2C+Weihang+Huang%2C+Jeannette+Littlemore%2C+Sarah+Turner%2C+and+Ellen+Wilding%2E+2025%2E+Metaphor+identification+using+large+language+models%3A+A+comparison+of+rag%2C+prompt+engineering%2C+and+fine%2Dtuning%2E) 16. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and Aiesha Letman et al. 2024. The llama 3 herd of models. [find](https://scholar.google.com/scholar?q=Aaron+Grattafiori%2C+Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+and+Aiesha+Letman+et+al%2E+2024%2E+The+llama+3+herd+of+models%2E) 17. Sunny Rai and Shampa Chakraverty. 2021. A survey on computational metaphor processing. ACM Computing Surveys, 53(2):1–37. [find](https://scholar.google.com/scholar?q=Sunny+Rai+and+Shampa+Chakraverty%2E+2021%2E+A+survey+on+computational+metaphor+processing%2E+ACM+Computing+Surveys%2C+53%282%29%3A1%E2%80%9337%2E) 18. Manisha Rana, Rita Chhikara, and Srishti Sharma. 2025. A SURRVEY ON METAPHOR DE-TECTION AND INTERPRETATION. International Journal For Multidisciplinary Research, 7(4):54959. [find](https://scholar.google.com/scholar?q=Manisha+Rana%2C+Rita+Chhikara%2C+and+Srishti+Sharma%2E+2025%2E+A+SURRVEY+ON+METAPHOR+DE%2DTECTION+AND+INTERPRETATION%2E+International+Journal+For+Multidisciplinary+Research%2C+7%284%29%3A54959%2E) 19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 20. Elisa Sanchez-Bayona and Rodrigo Agerri. 2025. Meta4xnli: A crosslingual parallel corpus for metaphor detection and interpretation. [find](https://scholar.google.com/scholar?q=Elisa+Sanchez%2DBayona+and+Rodrigo+Agerri%2E+2025%2E+Meta4xnli%3A+A+crosslingual+parallel+corpus+for+metaphor+detection+and+interpretation%2E) 21. Taylor Webb, Keith J. Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526–1541. [find](https://scholar.google.com/scholar?q=Taylor+Webb%2C+Keith+J%2E+Holyoak%2C+and+Hongjing+Lu%2E+2023%2E+Emergent+analogical+reasoning+in+large+language+models%2E+Nature+Human+Behaviour%2C+7%289%29%3A1526%E2%80%931541%2E) 22. Meghdut Sengupta, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, Eyke Hüllermeier, Debanjan Ghosh, and Henning Wachsmuth. 2025. Investigating the impact of conceptual metaphors on LLM-based NLI through shapley interactions. In Findings of the Association for Computational Linguistics: EMNLP… [find](https://scholar.google.com/scholar?q=Meghdut+Sengupta%2C+Maximilian+Muschalik%2C+Fabian+Fumagalli%2C+Barbara+Hammer%2C+Eyke+H%C3%BCllermeier%2C+Debanjan+Ghosh%2C+and+Henning+Wachsmuth%2E+2025%2E+Investigating+the+impact+of+conceptual+metaphors+on+LLM%2Dbased+N) 23. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E)