--- title: "Memorization or Reasoning? Exploring the Idiom — Kim 2025" description: "The paper asks whether large language models understand idioms by recalling them or by working out their figurative meaning." authors: ["Jisu Kim", "Youngwoo Shin", "Uiji Hwang", "Jihun Choi", "Richeng Xuan", "Taeuk Kim"] year: 2025 venue: "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/2025.emnlp-main.1099/" doi: "10.18653/v1/2025.emnlp-main.1099" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/kim-2025-memorization-or-reasoning-exploring-idiom/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/kim-2025-memorization-or-reasoning-exploring-idiom/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2025](https://md.research.elamur.ai/free/papers/2025/page/2/) / Memorization or Reasoning? Exploring the Idiom Understanding of LLMs # Memorization or Reasoning? Exploring the Idiom Understanding of LLMs **Authors:** Jisu Kim, Youngwoo Shin, Uiji Hwang, Jihun Choi, Richeng Xuan, Taeuk Kim · **Year:** 2025 · **Venue:** Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2025.emnlp-main.1099/) · [PDF](https://aclanthology.org/2025.emnlp-main.1099.pdf) · [DOI 10.18653/v1/2025.emnlp-main.1099](https://doi.org/10.18653/v1/2025.emnlp-main.1099) · [OpenAlex](https://openalex.org/W4416036058) **Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [10](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether large language models understand idioms by recalling them or by working out their figurative meaning. It frames this as a multilingual question, since evaluation resources outside English and Chinese are scarce. The authors build MIDAS, a dataset of 64,660 idioms in English, German, Chinese, Korean, Arabic and Turkish. The idioms come from authoritative dictionaries and collections. GPT-4o cleaned the meanings (GPT-4o mini for English), and native speakers then checked them. Example sentences and surface variants are kept where available. Understanding is tested with five-option multiple-choice questions about an idiom's meaning. Further probes: - separate memorized from unmemorized idioms with a final-word completion test; - group idioms by the model's own compositionality estimate; - compare results with and without example sentences; - examine chain-of-thought and reasoning-focused models. Models perform clearly better in high-resource languages like English than in Korean. Their strategy appears hybrid: memory matters, but compositional idioms and context also help. Gains from reasoning differ across languages. In practice, supplying explicit idiom meanings improves generation, translation and LLM-based evaluation. ## Key points - **MIDAS size and sources:** EN 9,766, DE 10,097, ZH 11,851, KO 11,316, AR 8,051, TR 13,579 idioms. Sources include Wiktionary, Duden–Redewendungen, the Xinhua dictionary and the Korean Standard Dictionary. The dataset is released at github.com/HYU-NLP/MIDAS. - **Question design:** each MCQ has one correct meaning and four distractors. Two distractors resemble the idiom's surface form and two resemble its figurative meaning. - **Context and memorization:** with Qwen2.5-32B-Instruct, example sentences help most on idioms the model has not memorized. Higher compositionality seems to go with easier memorization. - **GPT-4o as a judge:** agreement with human scores rises sharply when the idiom's meaning is provided. - Sentence generation: Kendall's τ 0.739 with meaning vs. 0.118 without. - Translation: 0.615 vs. 0.403. - **Downstream tasks:** for Korean and Turkish, MIDAS meanings improved idiom sentence generation and translation into English. This held for both Qwen2.5-32B-Instruct and Aya-Expanse-32B. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet. - [ID10M-JAM: Stress-Testing Idiom Identification Under Challenging Context](https://md.research.elamur.ai/free/papers/hashiloni-2026-id10m-jam-stress-testing-idiom/) - [ITUNLP at MWE-2026 AdMIRe 2: A Zero-Shot LLM Pipeline for Multimodal Idiom Understanding and Ranking](https://md.research.elamur.ai/free/papers/site-2026-itunlp-mwe-2026-admire-2/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{kim2025memorization, title = {{Memorization or Reasoning? Exploring the Idiom Understanding of LLMs}}, author = {Kim, Jisu and Shin, Youngwoo and Hwang, Uiji and Choi, Jihun and Xuan, Richeng and Kim, Taeuk}, booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}, year = {2025}, doi = {10.18653/v1/2025.emnlp-main.1099}, url = {https://doi.org/10.18653/v1/2025.emnlp-main.1099} } ``` - [← previous Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://md.research.elamur.ai/free/papers/chhikara-2025-mem0-building-production-ready-ai/) - [next → METAL: A Multi-Agent Framework for Chart Generation with Test-Time Scaling](https://md.research.elamur.ai/free/papers/li-2025-metal-multi-agent-framework-chart/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/papers/2025/page/2/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. We outline several directions for future research to build upon this work. [find](https://scholar.google.com/scholar?q=We+outline+several+directions+for+future+research+to+build+upon+this+work%2E) 2. Tosin Adewumi, Roshanak Vadoodi, Aparajita Tripathy, Konstantina Nikolaido, Foteini Liwicki, and Marcus Liwicki. 2022. Potential idiomatic expression (pie)-english: Corpus for classes of idioms. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 689–696. [find](https://scholar.google.com/scholar?q=Tosin+Adewumi%2C+Roshanak+Vadoodi%2C+Aparajita+Tripathy%2C+Konstantina+Nikolaido%2C+Foteini+Liwicki%2C+and+Marcus+Liwicki%2E+2022%2E+Potential+idiomatic+expression+%28pie%29%2Denglish%3A+Corpus+for+classes+of+idioms%2E+In+Pr)