--- title: "It's not Rocket Science — Chakrabarty 2022" description: "Transformer-based language models handle literal text well but often fail on figurative expressions." authors: ["Tuhin Chakrabarty", "Yejin Choi", "Vered Shwartz"] year: 2022 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2022.tacl-1.34/" doi: "10.1162/tacl_a_00478" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/papers/chakrabarty-2022-it-s-not-rocket-science/" --- [Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/chakrabarty-2022-it-s-not-rocket-science/) [home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2022](https://md.research.elamur.ai/free/papers/2022/page/2/) / It's not Rocket Science: Interpreting Figurative Language in Narratives # It's not Rocket Science: Interpreting Figurative Language in Narratives **Authors:** [Tuhin Chakrabarty](https://md.research.elamur.ai/free/authors/tuhin-chakrabarty/), [Yejin Choi](https://md.research.elamur.ai/free/authors/yejin-choi/), [Vered Shwartz](https://md.research.elamur.ai/free/authors/vered-shwartz/) · **Year:** 2022 · **Venue:** Transactions of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.34/) · [PDF](https://aclanthology.org/2022.tacl-1.34.pdf) · [DOI 10.1162/tacl\_a\_00478](https://doi.org/10.1162/tacl_a_00478) · [OpenAlex](https://openalex.org/W3198711991) **Citations:** 35 citations (openalex-snapshot · 2026-10-14) · [56](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Transformer-based language models handle literal text well but often fail on figurative expressions. Idioms and similes are hard because their meaning cannot be built from the meanings of their parts. The authors frame this as a story-continuation problem, following the Story Cloze setup. They took short passages from the Toronto Book corpus, each containing an idiom or a simile. Crowd workers then wrote continuations that make sense, or do not, depending on whether the expression is understood correctly. Two tasks are defined: choosing the plausible ending from two candidates, and generating one. Pre-trained models, GPT-3 included, perform poorly in zero-shot and few-shot settings. Supervised models still trail human performance. To narrow this gap, the authors add knowledge that mirrors two human interpretation strategies. ParaCOMET supplies inferences from the surrounding context, and COMET-ConceptNet supplies literal knowledge about the constituent words. These knowledge-enhanced models beat the baselines on both tasks, with the clearest gains in generation. The work offers a benchmark and evidence that cognitively motivated knowledge helps machines understand figurative language. ## Key points - Two datasets of fictional narratives, one for idioms and one for similes, drawn from the Toronto Book corpus. Each comes with crowdsourced plausible and implausible continuations. Code and data are on GitHub (FigurativeNarrativeBenchmark), with a leaderboard at AllenAI. - Even supervised models fall short of humans. On the discriminative task, the accuracy gap is 10 points for idioms and 14.6 for similes. On the generative task, human-judged plausibility trails by about 24 and 28 points. - The context model (ParaCOMET) works best for idioms, and the literal model (COMET-ConceptNet) works best for similes. This matches Cooper's (1999) findings on how people interpret idioms and how similes rely on attributes of the vehicle. - In the absolute human evaluation, GPT2-XL continuations were rated plausible 56% (idioms) and 60% (similes) of the time. The best knowledge-enhanced models reached 68% and 76%, and human references reached 80% and 88%. - Rouge-L and BERT-Score show little difference between the knowledge-enhanced models and the GPT-2 XL baseline, unlike the human judgments. Human-evaluation agreement was moderate (Krippendorff's α = 0.68 for the absolute evaluation, 0.63 for the comparative one). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) ## Cited by Cited by 56 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 50 not yet. - [No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models](https://md.research.elamur.ai/free/papers/cheng-2024-no-context-needed-contextual-quandary/) - [Understanding Transformer Memorization Recall Through Idioms](https://md.research.elamur.ai/free/papers/haviv-2023-understanding-transformer-memorization-recall-through/) - [Comparative Study of Multilingual Idioms and Similes in Large Language Models](https://md.research.elamur.ai/free/papers/khoshtab-2025-comparative-study-multilingual-idioms-similes/) - [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://md.research.elamur.ai/free/papers/miletic-2024-semantics-multiword-expressions-transformer-based/) - [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://md.research.elamur.ai/free/papers/ravi-2023-what-happens-before-after-multi/) - [IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions](https://md.research.elamur.ai/free/papers/zeng-2023-iekg-commonsense-knowledge-graph-idiomatic/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{chakrabarty2022it, title = {{It's not Rocket Science: Interpreting Figurative Language in Narratives}}, author = {Chakrabarty, Tuhin and Choi, Yejin and Shwartz, Vered}, journal = {Transactions of the Association for Computational Linguistics}, year = {2022}, doi = {10.1162/tacl\_a\_00478}, url = {https://doi.org/10.1162/tacl_a_00478} } ``` - [← previous It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books](https://md.research.elamur.ai/free/papers/yao-2022-it-is-ai-s-turn/) - [next → JGLUE: Japanese General Language Understanding Evaluation](https://md.research.elamur.ai/free/papers/kurihara-2022-jglue-japanese-general-language-understanding/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/papers/2022/page/2/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. understanding its actual meaning ‘‘closest of friends’’. [find](https://scholar.google.com/scholar?q=understanding+its+actual+meaning+%E2%80%98%E2%80%98closest+of+friends%E2%80%99%E2%80%99%2E) 2. Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O. Riedl. 2020. Automated storytelling via causal, commonsense plot ordering. arXiv preprint arXiv:2009.00829. — [arXiv:2009.00829](https://arxiv.org/abs/2009.00829) [find](https://scholar.google.com/scholar?q=Prithviraj+Ammanabrolu%2C+Wesley+Cheung%2C+William+Broniec%2C+and+Mark+O%2E+Riedl%2E+2020%2E+Automated+storytelling+via+causal%2C+commonsense+plot+ordering%2E+arXiv+preprint+arXiv%3A2009%2E00829%2E+%E2%80%94+%5BarXiv%3A2009%2E00829%5D%28htt)