---
title: "Textual Entailment as an Evaluation Framework for Metaphor"
description: "Metaphor is common in everyday text, yet applied NLP has mostly ignored it. 15 references, cited by 5 papers here."
authors: ["Rodrigo Agerri", "John Barnden", "Mark Lee", "Alan Wallington"]
year: 2008
venue: "Semantics in Text Processing. STEP 2008 Conference Proceedings"
source: "https://aclanthology.org/W08-2228/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/agerri-2008-textual-entailment-as-evaluation-framework/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agerri-2008-textual-entailment-as-evaluation-framework/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2008](https://md.research.elamur.ai/free/en/papers/2008/page/2/) / Textual Entailment as an Evaluation Framework for Metaphor Resolution: A Proposal

# Textual Entailment as an Evaluation Framework for Metaphor Resolution: A Proposal

**Authors:** [Rodrigo Agerri](https://md.research.elamur.ai/free/en/authors/rodrigo-agerri/), [John Barnden](https://md.research.elamur.ai/free/en/authors/john-barnden/), [Mark Lee](https://md.research.elamur.ai/free/en/authors/mark-lee/) and Alan Wallington · **Year:** 2008 · **Venue:** [Semantics in Text Processing. STEP 2008 Conference Proceedings](https://md.research.elamur.ai/free/en/venues/semantics-in-text-processing-step-conference-proceedings/) ([Semantics in Text Processing. STEP Conference Proceedings 2008](https://md.research.elamur.ai/free/en/venues/semantics-in-text-processing-step-conference-proceedings/2008/)) · **Publisher:** [College Publications](https://md.research.elamur.ai/free/en/publishers/college-publications/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W08-2228/) · [PDF](https://aclanthology.org/W08-2228.pdf) · [OpenAlex](https://openalex.org/W2081593112)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/agerri-2008-textual-entailment-as-evaluation-framework/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Metaphor is common in everyday text, yet applied NLP has mostly ignored it. Metaphor-understanding research, for its part, has stayed small-scale: specialised systems receive a few hand-picked inputs, and the researchers themselves decide what the right outputs are. This proposal, adapted from a grant application, suggests evaluating metaphor resolution inside the Recognizing Textual Entailment (RTE) framework. In RTE, a system judges whether a Hypothesis plausibly follows from a Text, checked against human judgments.

The authors plan two deliverables:
- **A dataset:** a public, annotated, metaphor-focused collection fit for RTE. Sources are past RTE data, a 242K-word metaphor corpus derived from the British National Corpus, and seed-based mining of corpora and the web.
- **A system:** a prototype RTE system that spans shallow and deep processing. Shallow detection extends the E-Drama methods (Rasp parsing plus WordNet). Deep reasoning follows ATT-Meta-style source-domain inference, broadened with resources like ConceptNet.

Their analysis of RTE-1 and RTE-2 submissions shows that systems, including the strongest ones, do measurably worse on metaphorical pairs. The proposal matters because it would give metaphor research shared data and a uniform, human-grounded evaluation.

## Key points
- An earlier study by the group (Tech. Rept. CSRP-03-05) counted about one metaphorical term per 17.3 words across genres. Gedigian et al. (2006) found 90% of certain motion, manipulation and health verbs in a Wall Street Journal corpus used metaphorically.
- RTE-1 results:
  - The authors examined 4 of 16 systems, which included the 1st, 3rd and 4th most accurate.
  - On the "gastronomic capital" example, only one system run was correct.
  - On 11 metaphor cases, runs mostly did worse than on the full dataset, significant at 0.05 by Fisher's test.
- RTE-2 results: across 9 metaphorical pairs, 10 runs from 23 systems all underperformed relative to the rest of the data. In 7 of those runs the drop was significant at \< 0.05.
- Existing RTE datasets tended to avoid harder metaphor, because organisers favoured pairs with crisp, uncontroversial judgments.
- Planned resources and tools: ATT-Meta, E-Drama, Rasp, WordNet 3.0, ConceptNet 3.0, and WordNets enriched with web-mined knowledge (Veale and Hao, 2008).

## Interesting ideas
- Treat the RTE Hypothesis as the goal for backward-chaining inference, so reasoning stays focused on what is relevant instead of expanding everything the Text could imply.
- View-Neutral Mapping Adjuncts carry over only abstract structure from the source domain to the target: process shape, timing, causation, functioning, mental and emotional states, and value judgments. For "incubate an idea", this yields slow protected development, not literal warmth.
- Measure a phenomenon-specific slice (here, metaphorical pairs) against the rest of the benchmark with a significance test such as Fisher's. This shows a targeted weakness instead of letting it disappear into overall accuracy.
- Model stock figurative phrases as templates with an open slot for any compatible qualifier, rather than as fixed strings. In "the deepest recesses of the mind", the slot could hold dark, hidden or dim, and semantic reasoning about the qualifier supplies the emphasis.
- Run a graded spectrum from shallow to deep analysis and decide per case how much inference to attempt and how to combine results.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): A reader could turn the question into a goal and search the text's world only along paths that could support or refute it, keeping each answer traceable.
- **I2** (frames): When a VerbNet frame is used figuratively, keep its aspectual, causal and affective content and drop the physical role restrictions, recording the transfer as an explicit step.
- **I3** (learn): Evaluation could report accuracy on tagged slices (metaphor, idiom, irony) against the remainder, with significance, to reveal blind spots that a single global score hides.
- **I4** (frames): Idiom frames could match a lemma pattern with a constrained modifier slot whose WordNet category is checked, so productive variants still trigger the right figurative reading.
- **I5** (world): Building the world of a text could try cheap lexicon-listed senses first and fall back to deeper source-domain reasoning only when the cheap reading fails to fit the context.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Birmingham](https://md.research.elamur.ai/free/en/universities/university-of-birmingham/) — University, Birmingham, United Kingdom · 27 works · 62,564 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [💭 Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/) · 1,649 works · [💭 Experimental and Cognitive Psychology](https://md.research.elamur.ai/free/en/shelves/psychology/experimental-and-cognitive-psychology/) · 930 works

🧭 Topics (3): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [🧷 Idioms & multiword expressions](https://md.research.elamur.ai/free/en/topics/multiword-expressions/) · 222 works

## Cite

Cite as: Rodrigo Agerri, John Barnden, Mark Lee and Alan Wallington. 2008. Textual Entailment as an Evaluation Framework for Metaphor Resolution: A Proposal. Semantics in Text Processing. STEP 2008 Conference Proceedings.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{agerri2008textual,
  title = {{Textual Entailment as an Evaluation Framework for Metaphor Resolution: A Proposal}},
  author = {Rodrigo Agerri, John Barnden, Mark Lee and Alan Wallington},
  booktitle = {Semantics in Text Processing. STEP 2008 Conference Proceedings},
  year = {2008},
  doi = {10.3115/1626481.1626509},
  url = {https://doi.org/10.3115/1626481.1626509}
}
```

- [← previous Identification of Naturally Occurring Numerical Expressions in Arabic](https://md.research.elamur.ai/free/en/papers/habash-2008-identification-naturally-occurring-numerical-expressions/)
- [next → A Method for Automatically Constructing Case Frames for English](https://md.research.elamur.ai/free/en/papers/kawahara-2008-method-automatically-constructing-case-frames/)
- [↑ back to papers of 2008](https://md.research.elamur.ai/free/en/papers/2008/page/2/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Agerri, R., J. Barnden, M. Lee, and A. Wallington (2007). Metaphor, inference and domain independent mappings. In Proceedings of Research Advances in Natural Language Processing (RANLP 2007), Borovets, Bulgaria, pp. 17–24. [find](https://scholar.google.com/scholar?q=Agerri%2C+R%2E%2C+J%2E+Barnden%2C+M%2E+Lee%2C+and+A%2E+Wallington+%282007%29%2E+Metaphor%2C+inference+and+domain+independent+mappings%2E+In+Proceedings+of+Research+Advances+in+Natural+Language+Processing+%28RANLP+2007%29%2C+Borovets)
2. Barnden, J., S. Glasbey, M. Lee, and A. Wallington (2003). Domain-transcending mappings in a system for metaphorical reasoning. In Companion Proceedings of the 10th Conference on the European Chapter of the Association for Computational Linguistics (EACL-03), pp. 57–61. [find](https://scholar.google.com/scholar?q=Barnden%2C+J%2E%2C+S%2E+Glasbey%2C+M%2E+Lee%2C+and+A%2E+Wallington+%282003%29%2E+Domain%2Dtranscending+mappings+in+a+system+for+metaphorical+reasoning%2E+In+Companion+Proceedings+of+the+10th+Conference+on+the+European+Chapter+)
3. Birke, J. and A. Sarkar (2006). A clustering approach for the nearly unsupervised recognition of nonliteral language. In Proceedings of the 11th Conference on the European Chapter of the Association for Computational Linguistics (EACL 2006), Trento, Italy, pp. 329–336. [find](https://scholar.google.com/scholar?q=Birke%2C+J%2E+and+A%2E+Sarkar+%282006%29%2E+A+clustering+approach+for+the+nearly+unsupervised+recognition+of+nonliteral+language%2E+In+Proceedings+of+the+11th+Conference+on+the+European+Chapter+of+the+Association+f)
4. Bos, J. and K. Markert (2006). Recognizing textual entailment with robust logical inference. In J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d’Alché Buc (Eds.), MLCW 2005, Volume 3944 of LNAI, pp. 404–426. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Bos%2C+J%2E+and+K%2E+Markert+%282006%29%2E+Recognizing+textual+entailment+with+robust+logical+inference%2E+In+J%2E+Qui%C3%B1onero%2DCandela%2C+I%2E+Dagan%2C+B%2E+Magnini%2C+and+F%2E+d%E2%80%99Alch%C3%A9+Buc+%28Eds%2E%29%2C+MLCW+2005%2C+Volume+3944+of+LNAI%2C+p)
5. Clark, P., W. Murray, J. Thompson, P. Harrison, J. Hobbs, and C. Fellbaum (2007). On the role of lexical and world knowledge in RTE3. In Proceedings of the Workshop on Textual Entailment and Paraphrasing, Prague, pp. 54–59. ACL 2007. [find](https://scholar.google.com/scholar?q=Clark%2C+P%2E%2C+W%2E+Murray%2C+J%2E+Thompson%2C+P%2E+Harrison%2C+J%2E+Hobbs%2C+and+C%2E+Fellbaum+%282007%29%2E+On+the+role+of+lexical+and+world+knowledge+in+RTE3%2E+In+Proceedings+of+the+Workshop+on+Textual+Entailment+and+Paraphras)
6. Dagan, I., O. Glickman, and B. Magnini (2007). The PASCAL Recognising Textual Entailment challenge. In J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d’Alché Buc (Eds.), MLCW 2005, Volume 3944 of LNAI, pp. 177–190. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+O%2E+Glickman%2C+and+B%2E+Magnini+%282007%29%2E+The+PASCAL+Recognising+Textual+Entailment+challenge%2E+In+J%2E+Qui%C3%B1onero%2DCandela%2C+I%2E+Dagan%2C+B%2E+Magnini%2C+and+F%2E+d%E2%80%99Alch%C3%A9+Buc+%28Eds%2E%29%2C+MLCW+2005%2C+Volume+3944+of+)
7. Falkenhainer, B., K. Forbus, and D. Gentner (1989). The structure-mapping engine: algorithm and examples. Artificial Intelligence 41(1), 1–63. [find](https://scholar.google.com/scholar?q=Falkenhainer%2C+B%2E%2C+K%2E+Forbus%2C+and+D%2E+Gentner+%281989%29%2E+The+structure%2Dmapping+engine%3A+algorithm+and+examples%2E+Artificial+Intelligence+41%281%29%2C+1%E2%80%9363%2E)
8. Fass, D. (1997). Processing metaphor and metonymy. Greenwich, Connecticut: Ablex. [find](https://scholar.google.com/scholar?q=Fass%2C+D%2E+%281997%29%2E+Processing+metaphor+and+metonymy%2E+Greenwich%2C+Connecticut%3A+Ablex%2E)
9. Gedigian, M., J. Bryant, S. Narayanan, and B. Ciric (2006). Catching metaphors. In Proceedings of the 3rd Workshop on Scalable Natural Language Understanding, New York, pp. 41–48. [find](https://scholar.google.com/scholar?q=Gedigian%2C+M%2E%2C+J%2E+Bryant%2C+S%2E+Narayanan%2C+and+B%2E+Ciric+%282006%29%2E+Catching+metaphors%2E+In+Proceedings+of+the+3rd+Workshop+on+Scalable+Natural+Language+Understanding%2C+New+York%2C+pp%2E+41%E2%80%9348%2E)
10. Goatly, A. (1997). The Language of Metaphors. Routledge. Agerri, Barnden, Lee, and Wallington Textual Entailment as an Evaluation Framework for Metaphor Resolution Markert, K. and M. Nissim (2007, June). Semeval-2007: Metonymy resolution at semeval-2007. In International Workshop on Semantic Evaluations (SemEval-2007), Prague, pp. 36–41. Association for Computational Linguistics (ACL-07). [find](https://scholar.google.com/scholar?q=Goatly%2C+A%2E+%281997%29%2E+The+Language+of+Metaphors%2E+Routledge%2E+Agerri%2C+Barnden%2C+Lee%2C+and+Wallington+Textual+Entailment+as+an+Evaluation+Framework+for+Metaphor+Resolution+Markert%2C+K%2E+and+M%2E+Nissim+%282007%2C+Jun)
11. Martin, J. (1990). [A computational model of metaphor interpretation](https://md.research.elamur.ai/free/en/papers/fass-1991-computational-model-metaphor-interpretation/). New York: Academic Press.
12. Mason, Z. (2004). [CorMet: a computational, corpus-based conventional metaphor extraction system](https://md.research.elamur.ai/free/en/papers/mason-2004-cormet-computational-corpus-based-conventional/). Computational Linguistics 30(1), 23–44.
13. Veale, T. and Y. Hao (2008). Enriching WordNet with folk knowledge and stereotypes. In Proceedings of the 4th Global WordNet Conference, Szeged, Hungary. [find](https://scholar.google.com/scholar?q=Veale%2C+T%2E+and+Y%2E+Hao+%282008%29%2E+Enriching+WordNet+with+folk+knowledge+and+stereotypes%2E+In+Proceedings+of+the+4th+Global+WordNet+Conference%2C+Szeged%2C+Hungary%2E)
14. Wallington, A., R. Agerri, J. Barnden, M. Lee, and T. Rumbell (2008, May). Affect transfer by metaphor for an intelligent conversational agent. In Proceedings of the LREC 2008 Workshop on Sentiment Analysis: Emotion, Metaphor, Ontology and Terminology (EMOT 08), Marrakech, Morocco, pp. 102–107. [find](https://scholar.google.com/scholar?q=Wallington%2C+A%2E%2C+R%2E+Agerri%2C+J%2E+Barnden%2C+M%2E+Lee%2C+and+T%2E+Rumbell+%282008%2C+May%29%2E+Affect+transfer+by+metaphor+for+an+intelligent+conversational+agent%2E+In+Proceedings+of+the+LREC+2008+Workshop+on+Sentiment+An)
15. Zaenen, A., L. Karttunen, and R. Crouch (2005). Local textual inference: Can it be defined or circumscribed? In Proceedings of the ACL 05 Workshop on Empirical Modelling of Semantic Equivalence and Entailment, pp. 31–36. [find](https://scholar.google.com/scholar?q=Zaenen%2C+A%2E%2C+L%2E+Karttunen%2C+and+R%2E+Crouch+%282005%29%2E+Local+textual+inference%3A+Can+it+be+defined+or+circumscribed%3F+In+Proceedings+of+the+ACL+05+Workshop+on+Empirical+Modelling+of+Semantic+Equivalence+and+En)

*Updated 2 November 2026*