---
title: "“Ask Not What Textual Entailment Can Do for You...” (2010)"
description: "The authors then propose an annotation model in which humans resolve an… 0.735 accuracy. 14 references, cited by 36 papers here."
authors: ["Mark Sammons", "V.G.Vinod Vydiswaran", "Dan Roth"]
year: 2010
venue: "Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P10-1122/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sammons-2010-ask-not-what-textual-entailment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sammons-2010-ask-not-what-textual-entailment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2010](https://md.research.elamur.ai/free/en/papers/2010/) / “Ask Not What Textual Entailment Can Do for You...”

# “Ask Not What Textual Entailment Can Do for You...”

**Authors:** [Mark Sammons](https://md.research.elamur.ai/free/en/authors/mark-sammons/), V.G.Vinod Vydiswaran and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2010 · **Venue:** [Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2010](https://md.research.elamur.ai/free/en/venues/acl/2010/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P10-1122/) · [PDF](https://aclanthology.org/P10-1122.pdf) · [OpenAlex](https://openalex.org/W2169954261)

**Citations:** 72 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/sammons-2010-ask-not-what-textual-entailment/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper argues that Recognizing Textual Entailment (RTE) should become a shared testbed for the component tasks of NLP, such as NER, coreference and parsing. The authors' main complaint is that each RTE pair receives only one global label (entailed, contradicted, or neither). This hides which local inference steps a system handled correctly and makes it impossible to credit individual components. Using all system outputs released for the RTE 5 two-way task, they compare participants with a strong lexical baseline. The best systems beat this baseline clearly, and they disagree both with it and with each other, so they are not just elaborated word-matchers. The authors then propose an annotation model in which humans resolve an entailment decision step by step, tagging the domain and the linguistic phenomena involved. A pilot annotation over a sample of the NIST TAC RTE 5 corpus suggests that this mark-up is feasible and immediately useful. The goal is a community-wide, extensible resource for judging NLP components inside a realistic inference application.

## Key points
- In the RTE 5 two-way task, the top system reached 0.735 accuracy. The "smart" lexical baseline (Do et al., 2010) reached 0.612; it combines lexical match with WordNet similarity, stemming and a large stopword list.
- Most systems agree with the baseline on only about 67% of examples. With chance agreement at 0.5, the Cohen's Kappa values fall roughly between 0.3 and 0.4, which shows real diversity among approaches.
- The official RTE 5 ablation of WordNet, VerbOcean and DIRT found little or no contribution. The authors caution that a single global label cannot show where such resources actually help.
- Successful systems typically used augmented dependency trees, semantic role labels, NER, coreference resolvers and ad-hoc modules for specific phenomena such as numeric reasoning.
- The proposed taxonomy of domains and phenomena is explicitly open. New annotators can add categories, or declare merges and splits, so that annotation sources stay compatible.

## Interesting ideas
- Annotating each test item with the local phenomena it requires turns a single pass/fail score into per-skill evidence, so credit and blame can be assigned to specific components.
- A pairwise agreement matrix between systems and a baseline, read against chance agreement, shows whether systems are truly different or are minor variants of one idea.
- A carefully tuned lexical baseline is a hard negative control, because beating it is the minimum proof that structure is being used.
- Contradiction can come from two mentions that most likely describe the same event but give it different values, such as the purchase price in the LexCorp example.
- Nominalized relations like "purchase of X by Y" must be recovered as verb-style relations with swapped surface roles before they can be matched.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Tag evaluation items with phenomena such as nominalization, numeric mismatch or coreference, so that every wrong answer points to the pipeline stage responsible.
- **I2** (learn): Before trusting silver data from several witness parsers, measure their chance-corrected agreement; correlated witnesses give false confidence.
- **I3** (answer): Run a bag-of-words baseline on the school-test and fable QA sets; items it already solves should not count as evidence that Mova understands the text.
- **I4** (world): When two frames resolve to one transaction but carry conflicting measure or value slots, record an explicit contradiction rather than two separate facts.
- **I5** (frames): Map nominal event heads to VerbNet frames and assign Agent and Theme from the "by" and "of" dependents, so that noun and verb phrasings yield the same frame.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Mark Sammons, V.G.Vinod Vydiswaran and Dan Roth. 2010. “Ask Not What Textual Entailment Can Do for You...”. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sammons2010ask,
  title = {{“Ask Not What Textual Entailment Can Do for You...”}},
  author = {Mark Sammons, V.G.Vinod Vydiswaran and Dan Roth},
  booktitle = {Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://aclanthology.org/P10-1122/}
}
```

- [← previous Painless Unsupervised Learning with Features](https://md.research.elamur.ai/free/en/papers/kirkpatrick-2010-painless-unsupervised-learning-features/)
- [next → "cba to check the spelling": Investigating Parser Performance on Discussion Forum Posts](https://md.research.elamur.ai/free/en/papers/foster-2010-cba-check-spelling-investigating-parser/)
- [↑ back to papers of 2010](https://md.research.elamur.ai/free/en/papers/2010/)

## References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernando Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In Notebook papers and Results, Text Analysis Conference (TAC), pages 14–24. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernando+Magnini%2E+2009%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E+In+Notebook+papers+and+Results%2C+Text+Analysis+Co)
2. Timothy Chklovski and Patrick Pantel. 2004. VerbOcean: Mining the Web for Fine-Grained Semantic Verb Relations. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP-04), pages 33–40. [find](https://scholar.google.com/scholar?q=Timothy+Chklovski+and+Patrick+Pantel%2E+2004%2E+VerbOcean%3A+Mining+the+Web+for+Fine%2DGrained+Semantic+Verb+Relations%2E+In+Proceedings+of+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%2D)
3. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104)
4. I. Dagan, O. Glickman, and B. Magnini, editors. 2006. The PASCAL Recognising Textual Entailment Challenge., volume 3944. Springer-Verlag, Berlin. [find](https://scholar.google.com/scholar?q=I%2E+Dagan%2C+O%2E+Glickman%2C+and+B%2E+Magnini%2C+editors%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E%2C+volume+3944%2E+Springer%2DVerlag%2C+Berlin%2E)
5. Marie-Catherine de Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. [Finding contradictions in text](https://md.research.elamur.ai/free/en/papers/marneffe-2008-finding-contradictions-text/). In Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio, June. Association for Computational Linguistics.
6. Quang Do, Dan Roth, Mark Sammons, Yuancheng Tu, and V.G.Vinod Vydiswaran. 2010. Robust, Light-weight Approaches to compute Lexical Similarity. Computer Science Research and Technical Reports, University of Illinois. [http://L2R.cs.uiuc.edu/∼danr/Papers/DRSTV10.pdf](http://L2R.cs.uiuc.edu/∼danr/Papers/DRSTV10.pdf). [↗](http://L2R.cs.uiuc.edu/∼danr/Papers/DRSTV10.pdf)
7. C. Fellbaum. 1998. WordNet: An Electronic Lexical Database. MIT Press. [find](https://scholar.google.com/scholar?q=C%2E+Fellbaum%2E+1998%2E+WordNet%3A+An+Electronic+Lexical+Database%2E+MIT+Press%2E)
8. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+pascal+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+ACL%2DPASCAL+Workshop+on+Textual+Entailment+and+Pa)
9. Sanda Harabagiu and Andrew Hickl. 2006. [Methods for Using Textual Entailment in Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/harabagiu-2006-methods-textual-entailment-open-domain/). In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 905–912, Sydney, Australia, July. Association for Computational Linguistics.
10. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of HLT/NAACL, New York. [find](https://scholar.google.com/scholar?q=Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Lance+Ramshaw%2C+and+Ralph+Weischedel%2E+2006%2E+Ontonotes%3A+The+90%25+solution%2E+In+Proceedings+of+HLT%2FNAACL%2C+New+York%2E)
11. D. Lin and P. Pantel. 2001. DIRT: discovery of inference rules from text. In Proc. of ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2001, pages 323–328. [find](https://scholar.google.com/scholar?q=D%2E+Lin+and+P%2E+Pantel%2E+2001%2E+DIRT%3A+discovery+of+inference+rules+from+text%2E+In+Proc%2E+of+ACM+SIGKDD+Conference+on+Knowledge+Discovery+and+Data+Mining+2001%2C+pages+323%E2%80%93328%2E)
12. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In The Eighth International Conference on Computational Semantics (IWCS-8), Tilburg, Netherlands. [find](https://scholar.google.com/scholar?q=Bill+MacCartney+and+Christopher+D%2E+Manning%2E+2009%2E+An+extended+model+of+natural+logic%2E+In+The+Eighth+International+Conference+on+Computational+Semantics+%28IWCS%2D8%29%2C+Tilburg%2C+Netherlands%2E)
13. Shachar Mirkin, Lucia Specia, Nicola Cancedda, Ido Dagan, Marc Dymetman, and Idan Szpektor. 2009. Source-language entailment modeling for translating unknown terms. In ACL/AFNLP, pages 791– 799, Suntec, Singapore, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shachar+Mirkin%2C+Lucia+Specia%2C+Nicola+Cancedda%2C+Ido+Dagan%2C+Marc+Dymetman%2C+and+Idan+Szpektor%2E+2009%2E+Source%2Dlanguage+entailment+modeling+for+translating+unknown+terms%2E+In+ACL%2FAFNLP%2C+pages+791%E2%80%93+799%2C+Sunte)
14. Sebastian Pado, Michel Galley, Dan Jurafsky, and Christopher D. Manning. 2009. Robust machine translation evaluation with entailment features. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 297–305, Suntec, Singapore, August. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sebastian+Pado%2C+Michel+Galley%2C+Dan+Jurafsky%2C+and+Christopher+D%2E+Manning%2E+2009%2E+Robust+machine+translation+evaluation+with+entailment+features%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual)

*Updated 24 October 2026*