--- title: "The Third PASCAL Recognizing Textual Entailment Challenge" description: "The paper summarizes the Third PASCAL Recognizing Textual Entailment Challenge (RTE-3). 5 references, cited by 384 papers here." authors: ["Danilo Giampiccolo", "Bernardo Magnini", "Ido Dagan", "Bill Dolan"] year: 2007 venue: "Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing" source: "https://aclanthology.org/W07-1401/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2007](https://md.research.elamur.ai/free/en/papers/2007/) / The Third PASCAL Recognizing Textual Entailment Challenge # The Third PASCAL Recognizing Textual Entailment Challenge **Authors:** [Danilo Giampiccolo](https://md.research.elamur.ai/free/en/authors/danilo-giampiccolo/), [Bernardo Magnini](https://md.research.elamur.ai/free/en/authors/bernardo-magnini/), [Ido Dagan](https://md.research.elamur.ai/free/en/authors/ido-dagan/) and [Bill Dolan](https://md.research.elamur.ai/free/en/authors/bill-dolan/) · **Year:** 2007 · **Venue:** [Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing](https://md.research.elamur.ai/free/en/venues/acl-pascal-workshop-on-textual-entailment-and-paraphrasing/) ([ACL-PASCAL Workshop on Textual Entailment and Paraphrasing 2007](https://md.research.elamur.ai/free/en/venues/acl-pascal-workshop-on-textual-entailment-and-paraphrasing/2007/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W07-1401/) · [PDF](https://aclanthology.org/W07-1401.pdf) · [OpenAlex](https://openalex.org/W4253067820) **Citations:** 376 citations (OpenAlex, 2026-10-14) · [384](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper summarizes the Third PASCAL Recognizing Textual Entailment Challenge (RTE-3). Participants had to determine whether the meaning of a short hypothesis h can be inferred from a text t. Such semantic inference is needed by many NLP applications, including QA, IE, summarization and IR, so the organizers made it a separate task. They hope that a generic entailment "engine" will eventually emerge that can be plugged in as a standard module. As before, the corpus contains 1600 t-h pairs, created manually in four application settings. This time, longer texts of up to a paragraph were added for the first time. A shared RTE Resource Pool was also introduced for exchanging tools and knowledge. NIST launched a pilot task with three-way labeling and justifications of decisions. In total, 26 teams submitted 44 runs, and the results mostly exceeded those of previous rounds. The work shows that the data creation methodology is reliable. At the same time, the authors acknowledge weaknesses: progress in knowledge acquisition is small, and the distribution of examples in the corpus is largely arbitrary. ## Key points - The RTE-3 corpus has 1600 pairs, split equally into development and test. Each of the IE, IR, QA and SUM settings accounts for 200 pairs in each part. Negative examples are slightly more numerous: 51.50% and 51.25%. - Texts longer than 270 bytes are marked as L. About 17% of the test part are such texts. Accuracy on them (58.72%) is only slightly lower than on short ones (61.93%). - The pairs were built from the outputs of real systems and from publicly available data: ACE-2004, MUC-4, TREC, CLEF, WikiNews, Wikipedia. - Average accuracy depends markedly on the setting: QA — 0.71, IR — 0.66, SUM — 0.58, IE — 0.52. - The NIST pilot task required YES/NO/UNKNOWN decisions, that is, distinguishing contradiction from unknown, as well as justifying the decisions. Among the new approaches, the authors highlight Hickl's commitment-based model, Bar-Haim's proof system and Stanford's use of Natural Logic. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Microsoft](https://md.research.elamur.ai/free/en/companies/microsoft/) — Company, Redmond, United States · 325 works · 142,931 citations - [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 141 works · 23,225 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works 💡 Concepts (1): [💡 Natural language understanding](https://md.research.elamur.ai/free/en/concepts/natural-language-understanding/) · 51 works ## Cite Cite as: Danilo Giampiccolo, Bernardo Magnini, Ido Dagan and Bill Dolan. 2007. The Third PASCAL Recognizing Textual Entailment Challenge. Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{giampiccolo2007third, title = {{The Third PASCAL Recognizing Textual Entailment Challenge}}, author = {Danilo Giampiccolo, Bernardo Magnini, Ido Dagan and Bill Dolan}, booktitle = {Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing}, year = {2007}, doi = {10.3115/1654536.1654538}, url = {https://doi.org/10.3115/1654536.1654538} } ``` - [← previous Instance Weighting for Domain Adaptation in NLP](https://md.research.elamur.ai/free/en/papers/jiang-2007-instance-weighting-domain-adaptation-nlp/) - [next → Large Language Models in Machine Translation](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/) - [↑ back to papers of 2007](https://md.research.elamur.ai/free/en/papers/2007/) ## References 5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini and Idan Szpektor. 2006. The Second PASCAL Recognizing Textual Entailment Challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognizing Textual Entailment, Venice, Italy. [find](https://scholar.google.com/scholar?q=Roy+Bar%2DHaim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini+and+Idan+Szpektor%2E+2006%2E+The+Second+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+In+Proceedings+of+the+Second+) 2. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognizing Textual Entailment Challenge. In Quiñonero-Candela et al., editors, MLCW 2005, LNAI Volume 3944, pages 177-190. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+In+Qui%C3%B1onero%2DCandela+et+al%2E%2C+editors%2C+MLCW+2005%2C+LNAI+Volume+3944%2C+pages+177%2D190%2E+Springer%2DVe) 3. J. R. Landis and G. G. Koch. 1997. The measurements of observer agreement for categorical data. Biometrics, 33:159–174. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Landis+and+G%2E+G%2E+Koch%2E+1997%2E+The+measurements+of+observer+agreement+for+categorical+data%2E+Biometrics%2C+33%3A159%E2%80%93174%2E) 4. Rebecca Passonneau, Ani Nenkova., Kathleen McKeown, and Sergey Sigleman. 2005. Applying the pyramid method in DUC 2005. In Proceedings of the Document Understanding Conference (DUC 05), Vancouver, B.C., Canada. [↗](https://doi.org/10.7916/d8tx3pvd) 5. Ellen M. Voorhees and Donna Harman. 1999. Overview of the seventh text retrieval conference. In Proceedings of the Seventh Text Retrieval Conference (TREC-7). NIST Special Publication. [find](https://scholar.google.com/scholar?q=Ellen+M%2E+Voorhees+and+Donna+Harman%2E+1999%2E+Overview+of+the+seventh+text+retrieval+conference%2E+In+Proceedings+of+the+Seventh+Text+Retrieval+Conference+%28TREC%2D7%29%2E+NIST+Special+Publication%2E) *Updated 1 November 2026*