---
title: "SemEval-2013 Task 7 — Dzikovska 2013"
description: "The paper reports on a SemEval-2013 shared task that joined educational NLP with the Recognizing Textual Entailment (RTE) series."
authors: ["Myroslava Dzikovska", "Rodney Nielsen", "Chris Brew", "Claudia Leacock", "Danilo Giampiccolo", "Luisa Bentivogli", "Peter Clark", "Ido Dagan", "Hoa Trang Dang"]
year: 2013
venue: "Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)"
source: "https://aclanthology.org/S13-2045/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/dzikovska-2013-semeval-2013-task-7-joint/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/dzikovska-2013-semeval-2013-task-7-joint/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2013](https://md.research.elamur.ai/free/papers/2013/) / SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

# SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

**Authors:** Myroslava Dzikovska, [Rodney Nielsen](https://md.research.elamur.ai/free/authors/rodney-nielsen/), Chris Brew, Claudia Leacock, Danilo Giampiccolo, [Luisa Bentivogli](https://md.research.elamur.ai/free/authors/luisa-bentivogli/), [Peter Clark](https://md.research.elamur.ai/free/authors/peter-clark/), [Ido Dagan](https://md.research.elamur.ai/free/authors/ido-dagan/), [Hoa Trang Dang](https://md.research.elamur.ai/free/authors/hoa-trang-dang/) · **Year:** 2013 · **Venue:** Second Joint Conference on Lexical and Computational Semantics (\*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/S13-2045/) · [PDF](https://aclanthology.org/S13-2045.pdf) · [OpenAlex](https://openalex.org/W2115584598)

**Citations:** 165 citations (openalex-snapshot · 2026-10-14) · [70](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
The paper reports on a SemEval-2013 shared task that joined educational NLP with the Recognizing Textual Entailment (RTE) series. The problem is automatically judging short student answers against an instructor's reference answer, so that a tutoring system can choose suitable feedback. The organizers built the data from the SRA corpus, which has two parts: BEETLE (electricity and electronics tutoring dialogues) and SciEntsBank (science assessment answers). They removed questions that depended on charts or allowed several unrelated correct answers, and they re-checked parts of the labels.

Systems could tackle a 5-way labeling task or simpler 3-way and 2-way entailment-style versions, which were derived by merging categories. Test data was split by how far it lay from the training material. A pilot task on partial entailment at the level of answer facets was also run. Nine teams took part.

The best systems usually clearly beat a lexical overlap baseline, although that baseline proved surprisingly strong, especially in the 2-way setting. The results suggest that RTE methods are useful for answer assessment and that educational data is a meaningful testbed for RTE.

## Key points
- The SRA corpus labels each answer with one of five categories: correct, partially correct incomplete, contradictory, irrelevant, and non domain. The 3-way version merges the last three into "incorrect", and the 2-way version further merges contradictory with incorrect.
- BEETLE contains 56 questions with roughly 3,000 answers. SciEntsBank, after filtering, has about 10,000 answers to 197 questions across 15 science domains.
- Test sets were organized as Unseen Answers (new answers to training questions), Unseen Questions (new questions in known domains) and Unseen Domains.
- Systems were scored with overall accuracy and with macro-averaged and weighted-average precision, recall and F1.
- The Partial Entailment Pilot had one participant, UKP-BIU, whose system combined semantic textual similarity features with BIUTEE.
  - Training data held 13,145 facets and the test set 16,263.
  - Its best run reached weighted F1 of 0.782, 0.765 and 0.816 on UA, UQ and UD.
  - The majority baseline scored roughly 0.48 to 0.55.

## Interesting ideas
- **Include the question in the entailing text.** The Text is defined as the question plus the student answer, while the Hypothesis is the reference answer. This handles students leaving out details the question already supplies, and it carries over to any QA evaluation where answers are elliptical.
- **Collapse one fine-grained annotation into several task granularities.** Deriving 5-, 3- and 2-way versions from the same labels lets different research communities compete on the same data. It also shows where performance drops as the categories become finer.
- **Split test sets by generalization distance.** Separating unseen answers, unseen questions and unseen domains shows whether a system memorizes question-specific patterns or truly generalizes. This is a cheap but revealing protocol for any domain-transfer evaluation.
- **Break the reference into small facets for partial entailment.** Decomposing a reference answer into term pairs and asking whether each is expressed allows credit for partially correct answers. It also gives diagnostic feedback rather than a single verdict.
- **Check task fit with a small pilot annotation first.** Before committing, the organizers annotated a sample with entailment judgments to confirm that these align well with teacher grades. This is a useful sanity step whenever an existing task framing is reused for a new application.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cited by

Cited by

70 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 68 not yet.

- [A guide to the dataset explosion in QA, NLI, and commonsense reasoning](https://md.research.elamur.ai/free/papers/rogers-2020-guide-dataset-explosion-qa-nli/)
- [Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://md.research.elamur.ai/free/papers/storks-2019-recent-advances-natural-language-inference/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{dzikovska2013semeval,
  title = {{SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge}},
  author = {Dzikovska, Myroslava and Nielsen, Rodney and Brew, Chris and Leacock, Claudia and Giampiccolo, Danilo and Bentivogli, Luisa and Clark, Peter and Dagan, Ido and Dang, Hoa Trang},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)},
  year = {2013},
  url = {https://aclanthology.org/S13-2045/}
}
```

- [← previous Semantic Relations between Events and their Time, Locations and Participants for Event Coreference Resolution](https://md.research.elamur.ai/free/papers/cybulska-2013-semantic-relations-between-events-their/)
- [next → Squibs: What Is a Paraphrase?](https://md.research.elamur.ai/free/papers/bhagat-2013-squibs-what-is-paraphrase/)
- [↑ back to papers of 2013](https://md.research.elamur.ai/free/papers/2013/)

## References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2. The Journal of Technology, Learning, and Assessment, 4(3), February. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater+v%2E2%2E+The+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%2C+February%2E)
2. Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. 2012. Ukp: Computing semantic textual similarity by combining multiple content similarity measures. In Proceedings of the 6th International Workshop on Semantic Evaluation, held in conjunction with the 1st Joint Conference on Lexical and Computational Semantics, pages 435–440, Montreal, Canada, June. [find](https://scholar.google.com/scholar?q=Daniel+Ba%CC%88r%2C+Chris+Biemann%2C+Iryna+Gurevych%2C+and+Torsten+Zesch%2E+2012%2E+Ukp%3A+Computing+semantic+textual+similarity+by+combining+multiple+content+similarity+measures%2E+In+Proceedings+of+the+6th+Internation)
3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernando Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of Text Analysis Conference (TAC) 2009. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernando+Magnini%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+Text+Analysis+Conference+%28TAC%29)
4. Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2010. Thesixth PAS-CAL recognizing textual entailment challenge. In Notebook papers and results, Text Analysis Conference (TAC). [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+and+Danilo+Giampiccolo%2E+2010%2E+Thesixth+PAS%2DCAL+recognizing+textual+entailment+challenge%2E+In+Notebook+papers+and+results%2C+Text+Analysis+Confere)
5. Luisa Bentivogli, Peter Clark, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2011. The seventh PASCAL recognizing textual entailment challenge. In Notebook papers and results, Text Analysis Conference (TAC). [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Peter+Clark%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+and+Danilo+Giampiccolo%2E+2011%2E+The+seventh+PASCAL+recognizing+textual+entailment+challenge%2E+In+Notebook+papers+and+results%2C+Text+Analysis+Confe)
6. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In J. Quin̄onero-Candela, I. Dagan, B. Magnini, and F. d’Alché Buc, editors, Machine Learning Challenges, volume 3944 of Lecture Notes in Computer Science. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+J%2E+Quin%CC%84onero%2DCandela%2C+I%2E+Dagan%2C+B%2E+Magnini%2C+and+F%2E+d%E2%80%99Alche%CC%81+Buc%2C+editors%2C+Machine+Learnin)
7. Myroslava O. Dzikovska, Johanna D. Moore, Natalie Steinhauser, Gwendolyn Campbell, Elaine Farrow, and Charles B. Callaway. 2010. Beetle II: a system for tutoring and computational linguistics experimentation. In Proc. of ACL 2010 System Demonstrations, pages 13–18. [find](https://scholar.google.com/scholar?q=Myroslava+O%2E+Dzikovska%2C+Johanna+D%2E+Moore%2C+Natalie+Steinhauser%2C+Gwendolyn+Campbell%2C+Elaine+Farrow%2C+and+Charles+B%2E+Callaway%2E+2010%2E+Beetle+II%3A+a+system+for+tutoring+and+computational+linguistics+experime)
8. Myroslava O. Dzikovska, Rodney D. Nielsen, and Chris Brew. 2012. Towards effective tutorial feedback for explanation questions: A dataset and baselines. In Proc. of 2012 Conference of NAACL: Human Language Technologies, pages 200–210. [find](https://scholar.google.com/scholar?q=Myroslava+O%2E+Dzikovska%2C+Rodney+D%2E+Nielsen%2C+and+Chris+Brew%2E+2012%2E+Towards+effective+tutorial+feedback+for+explanation+questions%3A+A+dataset+and+baselines%2E+In+Proc%2E+of+2012+Conference+of+NAACL%3A+Human+Lan)
9. Danilo Giampiccolo, Hoa Trang Dang, Bernardo Magnini, Ido Dagan, Elena Cabrio, and Bill Dolan. 2008. The fourth PASCAL recognizing textual entailment challenge. In Proceedings of Text Analysis Conference (TAC) 2008, Gaithersburg, MD, November. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Hoa+Trang+Dang%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+Elena+Cabrio%2C+and+Bill+Dolan%2E+2008%2E+The+fourth+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+Text+Analysis+Confere)
10. Michael Glass. 2000. Processing language input in the CIRCSIM-Tutor intelligent tutoring system. In Papers from the 2000 AAAI Fall Symposium, Available as AAAI technical report FS-00-01, pages 74–79. [find](https://scholar.google.com/scholar?q=Michael+Glass%2E+2000%2E+Processing+language+input+in+the+CIRCSIM%2DTutor+intelligent+tutoring+system%2E+In+Papers+from+the+2000+AAAI+Fall+Symposium%2C+Available+as+AAAI+technical+report+FS%2D00%2D01%2C+pages+74%E2%80%9379%2E)
11. A. C. Graesser, K. Wiemer-Hastings, P. Wiemer-Hastings, and R. Kreuz. 1999. Autotutor: A simulation of a human tutor. Cognitive Systems Research, 1:35–51. [↗](https://doi.org/10.1016/s1389-0417(99)00005-4)
12. Xiangen Hu, Zhiqiang Cai, Max Louwerse, Andrew Olney, Phanni Penumatsa, and Art Graesser. 2003. A revised algorithm for latent semantic analysis. In Proceedings of the 18th International Joint Conference on Artificial intelligence (IJCAI’03), pages 1489–1491, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. [find](https://scholar.google.com/scholar?q=Xiangen+Hu%2C+Zhiqiang+Cai%2C+Max+Louwerse%2C+Andrew+Olney%2C+Phanni+Penumatsa%2C+and+Art+Graesser%2E+2003%2E+A+revised+algorithm+for+latent+semantic+analysis%2E+In+Proceedings+of+the+18th+International+Joint+Confere)
13. Pamela W. Jordan, Maxim Makatchev, and Kurt Van-Lehn. 2004. Combining competing language understanding approaches in an intelligent tutoring system. In Proc. of Intelligent Tutoring Systems Conference, pages 346–357. [find](https://scholar.google.com/scholar?q=Pamela+W%2E+Jordan%2C+Maxim+Makatchev%2C+and+Kurt+Van%2DLehn%2E+2004%2E+Combining+competing+language+understanding+approaches+in+an+intelligent+tutoring+system%2E+In+Proc%2E+of+Intelligent+Tutoring+Systems+Conference)
14. Pamela Jordan, Maxim Makatchev, Umarani Pappuswamy, Kurt VanLehn, and Patricia Albacete. 2006. A natural language tutorial dialogue system for physics. In Proc. of 19th Intl. FLAIRS conference, pages 521–527. [find](https://scholar.google.com/scholar?q=Pamela+Jordan%2C+Maxim+Makatchev%2C+Umarani+Pappuswamy%2C+Kurt+VanLehn%2C+and+Patricia+Albacete%2E+2006%2E+A+natural+language+tutorial+dialogue+system+for+physics%2E+In+Proc%2E+of+19th+Intl%2E+FLAIRS+conference%2C+pages+)
15. Claudia Leacock and Martin Chodorow. 2003. C-rater: Automated scoring of short-answer questions. Computers and the Humanities, 37(4):389–405. [↗](https://doi.org/10.1023/a:1025779619903)
16. Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel R. Tetreault. 2010. Automated Grammatical Error Detection for Language Learners. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. [↗](https://doi.org/10.2200/s00275ed1v01y201006hlt009)
17. Philip M. McCarthy, Vasile Rus, Scott A. Crossley, Arthur C. Graesser, and Danielle S. McNamara. 2008. Assessing forward-, reverse-, and average-entailment indices on natural language input from the intelligent tutoring system, iSTART. In Proc. of 21st Intl. FLAIRS conference, pages 165–170. [find](https://scholar.google.com/scholar?q=Philip+M%2E+McCarthy%2C+Vasile+Rus%2C+Scott+A%2E+Crossley%2C+Arthur+C%2E+Graesser%2C+and+Danielle+S%2E+McNamara%2E+2008%2E+Assessing+forward%2D%2C+reverse%2D%2C+and+average%2Dentailment+indices+on+natural+language+input+from+the+i)
18. Michael Mohler, Razvan Bunescu, and Rada Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 752–762, Portland, Oregon, USA, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Michael+Mohler%2C+Razvan+Bunescu%2C+and+Rada+Mihalcea%2E+2011%2E+Learning+to+grade+short+answer+questions+using+semantic+similarity+measures+and+dependency+graph+alignments%2E+In+Proceedings+of+the+49th+Annual+)
19. Jessica Nelson, Charles Perfetti, David Liben, and Meredith Liben. 2012. Measures of text difficulty: Testing their predictive value for grade levels and student performance. Technical report, Student Achievement Partners. [http://www.ccsso.org/Documents/2012/Measures%20ofText%](http://www.ccsso.org/Documents/2012/Measures%20ofText%) 20Difficulty_fina%l.2012.pdf. [↗](http://www.ccsso.org/Documents/2012/Measures%20ofText%)
20. Rodney D. Nielsen, Wayne Ward, and James H. Martin. 2008a. Learning to assess low-level conceptual understanding. In Proc. of 21st Intl. FLAIRS Conference, pages 427–432. [find](https://scholar.google.com/scholar?q=Rodney+D%2E+Nielsen%2C+Wayne+Ward%2C+and+James+H%2E+Martin%2E+2008a%2E+Learning+to+assess+low%2Dlevel+conceptual+understanding%2E+In+Proc%2E+of+21st+Intl%2E+FLAIRS+Conference%2C+pages+427%E2%80%93432%2E)
21. Rodney D. Nielsen, Wayne Ward, James H. Martin, and Martha Palmer. 2008b. Annotating students’ understanding of science concepts. In Proceedings of the Sixth International Language Resources and Evaluation Conference, (LREC08), Marrakech, Morocco. [find](https://scholar.google.com/scholar?q=Rodney+D%2E+Nielsen%2C+Wayne+Ward%2C+James+H%2E+Martin%2C+and+Martha+Palmer%2E+2008b%2E+Annotating+students%E2%80%99+understanding+of+science+concepts%2E+In+Proceedings+of+the+Sixth+International+Language+Resources+and+Evalu)
22. Sarah Petersen and Mari Ostendorf. 2009. A machine learning approach to reading level assessment. Computer, Speech and Language, 23(1):89–106. [↗](https://doi.org/10.1016/j.csl.2008.04.003)
23. Heather Pon-Barry, Brady Clark, Karl Schultz, Elizabeth Owen Bratt, and Stanley Peters. 2004. Advantages of spoken language interaction in dialogue-based intelligent tutoring systems. In Proc. of ITS-2004 Conference, pages 390–400. [find](https://scholar.google.com/scholar?q=Heather+Pon%2DBarry%2C+Brady+Clark%2C+Karl+Schultz%2C+Elizabeth+Owen+Bratt%2C+and+Stanley+Peters%2E+2004%2E+Advantages+of+spoken+language+interaction+in+dialogue%2Dbased+intelligent+tutoring+systems%2E+In+Proc%2E+of+ITS%2D)
24. Stephen G Pulman and Jana Z Sukkarieh. 2005. Automatic short answer marking. In Proceedings of the Second Workshop on Building Educational Applications Using NLP, pages 9–16, Ann Arbor, Michigan, June. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Stephen+G+Pulman+and+Jana+Z+Sukkarieh%2E+2005%2E+Automatic+short+answer+marking%2E+In+Proceedings+of+the+Second+Workshop+on+Building+Educational+Applications+Using+NLP%2C+pages+9%E2%80%9316%2C+Ann+Arbor%2C+Michigan%2C+June)
25. Kathryn M. Sheehan, Irene Kostin, Yoko Futagi, and Michael Flor. 2010. Generating automated text complexity classifications that are aligned with targeted text complexity standards. Technical Report RR-10-28, Educational Testing Service. [↗](https://doi.org/10.1002/j.2333-8504.2010.tb02235.x)
26. Mark D. Shermis and Jill Burstein, editors. 2013. Handbook on Automated Essay Evaluation: Current Applications and New Directions. Routledge. [find](https://scholar.google.com/scholar?q=Mark+D%2E+Shermis+and+Jill+Burstein%2C+editors%2E+2013%2E+Handbook+on+Automated+Essay+Evaluation%3A+Current+Applications+and+New+Directions%2E+Routledge%2E)
27. Asher Stern and Ido Dagan. 2011. A confidence model for syntactically-motivated entailment proofs. In Recent Advances in Natural Language Processing (RANLP 2011), pages 455–462, Hissar, Bulgaria, September. [find](https://scholar.google.com/scholar?q=Asher+Stern+and+Ido+Dagan%2E+2011%2E+A+confidence+model+for+syntactically%2Dmotivated+entailment+proofs%2E+In+Recent+Advances+in+Natural+Language+Processing+%28RANLP+2011%29%2C+pages+455%E2%80%93462%2C+Hissar%2C+Bulgaria%2C+Sept)
28. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2010. Mining multi-label data. In Oded Maimon and Lior Rokach, editors, Data Mining and Knowledge Discovery Handbook, pages 667–685. Springer US. [↗](https://doi.org/10.1007/978-0-387-09823-4_34)
29. Kurt VanLehn, Pamela Jordan, and Diane Litman. 2007. Developing pedagogically effective tutorial dialogue tactics: Experiments and a testbed. In Proc. of SLaTE Workshop on Speech and Language Technology in Education, Farmington, PA, October. [find](https://scholar.google.com/scholar?q=Kurt+VanLehn%2C+Pamela+Jordan%2C+and+Diane+Litman%2E+2007%2E+Developing+pedagogically+effective+tutorial+dialogue+tactics%3A+Experiments+and+a+testbed%2E+In+Proc%2E+of+SLaTE+Workshop+on+Speech+and+Language+Technolo)
30. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th International Conference on Computational linguistics (COLING 2000), pages 947–953, Stroudsburg, PA, USA. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.cs/0008005)