---
title: "Learning to Grade Short Answer Questions using Semantic Similarity Measures — Mohler 2011"
description: "The authors build a three-stage pipeline: Both answers are parsed with the Stanford Dependency Parser. 32 references, cited by 63 papers here."
authors: ["Michael Mohler", "Razvan Bunescu", "Rada Mihalcea"]
year: 2011
venue: "Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/P11-1076/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mohler-2011-learning-grade-short-answer-questions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohler-2011-learning-grade-short-answer-questions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments

# Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments

**Authors:** Michael Mohler, [Razvan Bunescu](https://md.research.elamur.ai/free/en/authors/razvan-bunescu/) and [Rada Mihalcea](https://md.research.elamur.ai/free/en/authors/rada-mihalcea/) · **Year:** 2011 · **Venue:** Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P11-1076/) · [PDF](https://aclanthology.org/P11-1076.pdf) · [OpenAlex](https://openalex.org/W2133436118)

**Citations:** 247 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/mohler-2011-learning-grade-short-answer-questions/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper addresses automatic grading of short free-text student answers. Each student answer is compared with an instructor's reference answer, and the system predicts a grade on a 0–5 scale rather than a yes/no verdict as in textual entailment or paraphrase detection.

The authors build a three-stage pipeline:
1. Both answers are parsed with the Stanford Dependency Parser. An averaged perceptron, trained on a small hand-annotated set of node pairs, scores how well each instructor node matches each student node. It uses lexico-syntactic features plus WordNet and LSA similarity computed over the nodes and their surrounding subgraphs.
2. The Hungarian algorithm finds the best one-to-one alignment between the two graphs. Question demoting reduces credit for words simply copied from the question.
3. Alignment scores and several bag-of-words (BOW) similarity measures feed SVMRank or SVR. Isotonic regression maps the outputs back to the grade scale.

Learned combinations of BOW measures beat every single measure, and adding alignment features helps further (best Pearson 0.518 against inter-annotator agreement of 0.586). The results suggest that even rough syntactic structure adds value over pure BOW grading.

## Key points
- Node matching uses 68 features in total. Of these, 36 are semantic similarities (eight WordNet measures plus LSA) computed over four nested subgraph definitions around each node. The other 32 are lexico-syntactic features of the node itself.
- Training data for the perceptron came from hand-annotating 32 student answers to 21 questions, giving 7303 node pairs (656 matches, 6647 non-matches). These answers were then excluded from the rest of the pipeline and from evaluation.
- SVMRank achieved higher correlation, while SVR achieved lower RMSE:
  - With BOW features only, SVMRank improved Pearson's ρ from 0.462 (best single BOW measure) to 0.480.
  - The hybrid model with normalized alignment features reached 0.518.
  - SVR with the hybrid model and all alignment features reached the lowest RMSE, 0.978.
- Alignment features alone were too weak to serve as a standalone grader (ρ as low as 0.167–0.266 for unnormalized ones).
- The simple tf*idf baseline correlated poorly but had competitive error rates. This is because the dataset is skewed toward high grades: 46% of its outputs fell in the 4–4.5 range.

## Interesting ideas
- **Matching subgraphs instead of single words.** Similarity is computed between nodes together with progressively narrower sets of their dependents, controlled by which edge types may be followed. A descriptive phrase can thus match a single word, for example a multiword paraphrase versus a compound noun. The idea transfers to any alignment task where paraphrases differ in syntactic granularity.
- **Question demoting.** Words that appear in the question are discounted before answers are compared, so restating the prompt earns no credit. This is a cheap fix for any QA or response-evaluation setup where inputs naturally echo the prompt.
- **Learned scorer plus optimal assignment.** A learned scorer first rates every node pair, and the Hungarian algorithm then extracts a globally optimal one-to-one alignment. Local scoring stays flexible while the final alignment stays consistent.
- **Isotonic regression as calibration.** A ranking model's raw outputs are monotonically mapped onto the target grade scale using a separate held-out fold. This lets a ranker be used wherever scale-specific error metrics are needed.
- **Evaluating with both correlation and error.** Reporting both Pearson's ρ and RMSE showed that a trivial baseline can look good on error simply by predicting near the mean of a skewed label distribution. This is a useful caution for any evaluation on imbalanced data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Institutions

- [University of North Texas](https://md.research.elamur.ai/free/en/universities/university-of-north-texas/) — University, Denton, United States · 13 works · 3,921 citations

Also on the byline: [Ohio University](https://ror.org/01jr3y717).

## Cite

Cite as: Michael Mohler, Razvan Bunescu and Rada Mihalcea. 2011. Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mohler2011learning,
  title = {{Learning to Grade Short Answer Questions using Semantic Similarity Measures and Dependency Graph Alignments}},
  author = {Michael Mohler, Razvan Bunescu and Rada Mihalcea},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1076/}
}
```

- [← previous Using Cross-Entity Inference to Improve Event Extraction](https://md.research.elamur.ai/free/en/papers/hong-2011-cross-entity-inference-improve-event/)
- [next → The Arabic Online Commentary Dataset: an Annotated Dataset of Informal Arabic with High Dialectal Content](https://md.research.elamur.ai/free/en/papers/zaidan-2011-arabic-online-commentary-dataset-annotated/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/)

## References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. N. Chambers, D. Cer, T. Grenager, D. Hall, C. Kiddon, B. MacCartney, M.C. de Marneffe, D. Ramage, E. Yeh, and C.D. Manning. 2007. Learning alignments and leveraging natural logic. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 165–170. Association for Computa… [find](https://scholar.google.com/scholar?q=N%2E+Chambers%2C+D%2E+Cer%2C+T%2E+Grenager%2C+D%2E+Hall%2C+C%2E+Kiddon%2C+B%2E+MacCartney%2C+M%2EC%2E+de+Marneffe%2C+D%2E+Ramage%2C+E%2E+Yeh%2C+and+C%2ED%2E+Manning%2E+2007%2E+Learning+alignments+and+leveraging+natural+logic%2E+In+Proceedings+of+th)
2. M. Collins. 2002. Discriminative training methods for hidden Markov models: Theory and experiments with perceptron algorithms. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP-02), Philadelphia, PA, July. [our card](https://md.research.elamur.ai/free/en/papers/collins-2002-discriminative-training-methods-hidden-markov/)
3. I. Dagan, O. Glickman, and B. Magnini. 2005. The PAS-CAL recognising textual entailment challenge. In Proceedings of the PASCAL Workshop. [find](https://scholar.google.com/scholar?q=I%2E+Dagan%2C+O%2E+Glickman%2C+and+B%2E+Magnini%2E+2005%2E+The+PAS%2DCAL+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+PASCAL+Workshop%2E)
4. M.C. de Marneffe, B. MacCartney, and C.D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC 2006. [our card](https://md.research.elamur.ai/free/en/papers/marneffe-2006-generating-typed-dependency-parses-phrase/)
5. M.C. de Marneffe, T. Grenager, B. MacCartney, D. Cer, D. Ramage, C. Kiddon, and C.D. Manning. 2007. Aligning semantic graphs for textual inference and machine reading. In Proceedings of the AAAI Spring Symposium. Citeseer. [find](https://scholar.google.com/scholar?q=M%2EC%2E+de+Marneffe%2C+T%2E+Grenager%2C+B%2E+MacCartney%2C+D%2E+Cer%2C+D%2E+Ramage%2C+C%2E+Kiddon%2C+and+C%2ED%2E+Manning%2E+2007%2E+Aligning+semantic+graphs+for+textual+inference+and+machine+reading%2E+In+Proceedings+of+the+AAAI+Sprin)
6. Y. Freund and R. Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37:277–296. [find](https://scholar.google.com/scholar?q=Y%2E+Freund+and+R%2E+Schapire%2E+1999%2E+Large+margin+classification+using+the+perceptron+algorithm%2E+Machine+Learning%2C+37%3A277%E2%80%93296%2E)
7. E. Gabrilovich and S. Markovitch. 2007. Computing Semantic Relatedness using Wikipedia-based Explicit Semantic Analysis. Proceedings of the 20th International Joint Conference on Artificial Intelligence, pages 6–12. [find](https://scholar.google.com/scholar?q=E%2E+Gabrilovich+and+S%2E+Markovitch%2E+2007%2E+Computing+Semantic+Relatedness+using+Wikipedia%2Dbased+Explicit+Semantic+Analysis%2E+Proceedings+of+the+20th+International+Joint+Conference+on+Artificial+Intelligen)
8. A.D. Haghighi, A.Y. Ng, and C.D. Manning. 2005. Robust textual inference via graph matching. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 387–394. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=A%2ED%2E+Haghighi%2C+A%2EY%2E+Ng%2C+and+C%2ED%2E+Manning%2E+2005%2E+Robust+textual+inference+via+graph+matching%2E+In+Proceedings+of+the+conference+on+Human+Language+Technology+and+Empirical+Methods+in+Natural+Language+Pro)
9. D. Higgins, J. Burstein, D. Marcu, and C. Gentile. 2004. Evaluating multiple aspects of coherence in student essays. In Proceedings of the annual meeting of the North American Chapter of the Association for Computational Linguistics, Boston, MA. [find](https://scholar.google.com/scholar?q=D%2E+Higgins%2C+J%2E+Burstein%2C+D%2E+Marcu%2C+and+C%2E+Gentile%2E+2004%2E+Evaluating+multiple+aspects+of+coherence+in+student+essays%2E+In+Proceedings+of+the+annual+meeting+of+the+North+American+Chapter+of+the+Associati)
10. G. Hirst and D. St-Onge, 1998. Lexical chains as representations of contexts for the detection and correction of malaproprisms. The MIT Press. [find](https://scholar.google.com/scholar?q=G%2E+Hirst+and+D%2E+St%2DOnge%2C+1998%2E+Lexical+chains+as+representations+of+contexts+for+the+detection+and+correction+of+malaproprisms%2E+The+MIT+Press%2E)
11. J. Jiang and D. Conrath. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the International Conference on Research in Computational Linguistics, Taiwan. [find](https://scholar.google.com/scholar?q=J%2E+Jiang+and+D%2E+Conrath%2E+1997%2E+Semantic+similarity+based+on+corpus+statistics+and+lexical+taxonomy%2E+In+Proceedings+of+the+International+Conference+on+Research+in+Computational+Linguistics%2C+Taiwan%2E)
12. T.K. Landauer and S.T. Dumais. 1997. A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104. [find](https://scholar.google.com/scholar?q=T%2EK%2E+Landauer+and+S%2ET%2E+Dumais%2E+1997%2E+A+solution+to+plato%E2%80%99s+problem%3A+The+latent+semantic+analysis+theory+of+acquisition%2C+induction%2C+and+representation+of+knowledge%2E+Psychological+Review%2C+104%2E)
13. C. Leacock and M. Chodorow. 1998. Combining local context and WordNet sense similarity for word sense identification. In WordNet, An Electronic Lexical Database. The MIT Press. [find](https://scholar.google.com/scholar?q=C%2E+Leacock+and+M%2E+Chodorow%2E+1998%2E+Combining+local+context+and+WordNet+sense+similarity+for+word+sense+identification%2E+In+WordNet%2C+An+Electronic+Lexical+Database%2E+The+MIT+Press%2E)
14. C. Leacock and M. Chodorow. 2003. C-rater: Automated Scoring of Short-Answer Questions. Computers and the Humanities, 37(4):389–405. [find](https://scholar.google.com/scholar?q=C%2E+Leacock+and+M%2E+Chodorow%2E+2003%2E+C%2Drater%3A+Automated+Scoring+of+Short%2DAnswer+Questions%2E+Computers+and+the+Humanities%2C+37%284%29%3A389%E2%80%93405%2E)
15. M.E. Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the SIGDOC Conference 1986, Toronto, June. [find](https://scholar.google.com/scholar?q=M%2EE%2E+Lesk%2E+1986%2E+Automatic+sense+disambiguation+using+machine+readable+dictionaries%3A+How+to+tell+a+pine+cone+from+an+ice+cream+cone%2E+In+Proceedings+of+the+SIGDOC+Conference+1986%2C+Toronto%2C+June%2E)
16. D. Lin. 1998. An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, Madison, WI. [find](https://scholar.google.com/scholar?q=D%2E+Lin%2E+1998%2E+An+information%2Dtheoretic+definition+of+similarity%2E+In+Proceedings+of+the+15th+International+Conference+on+Machine+Learning%2C+Madison%2C+WI%2E)
17. B. MacCartney, T. Grenager, M.C. de Marneffe, D. Cer, and C.D. Manning. 2006. Learning to recognize features of valid textual entailments. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, page… [find](https://scholar.google.com/scholar?q=B%2E+MacCartney%2C+T%2E+Grenager%2C+M%2EC%2E+de+Marneffe%2C+D%2E+Cer%2C+and+C%2ED%2E+Manning%2E+2006%2E+Learning+to+recognize+features+of+valid+textual+entailments%2E+In+Proceedings+of+the+main+conference+on+Human+Language+Techn)
18. K.I. Malatesta, P. Wiemer-Hastings, and J. Robertson. 2002. Beyond the Short Answer Question with Research Methods Tutor. In Proceedings of the Intelligent Tutoring Systems Conference. [find](https://scholar.google.com/scholar?q=K%2EI%2E+Malatesta%2C+P%2E+Wiemer%2DHastings%2C+and+J%2E+Robertson%2E+2002%2E+Beyond+the+Short+Answer+Question+with+Research+Methods+Tutor%2E+In+Proceedings+of+the+Intelligent+Tutoring+Systems+Conference%2E)
19. R. Mihalcea, C. Corley, and C. Strapparava. 2006. Corpus-based and knowledge-based approaches to text semantic similarity. In Proceedings of the American Association for Artificial Intelligence (AAAI 2006), Boston. [find](https://scholar.google.com/scholar?q=R%2E+Mihalcea%2C+C%2E+Corley%2C+and+C%2E+Strapparava%2E+2006%2E+Corpus%2Dbased+and+knowledge%2Dbased+approaches+to+text+semantic+similarity%2E+In+Proceedings+of+the+American+Association+for+Artificial+Intelligence+%28AAAI+)
20. T. Mitchell, T. Russell, P. Broomhead, and N. Aldridge. 2002. Towards robust computerised marking of free-text responses. Proceedings of the 6th International Computer Assisted Assessment (CAA) Conference. [find](https://scholar.google.com/scholar?q=T%2E+Mitchell%2C+T%2E+Russell%2C+P%2E+Broomhead%2C+and+N%2E+Aldridge%2E+2002%2E+Towards+robust+computerised+marking+of+free%2Dtext+responses%2E+Proceedings+of+the+6th+International+Computer+Assisted+Assessment+%28CAA%29+Confer)
21. M. Mohler and R. Mihalcea. 2009. Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the European Association for Computational Linguistics (EACL 2009), Athens, Greece. [find](https://scholar.google.com/scholar?q=M%2E+Mohler+and+R%2E+Mihalcea%2E+2009%2E+Text%2Dto%2Dtext+semantic+similarity+for+automatic+short+answer+grading%2E+In+Proceedings+of+the+European+Association+for+Computational+Linguistics+%28EACL+2009%29%2C+Athens%2C+Gree)
22. R.D. Nielsen, W. Ward, and J.H. Martin. 2009. Recognizing entailment in intelligent tutoring systems. Natural Language Engineering, 15(04):479–501. [find](https://scholar.google.com/scholar?q=R%2ED%2E+Nielsen%2C+W%2E+Ward%2C+and+J%2EH%2E+Martin%2E+2009%2E+Recognizing+entailment+in+intelligent+tutoring+systems%2E+Natural+Language+Engineering%2C+15%2804%29%3A479%E2%80%93501%2E)
23. T. Pedersen, S. Patwardhan, and J. Michelizzi. 2004. WordNet:: Similarity-Measuring the Relatedness of Concepts. Proceedings of the National Conference on Artificial Intelligence, pages 1024–1025. [find](https://scholar.google.com/scholar?q=T%2E+Pedersen%2C+S%2E+Patwardhan%2C+and+J%2E+Michelizzi%2E+2004%2E+WordNet%3A%3A+Similarity%2DMeasuring+the+Relatedness+of+Concepts%2E+Proceedings+of+the+National+Conference+on+Artificial+Intelligence%2C+pages+1024%E2%80%931025%2E)
24. S.G. Pulman and J.Z. Sukkarieh. 2005. Automatic Short Answer Marking. ACL WS Bldg Ed Apps using NLP. [find](https://scholar.google.com/scholar?q=S%2EG%2E+Pulman+and+J%2EZ%2E+Sukkarieh%2E+2005%2E+Automatic+Short+Answer+Marking%2E+ACL+WS+Bldg+Ed+Apps+using+NLP%2E)
25. R. Raina, A. Haghighi, C. Cox, J. Finkel, J. Michels, K. Toutanova, B. MacCartney, M.C. de Marneffe, C.D. Manning, and A.Y. Ng. 2005. Robust textual inference using diverse knowledge sources. Recognizing Textual Entailment, page 57. [find](https://scholar.google.com/scholar?q=R%2E+Raina%2C+A%2E+Haghighi%2C+C%2E+Cox%2C+J%2E+Finkel%2C+J%2E+Michels%2C+K%2E+Toutanova%2C+B%2E+MacCartney%2C+M%2EC%2E+de+Marneffe%2C+C%2ED%2E+Manning%2C+and+A%2EY%2E+Ng%2E+2005%2E+Robust+textual+inference+using+diverse+knowledge+sources%2E+Recogniz)
26. P. Resnik. 1995. Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada. [find](https://scholar.google.com/scholar?q=P%2E+Resnik%2E+1995%2E+Using+information+content+to+evaluate+semantic+similarity%2E+In+Proceedings+of+the+14th+International+Joint+Conference+on+Artificial+Intelligence%2C+Montreal%2C+Canada%2E)
27. V. Rus, A. Graesser, and K. Desai. 2007. Lexico-syntactic subsumption for textual entailment. Recent Advances in Natural Language Processing IV: Selected Papers from RANLP 2005, page 187. [find](https://scholar.google.com/scholar?q=V%2E+Rus%2C+A%2E+Graesser%2C+and+K%2E+Desai%2E+2007%2E+Lexico%2Dsyntactic+subsumption+for+textual+entailment%2E+Recent+Advances+in+Natural+Language+Processing+IV%3A+Selected+Papers+from+RANLP+2005%2C+page+187%2E)
28. J.Z. Sukkarieh, S.G. Pulman, and N. Raikes. 2004. Auto-Marking 2: An Update on the UCLES-Oxford University research into using Computational Linguistics to Score Short, Free Text Responses. International Association of Educational Assessment, Philadephia. [find](https://scholar.google.com/scholar?q=J%2EZ%2E+Sukkarieh%2C+S%2EG%2E+Pulman%2C+and+N%2E+Raikes%2E+2004%2E+Auto%2DMarking+2%3A+An+Update+on+the+UCLES%2DOxford+University+research+into+using+Computational+Linguistics+to+Score+Short%2C+Free+Text+Responses%2E+Internatio)
29. P. Wiemer-Hastings, K. Wiemer-Hastings, and A. Graesser. 1999. Improving an intelligent tutor’s comprehension of students with Latent Semantic Analysis. Artificial Intelligence in Education, pages 535–542. [find](https://scholar.google.com/scholar?q=P%2E+Wiemer%2DHastings%2C+K%2E+Wiemer%2DHastings%2C+and+A%2E+Graesser%2E+1999%2E+Improving+an+intelligent+tutor%E2%80%99s+comprehension+of+students+with+Latent+Semantic+Analysis%2E+Artificial+Intelligence+in+Education%2C+pages+535)
30. P. Wiemer-Hastings, E. Arnott, and D. Allbritton. 2005. Initial results and mixed directions for research methods tutor. In AIED2005 - Supplementary Proceedings of the 12th International Conference on Artificial Intelligence in Education, Amsterdam. [find](https://scholar.google.com/scholar?q=P%2E+Wiemer%2DHastings%2C+E%2E+Arnott%2C+and+D%2E+Allbritton%2E+2005%2E+Initial+results+and+mixed+directions+for+research+methods+tutor%2E+In+AIED2005+%2D+Supplementary+Proceedings+of+the+12th+International+Conference+on)
31. Z. Wu and M. Palmer. 1994. Verb semantics and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics, Las Cruces, New Mexico. [our card](https://md.research.elamur.ai/free/en/papers/wu-1994-verb-semantics-lexical-selection/)
32. B. Zadrozny and C. Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. Edmonton, Alberta. [find](https://scholar.google.com/scholar?q=B%2E+Zadrozny+and+C%2E+Elkan%2E+2002%2E+Transforming+classifier+scores+into+accurate+multiclass+probability+estimates%2E+Edmonton%2C+Alberta%2E)