---
title: "Automated Essay Scoring Based on Finite State Transducer:"
description: "The paper addresses automatic content scoring of spoken ESL picture-description answers, where the text to be graded… 25 references."
authors: ["Xingyuan Peng", "Dengfeng Ke", "Bo Xu"]
year: 2012
venue: "Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/P12-1006/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peng-2012-automated-essay-scoring-based-finite/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peng-2012-automated-essay-scoring-based-finite/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/) / Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech

# Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech

**Authors:** Xingyuan Peng, Dengfeng Ke and [Bo Xu](https://md.research.elamur.ai/free/en/authors/bo-xu/) · **Year:** 2012 · **Venue:** [Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2012](https://md.research.elamur.ai/free/en/venues/acl/2012/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P12-1006/) · [PDF](https://aclanthology.org/P12-1006.pdf) · [OpenAlex](https://openalex.org/W2127622484)

**Citations:** 11 citations (OpenAlex, 2026-10-14)

## Summary
The paper addresses automatic content scoring of spoken ESL picture-description answers, where the text to be graded comes from an ASR system rather than from the learner's own writing. Standard AES techniques fit this setting poorly. Spelling, punctuation and grammar features become meaningless on noisy transcripts. Bag-of-words content models are also misled, because a strongly constrained language model tends to turn mumbled or off-topic speech into on-topic keywords. The authors instead model each exam topic with a Finite State Transducer. The FST maps word sequences to the picture they describe, so it captures both the order of the four pictures and the link between surface phrasing and meaning. Several FST-based features are combined by linear regression into a score. On 417 graded speeches, the FST approach clearly beats an LSA-SVR baseline, especially on ASR output (about 0.80 vs 0.58 correlation). Adding WordNet synonym similarity to substitution costs helps on manual transcripts. The result shows that sequence-aware matching holds up under recognition noise, while order-free content measures break down.

## Key points
- **Data:** 417 valid one-minute oral picture compositions from three ESL classes. Each was scored 0–20 (7 levels) by three raters, with an average inter-rater correlation of about 0.87.
- **Two transcript conditions:**
  - Manual transcripts (CRR).
  - HTK-based ASR output, with an overall WER of 58.86% and a match rate of 72.88%. WER drops sharply for higher-scoring speakers, while the match rate stays fairly stable.
- **Features and scoring:** 12 FST-derived features (ED, NED, MN, MR, CMV, SSF), each computed from two different FST building sets, are combined via linear regression trained with cross-validation.
- **Average correlations with expert scores:**

  | Method | CRR | ASR |
  |---|---|---|
  | FST | 0.898 | 0.804 |
  | LSA-SVR | 0.805 | 0.584 |
  | Length feature | 0.796 | 0.187 |

- **Improved FST:** Synonym similarity from WordNet::Similarity, corrected by hand, raises the CRR score to 0.906. The ASR score barely changes (0.8045).

## Interesting ideas
- Scoring content by aligning against an ordered reference structure, rather than counting keywords, penalises topic words strung together without sense. This matters whenever an upstream model may hallucinate plausible vocabulary.
- The FST layers surface wording under sense units (one per picture), separating how something is said from which content slot it fills.
- Synonym similarity enters as a softer substitution cost in the edit-distance-style match, not as a hard equivalence.
- The same system is evaluated on clean and noisy input side by side, with a trivial length feature as a control that exposes which methods collapse under noise.
- The authors note that strong language-model constraints make errors look topical, a reminder that noise from a confident upstream component is systematic rather than random.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Graders of student answers to tale or school questions could align the answer's event sequence with the expected one, instead of rewarding keyword overlap.
- **I2** (world): Event-order checks in the world of a text could match recovered events to expected story slots, tolerating varied phrasing while still enforcing temporal order.
- **I3** (frames): When matching frames, VerbNet-class or WordNet-close predicates could get graded substitution costs, keeping the match explainable as an edit path.
- **I4** (learn): Evaluations could routinely include a degraded-input condition and a shallow baseline as a negative control, to show that gains do not come from surface artefacts.
- **I5** (tree): Parser-output consumers should expect biased errors (e.g. attachments drifting toward frequent patterns) and treat agreement with priors as weaker evidence.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [Institute of Automation](https://md.research.elamur.ai/free/en/labs/institute-of-automation/) — Research institute, Beijing, China · 70 works · 14,251 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (3): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works · [🎙️ Speech & spoken language](https://md.research.elamur.ai/free/en/topics/speech/) · 480 works · [🖍️ Grammatical error correction & learner language](https://md.research.elamur.ai/free/en/topics/grammatical-error-correction/) · 156 works

## Cite

Cite as: Xingyuan Peng, Dengfeng Ke and Bo Xu. 2012. Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peng2012automated,
  title = {{Automated Essay Scoring Based on Finite State Transducer: towards ASR Transcription of Oral English Speech}},
  author = {Xingyuan Peng, Dengfeng Ke and Bo Xu},
  booktitle = {Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2012},
  url = {https://aclanthology.org/P12-1006/}
}
```

- [← previous Domain Adaptation of Coreference Resolution for Radiology Reports](https://md.research.elamur.ai/free/en/papers/apostolova-2012-domain-adaptation-coreference-resolution-radiology/)
- [next → Joint English Spelling Error Correction and POS Tagging for Language Learners Writing](https://md.research.elamur.ai/free/en/papers/sakaguchi-2012-joint-english-spelling-error-correction/)
- [↑ back to papers of 2012](https://md.research.elamur.ai/free/en/papers/2012/page/2/)

## References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut and Mehryar Mohri. 2007. OpenFst: a general and efficient weighted finite-state transducer library. In Proceedings of International Conference on Implementation and Application of Automata, 4783: 11-23. [find](https://scholar.google.com/scholar?q=Cyril+Allauzen%2C+Michael+Riley%2C+Johan+Schalkwyk%2C+Wojciech+Skut+and+Mehryar+Mohri%2E+2007%2E+OpenFst%3A+a+general+and+efficient+weighted+finite%2Dstate+transducer+library%2E+In+Proceedings+of+International+Confer)
2. Yigal Attali. 2011. A differential word use measure for content analysis in automated essay scoring. ETS research report, ETS RR-11-36. [↗](https://doi.org/10.1002/j.2333-8504.2011.tb02272.x)
3. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater R V.2. The Journal of Technology, Learning, and Assessment, 4(3), 1-34. [find](https://scholar.google.com/scholar?q=Yigal+Attali+and+Jill+Burstein%2E+2006%2E+Automated+essay+scoring+with+e%2Drater+R+V%2E2%2E+The+Journal+of+Technology%2C+Learning%2C+and+Assessment%2C+4%283%29%2C+1%2D34%2E)
4. Jill Burstein, Joel Tetreault, and Slava Andreyev. 2010. Using entity-based features to model coherence in student essays. In Human Language Technologies: The Annual Conference of the North American Chapter of the ACL, 681-684. [find](https://scholar.google.com/scholar?q=Jill+Burstein%2C+Joel+Tetreault%2C+and+Slava+Andreyev%2E+2010%2E+Using+entity%2Dbased+features+to+model+coherence+in+student+essays%2E+In+Human+Language+Technologies%3A+The+Annual+Conference+of+the+North+American+C)
5. Chih-Chung Chang, Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, Vol. 2. [find](https://scholar.google.com/scholar?q=Chih%2DChung+Chang%2C+Chih%2DJen+Lin%2E+2011%2E+LIBSVM%3A+a+library+for+support+vector+machines%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology%2C+Vol%2E+2%2E)
6. Yen-Yu Chen, Chien-Liang Liu, Chia-Hoang Lee, and Tao-Hsing Chang. 2010. An unsupervised automated essay scoring system. IEEE Intelligent Systems, 61-67. [↗](https://doi.org/10.1109/mis.2010.3)
7. Catia Cucchiarini, Helmer Strik, and Lou Boves. 2000. Quantitative assessment of second language learners’ fluency by means of automatic speech recognition technology. Acoustical Society of America, 107(2): 989-999. [find](https://scholar.google.com/scholar?q=Catia+Cucchiarini%2C+Helmer+Strik%2C+and+Lou+Boves%2E+2000%2E+Quantitative+assessment+of+second+language+learners%E2%80%99+fluency+by+means+of+automatic+speech+recognition+technology%2E+Acoustical+Society+of+America%2C+1)
8. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science. 41(6): 391-407. [find](https://scholar.google.com/scholar?q=Scott+Deerwester%2C+Susan+T%2E+Dumais%2C+George+W%2E+Furnas%2C+Thomas+K%2E+Landauer+and+Richard+Harshman%2E+1990%2E+Indexing+by+latent+semantic+analysis%2E+Journal+of+the+American+Society+for+Information+Science%2E+41%286%29)
9. Shen Huang, Hongyan Li, Shijin Wang, Jiaen Liang and Bo Xu. 2010. Automatic reference independent evaluation of prosody quality using multiple knowledge fusions. In INTERSPEECH, 610-613. [find](https://scholar.google.com/scholar?q=Shen+Huang%2C+Hongyan+Li%2C+Shijin+Wang%2C+Jiaen+Liang+and+Bo+Xu%2E+2010%2E+Automatic+reference+independent+evaluation+of+prosody+quality+using+multiple+knowledge+fusions%2E+In+INTERSPEECH%2C+610%2D613%2E)
10. Tsunenori Ishioka and Masayuki Kameda. 2004. Automated Japanese essay scoring system: jess. In Proceedings of the International Workshop on database and Expert Systems applications. [↗](https://doi.org/10.1109/dexa.2004.42)
11. Tuomo Kakkonen, Niko Myller, Jari Timonen, and Erkki Sutinen. 2005. Automatic essay grading with probabilistic latent semantic analysis. In Proceedings of Workshop on Building Educational Applications Using NLP, 29-36. [find](https://scholar.google.com/scholar?q=Tuomo+Kakkonen%2C+Niko+Myller%2C+Jari+Timonen%2C+and+Erkki+Sutinen%2E+2005%2E+Automatic+essay+grading+with+probabilistic+latent+semantic+analysis%2E+In+Proceedings+of+Workshop+on+Building+Educational+Applications)
12. Thomas K. Landauer, Darrell Laham and Peter Foltz. 2003. Automatic essay assessment. Assessment in Education: Principles, Policy and Practice (10:3), 295-309. [↗](https://doi.org/10.1080/0969594032000148154)
13. Deryle Lonsdale and Diane Strong-Krause. 2003. Automated rating of ESL essays. In Proceedings of the HLT-NAACL Workshop: Building Educational Applications Using NLP. [find](https://scholar.google.com/scholar?q=Deryle+Lonsdale+and+Diane+Strong%2DKrause%2E+2003%2E+Automated+rating+of+ESL+essays%2E+In+Proceedings+of+the+HLT%2DNAACL+Workshop%3A+Building+Educational+Applications+Using+NLP%2E)
14. Andreas Maier, F. Hönig, V. Zeissler, Anton Batliner, E. Körner, N. Yamanaka, P. Ackermann, Elmar Nöth 2009. A language-independent feature set for the automatic evaluation of prosody. In INTERSPEECH, 600-603. [find](https://scholar.google.com/scholar?q=Andreas+Maier%2C+F%2E+Ho%CC%88nig%2C+V%2E+Zeissler%2C+Anton+Batliner%2C+E%2E+Ko%CC%88rner%2C+N%2E+Yamanaka%2C+P%2E+Ackermann%2C+Elmar+No%CC%88th+2009%2E+A+language%2Dindependent+feature+set+for+the+automatic+evaluation+of+prosody%2E+In+INTERSPEE)
15. Mehryar Mohri. 2004. Weighted finite-state transducer algorithms: an overview. Formal Languages and Applications, 148 (620): 551-564. [↗](https://doi.org/10.1007/978-3-540-39886-8_29)
16. Leonardo Neumeyer, Horacio Franco, Vassilios Digalakis, Mitchel Weintraub. 2000. Automatic scoring of pronunciation quality. Speech Communication, 30(2-3): 83-94. [↗](https://doi.org/10.1016/s0167-6393(99)00046-1)
17. Ted Pedersen, Siddharth Patwardhan and Jason Michelizzi. 2004. WordNet::Similarity - measuring the relatedness of concepts. In Proceedings of the National Conference on Artificial Intelligence, 144-152. [find](https://scholar.google.com/scholar?q=Ted+Pedersen%2C+Siddharth+Patwardhan+and+Jason+Michelizzi%2E+2004%2E+WordNet%3A%3ASimilarity+%2D+measuring+the+relatedness+of+concepts%2E+In+Proceedings+of+the+National+Conference+on+Artificial+Intelligence%2C+144%2D15)
18. Xingyuan Peng, Dengfeng Ke, Zhenbiao Chen and Bo Xu. 2010. Automated Chinese essay scoring using vector space models. In Proceedings of IUCS, 149-153. [find](https://scholar.google.com/scholar?q=Xingyuan+Peng%2C+Dengfeng+Ke%2C+Zhenbiao+Chen+and+Bo+Xu%2E+2010%2E+Automated+Chinese+essay+scoring+using+vector+space+models%2E+In+Proceedings+of+IUCS%2C+149%2D153%2E)
19. Isaac Persing, Alan Davis and Vincent Ng. 2010. [Modeling organization in student essays](https://md.research.elamur.ai/free/en/papers/persing-2010-modeling-organization-student-essays/). In Proceedings of EMNLP, 229-239.
20. Lawrence M. Rudner and Tahung Liang. 2002. Automated essay scoring using Bayes’ theorem. The Journal of Technology, Learning and Assessment, 1(2):3-21. [find](https://scholar.google.com/scholar?q=Lawrence+M%2E+Rudner+and+Tahung+Liang%2E+2002%2E+Automated+essay+scoring+using+Bayes%E2%80%99+theorem%2E+The+Journal+of+Technology%2C+Learning+and+Assessment%2C+1%282%29%3A3%2D21%2E)
21. G. Salton, C. Yang, A. Wong. 1975. A vector space model for automatic indexing. Communications of the ACM, 18(11): 613-620. [↗](https://doi.org/10.1145/361219.361220)
22. Alex J. Smola and Bernhard Schölkopf. 2004. A tutorial on support vector regression. Statistics and Computing 14(3): 199-222. [↗](https://doi.org/10.1023/b:stco.0000035301.49549.88)
23. Robert A. Wagner and Michael J. Fischer. 1974. The string-to-string correction problem. Journal of the ACM, 21(1):168-173. [↗](https://doi.org/10.1145/321796.321811)
24. Helen Yannakoudakis, Ted Briscoe and Ben Medlock. 2011. [A new dataset and method for automatically grading ESOL texts](https://md.research.elamur.ai/free/en/papers/yannakoudakis-2011-new-dataset-method-automatically-grading/). In Proceedings of ACL, 180-189.
25. Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, Valtcho Valtchev, Phil Woodland. 2006. The HTK book (for HTK version 3.4). Cambridge University Engineering Department. [find](https://scholar.google.com/scholar?q=Steve+Young%2C+Gunnar+Evermann%2C+Mark+Gales%2C+Thomas+Hain%2C+Dan+Kershaw%2C+Xunying+Liu%2C+Gareth+Moore%2C+Julian+Odell%2C+Dave+Ollason%2C+Dan+Povey%2C+Valtcho+Valtchev%2C+Phil+Woodland%2E+2006%2E+The+HTK+book+%28for+HTK+versi)

*Updated 1 November 2026*