---
title: "Application of an Automatic Plagiarism Detection System in a"
description: "Test takers in high-stakes English speaking exams sometimes recite memorized material from online… 27 references, cited by 1 paper here."
authors: ["Xinhao Wang", "Keelan Evanini", "Matthew Mulholland", "Yao Qian", "James V. Bruno"]
year: 2019
venue: "Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications"
source: "https://aclanthology.org/W19-4445/"
doi: "10.18653/v1/w19-4445"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2019-application-automatic-plagiarism-detection-system/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2019-application-automatic-plagiarism-detection-system/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/) / Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

# Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency

**Authors:** [Xinhao Wang](https://md.research.elamur.ai/free/en/authors/xinhao-wang/), [Keelan Evanini](https://md.research.elamur.ai/free/en/authors/keelan-evanini/), Matthew Mulholland, [Yao Qian](https://md.research.elamur.ai/free/en/authors/yao-qian/) and James V. Bruno · **Year:** 2019 · **Venue:** [Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications](https://md.research.elamur.ai/free/en/venues/bea/) ([BEA 2019](https://md.research.elamur.ai/free/en/venues/bea/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W19-4445/) · [PDF](https://aclanthology.org/W19-4445.pdf) · [DOI 10.18653/v1/w19-4445](https://doi.org/10.18653/v1/w19-4445) · [OpenAlex](https://openalex.org/W2970238989)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/wang-2019-application-automatic-plagiarism-detection-system/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Test takers in high-stakes English speaking exams sometimes recite memorized material from online sources instead of speaking spontaneously. This distorts scores from both human raters and automated scorers. The authors, from ETS, built a classifier that separates such canned spoken responses from genuine ones. Responses are first transcribed with a Kaldi-based ASR system whose language model is adapted to new prompts without supervision. Two feature families are then compared. The first measures text similarity between a transcript and a pool of known source passages, using n-gram overlap and Word Mover's Distance. The second contrasts proficiency measures, such as fluency, across the responses of a single speaker, and needs no source passages at all. The data set is deliberately skewed, with about 43 control responses for each plagiarized one. On it, n-gram overlap features performed best, with an F1 of 0.761. The authors also ran the system on a full live administration, where it caught nearly 90% of confirmed cases. Because experts then only had to review a small flagged subset, the system greatly reduces manual effort.

## Key points
- **Data:**
  - 1,557 confirmed plagiarized responses to independent speaking tasks, gathered over several years.
  - 224 source passages identified by human raters.
  - A control set of 66,400 responses from 33,200 speakers: 166 test forms with 200 speakers each, giving a 43:1 imbalance.
- **Feature comparison:**
  - n-gram overlap features alone reached F1 = 0.761.
  - WMD features alone reached only 0.649.
  - Combining the two gave 0.76, so adding WMD brought no gain.
- **Proficiency features:** the source-free speaking-proficiency features were very precise (0.8) but recalled almost nothing (0.009). Canned delivery proved hard to tell apart from spontaneous speech.
- **Operational check:**
  - The system was run on one administration of 13,516 responses from 6,758 speakers, of which only 39 (0.29%) were confirmed plagiarized.
  - A union ensemble of random forest and AdaBoost models flagged 850 responses and found 35 of the 39 cases (recall 0.897).
- **Earlier baseline:** the authors' previous system reached F1 = 70.6% on simulated data. This work adds the improved ASR, the refined similarity features and the live validation.

## Interesting ideas
- Exact shared word sequences beat semantic-similarity measures here, because copied text keeps idiosyncratic phrasings that paraphrase-tolerant metrics tend to blur.
- The authors evaluated with a very large pool of negative controls at a realistic base rate, then confirmed the result on a complete live batch rather than trusting balanced test sets.
- For triage, a response was flagged if any model in the ensemble flagged it, which deliberately maximizes recall because a human reviews everything flagged anyway.
- Each flagged response came with a label naming the source it probably drew on, which made expert review faster and the decision explainable.
- The source-free features compared a speaker against their own other responses instead of against external references; this weak but precise signal could still complement content features.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): when grading student answers on fables, an n-gram overlap check against the source text could separate copied sentences from answers that were actually inferred from the world model.
- **I2** (learn): negative-control evaluations should be run at realistic error base rates, so that precision claims hold up when the rare target class is drowned in normal cases.
- **I3** (learn): disagreement or any-witness flags from the several parsers could route sentences to the LLM teacher tier, trading cheap false positives for fewer silent errors in silver data.
- **I4** (answer): every answer could carry a pointer to the text span or world fact it relied on, so reviewers can verify it quickly.
- **I5** (frames): register or style shifts within one author's text, measured against that same text's baseline, could serve as a high-precision cue for quoted or borrowed material when attributing speech acts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Educational Testing Service](https://md.research.elamur.ai/free/en/ngos/educational-testing-service/) — Nonprofit, Princeton, United States · 27 works · 2,212 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian and James V. Bruno. 2019. Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2019application,
  title = {{Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency}},
  author = {Xinhao Wang, Keelan Evanini, Matthew Mulholland, Yao Qian and James V. Bruno},
  booktitle = {Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2019},
  doi = {10.18653/v1/w19-4445},
  url = {https://doi.org/10.18653/v1/w19-4445}
}
```

- [← previous Subjective Sentiment Analysis for Arabic Newswire Comments](https://md.research.elamur.ai/free/en/papers/bessou-2019-subjective-sentiment-analysis-arabic-newswire/)
- [next → Classifying Arabic dialect text in the Social Media Arabic Dialect Corpus (SMADC)](https://md.research.elamur.ai/free/en/papers/alshutayri-2019-classifying-arabic-dialect-text-social/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/7/)

## References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. This study proposed a system which can benefit a high-stakes assessment of English speaking proficiency by automatically detecting potentially plagiarized spoken responses, and investigated the empirical effectiveness of two different types of features. One is based on automatic plagiarism S. M. Alzahrani, N. Salim, and A. Abraham. 2012. Understanding plagiarism linguistic patterns, textual features, and detection methods. IEEE Transaction on Systems, Man, and Cybernetics-Part C: Applications and Reviews, 42(2):133–149. [find](https://scholar.google.com/scholar?q=This+study+proposed+a+system+which+can+benefit+a+high%2Dstakes+assessment+of+English+speaking+proficiency+by+automatically+detecting+potentially+plagiarized+spoken+responses%2C+and+investigated+the+empiri)
2. S. Brin, J. Davis, and H. Garcia-Molina. 1995. [find](https://scholar.google.com/scholar?q=S%2E+Brin%2C+J%2E+Davis%2C+and+H%2E+Garcia%2DMolina%2E+1995%2E)
3. Copy Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. detection mechanisms for digital documents. In Proceedings of the ACM SIGMOD Annual Conference, pages 398–409. [↗](https://doi.org/10.48550/arxiv.1201.0490)
4. C. Chen, J. Yeh, and H. Ke. 2010. Plagiarism detection using ROUGE and WordNet. Journal of Computing, 2(3):34–44. [↗](https://doi.org/10.48550/arxiv.1003.4065)
5. M. Potthast, M. Hagen, T. Gollub, M. Tippmann, J. Kiesel, P. Rosso, E. Stamatatos, and B. Stein. 2013. Overview of the 5th International Competition on Plagiarism Detection. In CLEF 2013 Evaluation Labs and Workshop – Working Notes Papers. [find](https://scholar.google.com/scholar?q=M%2E+Potthast%2C+M%2E+Hagen%2C+T%2E+Gollub%2C+M%2E+Tippmann%2C+J%2E+Kiesel%2C+P%2E+Rosso%2C+E%2E+Stamatatos%2C+and+B%2E+Stein%2E+2013%2E+Overview+of+the+5th+International+Competition+on+Plagiarism+Detection%2E+In+CLEF+2013+Evaluation+La)
6. P. Cullen, A. French, and V. Jakeman. 2014. The Official Cambridge Guide to IELTS. Cambridge University Press. [find](https://scholar.google.com/scholar?q=P%2E+Cullen%2C+A%2E+French%2C+and+V%2E+Jakeman%2E+2014%2E+The+Official+Cambridge+Guide+to+IELTS%2E+Cambridge+University+Press%2E)
7. ETS. 2012. The Official Guide to the TOEFL R Test, Fourth Edition. McGraw-Hill. [find](https://scholar.google.com/scholar?q=ETS%2E+2012%2E+The+Official+Guide+to+the+TOEFL+R+Test%2C+Fourth+Edition%2E+McGraw%2DHill%2E)
8. M. Potthast, B. Stein, A. Barrón-Cedeño, and P. Rosso. 2010. An evaluation framework for plagiarism detection. In Proceedings of the 23rd International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=M%2E+Potthast%2C+B%2E+Stein%2C+A%2E+Barro%CC%81n%2DCeden%CC%83o%2C+and+P%2E+Rosso%2E+2010%2E+An+evaluation+framework+for+plagiarism+detection%2E+In+Proceedings+of+the+23rd+International+Conference+on+Computational+Linguistics%2E)
9. Keelan Evanini and Xinhao Wang. 2014. Automatic detection of plagiarized spoken responses. In Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications, pages 22–27. [find](https://scholar.google.com/scholar?q=Keelan+Evanini+and+Xinhao+Wang%2E+2014%2E+Automatic+detection+of+plagiarized+spoken+responses%2E+In+Proceedings+of+the+Ninth+Workshop+on+Innovative+Use+of+NLP+for+Building+Educational+Applications%2C+pages+22)
10. Y. Rubner, C. Tomasi, and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In Proceedings of the Sixth International Conference on Computer Vision. [find](https://scholar.google.com/scholar?q=Y%2E+Rubner%2C+C%2E+Tomasi%2C+and+L%2E+J%2E+Guibas%2E+1998%2E+A+metric+for+distributions+with+applications+to+image+databases%2E+In+Proceedings+of+the+Sixth+International+Conference+on+Computer+Vision%2E)
11. A. Finch, Y. Hwang, and E. Sumita. 2005. Using machine translation evaluation techniques to determine sentence-level semantic equivalence. In Proceedings of the Third International Workshop on Paraphrasing, pages 17–24. [find](https://scholar.google.com/scholar?q=A%2E+Finch%2C+Y%2E+Hwang%2C+and+E%2E+Sumita%2E+2005%2E+Using+machine+translation+evaluation+techniques+to+determine+sentence%2Dlevel+semantic+equivalence%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphr)
12. N. Shivakumar and H. Garcia-Molina. 1995. SCAM: A copy detection mechanism for digital documents. In Proceedings of the Second Annual Conference on the Theory and Practice of Digital Libraries. [find](https://scholar.google.com/scholar?q=N%2E+Shivakumar+and+H%2E+Garcia%2DMolina%2E+1995%2E+SCAM%3A+A+copy+detection+mechanism+for+digital+documents%2E+In+Proceedings+of+the+Second+Annual+Conference+on+the+Theory+and+Practice+of+Digital+Libraries%2E)
13. A. Hauptmann. 2006. Automatic spoken document retrieval. In Ketih Brown, editor, Encylclopedia of Language and Linguistics (Second Edition), pages 95–103. Elsevier Science. [find](https://scholar.google.com/scholar?q=A%2E+Hauptmann%2E+2006%2E+Automatic+spoken+document+retrieval%2E+In+Ketih+Brown%2C+editor%2C+Encylclopedia+of+Language+and+Linguistics+%28Second+Edition%29%2C+pages+95%E2%80%93103%2E+Elsevier+Science%2E)
14. E. Stamatatos. 2011. Plagiarism detection using stopword n-grams. American Society for Information Science and Technology, 62(12):2512–2527. [↗](https://doi.org/10.1002/asi.21630)
15. T. C. Hoad and J. Zobel. 2003. Methods for identifying versioned and plagiarised documents. Journal of the American Society for Information Science and Technology, 54:203–215. [find](https://scholar.google.com/scholar?q=T%2E+C%2E+Hoad+and+J%2E+Zobel%2E+2003%2E+Methods+for+identifying+versioned+and+plagiarised+documents%2E+Journal+of+the+American+Society+for+Information+Science+and+Technology%2C+54%3A203%E2%80%93215%2E)
16. B. Stein, N. Lipka, and P. Prettenhofer. 2011. Intrinsic plagiarism analysis. Language Resources and Evaluation, 45(1):63–82. [↗](https://doi.org/10.1007/s10579-010-9115-y)
17. Ö. Uzuner, B. Katz, and T. Nahnsen. 2005. Using syntactic information to identify plagiarism. In Proceedings of the 2nd Workshop on Building Educational Applications using NLP. Ann Arbor. [find](https://scholar.google.com/scholar?q=O%CC%88%2E+Uzuner%2C+B%2E+Katz%2C+and+T%2E+Nahnsen%2E+2005%2E+Using+syntactic+information+to+identify+plagiarism%2E+In+Proceedings+of+the+2nd+Workshop+on+Building+Educational+Applications+using+NLP%2E+Ann+Arbor%2E)
18. Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957–966, Lille, France. [find](https://scholar.google.com/scholar?q=Matt+Kusner%2C+Yu+Sun%2C+Nicholas+Kolkin%2C+and+Kilian+Weinberger%2E+2015%2E+From+word+embeddings+to+document+distances%2E+In+Proceedings+of+the+32nd+International+Conference+on+Machine+Learning%2C+volume+37+of+Pro)
19. X. Wang, K. Evanini, J. Bruno, and M. Mulholland. 2016. Automatic plagiarism detection for spoken responses in an assessment of english language proficiency. In Proceedings of the IEEE Spoken Language Technology Workshop, pages 121–128. [find](https://scholar.google.com/scholar?q=X%2E+Wang%2C+K%2E+Evanini%2C+J%2E+Bruno%2C+and+M%2E+Mulholland%2E+2016%2E+Automatic+plagiarism+detection+for+spoken+responses+in+an+assessment+of+english+language+proficiency%2E+In+Proceedings+of+the+IEEE+Spoken+Language)
20. P. Longman. 2010. The Official Guide to Pearson Test of English Academic. Pearson Education ESL. [find](https://scholar.google.com/scholar?q=P%2E+Longman%2E+2010%2E+The+Official+Guide+to+Pearson+Test+of+English+Academic%2E+Pearson+Education+ESL%2E)
21. C. Lyon, R. Barrett, and J. Malcolm. 2006. Plagiarism is easy, but also easy to detect. Plagiary, 1:57–65. [find](https://scholar.google.com/scholar?q=C%2E+Lyon%2C+R%2E+Barrett%2C+and+J%2E+Malcolm%2E+2006%2E+Plagiarism+is+easy%2C+but+also+easy+to+detect%2E+Plagiary%2C+1%3A57%E2%80%9365%2E)
22. K. Zechner, D. Higgins, and X. Xi. 2007. Speechrater SM : A construct-driven approach to scoring spontaneous non-native speech. In Proceedings of the International Speech Communication Association Special Interest Group on Speech and Language Technology in Education, pages 128–131. [find](https://scholar.google.com/scholar?q=K%2E+Zechner%2C+D%2E+Higgins%2C+and+X%2E+Xi%2E+2007%2E+Speechrater+SM+%3A+A+construct%2Ddriven+approach+to+scoring+spontaneous+non%2Dnative+speech%2E+In+Proceedings+of+the+International+Speech+Communication+Association+Spe)
23. N. Madnani, J. Tetreault, and M. Chodorow. 2012. [Re-examining machine translation metrics for paraphrase identification](https://md.research.elamur.ai/free/en/papers/madnani-2012-re-examining-machine-translation-metrics/). In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 182–190, Montréal, Canada. Association for Computational Linguistics.
24. K. Zechner, D. Higgins, X. Xi, and D. M. Williamson. 2009. Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10):883–895. [↗](https://doi.org/10.1016/j.specom.2009.04.009)
25. M. Mozgovoy, T. Kakkonen, and E. Sutinen. 2007. Using natural language parsers in plagiarism detection. In Proceedings of the ISCA Workshop on Speech and Language Technology in Education (SLaTE). [find](https://scholar.google.com/scholar?q=M%2E+Mozgovoy%2C+T%2E+Kakkonen%2C+and+E%2E+Sutinen%2E+2007%2E+Using+natural+language+parsers+in+plagiarism+detection%2E+In+Proceedings+of+the+ISCA+Workshop+on+Speech+and+Language+Technology+in+Education+%28SLaTE%29%2E)
26. K. Zechner and X. Wang. 2013. [Automated content scoring of spoken responses in an assessment for teachers of english](https://md.research.elamur.ai/free/en/papers/zechner-2013-automated-content-scoring-spoken-responses/). In Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, page 73–81.
27. T. Nahnsen, Ö. Uzuner, and B. Katz. 2005. Lexical chains and sliding locality windows in content-based text similarity detection. CSAIL Technical Report, MIT-CSAIL-TR-2005-034. [find](https://scholar.google.com/scholar?q=T%2E+Nahnsen%2C+O%CC%88%2E+Uzuner%2C+and+B%2E+Katz%2E+2005%2E+Lexical+chains+and+sliding+locality+windows+in+content%2Dbased+text+similarity+detection%2E+CSAIL+Technical+Report%2C+MIT%2DCSAIL%2DTR%2D2005%2D034%2E)

*Updated 1 November 2026*