---
title: "Responsible AI for Test Equity and Quality: The Duolingo English"
description: "Using the Duolingo English Test (DET), a high-stakes, computer-adaptive English proficiency exam, as a case study, the… 77 references."
authors: ["Jill Burstein", "Geoffrey T. LaFlair", "Kevin Yancey", "Alina A. von Davier", "Ravit Dotan"]
year: 2024
venue: "arXiv:2409.07476"
source: "https://arxiv.org/abs/2409.07476"
doi: "10.48550/arxiv.2409.07476"
arxiv: "2409.07476"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/burstein-2024-responsible-ai-test-equity-quality/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/burstein-2024-responsible-ai-test-equity-quality/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study

# Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study

**Authors:** [Jill Burstein](https://md.research.elamur.ai/free/en/authors/jill-burstein/), Geoffrey T. LaFlair, Kevin Yancey, Alina A. von Davier and Ravit Dotan · **Year:** 2024 · **Venue:** arXiv:2409.07476 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2409.07476) · [PDF](https://arxiv.org/pdf/2409.07476) · [DOI 10.48550/arxiv.2409.07476](https://doi.org/10.48550/arxiv.2409.07476) · [arXiv 2409.07476](https://arxiv.org/abs/2409.07476) · [OpenAlex](https://openalex.org/W4403662745)

**Citations:** 6 citations (OpenAlex, 2026-10-14)

## Summary
This chapter addresses a gap: older test-validity and fairness frameworks were designed for paper or early computer tests. They say little about the new risks of AI-driven assessment, such as biased generated items, AI literacy of test developers, or the carbon cost of LLMs. Using the Duolingo English Test (DET), a high-stakes, computer-adaptive English proficiency exam, as a case study, the authors set out the four pillars of its Responsible AI (RAI) framework:
- Reliability, together with validity
- Fairness
- Security, together with privacy
- Transparency, together with accountability

They describe how the standards were built: a literature review of AI ethics principles, a review of assessment standards and guidelines, and consultation with internal and external experts. They then check the standards against the domain-agnostic NIST AI RMF and show concrete practices for each standard. One example is human-in-the-loop review of generated items, labelling, and proctoring. The contribution is not an experiment but a worked template. It shows how abstract ethics principles become auditable practices that protect score interpretation. The authors also openly list gaps the standards do not yet cover.

## Key points
- The DET uses AI in three areas:
  - test design, through automated item generation;
  - measurement, through automated scoring of writing and speaking;
  - security, through plagiarism detection.
- AI also powers a free practice test that gives an instant score estimate.
- Validity is framed through the argument-based chain of inferences of Kane and Chapelle et al.: domain definition, evaluation, generalization, explanation, extrapolation, and utilization.
- The four DET RAI standards are mapped onto the trustworthiness characteristics of the NIST AI RMF (NIST, 2023). They are also checked against AERA/APA/NCME (2014), OECD, ITC-ATP, and US Department of Education guidance.
- New scoring models and new items need an evidence document (ECP) covering validity, reliability, and fairness before launch. Several senior assessment and AI experts must approve it.
- Acknowledged limitations:
  - GPT-4o-era risks, such as deep fakes;
  - possible bias in the plagiarism detectors themselves;
  - no coverage yet of environmental and labour impacts.
- The authors estimate that replacing DET sessions with in-person tests would add tens of millions of km of travel per year.

## Interesting ideas
- Nothing ships without a written evidence dossier on accuracy, consistency, and fairness, signed off by several reviewers; this turns quality claims into a reviewable artifact.
- Validity is argued as a chain of separate inferences (task covers the domain, score reflects ability, score is stable, and so on), so a weak link can be located precisely.
- Humans are placed at specific decision points (reviewing generated items, labelling training data, confirming automatic flags) rather than everywhere, which keeps oversight cheap but targeted.
- Fairness is treated as a theme that cuts across all standards, so that bias in auxiliary tools such as detectors is not overlooked.
- A domain-specific rule set is cross-checked against a general framework to find missing coverage.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Every new perceptron or gparse model, or new repair-rule set, could ship with a release record of LAS on held-out treebanks, negative-control results, and per-genre breakdowns, reviewed before replacing the old one.
- **I2** (answer): School-test and fable-QA evaluation could be organised as such a chain, checking separately that the questions cover the skill, that correct answers come from real comprehension, and that results hold on new texts.
- **I3** (back): Generated text from the graph → text path could be sampled at fixed checkpoints for human or strong-LLM review, with only flagged outputs escalated.
- **I4** (tree): Parser and conversion quality could be reported per dialect, register, and genre at every stage, not only at final evaluation, so systematic weaknesses on non-standard English surface early.
- **I5** (learn): Mova's own evaluation checklist could be mapped against a general AI risk framework like NIST AI RMF to spot unaddressed areas, such as data provenance or explainability gaps in a stage.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Duolingo](https://md.research.elamur.ai/free/en/companies/duolingo/) — Company, Pittsburgh, United States · 2 works · 437 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 543 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 108 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Jill Burstein, Geoffrey T. LaFlair, Kevin Yancey, Alina A. von Davier and Ravit Dotan. 2024. Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study. arXiv:2409.07476.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{burstein2024responsible,
  title = {{Responsible AI for Test Equity and Quality: The Duolingo English Test as a Case Study}},
  author = {Jill Burstein, Geoffrey T. LaFlair, Kevin Yancey, Alina A. {von Davier} and Ravit Dotan},
  year = {2024},
  doi = {10.48550/arxiv.2409.07476},
  eprint = {2409.07476},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.07476}
}
```

- [← previous QCAW 1.0: Building a Qatari Corpus of Student Argumentative Writing](https://md.research.elamur.ai/free/en/papers/zaghouani-2024-qcaw-1-0-building-qatari/)
- [next → SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection](https://md.research.elamur.ai/free/en/papers/hariri-2024-smash-stanceeval-2024-prompt-engineering/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

77 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational & psychological testing. American Educational Research Association. [https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf](https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf) [↗](https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf)
2. Association of Test Publishers. (2024). Creating Responsible and Ethical AI Policies for Assessment Organizations, July 12, 2024. Attali, Y., Runge, A., LaFlair, G. T., Yancey, K., Goodwin, S., Park, Y., & von Davier, A. A. [find](https://scholar.google.com/scholar?q=Association+of+Test+Publishers%2E+%282024%29%2E+Creating+Responsible+and+Ethical+AI+Policies+for+Assessment+Organizations%2C+July+12%2C+2024%2E+Attali%2C+Y%2E%2C+Runge%2C+A%2E%2C+LaFlair%2C+G%2E+T%2E%2C+Yancey%2C+K%2E%2C+Goodwin%2C+S%2E%2C+Park%2C+)
3. (2022). The interactive reading task: Transformer-based automatic item generation. Frontiers in Artificial Intelligence, 5. https://doi.org/10.3389/frai.2022.903077 [↗](https://doi.org/10.3389/frai.2022.903077)
4. Auernhammer, J. (2020). Human-centered AI: The role of human-centered design research in the development of AI. In S. Boess, M. Cheung, & R. Cain (Eds.), Synergy - DRS international conference 2020 (pp. 1315–1333). https://doi.org/10.21606/drs.2020.282 [↗](https://doi.org/10.21606/drs.2020.282)
5. Belzak, W. C., Naismith, B., & Burstein, J. (2023, June). Ensuring fairness of human-and AI-generated test items. In International Conference on Artificial Intelligence in Education (pp. 701-707). Cham: Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Belzak%2C+W%2E+C%2E%2C+Naismith%2C+B%2E%2C+%26+Burstein%2C+J%2E+%282023%2C+June%29%2E+Ensuring+fairness+of+human%2Dand+AI%2Dgenerated+test+items%2E+In+International+Conference+on+Artificial+Intelligence+in+Education+%28pp%2E+701%2D707%29%2E+Cha)
6. Burstein, J. (2023). Responsible AI standards. Duolingo English Test. [https://duolingo-papers.s3.amazonaws.com/other/DET+Responsible+AI+033123.pdf](https://duolingo-papers.s3.amazonaws.com/other/DET+Responsible+AI+033123.pdf) [↗](https://duolingo-papers.s3.amazonaws.com/other/DET+Responsible+AI+033123.pdf)
7. Burstein, J., LaFlair, G. T., Kunnan, A. J., & von Davier, A. A. (2022). A theoretical assessment ecosystem for a digital-first assessment—The Duolingo English Test [Research report]. Duolingo English Test. https://duolingo-papers.s3.amazonaws.com/other/det-assessment-ecosystem-mpr.pdf Burstein, J., Kukich, K., Wolff, S., Lu, C., Chodorow, M., Braden-Harder, L., & Harris, M. D. [↗](https://doi.org/10.46999/kiqf4328)
8. (1998). Automated scoring using a hybrid feature identification technique. In 36th annual meeting of the Association for Computational Linguistics and 17th international conference on Computational Linguistics: Vol. 1 (pp. 206–210). Association for Computational Linguistics. http://dx.doi.org/10.3115/980845.980879 [↗](https://doi.org/10.3115/980845.980879)
9. Bryant, C., Felice, M., & Briscoe, T. (2017). [Automatic annotation and evaluation of error types for grammatical error correction](https://md.research.elamur.ai/free/en/papers/bryant-2017-automatic-annotation-evaluation-error-types/). In R. Barzilay & M.-Y. Kan (Eds.), Proceedings of the 55th annual meeting of the association for computational linguistics: Vol. 1. Long papers (pp. 793–805). Association for Computational Linguistics. http://dx.doi.org/10.18653/v1/P17-1074 https://www.cambridgeenglish.org/Images/650585-linguaskill-writing-assessment-criteria.pdf [↗](https://doi.org/10.18653/v1/p17-1074)
10. Cardwell, R., Naismith, B., LaFlair, G. T., & Nydick, S. (2024) Duolingo English Test: Technical manual [Research report]. Duolingo English Test. [http://duolingo-papers.s3.amazonaws.com/other/technical_manual.pdf](http://duolingo-papers.s3.amazonaws.com/other/technical_manual.pdf) [↗](http://duolingo-papers.s3.amazonaws.com/other/technical_manual.pdf)
11. Chapelle, C. A. (2008). The TOEFL validity argument. In C. A. Chapelle, M. K. Enright, & J. Jamieson (Eds.), Building a validity argument for the Test of English as a Foreign Language (pp. 319–352). London : Routledge. [find](https://scholar.google.com/scholar?q=Chapelle%2C+C%2E+A%2E+%282008%29%2E+The+TOEFL+validity+argument%2E+In+C%2E+A%2E+Chapelle%2C+M%2E+K%2E+Enright%2C+%26+J%2E+Jamieson+%28Eds%2E%29%2C+Building+a+validity+argument+for+the+Test+of+English+as+a+Foreign+Language+%28pp%2E+319%E2%80%93352%29%2E+L)
12. Chandrasoma, R., Thompson, C., & Pennycook, A. (2004). Beyond plagiarism: Transgressive and nontransgressive intertextuality [Publisher: Taylor & Francis]. Journal of Language, Identity, and Education, 3(3), 171–193. [↗](https://doi.org/10.1207/s15327701jlie0303_1)
13. Chen, S., Ankenmann, R. D., & Spray, J. A. (2003). Exploring the relationship between item exposure rate and item overlap rate in computerized adaptive testing. Journal of Educational Measurement, 40(2), 129–145. https://doi.org/10.1111/j.1745-3984.2003.tb01100.x [↗](https://doi.org/10.1111/j.1745-3984.2003.tb01100.x)
14. Chen, T., & Guestrin, C. (2016, August). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining (pp. 785-794). [find](https://scholar.google.com/scholar?q=Chen%2C+T%2E%2C+%26+Guestrin%2C+C%2E+%282016%2C+August%29%2E+Xgboost%3A+A+scalable+tree+boosting+system%2E+In+Proceedings+of+the+22nd+acm+sigkdd+international+conference+on+knowledge+discovery+and+data+mining+%28pp%2E+785%2D794%29%2E)
15. Congressional Research Service. (2023). Generative artificial intelligence and copyright law (Report no. LSB10922, Version 4) . [https://crsreports.congress.gov/product/pdf/LSB/LSB10922](https://crsreports.congress.gov/product/pdf/LSB/LSB10922) [↗](https://crsreports.congress.gov/product/pdf/LSB/LSB10922)
16. Council of Europe. (2020). Common European framework of reference for languages: Learning, teaching, assessment–companion volume. Council of Europe Publishing. www.coe.int/lang-cefr. [find](https://scholar.google.com/scholar?q=Council+of+Europe%2E+%282020%29%2E+Common+European+framework+of+reference+for+languages%3A+Learning%2C+teaching%2C+assessment%E2%80%93companion+volume%2E+Council+of+Europe+Publishing%2E+www%2Ecoe%2Eint%2Flang%2Dcefr%2E)
17. de Chazal, E. (2014). English for academic purposes. Oxford University Press. [find](https://scholar.google.com/scholar?q=de+Chazal%2C+E%2E+%282014%29%2E+English+for+academic+purposes%2E+Oxford+University+Press%2E)
18. Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., Li, L., & Sui, Z. (2022). [A survey for in-context learning](https://md.research.elamur.ai/free/en/papers/dong-2023-survey-context-learning/). arXiv preprint arXiv:2301.00234 [↗](https://arxiv.org/abs/2301.00234)
19. Duolingo English Test. (2021). Duolingo English Test: Security, proctoring, and accommodations.[White Paper]. [https://duolingo-papers.s3.amazonaws.com/other/det-security-proctoring-whitepaper-202](https://duolingo-papers.s3.amazonaws.com/other/det-security-proctoring-whitepaper-202) 1-11.pdf Faiz, A., Kaneda, S., Wang, R., Osi, R., Sharma, P., Chen, F., & Jiang, L. (2023). LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Models. The Twelfth International Conference on Learning Representations, https://openreview.net/forum?id=aIok3ZD9to [↗](https://duolingo-papers.s3.amazonaws.com/other/det-security-proctoring-whitepaper-202)
20. Fiesler, Casey and Garrett, Natalie .(16 Sept 2020). Ethical Tech Starts with Addressing Ethical Debt, Wired Ideas: [https://www.wired.com/story/opinion-ethical-tech-starts-with-addressing-ethical-debt/](https://www.wired.com/story/opinion-ethical-tech-starts-with-addressing-ethical-debt/) [↗](https://www.wired.com/story/opinion-ethical-tech-starts-with-addressing-ethical-debt/)
21. Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., & Srikumar, M. (2020). Principled artificial intelligence: Mapping consensus in ethical and rights-based approaches to principles for AI [Research report]. Berkman Klein Center for Internet & Society at Harvard University. https://dx.doi.org/10.2139/ssrn.3518482 [↗](https://doi.org/10.2139/ssrn.3518482)
22. Floridi, L., & Cowls, J. (2022). A unified framework of five principles for AI in society. In S. Carta (Ed.), Machine learning and the city: Applications in architecture and urban design (pp. 535–545). John Wiley & Sons Ltd. https://doi.org/10.1002/9781119815075.ch45 Foltýnek, T., Bjelobaba, S., Glendinning, I., Khan, Z. R., Santos, R., Pavletic, P., & Kravjar, J. [↗](https://doi.org/10.1002/9781119815075.ch45)
23. (2023). ENAI recommendations on the ethical use of Artificial Intelligence in education. International Journal for Educational Integrity, 19(1). [find](https://scholar.google.com/scholar?q=%282023%29%2E+ENAI+recommendations+on+the+ethical+use+of+Artificial+Intelligence+in+education%2E+International+Journal+for+Educational+Integrity%2C+19%281%29%2E)
24. Foltýnek, T., Dlabolová, D., Anohina-Naumeca, A., Razı, S., Kravjar, J., Kamzola, L., Guerrero-Dib, J., Çelik, Ö., & Weber-Wulff, D. (2020). Testing of support tools for plagiarism detection. International Journal of Educational Technology in Higher Education, 17(1), 46. https://doi.org/10.1186/s41239-020-00192-4 [↗](https://doi.org/10.1186/s41239-020-00192-4)
25. Foltz, P. W., Laham, D., & Landauer, T. K. (1999). The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2). [http://imej.wfu.edu/articles/1999/2/04/index.asp](http://imej.wfu.edu/articles/1999/2/04/index.asp) [↗](http://imej.wfu.edu/articles/1999/2/04/index.asp)
26. Giulia Cataldo, M., & Oakhill, J. (2000). Why are poor comprehenders inefficient searchers? An investigation into the effects of text representation and spatial memory on the ability to locate information in text. Journal of Educational Psychology, 92(4), 791–799. https://doi.org/10.1037/0022-0663.92.4.791 [↗](https://doi.org/10.1037/0022-0663.92.4.791)
27. Grabe, W. (2008). Reading in a second language: Moving from theory to practice. Cambridge University Press. https://doi.org/10.1017/CBO9781139150484 [↗](https://doi.org/10.1017/cbo9781139150484)
28. Grabe, W., & Stoller, F. L. (2020). Teaching and researching reading (3rd ed.). Routledge. [find](https://scholar.google.com/scholar?q=Grabe%2C+W%2E%2C+%26+Stoller%2C+F%2E+L%2E+%282020%29%2E+Teaching+and+researching+reading+%283rd+ed%2E%29%2E+Routledge%2E)
29. Guthrie, J. T. (1988). Locating information in documents: Examination of a cognitive model. Reading Research Quarterly, 23(2), 178. https://doi.org/10.2307/747801 [↗](https://doi.org/10.2307/747801)
30. Guthrie, J. T., & Kirsch, I. S. (1987). Distinctions between reading comprehension and locating information in text. Journal of Educational Psychology, 79(3), 220–227. https://doi.org/10.1037/0022-0663.79.3.220 [↗](https://doi.org/10.1037/0022-0663.79.3.220)
31. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5 [↗](https://doi.org/10.1207/s15324818ame1503_5)
32. Head, A. J., & Eisenberg, M. B. (2009). Lessons learned: How college students seek information in the digital age [Progress report]. University of Washington, The Information School. [http://www.ssrn.com/abstract=2281478](http://www.ssrn.com/abstract=2281478) [↗](http://www.ssrn.com/abstract=2281478)
33. Holland, P. W., & Wainer, H. (2012). Differential item functioning. Routledge. [↗](https://doi.org/10.4324/9780203357811)
34. International Organization for Standardization. (2022). Information technology/Artificial Intelligence: Overview of ethical and societal concerns. (ISO/IEC TR 24368:2022). [https://www.iso.org/standard/78507.html](https://www.iso.org/standard/78507.html). [↗](https://www.iso.org/standard/78507.html)
35. The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. (2017). Ethically aligned design: A vision for prioritizing human well-being with autonomous and intelligent systems (Version 2). IEEE. [https://standards.ieee.org/wp-content/uploads/import/documents/other/ead_v2.pdf](https://standards.ieee.org/wp-content/uploads/import/documents/other/ead_v2.pdf) [↗](https://standards.ieee.org/wp-content/uploads/import/documents/other/ead_v2.pdf)
36. The International Privacy Subcommittee of the ATP Security Committee. (2021, July 6). Artificial intelligence and the testing industry: A primer. Association of Test Publishers. [find](https://scholar.google.com/scholar?q=The+International+Privacy+Subcommittee+of+the+ATP+Security+Committee%2E+%282021%2C+July+6%29%2E+Artificial+intelligence+and+the+testing+industry%3A+A+primer%2E+Association+of+Test+Publishers%2E)
37. International Test Commission & Association of Test Publishers (2022). Guidelines for technology-based assessment. Association of Test Publishers; International Test Commission. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. [find](https://scholar.google.com/scholar?q=International+Test+Commission+%26+Association+of+Test+Publishers+%282022%29%2E+Guidelines+for+technology%2Dbased+assessment%2E+Association+of+Test+Publishers%3B+International+Test+Commission%2E+Ji%2C+Z%2E%2C+Lee%2C+N%2E%2C+Fries)
38. (2023). [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12), 1–38.
39. Jin, Y. (2023). Test-taker insights for language assessment policies and practices. Language Testing, 40(1), 193–203. https://doi.org/10.1177/02655322221117136 [↗](https://doi.org/10.1177/02655322221117136)
40. Jin, K.-Y., & Eckes, T. (2021). Detecting differential rater functioning in severity and centrality: The dual DRF facets model. Educational and Psychological Measurement 82(4), 757–781. https://doi.org/10.1177/00131644211043207 [↗](https://doi.org/10.1177/00131644211043207)
41. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1, 389-399. https://doi.org/10.1038/s42256-019-0088-2 [↗](https://doi.org/10.1038/s42256-019-0088-2)
42. Johnson, M. S., Liu, X., & McCaffrey, D. F. (2022). Psychometric methods to evaluate measurement and algorithmic bias in automated scoring. Journal of Educational Measurement, 59(3), 338-361. [↗](https://doi.org/10.1111/jedm.12335)
43. Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. https://psycnet.apa.org/doi/10.1037/0033-2909.112.3.527 [↗](https://doi.org/10.1037/0033-2909.112.3.527)
44. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73. https://doi.org/10.1111/jedm.12000 [↗](https://doi.org/10.1111/jedm.12000)
45. Khalil, M., & Er, E. (2023, June). Will ChatGPT Get You Caught? Rethinking of Plagiarism Detection. In International Conference on Human-Computer Interaction (pp. 475-487). Cham: Springer Nature Switzerland. [find](https://scholar.google.com/scholar?q=Khalil%2C+M%2E%2C+%26+Er%2C+E%2E+%282023%2C+June%29%2E+Will+ChatGPT+Get+You+Caught%3F+Rethinking+of+Plagiarism+Detection%2E+In+International+Conference+on+Human%2DComputer+Interaction+%28pp%2E+475%2D487%29%2E+Cham%3A+Springer+Nature+Switz)
46. Khan, S., Hamer, J., & Almeida, T. (2021). Generate: A NLG system for educational content creation. In I.-H. Hsiao, S. Sahebi, F. Bouchet, J. -J.Vie (Eds.), Proceedings of the 14th international conference on educational data mining (pp. 736–740). Educational Data Mining. [find](https://scholar.google.com/scholar?q=Khan%2C+S%2E%2C+Hamer%2C+J%2E%2C+%26+Almeida%2C+T%2E+%282021%29%2E+Generate%3A+A+NLG+system+for+educational+content+creation%2E+In+I%2E%2DH%2E+Hsiao%2C+S%2E+Sahebi%2C+F%2E+Bouchet%2C+J%2E+%2DJ%2EVie+%28Eds%2E%29%2C+Proceedings+of+the+14th+international+confe)
47. Kunnan, A. J. (2000). Fairness and justice for all. In A. J. Kunnan (Ed.), Studies in language testing: Vol. 9. Fairness and validation in language assessment: Selected papers from the 19th Language Testing colloquium, Orlando, Florida (pp. 1–14). Cambridge University Press. [find](https://scholar.google.com/scholar?q=Kunnan%2C+A%2E+J%2E+%282000%29%2E+Fairness+and+justice+for+all%2E+In+A%2E+J%2E+Kunnan+%28Ed%2E%29%2C+Studies+in+language+testing%3A+Vol%2E+9%2E+Fairness+and+validation+in+language+assessment%3A+Selected+papers+from+the+19th+Language+T)
48. Liao, M., Attali, Y., Lockwood, J. R., & von Davier, A. A. (2022). Maintaining and monitoring quality of a continuously administered digital assessment. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.857496 [↗](https://doi.org/10.3389/feduc.2022.857496)
49. Lottridge, S., Godek, B., Jafari, A., & Patel, M. (2021). Comparing the robustness of deep learning and classical automated scoring approaches to gaming strategies [Technical report]. Cambium Assessment Inc. [find](https://scholar.google.com/scholar?q=Lottridge%2C+S%2E%2C+Godek%2C+B%2E%2C+Jafari%2C+A%2E%2C+%26+Patel%2C+M%2E+%282021%29%2E+Comparing+the+robustness+of+deep+learning+and+classical+automated+scoring+approaches+to+gaming+strategies+%5BTechnical+report%5D%2E+Cambium+Assessme)
50. Lundberg, S. M., & Lee, S. I. (2017). [A unified approach to interpreting model predictions](https://md.research.elamur.ai/free/en/papers/lundberg-2017-unified-approach-interpreting-model-predictions/). In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems: Vol. 30. Curran Associates, Inc. [↗](https://doi.org/10.48550/arxiv.1705.07874)
51. McNamara, D. S., & A. C. Graesser. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing." Applied natural language processing: Identification, investigation and resolution (pp. 188–205). IGI Global. [find](https://scholar.google.com/scholar?q=McNamara%2C+D%2E+S%2E%2C+%26+A%2E+C%2E+Graesser%2E+%282012%29%2E+Coh%2DMetrix%3A+An+automated+tool+for+theoretical+and+applied+natural+language+processing%2E%22+Applied+natural+language+processing%3A+Identification%2C+investigation+an)
52. Memarian, B., & Doleck, T. (2023). Fairness, accountability, transparency, and ethics (FATE) in Artificial Intelligence (AI), and higher education: A systematic review. Computers and Education: Artificial Intelligence, 5. https://doi.org/10.1016/j.caeai.2023.100152 [↗](https://doi.org/10.1016/j.caeai.2023.100152)
53. Myford, C., & Wolfe, E. W. (2004) Detecting and measuring rater effects using many-facet Rasch measurement: Part II. Journal of Applied Measurement 5(2) 189–227. [find](https://scholar.google.com/scholar?q=Myford%2C+C%2E%2C+%26+Wolfe%2C+E%2E+W%2E+%282004%29+Detecting+and+measuring+rater+effects+using+many%2Dfacet+Rasch+measurement%3A+Part+II%2E+Journal+of+Applied+Measurement+5%282%29+189%E2%80%93227%2E)
54. National Institute of Standards and Technology (2023). Artificial intelligence risk management framework (AI RMF 1.0). U.S. Department of Commerce. https://doi.org/10.6028/NIST.AI.100-1 [↗](https://doi.org/10.6028/nist.ai.100-1)
55. OpenAI (2024). [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). Accessed Jun 20, 2024 . [↗](https://openai.com/index/hello-gpt-4o/)
56. OpenAI (2023). [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). https://arxiv.org/pdf/2303.08774.pdf [↗](https://arxiv.org/abs/2303.08774)
57. OECD (2023). Advancing accountability in AI: Governing and managing risks throughout the lifecycle for trustworthy AI (No. 349). OECD Publishing. https://doi.org/10.1787/2448f04b-en [↗](https://doi.org/10.1787/2448f04b-en)
58. Park, Y., LaFlair, G. T., Attali, Y., Runge, A., & Goodwin, S. (2022) Interactive reading—The Duolingo English Test [White paper]. Duolingo English Test. https://doi.org/10.46999/RCXB1889 [↗](https://doi.org/10.46999/rcxb1889)
59. Pecorari, D., & Petrić, B. (2014). Plagiarism in second-language writing. Language Teaching, 47(3), 269–302. https://doi.org/10.1017/S0261444814000056 [↗](https://doi.org/10.1017/s0261444814000056)
60. PriceWaterhouseCoopers (PwC) (2024). PwC’s 2024 US Responsible AI Survey. Accessed from: [https://www.pwc.com/us/en/tech-effect/ai-analytics/responsible-ai-survey.html](https://www.pwc.com/us/en/tech-effect/ai-analytics/responsible-ai-survey.html) [↗](https://www.pwc.com/us/en/tech-effect/ai-analytics/responsible-ai-survey.html)
61. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 1-18. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+%26+Sutskever%2C+I%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%2C+1%2D18%2E)
62. Rei, M., & Cummins, R. (2016). Sentence similarity measures for fine-grained estimation of topical relevance in learner essays. In J. Tetreault, J. Burstein, C. Leacock, & H. Yannakoudakis (Eds.), Proceedings of the 11th workshop on innovative use of NLP for building educational applications (pp. 283–288). Association for Computational Linguistics. http://dx.doi.org/10.18653/v1/W16-0533 [↗](https://doi.org/10.18653/v1/w16-0533)
63. Schwarm, S. & Ostendorf, M. (2005). [Reading Level Assessment Using Support Vector Machines and Statistical Language Models](https://md.research.elamur.ai/free/en/papers/schwarm-2005-reading-level-assessment-support-vector/). In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 523–530, Ann Arbor, Michigan. Association for Computational Linguistics. https://doi.org/10.17705/1thci.00131 [↗](https://doi.org/10.17705/1thci.00131)
64. Shermis, M. D., & Burstein, J. (2013). Handbook of automated essay evaluation. Routledge. [find](https://scholar.google.com/scholar?q=Shermis%2C+M%2E+D%2E%2C+%26+Burstein%2C+J%2E+%282013%29%2E+Handbook+of+automated+essay+evaluation%2E+Routledge%2E)
65. Shermis, M. D., & Burstein, J. C. (2003). Automated essay scoring: A cross-disciplinary perspective. Routledge. [find](https://scholar.google.com/scholar?q=Shermis%2C+M%2E+D%2E%2C+%26+Burstein%2C+J%2E+C%2E+%282003%29%2E+Automated+essay+scoring%3A+A+cross%2Ddisciplinary+perspective%2E+Routledge%2E)
66. Shohamy, E. (2001). Democratic assessment as an alternative. Language testing, 18(4), 373–391.http://dx.doi.org/10.1191/026553201682430094 [↗](https://doi.org/10.1191/026553201682430094)
67. Strubell, E., Ganesh, A., and McCallum, A. (2019, July). [Energy and policy considerations for deep learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In Proceedings of the Annual Meeting of the Association for Computational Linguistics, (pp. 3645–3650).
68. Thompson, C., Morton, J., & Storch, N. (2013). Where from, who, why and how? A study of the use of sources by first year L2 university students. Journal of English for Academic Purposes, 12(2), 99–109. https://doi.org/10.1016/j.jeap.2012.11.004 [↗](https://doi.org/10.1016/j.jeap.2012.11.004)
69. U.S. Department of Education, Office of Educational Technology (2024). Designing for Education with Artificial Intelligence: An Essential Guide for Developers, Washington, D.C. [find](https://scholar.google.com/scholar?q=U%2ES%2E+Department+of+Education%2C+Office+of+Educational+Technology+%282024%29%2E+Designing+for+Education+with+Artificial+Intelligence%3A+An+Essential+Guide+for+Developers%2C+Washington%2C+D%2EC%2E)
70. U.S. Department of Education, Office of Educational Technology (2023). Artificial intelligence and future of teaching and learning: Insights and recommendations [Report]. [https://www2.ed.gov/documents/ai-report/ai-report.pdf](https://www2.ed.gov/documents/ai-report/ai-report.pdf) [↗](https://www2.ed.gov/documents/ai-report/ai-report.pdf)
71. Wang, G. (2019, October 20). Humans in the Loop: The Design of Interactive AI Systems: [https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems](https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems). [↗](https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems)
72. Wang, X., Evanini, K., Mulholland, M., Qian, Y., & Bruno, J. V. (2019). [Application of an Automatic Plagiarism Detection System in a Large-scale Assessment of English Speaking Proficiency](https://md.research.elamur.ai/free/en/papers/wang-2019-application-automatic-plagiarism-detection-system/). Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, 435–443.
73. Way, W. D. (1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17, 17–27. https://doi.org/10.1111/j.1745-3992.1998.tb00632.x [↗](https://doi.org/10.1111/j.1745-3992.1998.tb00632.x)
74. Weir, C., Hawkey, R., Green, A., Unaldi, A., & Devi, S. (2009). The relationship between the academic reading construct as measured by IELTS and the reading experiences of students in their first year of study at a British university (vol. 9; pp. 97–156). British Council; IELTS Australia. [https://www.ielts.org/-/media/research-reports/ielts_rr_volume09_report3.ashx](https://www.ielts.org/-/media/research-reports/ielts_rr_volume09_report3.ashx) [↗](https://www.ielts.org/-/media/research-reports/ielts_rr_volume09_report3.ashx)
75. Xia, M., Kochmar, E., & Briscoe, T. (2016). [Text readability assessment for second language learners](https://md.research.elamur.ai/free/en/papers/xia-2016-text-readability-assessment-second-language/). In J. Tetreault, J. Burstein, C. Leacock, & H. Yannakoudakis (Eds.), Proceedings of the 11th workshop on innovative use of NLP for building educational applications (pp. 12–22). Association for Computational Linguistics. http://dx.doi.org/10.18653/v1/W16-0502 [↗](https://doi.org/10.18653/v1/w16-0502)
76. Zechner, K., & Evanini, K. (Eds.). (2019). Automated speaking assessment: Using language technologies to score spontaneous speech. Routledge. [find](https://scholar.google.com/scholar?q=Zechner%2C+K%2E%2C+%26+Evanini%2C+K%2E+%28Eds%2E%29%2E+%282019%29%2E+Automated+speaking+assessment%3A+Using+language+technologies+to+score+spontaneous+speech%2E+Routledge%2E)
77. Zieky, M. J. (2015). Developing fair tests. In S. Lane, M. R. Raymond, & T.M. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 81–99). Routledge. [↗](https://doi.org/10.4324/9780203102961.ch5)

*Updated 1 November 2026*