--- title: "Improving the Validity of Automatically Generated Feedback" description: "The authors build a framework for both generating and evaluating such feedback for… 76% accuracy. 7 references, cited by 16 papers here." authors: ["Alexander Scarlatos", "Digory Smith", "Simon Woodhead", "Andrew Lan"] year: 2024 venue: "Lecture Notes in Computer Science" source: "https://arxiv.org/abs/2403.01304" doi: "10.1007/978-3-031-64302-6_20" arxiv: "2403.01304" licence: "CC BY 4.0 (arXiv)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/scarlatos-2024-improving-validity-automatically-generated-feedback/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/scarlatos-2024-improving-validity-automatically-generated-feedback/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / Improving the Validity of Automatically Generated Feedback via Reinforcement Learning # Improving the Validity of Automatically Generated Feedback via Reinforcement Learning **Authors:** [Alexander Scarlatos](https://md.research.elamur.ai/free/en/authors/alexander-scarlatos/), Digory Smith, [Simon Woodhead](https://md.research.elamur.ai/free/en/authors/simon-woodhead/) and [Andrew Lan](https://md.research.elamur.ai/free/en/authors/andrew-lan/) · **Year:** 2024 · **Venue:** [Lecture Notes in Computer Science](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/) ([Lecture notes in computer science 2024](https://md.research.elamur.ai/free/en/venues/lecture-notes-in-computer-science/2024/)) · **Publisher:** [Springer Science+Business Media](https://md.research.elamur.ai/free/en/publishers/springer-science-business-media/) · **Licence:** [CC BY 4.0 (arXiv)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://doi.org/10.1007/978-3-031-64302-6_20) · [PDF](https://arxiv.org/pdf/2403.01304) · [DOI 10.1007/978-3-031-64302-6\_20](https://doi.org/10.1007/978-3-031-64302-6_20) · [arXiv 2403.01304](https://arxiv.org/abs/2403.01304) · [OpenAlex](https://openalex.org/W4400208028) **Citations:** 31 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/scarlatos-2024-improving-validity-automatically-generated-feedback/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary LLM-written feedback on students' wrong answers in math must be mathematically sound and must also follow good tutoring practice. Judging both properties automatically is hard, and text-overlap metrics measure them poorly. The authors build a framework for both generating and evaluating such feedback for incorrect options in multiple-choice math questions, using teacher-written messages as gold data. They define a five-part binary rubric and have GPT-4 apply it through zero-shot chain-of-thought prompting. They then expand the dataset with feedback produced by several prompting strategies, score everything with GPT-4, and turn the scores into preference pairs. These pairs are used to fine-tune Llama 2 7B with DPO, an offline RL method that avoids calling GPT-4 repeatedly during training. The fine-tuned model produces feedback that is significantly more correct and better aligned than baselines. On alignment metrics, it comes close to human and GPT-4 feedback despite being a far smaller open model. This suggests that affordable, open models can deliver pedagogically sound feedback in tutoring systems. ## Key points - **Rubric:** five binary labels: Correct, Revealing (does not give away the answer), Suggestion, Diagnostic, and Positive. Any incorrect feedback scores 0. Otherwise, each satisfied label adds 0.2 to the score. - **Augmented data:** the gold feedback is combined with three LLM-generated variants: few-shot with random examples (D_R), few-shot with the most similar examples (D_S), and zero-shot (D_Z). - **Annotator validation:** on 80 sampled messages, GPT-4 averaged 76% accuracy against human labels. Its rubric scores correlated with human scores at Pearson 0.56, compared with 0.40 for both ROUGE-L and BERTScore. Agreement between the two human annotators (Cohen's kappa) ranged from 0.29 (POS.) to 0.53 (COR., DIA.). - **GPT-4 weaknesses:** it tends to accept persuasive but wrong feedback, especially when checking requires calculation. It also hallucinates diagnostic content, producing false positives on that label. - **Availability:** published at AIED 2024, with code at github.com/umass-ml4ed/feedback-gen-dpo. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Massachusetts Amherst](https://md.research.elamur.ai/free/en/universities/university-of-massachusetts-amherst/) — University, Amherst Center, United States · 122 works · 69,574 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works ## Cite Cite as: Alexander Scarlatos, Digory Smith, Simon Woodhead and Andrew Lan. 2024. Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. Lecture Notes in Computer Science. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{scarlatos2024improving, title = {{Improving the Validity of Automatically Generated Feedback via Reinforcement Learning}}, author = {Alexander Scarlatos, Digory Smith, Simon Woodhead and Andrew Lan}, booktitle = {Lecture Notes in Computer Science}, year = {2024}, doi = {10.1007/978-3-031-64302-6\_20}, eprint = {2403.01304}, archivePrefix = {arXiv}, url = {https://doi.org/10.1007/978-3-031-64302-6_20} } ``` - [← previous Getting the most out of your tokenizer for pre-training and domain adaptation](https://md.research.elamur.ai/free/en/papers/dagan-2024-getting-most-out-tokenizer-pre/) - [next → LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://md.research.elamur.ai/free/en/papers/shojaee-2024-llm-sr-scientific-equation/) - [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) ## References 7 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 1. Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., Dorodchi, M.: [Socratic questioning of novice debuggers: A benchmark dataset and preliminary evaluations](https://md.research.elamur.ai/free/en/papers/hossami-2023-socratic-questioning-novice-debuggers-benchmark/). In: Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications (BEA2023@ACL). pp. 709–726 (Jul… 2. Improving the Validity of Automatically Generated Feedback 2. Boaler, J.: Ability and mathematics: The mindset revolution that is reshaping education. The Forum 55, 143–152 (2013) 3. Botelho, A., Baral, S., Erickson, J.A., Benachamardi, P., Heffernan, N.T.: Leveraging natural language processing to… [find](https://scholar.google.com/scholar?q=Improving+the+Validity+of+Automatically+Generated+Feedback+2%2E+Boaler%2C+J%2E%3A+Ability+and+mathematics%3A+The+mindset+revolution+that+is+reshaping+education%2E+The+Forum+55%2C+143%E2%80%93152+%282013%29+3%2E+Botelho%2C+A%2E%2C+Bara) 3. International Educational Data Mining Society (2022) 11. Kakarla, S., Thomas, D., Lin, J., Gupta, S., Koedinger, K.R.: Using large language models to assess tutors’ performance in reacting to students making math errors. arXiv preprint arXiv:2401.03238 (2024) 12. Kochmar, E., Vu, D.D., Belfer, R., G… — [Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System](https://md.research.elamur.ai/free/en/papers/kochmar-2020-automated-personalized-feedback-improves-learning/) 4. A. Scarlatos et al. 20. McNichols, H., Feng, W., Lee, J., Scarlatos, A., Smith, D., Woodhead, S., Lan, A.: [Automated distractor and feedback generation for math multiple-choice questions via in-context learning](https://md.research.elamur.ai/free/en/papers/mcnichols-2023-automated-distractor-feedback-generation-math/). NeurIPS’23 Workshop on Generative AI for Education (2023) 21. McNichols, H., Zhang, M.,… 5. Springer (2020) 26. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019) 27. Robinson, J.D., Chuang, C.Y., Sra, S., Je… — [Reflexion: Language Agents with Verbal Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/) 6. International Educational Data Mining Society (2022) [find](https://scholar.google.com/scholar?q=International+Educational+Data+Mining+Society+%282022%29) 7. Improving the Validity of Automatically Generated Feedback 37. Zhang, M., Wang, Z., Baraniuk, R., Lan, A.: Math operation embeddings for open-ended solution analysis and feedback. International Educational Data Mining Society (2021) 38. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Be… [find](https://scholar.google.com/scholar?q=Improving+the+Validity+of+Automatically+Generated+Feedback+37%2E+Zhang%2C+M%2E%2C+Wang%2C+Z%2E%2C+Baraniuk%2C+R%2E%2C+Lan%2C+A%2E%3A+Math+operation+embeddings+for+open%2Dended+solution+analysis+and+feedback%2E+International+Educat) *Updated 3 November 2026*