---
title: "Evaluating and Optimizing Educational Content with Large"
description: "The authors then build an optimization loop for math word problem worksheets. 51 references, cited by 7 papers here."
authors: ["Joy He-Yueya", "Noah D. Goodman", "Emma Brunskill"]
year: 2024
venue: "arXiv:2403.02795"
source: "https://arxiv.org/abs/2403.02795"
doi: "10.48550/arxiv.2403.02795"
arxiv: "2403.02795"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/he-yueya-2024-evaluating-optimizing-educational-content-large/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-yueya-2024-evaluating-optimizing-educational-content-large/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/) / Evaluating and Optimizing Educational Content with Large Language Model Judgments

# Evaluating and Optimizing Educational Content with Large Language Model Judgments

**Authors:** [Joy He-Yueya](https://md.research.elamur.ai/free/en/authors/joy-he-yueya/), [Noah D. Goodman](https://md.research.elamur.ai/free/en/authors/noah-d-goodman/) and [Emma Brunskill](https://md.research.elamur.ai/free/en/authors/emma-brunskill/) · **Year:** 2024 · **Venue:** arXiv:2403.02795 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.02795) · [PDF](https://arxiv.org/pdf/2403.02795) · [DOI 10.48550/arxiv.2403.02795](https://doi.org/10.48550/arxiv.2403.02795) · [arXiv 2403.02795](https://arxiv.org/abs/2403.02795) · [OpenAlex](https://openalex.org/W4392538204)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/he-yueya-2024-evaluating-optimizing-educational-content-large/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Testing whether teaching materials actually help learners usually requires slow, costly studies with real students. Earlier efforts to have language models role-play students struggled because the simulated knowledge levels drifted before and after instruction. The authors instead cast the model as a pedagogical expert. In their Simulated Expert Evaluation (SEE), an evaluator LM receives a student persona describing prior skills, a piece of instruction, and a series of post-test questions. It then estimates the chance that the student solves each item, and these estimates are aggregated across personas. With GPT-3.5 in this role, the judgments recover a pair of well-known findings from educational psychology, namely the Variability Effect and the Expertise Reversal Effect. The authors then build an optimization loop for math word problem worksheets. GPT-4 drafts new worksheets from earlier worksheet–score pairs, and GPT-3.5's predicted post-test scores act as the reward. Teachers' pairwise comparisons of the resulting worksheets correlate with the model's predictions. This suggests LMs could serve as inexpensive first-pass evaluators that help plan human experiments, though they cannot replace testing with actual students.

## Key points
- SEE feeds the evaluator a persona, an instruction and test items one after another. It averages the per-item success probabilities into a predicted score. The prompts are released at github.com/StanfordAI4HI/ed-expert-simulator.
- GPT-3.5 judgments replicate the Expertise Reversal Effect, where the benefit of instruction depends on the learner's prior knowledge, and the Variability Effect.
- GPT-3.5 served as the evaluator because it is cheaper for long prompts. GPT-4 served as the optimizer, generating worksheets for an 8th-grade student working on algebra word problems.
- In the human study, 56 teachers rated worksheet pairs on a slider. Their preference scores correlated with LM-predicted post-test scores (Pearson r = 0.661, p \< 0.05).
- The paper notes several limitations:
  - Teachers often could not separate worksheets that the LM scored differently, especially in the 60–90 predicted-score range.
  - The models may have seen the replicated studies during training.
  - Only mathematics and only GPT-3.5/GPT-4 were tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 567 works · 335,190 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 4,145 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,394 works

## Cite

Cite as: Joy He-Yueya, Noah D. Goodman and Emma Brunskill. 2024. Evaluating and Optimizing Educational Content with Large Language Model Judgments. arXiv:2403.02795.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{heyueya2024evaluating,
  title = {{Evaluating and Optimizing Educational Content with Large Language Model Judgments}},
  author = {Joy He-Yueya, Noah D. Goodman and Emma Brunskill},
  year = {2024},
  doi = {10.48550/arxiv.2403.02795},
  eprint = {2403.02795},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.02795}
}
```

- [← previous Eureka: Evaluating and Understanding Large Foundation Models](https://md.research.elamur.ai/free/en/papers/balachandran-2024-eureka-evaluating-understanding-large-foundation/)
- [next → FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation](https://md.research.elamur.ai/free/en/papers/ma-2024-fbi-llm-scaling-up-fully/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/15/)

## References

51 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774, 2023.
2. G. V. Aher, R. I. Arriaga, and A. T. Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337–371. PMLR, 2023. [find](https://scholar.google.com/scholar?q=G%2E+V%2E+Aher%2C+R%2E+I%2E+Arriaga%2C+and+A%2E+T%2E+Kalai%2E+Using+large+language+models+to+simulate+multiple+humans+and+replicate+human+subject+studies%2E+In+International+Conference+on+Machine+Learning%2C+pages+337%E2%80%93371%2E)
3. J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167–207, 1995. [find](https://scholar.google.com/scholar?q=J%2E+R%2E+Anderson%2C+A%2E+T%2E+Corbett%2C+K%2E+R%2E+Koedinger%2C+and+R%2E+Pelletier%2E+Cognitive+tutors%3A+Lessons+learned%2E+The+journal+of+the+learning+sciences%2C+4%282%29%3A167%E2%80%93207%2C+1995%2E)
4. L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. [Out of one, many: Using language models to simulate human samples](https://md.research.elamur.ai/free/en/papers/argyle-2022-out-one-many-using-language/). Political Analysis, 31(3):337–351, 2023.
5. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022.
6. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/). arXiv preprint arXiv:2303.12712, 2023.
7. P. Denny, H. Khosravi, A. Hellas, J. Leinonen, and S. Sarsa. Can we trust ai-generated educational content? comparative analysis of human and ai-generated learning resources. arXiv preprint arXiv:2306.10509, 2023. — [arXiv:2306.10509](https://arxiv.org/abs/2306.10509) [find](https://scholar.google.com/scholar?q=P%2E+Denny%2C+H%2E+Khosravi%2C+A%2E+Hellas%2C+J%2E+Leinonen%2C+and+S%2E+Sarsa%2E+Can+we+trust+ai%2Dgenerated+educational+content%3F+comparative+analysis+of+human+and+ai%2Dgenerated+learning+resources%2E+arXiv+preprint+arXiv%3A2306)
8. J. E. Greer and G. I. McCalla. Student modelling: the key to individualized knowledge-based instruction, volume 125. Springer Science & Business Media, 2013. [find](https://scholar.google.com/scholar?q=J%2E+E%2E+Greer+and+G%2E+I%2E+McCalla%2E+Student+modelling%3A+the+key+to+individualized+knowledge%2Dbased+instruction%2C+volume+125%2E+Springer+Science+%26+Business+Media%2C+2013%2E)
9. J. He-Yueya, G. Poesia, R. E. Wang, and N. D. Goodman. [Solving math word problems by combining language models with symbolic solvers](https://md.research.elamur.ai/free/en/papers/yueya-2023-solving-math-word-problems-combining/). arXiv preprint arXiv:2304.09102, 2023.
10. J. J. Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Horton%2E+Large+language+models+as+simulated+economic+agents%3A+What+can+we+learn+from+homo+silicus%3F+Technical+report%2C+National+Bureau+of+Economic+Research%2C+2023%2E)
11. H. Jin, S. Lee, H. Shin, and J. Kim. ” teach ai how to code”: Using large language models as teachable agents for programming education. arXiv preprint arXiv:2309.14534, 2023. — [arXiv:2309.14534](https://arxiv.org/abs/2309.14534) [find](https://scholar.google.com/scholar?q=H%2E+Jin%2C+S%2E+Lee%2C+H%2E+Shin%2C+and+J%2E+Kim%2E+%E2%80%9D+teach+ai+how+to+code%E2%80%9D%3A+Using+large+language+models+as+teachable+agents+for+programming+education%2E+arXiv+preprint+arXiv%3A2309%2E14534%2C+2023%2E+%E2%80%94+%5BarXiv%3A2309%2E14534%5D%28htt)
12. S. Jinxin, Z. Jiabao, W. Yilei, W. Xingjiao, L. Jiawen, and H. Liang. Cgmi: Configurable general multi-agent interaction framework. arXiv preprint arXiv:2308.12503, 2023. — [arXiv:2308.12503](https://arxiv.org/abs/2308.12503) [find](https://scholar.google.com/scholar?q=S%2E+Jinxin%2C+Z%2E+Jiabao%2C+W%2E+Yilei%2C+W%2E+Xingjiao%2C+L%2E+Jiawen%2C+and+H%2E+Liang%2E+Cgmi%3A+Configurable+general+multi%2Dagent+interaction+framework%2E+arXiv+preprint+arXiv%3A2308%2E12503%2C+2023%2E+%E2%80%94+%5BarXiv%3A2308%2E12503%5D%28https%3A%2F%2F)
13. S. Kalyuga. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review, 19:509–539, 2007. [find](https://scholar.google.com/scholar?q=S%2E+Kalyuga%2E+Expertise+reversal+effect+and+its+implications+for+learner%2Dtailored+instruction%2E+Educational+psychology+review%2C+19%3A509%E2%80%93539%2C+2007%2E)
14. S. Kalyuga, P. Ayres, P. Chandler, and J. Sweller. The expertise reversal effect. Educational Psychologist, 38(1):23–31, 2003. [find](https://scholar.google.com/scholar?q=S%2E+Kalyuga%2C+P%2E+Ayres%2C+P%2E+Chandler%2C+and+J%2E+Sweller%2E+The+expertise+reversal+effect%2E+Educational+Psychologist%2C+38%281%29%3A23%E2%80%9331%2C+2003%2E)
15. S. Kalyuga and J. Sweller. Measuring knowledge to optimize cognitive load factors during instruction. Journal of educational psychology, 96(3):558, 2004. [find](https://scholar.google.com/scholar?q=S%2E+Kalyuga+and+J%2E+Sweller%2E+Measuring+knowledge+to+optimize+cognitive+load+factors+during+instruction%2E+Journal+of+educational+psychology%2C+96%283%29%3A558%2C+2004%2E)
16. P. K. Komoski. Eric/avcr annual review paper: An imbalance of product quantity and instructional quality: The imperative of empiricism. AV Communication Review, 22(4):357–386, 1974. [find](https://scholar.google.com/scholar?q=P%2E+K%2E+Komoski%2E+Eric%2Favcr+annual+review+paper%3A+An+imbalance+of+product+quantity+and+instructional+quality%3A+The+imperative+of+empiricism%2E+AV+Communication+Review%2C+22%284%29%3A357%E2%80%93386%2C+1974%2E)
17. H. Kumar, D. M. Rothschild, D. G. Goldstein, and J. M. Hofman. Math education with large language models: Peril or promise? Available at SSRN 4641653, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Kumar%2C+D%2E+M%2E+Rothschild%2C+D%2E+G%2E+Goldstein%2C+and+J%2E+M%2E+Hofman%2E+Math+education+with+large+language+models%3A+Peril+or+promise%3F+Available+at+SSRN+4641653%2C+2023%2E)
18. R. Liu, H. Yen, R. Marjieh, T. L. Griffiths, and R. Krishna. Improving interpersonal communication by simulating audiences with language models. arXiv preprint arXiv:2311.00687, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Liu%2C+H%2E+Yen%2C+R%2E+Marjieh%2C+T%2E+L%2E+Griffiths%2C+and+R%2E+Krishna%2E+Improving+interpersonal+communication+by+simulating+audiences+with+language+models%2E+arXiv+preprint+arXiv%3A2311%2E00687%2C+2023%2E)
19. Q. Ma, H. Shen, K. Koedinger, and T. Wu. Hypocompass: Large-language-model-based tutor for hypothesis construction in debugging for novices. arXiv preprint arXiv:2310.05292, 2023. — [arXiv:2310.05292](https://arxiv.org/abs/2310.05292) [find](https://scholar.google.com/scholar?q=Q%2E+Ma%2C+H%2E+Shen%2C+K%2E+Koedinger%2C+and+T%2E+Wu%2E+Hypocompass%3A+Large%2Dlanguage%2Dmodel%2Dbased+tutor+for+hypothesis+construction+in+debugging+for+novices%2E+arXiv+preprint+arXiv%3A2310%2E05292%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E05292%5D)
20. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Madaan%2C+N%2E+Tandon%2C+P%2E+Gupta%2C+S%2E+Hallinan%2C+L%2E+Gao%2C+S%2E+Wiegreffe%2C+U%2E+Alon%2C+N%2E+Dziri%2C+S%2E+Prabhumoye%2C+Y%2E+Yang%2C+et+al%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Dfeedback%2E+arXiv+preprint+arXiv%3A2303%2E176)
21. J. M. Markel, S. G. Opferman, J. A. Landay, and C. Piech. Gpteach: Interactive ta training with gpt based students. 2023. [find](https://scholar.google.com/scholar?q=J%2E+M%2E+Markel%2C+S%2E+G%2E+Opferman%2C+J%2E+A%2E+Landay%2C+and+C%2E+Piech%2E+Gpteach%3A+Interactive+ta+training+with+gpt+based+students%2E+2023%2E)
22. N. Matsuda, W. W. Cohen, J. Sewall, G. Lacerda, and K. R. Koedinger. Evaluating a simulated student using real students data for training and testing. In International Conference on User Modeling, pages 107–116. Springer, 2007. [find](https://scholar.google.com/scholar?q=N%2E+Matsuda%2C+W%2E+W%2E+Cohen%2C+J%2E+Sewall%2C+G%2E+Lacerda%2C+and+K%2E+R%2E+Koedinger%2E+Evaluating+a+simulated+student+using+real+students+data+for+training+and+testing%2E+In+International+Conference+on+User+Modeling%2C+pag)
23. N. Matsuda, V. Keiser, R. Raizada, A. Tu, G. Stylianides, W. W. Cohen, and K. R. Koedinger. Learning by teaching simstudent: Technical accomplishments and an initial use with students. In Intelligent Tutoring Systems: 10th International Conference, ITS 2010, Pittsburgh, PA, USA, June 14-18, 2010, Pr… [find](https://scholar.google.com/scholar?q=N%2E+Matsuda%2C+V%2E+Keiser%2C+R%2E+Raizada%2C+A%2E+Tu%2C+G%2E+Stylianides%2C+W%2E+W%2E+Cohen%2C+and+K%2E+R%2E+Koedinger%2E+Learning+by+teaching+simstudent%3A+Technical+accomplishments+and+an+initial+use+with+students%2E+In+Intelligent+)
24. J. S. Mertz. Using a simulated student for instructional design. International Journal of Artificial Intelligence in Education (IJAIED), 8:116–141, 1997. [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Mertz%2E+Using+a+simulated+student+for+instructional+design%2E+International+Journal+of+Artificial+Intelligence+in+Education+%28IJAIED%29%2C+8%3A116%E2%80%93141%2C+1997%2E)
25. M. H. Nguyen, S. Tschiatschek, and A. Singla. Large language models for in-context student modeling: Synthesizing student’s behavior in visual programming from one-shot observation. arXiv preprint arXiv:2310.10690, 2023. — [arXiv:2310.10690](https://arxiv.org/abs/2310.10690) [find](https://scholar.google.com/scholar?q=M%2E+H%2E+Nguyen%2C+S%2E+Tschiatschek%2C+and+A%2E+Singla%2E+Large+language+models+for+in%2Dcontext+student+modeling%3A+Synthesizing+student%E2%80%99s+behavior+in+visual+programming+from+one%2Dshot+observation%2E+arXiv+preprint+arX)
26. N. Nieveen and E. Folmer. Formative evaluation in educational design research. Design Research, 153(1):152–169, 2013. [find](https://scholar.google.com/scholar?q=N%2E+Nieveen+and+E%2E+Folmer%2E+Formative+evaluation+in+educational+design+research%2E+Design+Research%2C+153%281%29%3A152%E2%80%93169%2C+2013%2E)
27. S. Ohlsson, A. M. Ernst, and E. Rees. The cognitive complexity of learning and doing arithmetic. Journal for Research in Mathematics Education, 23(5):441–467, 1992. [find](https://scholar.google.com/scholar?q=S%2E+Ohlsson%2C+A%2E+M%2E+Ernst%2C+and+E%2E+Rees%2E+The+cognitive+complexity+of+learning+and+doing+arithmetic%2E+Journal+for+Research+in+Mathematics+Education%2C+23%285%29%3A441%E2%80%93467%2C+1992%2E)
28. A. Opedal, A. Stolfo, H. Shirakami, Y. Jiao, R. Cotterell, B. Schölkopf, A. Saparov, and M. Sachan. Do language models exhibit the same cognitive biases in problem solving as human learners? arXiv preprint arXiv:2401.18070, 2024. — [arXiv:2401.18070](https://arxiv.org/abs/2401.18070) [find](https://scholar.google.com/scholar?q=A%2E+Opedal%2C+A%2E+Stolfo%2C+H%2E+Shirakami%2C+Y%2E+Jiao%2C+R%2E+Cotterell%2C+B%2E+Scho%CC%88lkopf%2C+A%2E+Saparov%2C+and+M%2E+Sachan%2E+Do+language+models+exhibit+the+same+cognitive+biases+in+problem+solving+as+human+learners%3F+arXiv+pr)
29. F. G. Paas and J. J. Van Merriënboer. Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of educational psychology, 86(1):122, 1994. [find](https://scholar.google.com/scholar?q=F%2E+G%2E+Paas+and+J%2E+J%2E+Van+Merrie%CC%88nboer%2E+Variability+of+worked+examples+and+transfer+of+geometrical+problem%2Dsolving+skills%3A+A+cognitive%2Dload+approach%2E+Journal+of+educational+psychology%2C+86%281%29%3A122%2C+1994%2E)
30. M. Pankiewicz and R. S. Baker. Large language models (gpt) for automating feedback on programming assignments. arXiv preprint arXiv:2307.00150, 2023. [find](https://scholar.google.com/scholar?q=M%2E+Pankiewicz+and+R%2E+S%2E+Baker%2E+Large+language+models+%28gpt%29+for+automating+feedback+on+programming+assignments%2E+arXiv+preprint+arXiv%3A2307%2E00150%2C+2023%2E)
31. Z. A. Pardos and S. Bhandari. [Learning gain differences between chatgpt and human tutor generated algebra hints](https://md.research.elamur.ai/free/en/papers/pardos-2023-learning-gain-differences-between-chatgpt/). arXiv preprint arXiv:2302.06871, 2023.
32. J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023. [find](https://scholar.google.com/scholar?q=J%2E+S%2E+Park%2C+J%2E+O%E2%80%99Brien%2C+C%2E+J%2E+Cai%2C+M%2E+R%2E+Morris%2C+P%2E+Liang%2C+and+M%2E+S%2E+Bernstein%2E+Generative+agents%3A+Interactive+simulacra+of+human+behavior%2E+In+Proceedings+of+the+36th+Annual+ACM+Symposium+on+User+Inte)
33. J. S. Park, L. Popowski, C. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. [Social simulacra: Creating populated prototypes for social computing systems](https://md.research.elamur.ai/free/en/papers/park-2022-social-simulacra-creating-populated-prototypes/). In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pages 1–18, 2022.
34. T. Phung, V.-A. Pădurean, A. Singh, C. Brooks, J. Cambronero, S. Gulwani, A. Singla, and G. Soares. Automating human tutor-style programming feedback: Leveraging gpt-4 tutor model for hint generation and gpt-3.5 student model for hint validation. arXiv preprint arXiv:2310.03780, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Phung%2C+V%2E%2DA%2E+Pa%CC%86durean%2C+A%2E+Singh%2C+C%2E+Brooks%2C+J%2E+Cambronero%2C+S%2E+Gulwani%2C+A%2E+Singla%2C+and+G%2E+Soares%2E+Automating+human+tutor%2Dstyle+programming+feedback%3A+Leveraging+gpt%2D4+tutor+model+for+hint+generation)
35. E. Prihar, M. Lee, M. Hopman, A. T. Kalai, S. Vempala, A. Wang, G. Wickline, A. Murray, and N. Heffernan. Comparing different approaches to generating mathematics explanations using large language models. In International Conference on Artificial Intelligence in Education, pages 290–295. Springer, 2… [find](https://scholar.google.com/scholar?q=E%2E+Prihar%2C+M%2E+Lee%2C+M%2E+Hopman%2C+A%2E+T%2E+Kalai%2C+S%2E+Vempala%2C+A%2E+Wang%2C+G%2E+Wickline%2C+A%2E+Murray%2C+and+N%2E+Heffernan%2E+Comparing+different+approaches+to+generating+mathematics+explanations+using+large+language+mod)
36. R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.[03495, 2023](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/).
37. A. V. Riabko and T. A. Vakaliuk. Physics on autopilot: exploring the use of an ai assistant for independent problem-solving practice. Educational Technology Quarterly, 2024. [find](https://scholar.google.com/scholar?q=A%2E+V%2E+Riabko+and+T%2E+A%2E+Vakaliuk%2E+Physics+on+autopilot%3A+exploring+the+use+of+an+ai+assistant+for+independent+problem%2Dsolving+practice%2E+Educational+Technology+Quarterly%2C+2024%2E)
38. S. Sarsa, P. Denny, A. Hellas, and J. Leinonen. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1, pages 27–43, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Sarsa%2C+P%2E+Denny%2C+A%2E+Hellas%2C+and+J%2E+Leinonen%2E+Automatic+generation+of+programming+exercises+and+code+explanations+using+large+language+models%2E+In+Proceedings+of+the+2022+ACM+Conference+on+Internatio)
39. R. Schmucker, M. Xia, A. Azaria, and T. Mitchell. Ruffle&riley: Towards the automated induction of conversational tutoring systems. arXiv preprint arXiv:2310.01420, 2023. — [arXiv:2310.01420](https://arxiv.org/abs/2310.01420) [find](https://scholar.google.com/scholar?q=R%2E+Schmucker%2C+M%2E+Xia%2C+A%2E+Azaria%2C+and+T%2E+Mitchell%2E+Ruffle%26riley%3A+Towards+the+automated+induction+of+conversational+tutoring+systems%2E+arXiv+preprint+arXiv%3A2310%2E01420%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E01420%5D%28https%3A%2F%2F)
40. O. Shaikh, V. Chai, M. J. Gelfand, D. Yang, and M. S. Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. arXiv preprint arXiv:2309.12309, 2023. [find](https://scholar.google.com/scholar?q=O%2E+Shaikh%2C+V%2E+Chai%2C+M%2E+J%2E+Gelfand%2C+D%2E+Yang%2C+and+M%2E+S%2E+Bernstein%2E+Rehearsal%3A+Simulating+conflict+to+teach+conflict+resolution%2E+arXiv+preprint+arXiv%3A2309%2E12309%2C+2023%2E)
41. P. L. Smith and T. J. Ragan. Instructional design. John Wiley & Sons, 2004. [find](https://scholar.google.com/scholar?q=P%2E+L%2E+Smith+and+T%2E+J%2E+Ragan%2E+Instructional+design%2E+John+Wiley+%26+Sons%2C+2004%2E)
42. J. Sweller, J. J. van Merriënboer, and F. Paas. Cognitive architecture and instructional design: 20 years later. Educational psychology review, 31:261–292, 2019. [find](https://scholar.google.com/scholar?q=J%2E+Sweller%2C+J%2E+J%2E+van+Merrie%CC%88nboer%2C+and+F%2E+Paas%2E+Cognitive+architecture+and+instructional+design%3A+20+years+later%2E+Educational+psychology+review%2C+31%3A261%E2%80%93292%2C+2019%2E)
43. J. E. Tuovinen and J. Sweller. A comparison of cognitive load associated with discovery learning and worked examples. Journal of educational psychology, 91(2):334, 1999. [find](https://scholar.google.com/scholar?q=J%2E+E%2E+Tuovinen+and+J%2E+Sweller%2E+A+comparison+of+cognitive+load+associated+with+discovery+learning+and+worked+examples%2E+Journal+of+educational+psychology%2C+91%282%29%3A334%2C+1999%2E)
44. K. VanLehn, S. Ohlsson, and R. Nason. Applications of simulated students: An exploration. Journal of artificial intelligence in education, 5:135–135, 1994. [find](https://scholar.google.com/scholar?q=K%2E+VanLehn%2C+S%2E+Ohlsson%2C+and+R%2E+Nason%2E+Applications+of+simulated+students%3A+An+exploration%2E+Journal+of+artificial+intelligence+in+education%2C+5%3A135%E2%80%93135%2C+1994%2E)
45. R. E. Wang and D. Demszky. [Is chatgpt a good teacher coach? measuring zero-shot performance for scoring and providing actionable insights on classroom instruction](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-teacher-coach-measuring/). arXiv preprint arXiv:2306.03090, 2023.
46. R. E. Wang, Q. Zhang, C. Robinson, S. Loeb, and D. Demszky. Step-by-step remediation of students’ mathematical mistakes. arXiv preprint arXiv:2310.10648, 2023. [find](https://scholar.google.com/scholar?q=R%2E+E%2E+Wang%2C+Q%2E+Zhang%2C+C%2E+Robinson%2C+S%2E+Loeb%2C+and+D%2E+Demszky%2E+Step%2Dby%2Dstep+remediation+of+students%E2%80%99+mathematical+mistakes%2E+arXiv+preprint+arXiv%3A2310%2E10648%2C+2023%2E)
47. Z. Wang, J. Valdez, D. Basu Mallick, and R. G. Baraniuk. Towards human-like educational question generation with large language models. In International conference on artificial intelligence in education, pages 153–166. Springer, 2022. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+J%2E+Valdez%2C+D%2E+Basu+Mallick%2C+and+R%2E+G%2E+Baraniuk%2E+Towards+human%2Dlike+educational+question+generation+with+large+language+models%2E+In+International+conference+on+artificial+intelligence+in+educat)
48. D. D. Williams, J. B. South, S. C. Yanchar, B. G. Wilson, and S. Allen. How do instructional designers evaluate? a qualitative study of evaluation in practice. Educational Technology Research and Development, 59:885–907, 2011. [find](https://scholar.google.com/scholar?q=D%2E+D%2E+Williams%2C+J%2E+B%2E+South%2C+S%2E+C%2E+Yanchar%2C+B%2E+G%2E+Wilson%2C+and+S%2E+Allen%2E+How+do+instructional+designers+evaluate%3F+a+qualitative+study+of+evaluation+in+practice%2E+Educational+Technology+Research+and+Deve)
49. C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). arXiv preprint arXiv:2309.03409, 2023.
50. E. Zelikman, E. Lorch, L. Mackey, and A. T. Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023. — [arXiv:2310.02304](https://arxiv.org/abs/2310.02304) [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+E%2E+Lorch%2C+L%2E+Mackey%2C+and+A%2E+T%2E+Kalai%2E+Self%2Dtaught+optimizer+%28stop%29%3A+Recursively+self%2Dimproving+code+generation%2E+arXiv+preprint+arXiv%3A2310%2E02304%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E02304%5D%28https%3A%2F%2Farxiv%2Eo)
51. E. Zelikman, W. A. Ma, J. E. Tran, D. Yang, J. D. Yeatman, and N. Haber. Generating and evaluating tests for k-12 students with language model simulations: A case study on sentence reading efficiency. arXiv preprint arXiv:2310.06837, 2023. [find](https://scholar.google.com/scholar?q=E%2E+Zelikman%2C+W%2E+A%2E+Ma%2C+J%2E+E%2E+Tran%2C+D%2E+Yang%2C+J%2E+D%2E+Yeatman%2C+and+N%2E+Haber%2E+Generating+and+evaluating+tests+for+k%2D12+students+with+language+model+simulations%3A+A+case+study+on+sentence+reading+efficiency%2E)

*Updated 3 November 2026*