---
title: "The AI teacher test — paper by Tack (2022) — summary and ideas"
description: "Tack and Piech propose a pilot \"AI teacher test\". 20 references, cited by 52 papers here. Among them: 36."
authors: ["Tack"]
year: 2022
venue: "arXiv (Cornell University)"
source: "https://arxiv.org/abs/2205.07540"
arxiv: "2205.07540"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "https://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tack-2022-ai-teacher-test-measuring-pedagogical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tack-2022-ai-teacher-test-measuring-pedagogical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / The AI teacher test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues

# The AI teacher test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues

**Authors:** Tack · **Year:** 2022 · **Venue:** arXiv (Cornell University) · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](https://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2205.07540) · [PDF](https://arxiv.org/pdf/2205.07540) · [arXiv 2205.07540](https://arxiv.org/abs/2205.07540) · [OpenAlex](https://openalex.org/W4280504808)

**Citations:** 36 citations (OpenAlex, 2026-10-14) · [52](https://md.research.elamur.ai/free/en/papers/tack-2022-ai-teacher-test-measuring-pedagogical/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks how to check whether large generative chatbots can act as competent teachers when responding to learners. Existing metrics such as perplexity, BLEU/F1 and MAUVE either ignore the dialogue context, require a single correct reply, or cannot compare two individual utterances. None of them score pedagogical qualities. Tack and Piech propose a pilot "AI teacher test". Blender and GPT-3 are placed in real student-teacher conversations at the point where the human teacher replied, and they produce alternative responses. Paid human raters then judge pairs of replies on three dimensions: sounding like a teacher, understanding the student, and helping the student. A Bayesian Bradley-Terry model turns these pairwise judgments into ability estimates with uncertainty. Both models score significantly below human teachers on all three dimensions, and the largest gap is in helpfulness. Blender does relatively well on understanding and on conversational uptake. The work matters because it offers an early, reproducible way to measure teaching quality. The authors release its code and data.

## Key points
- **Data sources:**
  - Teacher-Student Chatroom Corpus (TSCC): 102 second-language English lessons, 4,439 dialogic pairs after merging consecutive same-speaker turns.
  - Educational Uptake Dataset by Demszky et al.: 2,246 mathematics pairs drawn from NCTE transcripts.
- **Ability gaps versus human teachers (Δ ability):**
  - Blender 9B: −0.60 (speaking like a teacher), −0.55 (understanding), −0.75 (helping).
  - GPT-3 Davinci: −0.67, −0.67 and −0.93 on the same three dimensions.
  - All of these gaps are significant at p \< .001.
- **Blender versus GPT-3:** Blender is slightly ahead on understanding and helping, but the differences between the two models are not statistically significant.
- **Link to automated uptake:** The ability estimates correlate moderately with the BERT-based uptake predictor of Demszky et al. (Pearson r = .33–.38). The strongest correlation is for understanding the student.
- **Per-item results:** Human teacher replies receive positive estimates in 69–78% of contexts, so 22–31% are rated negatively. Generated replies are rated positively in 33–45% of contexts. This suggests model output could be used to propose better replies.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 654 works

## Cite

Cite as: Tack. 2022. The AI teacher test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tack2022ai,
  title = {{The AI teacher test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues}},
  author = {Tack},
  year = {2022},
  eprint = {2205.07540},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.07540}
}
```

- [← previous MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://md.research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/)
- [next → Multi-lingual Evaluation of Code Generation Models](https://md.research.elamur.ai/free/en/papers/athiwaratkun-2022-multi-lingual-evaluation-code-generation/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Bibauw, W. Van den Noortgate, T. François, and P. Desmet. Dialogue systems for language learning: A meta-analysis. Language Learning & Technology, 26(1): accepted, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Bibauw%2C+W%2E+Van+den+Noortgate%2C+T%2E+Franc%CC%A7ois%2C+and+P%2E+Desmet%2E+Dialogue+systems+for+language+learning%3A+A+meta%2Danalysis%2E+Language+Learning+%26+Technology%2C+26%281%29%3A+accepted%2C+2022%2E)
2. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Etha… — [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
3. R. A. Bradley and M. E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324, Dec. 1952. ISSN 00063444. doi: 10.2307/2334029. — [doi:10.2307/2334029](https://doi.org/10.2307/2334029) [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Bradley+and+M%2E+E%2E+Terry%2E+Rank+Analysis+of+Incomplete+Block+Designs%3A+I%2E+The+Method+of+Paired+Comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%2C+Dec%2E+1952%2E+ISSN+00063444%2E+doi%3A+10%2E2307%2F2334029%2E+%E2%80%94+%5Bdoi%3A10%2E2307%2F2)
4. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. A. Caines, H. Yannakoudakis, H. Edmondson, H. Allen, P. Pérez-Paredes, B. Byrne, and P. Buttery. The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10–20, Gothenburg, Sweden, Nov. 2020. LiU Electronic Press. [find](https://scholar.google.com/scholar?q=A%2E+Caines%2C+H%2E+Yannakoudakis%2C+H%2E+Edmondson%2C+H%2E+Allen%2C+P%2E+Pe%CC%81rez%2DParedes%2C+B%2E+Byrne%2C+and+P%2E+Buttery%2E+The+teacher%2Dstudent+chatroom+corpus%2E+In+Proceedings+of+the+9th+Workshop+on+NLP+for+Computer+Assisted+L)
6. D. Demszky, J. Liu, Z. Mancenido, J. Cohen, H. Hill, D. Jurafsky, and T. Hashimoto. Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… [find](https://scholar.google.com/scholar?q=D%2E+Demszky%2C+J%2E+Liu%2C+Z%2E+Mancenido%2C+J%2E+Cohen%2C+H%2E+Hill%2C+D%2E+Jurafsky%2C+and+T%2E+Hashimoto%2E+Measuring+Conversational+Uptake%3A+A+Case+Study+on+Student%2DTeacher+Interactions%2E+In+Proceedings+of+the+59th+Annual+Mee)
7. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sh…
8. S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, Sept. 1987. ISSN 0370-2693. doi: 10.1016/0370-2693(87)91197-X. — [doi:10.1016/0370-2693(87)91197-x](https://doi.org/10.1016/0370-2693%2887%2991197-x) [find](https://scholar.google.com/scholar?q=S%2E+Duane%2C+A%2E+D%2E+Kennedy%2C+B%2E+J%2E+Pendleton%2C+and+D%2E+Roweth%2E+Hybrid+Monte+Carlo%2E+Physics+Letters+B%2C+195%282%29%3A216%E2%80%93222%2C+Sept%2E+1987%2E+ISSN+0370%2D2693%2E+doi%3A+10%2E1016%2F0370%2D2693%2887%2991197%2DX%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2F0370%2D2693%28)
9. L. Goe, C. Bell, and O. Little. Approaches to Evaluating Teacher Effectiveness: A Research Synthesis. National Comprehensive Center for Teacher Quality, June 2008. [find](https://scholar.google.com/scholar?q=L%2E+Goe%2C+C%2E+Bell%2C+and+O%2E+Little%2E+Approaches+to+Evaluating+Teacher+Effectiveness%3A+A+Research+Synthesis%2E+National+Comprehensive+Center+for+Teacher+Quality%2C+June+2008%2E)
10. S. Heldsinger and S. Humphry. Using the Method of Pairwise Comparison to Obtain Reliable Teacher Assessments. Australian Educational Researcher, 37(2): 1–19, Aug. 2010. ISSN 0311-6999. [find](https://scholar.google.com/scholar?q=S%2E+Heldsinger+and+S%2E+Humphry%2E+Using+the+Method+of+Pairwise+Comparison+to+Obtain+Reliable+Teacher+Assessments%2E+Australian+Educational+Researcher%2C+37%282%29%3A+1%E2%80%9319%2C+Aug%2E+2010%2E+ISSN+0311%2D6999%2E)
11. M. D. Homan and A. Gelman. The No-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. The Journal of Machine Learning Research, 15 (1):1593–1623, Jan. 2014. ISSN 1532-4435. [find](https://scholar.google.com/scholar?q=M%2E+D%2E+Homan+and+A%2E+Gelman%2E+The+No%2DU%2Dturn+sampler%3A+Adaptively+setting+path+lengths+in+Hamiltonian+Monte+Carlo%2E+The+Journal+of+Machine+Learning+Research%2C+15+%281%29%3A1593%E2%80%931623%2C+Jan%2E+2014%2E+ISSN+1532%2D4435%2E)
12. M. Lesterhuis, S. Verhavert, L. Coertjens, V. Donche, and S. De Maeyer. Comparative judgement as a promising alternative to score competences. In E. Cano, G. Ion, and J. Keengwe, editors, Innovative Practices for Higher Education Assessment and Measurement:, Advances in Higher Education and Professi… — [doi:10.4018/978-1-5225-0531-0](https://doi.org/10.4018/978-1-5225-0531-0) [find](https://scholar.google.com/scholar?q=M%2E+Lesterhuis%2C+S%2E+Verhavert%2C+L%2E+Coertjens%2C+V%2E+Donche%2C+and+S%2E+De+Maeyer%2E+Comparative+judgement+as+a+promising+alternative+to+score+competences%2E+In+E%2E+Cano%2C+G%2E+Ion%2C+and+J%2E+Keengwe%2C+editors%2C+Innovative+P)
13. A. Miller, W. Feng, D. Batra, A. Bordes, A. Fisch, J. Lu, D. Parikh, and J. Weston. [ParlAI: A dialog research software platform](https://md.research.elamur.ai/free/en/papers/miller-2017-parlai-dialog-research-software-platform/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen, Denmark, Sept. 2017. Association…
14. D. Muijs. Measuring teacher effectiveness: Some methodological reflections. Educational Research and Evaluation, 12(1):53–74, Feb. 2006. ISSN 1380-3611, 1744-4187. doi: 10.1080/13803610500392236. — [doi:10.1080/13803610500392236](https://doi.org/10.1080/13803610500392236) [find](https://scholar.google.com/scholar?q=D%2E+Muijs%2E+Measuring+teacher+effectiveness%3A+Some+methodological+reflections%2E+Educational+Research+and+Evaluation%2C+12%281%29%3A53%E2%80%9374%2C+Feb%2E+2006%2E+ISSN+1380%2D3611%2C+1744%2D4187%2E+doi%3A+10%2E1080%2F13803610500392236%2E+%E2%80%94+%5Bd)
15. K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui. [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://md.research.elamur.ai/free/en/papers/pillutla-2021-mauve-measuring-gap-between-neural/). In Advances in Neural Information Processing Systems 34 Pre-Proceedings (NeurIPS 2021), pages 1–35, Nov. 2021.
16. A. Riddell, A. Hartikainen, and M. Carter. PyStan (3.3.0). PyPI, Mar. 2021. [find](https://scholar.google.com/scholar?q=A%2E+Riddell%2C+A%2E+Hartikainen%2C+and+M%2E+Carter%2E+PyStan+%283%2E3%2E0%29%2E+PyPI%2C+Mar%2E+2021%2E)
17. S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, K. Shuster, E. M. Smith, Y.-L. Boureau, and J. Weston. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). 2020. doi: 10.48550/ARXIV.2004.13637.
18. E. M. Smith, M. Williamson, K. Shuster, J. Weston, and Y.-L. Boureau. Can you put it all together: Evaluating conversational agents’ ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2021[–2030, Online, July 2020](https://md.research.elamur.ai/free/en/papers/smith-2020-can-you-put-it-all/). Association fo…
19. Stan Development Team. Stan Modeling Language Users Guide and Reference Manual (v2.29.0), 2022. [find](https://scholar.google.com/scholar?q=Stan+Development+Team%2E+Stan+Modeling+Language+Users+Guide+and+Reference+Manual+%28v2%2E29%2E0%29%2C+2022%2E)
20. S. Wollny, J. Schneider, D. Di Mitri, J. Weidlich, M. Rittberger, and H. Drachsler. Are We There Yet? - A Systematic Literature Review on Chatbots in Education. Frontiers in Artificial Intelligence, 4:654924, July 2021. ISSN 2624-8212. doi: 10.3389/frai.2021.654924. — [doi:10.3389/frai.2021.654924](https://doi.org/10.3389/frai.2021.654924) [find](https://scholar.google.com/scholar?q=S%2E+Wollny%2C+J%2E+Schneider%2C+D%2E+Di+Mitri%2C+J%2E+Weidlich%2C+M%2E+Rittberger%2C+and+H%2E+Drachsler%2E+Are+We+There+Yet%3F+%2D+A+Systematic+Literature+Review+on+Chatbots+in+Education%2E+Frontiers+in+Artificial+Intelligence%2C+4)

*Updated 2 November 2026*