---
title: "Benchmarking Foundation Models — paper by Bai et al. (2023)"
description: "Bai et al. propose Language-Model-as-an-Examiner, in which a strong LM both writes the questions… 64 references, cited by 78 papers here."
authors: ["Yushi Bai", "Jiahao Ying", "Yixin Cao", "Xin Lv", "Yuze He", "Xiaozhi Wang", "Jifan Yu", "Kaisheng Zeng", "Yijia Xiao", "Haozhe Lyu", "Jiayin Zhang", "Juanzi Li", "Lei Hou"]
year: 2023
venue: "arXiv:2306.04181"
source: "https://arxiv.org/abs/2306.04181"
doi: "10.48550/arxiv.2306.04181"
arxiv: "2306.04181"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/bai-2023-benchmarking-foundation-models-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bai-2023-benchmarking-foundation-models-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Benchmarking Foundation Models with Language-Model-as-an-Examiner

# Benchmarking Foundation Models with Language-Model-as-an-Examiner

**Authors:** [Yushi Bai](https://md.research.elamur.ai/free/en/authors/yushi-bai/), [Jiahao Ying](https://md.research.elamur.ai/free/en/authors/jiahao-ying/), [Yixin Cao](https://md.research.elamur.ai/free/en/authors/yixin-cao/), [Xin Lv](https://md.research.elamur.ai/free/en/authors/xin-lv/), [Yuze He](https://md.research.elamur.ai/free/en/authors/yuze-he/), [Xiaozhi Wang](https://md.research.elamur.ai/free/en/authors/xiaozhi-wang/), [Jifan Yu](https://md.research.elamur.ai/free/en/authors/jifan-yu/), [Kaisheng Zeng](https://md.research.elamur.ai/free/en/authors/kaisheng-zeng/), [Yijia Xiao](https://md.research.elamur.ai/free/en/authors/yijia-xiao/), Haozhe Lyu, Jiayin Zhang, [Juanzi Li](https://md.research.elamur.ai/free/en/authors/juanzi-li/) and [Lei Hou](https://md.research.elamur.ai/free/en/authors/lei-hou/) · **Year:** 2023 · **Venue:** [arXiv:2306.04181](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/) ([NeurIPS 2023](https://md.research.elamur.ai/free/en/venues/neural-information-processing-systems/2023/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.04181) · [PDF](https://arxiv.org/pdf/2306.04181) · [DOI 10.48550/arxiv.2306.04181](https://doi.org/10.48550/arxiv.2306.04181) · [arXiv 2306.04181](https://arxiv.org/abs/2306.04181) · [OpenAlex](https://openalex.org/W7133209743)

**Citations:** 28 citations (OpenAlex, 2026-10-14) · [78](https://md.research.elamur.ai/free/en/papers/bai-2023-benchmarking-foundation-models-language-model/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Open-ended question answering benchmarks face two obstacles. Test items may already appear in pre-training data, which inflates scores. Free-form answers are also hard to grade automatically because no single reference covers every acceptable response. Bai et al. propose Language-Model-as-an-Examiner, in which a strong LM both writes the questions and grades other models' answers without references. Since the examiner creates questions from its own knowledge, it is better equipped to judge them, and fresh question sets can be regenerated if leakage becomes a concern. With GPT-4 as examiner, the authors build LMExamQA. It contains questions drawn from many Google Trends domains, spread over three cognitive levels, plus multi-turn follow-ups that probe depth. Grading combines Likert-scale scores with response rankings, which the authors report match human judgments much better than earlier metrics. To reduce the bias of a single examiner, a decentralized peer-examination scheme lets several models examine one another. Eight popular foundation models are benchmarked, offering a scalable and renewable alternative to static, human-curated QA sets.

## Key points
- Motivating pilot study: GPT-4 acting only as a judge of LLaMA answers on Natural Questions made wrong calls in 18 of 100 cases. The authors attribute this mainly to gaps in the judge's own knowledge.
- LMExamQA uses the Google Trends Categories as its domain taxonomy. Questions are labeled by Bloom's taxonomy levels as memorization (22%), comprehension (43%) and analysis (35%). This mix is more balanced and harder than MS MARCO, SQuAD2.0, Natural Questions or ELI5.
- Leakage check: questions were compared with Google search results (100 sampled questions) and with the Flan collection (1,836 tasks, 15M instances) via BM25 retrieval. Average ROUGE-L to the closest match was only 0.293 for web data and 0.046 for Flan.
- Peer-examination produces more diverse questions and evens out individual examiners' preferences. However, only four accessible models were judged capable enough to act as examiners.
- Published at NeurIPS 2023 (Datasets and Benchmarks track). Data and results are available at lmexam.xlore.cn.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🎓 Education](https://md.research.elamur.ai/free/en/shelves/social-sciences/education/) · 212 works

## Cite

Cite as: Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li and Lei Hou. 2023. Benchmarking Foundation Models with Language-Model-as-an-Examiner. arXiv:2306.04181.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{bai2023benchmarking,
  title = {{Benchmarking Foundation Models with Language-Model-as-an-Examiner}},
  author = {Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li and Lei Hou},
  year = {2023},
  doi = {10.48550/arxiv.2306.04181},
  eprint = {2306.04181},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.04181}
}
```

- [← previous M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection](https://md.research.elamur.ai/free/en/papers/wang-2023-m4-multi-generator-multi-domain/)
- [next → Complementary Explanations for Effective In-Context Learning](https://md.research.elamur.ai/free/en/papers/ye-2023-complementary-explanations-effective-context-learning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

64 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., “[On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/),” arXiv preprint arXiv:2108.07258, 2021.
2. OpenAI, “Introducing chatgpt,” 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2C+%E2%80%9CIntroducing+chatgpt%2C%E2%80%9D+2022%2E)
3. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “[Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/),” arXiv preprint arXiv:2302.13971, 2023.
4. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al., “[Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/),” arXiv preprint arXiv:2204.02311, 2022.
5. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “[Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/),” ACM Computing Surveys, vol. 55, no. 12, pp. 1–38, 2023.
6. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “[Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/),” Advances in neural information processing systems, vol. 32, 2019.
7. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “[Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/),” in International Conference on Learning Representations, 2021.
8. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al., “Holistic evaluation of language models,” arXiv preprint arXiv:2211.09110, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+R%2E+Bommasani%2C+T%2E+Lee%2C+D%2E+Tsipras%2C+D%2E+Soylu%2C+M%2E+Yasunaga%2C+Y%2E+Zhang%2C+D%2E+Narayanan%2C+Y%2E+Wu%2C+A%2E+Kumar%2C+et+al%2E%2C+%E2%80%9CHolistic+evaluation+of+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2211%2E09110%2C+2022%2E)
9. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan, “[Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/),” arXiv preprint arXiv:2304.06364, 2023.
10. J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li, et al., “Kola: Carefully benchmarking world knowledge of large language models,” arXiv preprint arXiv:2306.09296, 2023. — [arXiv:2306.09296](https://arxiv.org/abs/2306.09296) [find](https://scholar.google.com/scholar?q=J%2E+Yu%2C+X%2E+Wang%2C+S%2E+Tu%2C+S%2E+Cao%2C+D%2E+Zhang%2DLi%2C+X%2E+Lv%2C+H%2E+Peng%2C+Z%2E+Yao%2C+X%2E+Zhang%2C+H%2E+Li%2C+et+al%2E%2C+%E2%80%9CKola%3A+Carefully+benchmarking+world+knowledge+of+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2306%2E09296%2C+2)
11. Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, “[Longbench: A bilingual, multitask benchmark for long context understanding](https://md.research.elamur.ai/free/en/papers/bai-2023-longbench-bilingual-multitask-benchmark/),” arXiv preprint arXiv:2308.14508, 2023.
12. A. Rogers, M. Gardner, and I. Augenstein, “Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension,” ACM Computing Surveys, vol. 55, no. 10, pp. 1–45, 2023. [find](https://scholar.google.com/scholar?q=A%2E+Rogers%2C+M%2E+Gardner%2C+and+I%2E+Augenstein%2C+%E2%80%9CQa+dataset+explosion%3A+A+taxonomy+of+nlp+resources+for+question+answering+and+reading+comprehension%2C%E2%80%9D+ACM+Computing+Surveys%2C+vol%2E+55%2C+no%2E+10%2C+pp%2E+1%E2%80%9345%2C+2023%2E)
13. C.-Y. Lin, “[Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/),” in Text summarization branches out, pp. 74–81, 2004.
14. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” in International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Zhang%2C+V%2E+Kishore%2C+F%2E+Wu%2C+K%2E+Q%2E+Weinberger%2C+and+Y%2E+Artzi%2C+%E2%80%9CBertscore%3A+Evaluating+text+generation+with+bert%2C%E2%80%9D+in+International+Conference+on+Learning+Representations%2C+2020%2E)
15. A. Chen, G. Stanovsky, S. Singh, and M. Gardner, “Evaluating question answering evaluation,” in Proceedings of the 2nd workshop on machine reading for question answering, pp. 119–124, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Chen%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2C+%E2%80%9CEvaluating+question+answering+evaluation%2C%E2%80%9D+in+Proceedings+of+the+2nd+workshop+on+machine+reading+for+question+answering%2C+pp%2E+119%E2%80%93124%2C+2019%2E)
16. K. Krishna, A. Roy, and M. Iyyer, “[Hurdles to progress in long-form question answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/),” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
17. J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” arXiv preprint arXiv:2302.04166, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Fu%2C+S%2E%2DK%2E+Ng%2C+Z%2E+Jiang%2C+and+P%2E+Liu%2C+%E2%80%9CGptscore%3A+Evaluate+as+you+desire%2C%E2%80%9D+arXiv+preprint+arXiv%3A2302%2E04166%2C+2023%2E)
18. Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “Gpteval: Nlg evaluation using gpt-4 with better human alignment,” arXiv preprint arXiv:2303.[16634, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/).
19. J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou, “[Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/),” arXiv preprint arXiv:2303.04048, 2023.
20. OpenAI, “Openai: Gpt-4,” 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2C+%E2%80%9COpenai%3A+Gpt%2D4%2C%E2%80%9D+2023%2E)
21. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al., “[Natural questions: a benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/),” Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019.
22. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “[Ms marco: A human generated machine reading comprehension dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/),” choice, vol. 2640, p. 660, 2016.
23. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “[Squad: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/),” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, 2016.
24. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784–789, 2018. — [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
25. J. Berant, A. Chou, R. Frostig, and P. Liang, “[Semantic parsing on freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/),” in Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533–1544, 2013.
26. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, 2018. [find](https://scholar.google.com/scholar?q=T%2E+Mihaylov%2C+P%2E+Clark%2C+T%2E+Khot%2C+and+A%2E+Sabharwal%2C+%E2%80%9CCan+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2C%E2%80%9D+in+Proceedings+of+the+2018+Conference+on+Empirical+Methods+)
27. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli, “[Eli5: Long form question answering](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558–3567, 2019.
28. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “[Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/),” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
29. S. Banerjee and A. Lavie, “[Meteor: An automatic metric for mt evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/),” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65–72, 2005.
30. A. Chen, G. Stanovsky, S. Singh, and M. Gardner, “Mocha: A dataset for training and evaluating generative reading comprehension metrics,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6521–6532, 2020. [find](https://scholar.google.com/scholar?q=A%2E+Chen%2C+G%2E+Stanovsky%2C+S%2E+Singh%2C+and+M%2E+Gardner%2C+%E2%80%9CMocha%3A+A+dataset+for+training+and+evaluating+generative+reading+comprehension+metrics%2C%E2%80%9D+in+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+)
31. C. Si, C. Zhao, and J. Boyd-Graber, “What’s in a name? answer equivalence for [open-domain question answering](https://md.research.elamur.ai/free/en/papers/chen-2020-open-domain-question-answering/),” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9623–9629, 2021.
32. T. Sellam, D. Das, and A. Parikh, “[Bleurt: Learning robust metrics for text generation](https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/),” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881–7892, 2020.
33. R. Rei, C. Stewart, A. C. Farinha, and A. Lavie, “[Comet: A neural framework for mt evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/),” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685–2702, 2020.
34. W. Yuan, G. Neubig, and P. Liu, “[Bartscore: Evaluating generated text as text generation](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/),” Advances in Neural Information Processing Systems, vol. 34, pp. 27263–27277, 2021.
35. A. B. Sai, A. K. Mohankumar, and M. M. Khapra, “A survey of evaluation metrics used for nlg systems,” ACM Computing Surveys (CSUR), vol. 55, no. 2, pp. 1–39, 2022. [find](https://scholar.google.com/scholar?q=A%2E+B%2E+Sai%2C+A%2E+K%2E+Mohankumar%2C+and+M%2E+M%2E+Khapra%2C+%E2%80%9CA+survey+of+evaluation+metrics+used+for+nlg+systems%2C%E2%80%9D+ACM+Computing+Surveys+%28CSUR%29%2C+vol%2E+55%2C+no%2E+2%2C+pp%2E+1%E2%80%9339%2C+2022%2E)
36. W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, and S. Eger, “[Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance](https://md.research.elamur.ai/free/en/papers/zhao-2019-moverscore-text-generation-evaluating-contextualized/),” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
37. S. Gehrmann, E. Clark, and T. Sellam, “Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text,” arXiv preprint arXiv:2202.06935, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Gehrmann%2C+E%2E+Clark%2C+and+T%2E+Sellam%2C+%E2%80%9CRepairing+the+cracked+foundation%3A+A+survey+of+obstacles+in+evaluation+practices+for+generated+text%2C%E2%80%9D+arXiv+preprint+arXiv%3A2202%2E06935%2C+2022%2E)
38. M. Zhong, Y. Liu, D. Yin, Y. Mao, Y. Jiao, P. Liu, C. Zhu, H. Ji, and J. Han, “[Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/),” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2023–2038, 2022.
39. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. [find](https://scholar.google.com/scholar?q=W%2E%2DL%2E+Chiang%2C+Z%2E+Li%2C+Z%2E+Lin%2C+Y%2E+Sheng%2C+Z%2E+Wu%2C+H%2E+Zhang%2C+L%2E+Zheng%2C+S%2E+Zhuang%2C+Y%2E+Zhuang%2C+J%2E+E%2E+Gonzalez%2C+I%2E+Stoica%2C+and+E%2E+P%2E+Xing%2C+%E2%80%9CVicuna%3A+An+open%2Dsource+chatbot+impressing+gpt%2D4+with+90%25%2A+chatgpt+qu)
40. Y. Chen, R. Wang, H. Jiang, S. Shi, and R. Xu, “Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study,” arXiv preprint arXiv:2304.00723, 2023. — [arXiv:2304.00723](https://arxiv.org/abs/2304.00723) [find](https://scholar.google.com/scholar?q=Y%2E+Chen%2C+R%2E+Wang%2C+H%2E+Jiang%2C+S%2E+Shi%2C+and+R%2E+Xu%2C+%E2%80%9CExploring+the+use+of+large+language+models+for+reference%2Dfree+text+quality+evaluation%3A+A+preliminary+empirical+study%2C%E2%80%9D+arXiv+preprint+arXiv%3A2304%2E00723%2C+)
41. W. Yidong, Y. Zhuohao, Z. Zhengran, Y. Linyi, H. Qiang, W. Cunxiang, C. Hao, J. Chaoya, X. Rui, W. Jindong, X. Xing, Y. Wei, Z. Shikun, and Z. Yue, “Pandalm: Reproducible and automated language model assessment.” [https://github.com/WeOpenML/PandaLM](https://github.com/WeOpenML/PandaLM), 2023. [↗](https://github.com/WeOpenML/PandaLM)
42. M. Gao, J. Ruan, R. Sun, X. Yin, S. Yang, and X. Wan, “[Human-like summarization evaluation with chatgpt](https://md.research.elamur.ai/free/en/papers/gao-2023-human-like-summarization-evaluation-chatgpt/),” arXiv preprint arXiv:2304.02554, 2023.
43. A. R. Fabbri, W. Kryściński, B. McCann, C. Xiong, R. Socher, and D. Radev, “Summeval: Re-evaluating summarization evaluation,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 391–409, 2021. [find](https://scholar.google.com/scholar?q=A%2E+R%2E+Fabbri%2C+W%2E+Krys%CC%81cin%CC%81ski%2C+B%2E+McCann%2C+C%2E+Xiong%2C+R%2E+Socher%2C+and+D%2E+Radev%2C+%E2%80%9CSummeval%3A+Re%2Devaluating+summarization+evaluation%2C%E2%80%9D+Transactions+of+the+Association+for+Computational+Linguistics%2C+vol%2E+9%2C+)
44. H. Chen, D. M. Vo, H. Takamura, Y. Miyao, and H. Nakayama, “Storyer: Automatic story evaluation via ranking, rating and reasoning,” Journal of Natural Language Processing, vol. 30, no. 1, pp. 243–249, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Chen%2C+D%2E+M%2E+Vo%2C+H%2E+Takamura%2C+Y%2E+Miyao%2C+and+H%2E+Nakayama%2C+%E2%80%9CStoryer%3A+Automatic+story+evaluation+via+ranking%2C+rating+and+reasoning%2C%E2%80%9D+Journal+of+Natural+Language+Processing%2C+vol%2E+30%2C+no%2E+1%2C+pp%2E+243%E2%80%93249%2C)
45. B. S. Bloom and D. R. Krathwohl, Taxonomy of educational objectives: The classification of educational goals. Book 1, Cognitive domain. longman, 2020. [find](https://scholar.google.com/scholar?q=B%2E+S%2E+Bloom+and+D%2E+R%2E+Krathwohl%2C+Taxonomy+of+educational+objectives%3A+The+classification+of+educational+goals%2E+Book+1%2C+Cognitive+domain%2E+longman%2C+2020%2E)
46. E. Choi, H. He, M. Iyyer, M. Yatskar, W.-t. Yih, Y. Choi, P. Liang, and L. Zettlemoyer, “[Quac: Question answering in context](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/),” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2174–2184, 2018.
47. S. Reddy, D. Chen, and C. D. Manning, “[Coqa: A conversational question answering challenge](https://md.research.elamur.ai/free/en/papers/reddy-2019-coqa-conversational-question-answering-challenge/),” Transactions of the Association for Computational Linguistics, vol. 7, pp. 249–266, 2019.
48. J. A. Campos, A. Otegi, A. Soroa, J. M. Deriu, M. Cieliebak, and E. Agirre, “Doqa-accessing domain-specific faqs via conversational qa,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7302–7314, 2020. [find](https://scholar.google.com/scholar?q=J%2E+A%2E+Campos%2C+A%2E+Otegi%2C+A%2E+Soroa%2C+J%2E+M%2E+Deriu%2C+M%2E+Cieliebak%2C+and+E%2E+Agirre%2C+%E2%80%9CDoqa%2Daccessing+domain%2Dspecific+faqs+via+conversational+qa%2C%E2%80%9D+in+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+fo)
49. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al., “Scaling instruction-finetuned language models,” arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+S%2E+Longpre%2C+B%2E+Zoph%2C+Y%2E+Tay%2C+W%2E+Fedus%2C+E%2E+Li%2C+X%2E+Wang%2C+M%2E+Dehghani%2C+S%2E+Brahma%2C+et+al%2E%2C+%E2%80%9CScaling+instruction%2Dfinetuned+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2210%2E11416%2C+2022%2E)
50. OpenAI, “[Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/),” arXiv preprint arXiv:2303.08774, 2023.
51. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al., “[Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/),” arXiv preprint arXiv:2303.12712, 2023.
52. Anthropic, “Anthropic: Claude,” 2023. [find](https://scholar.google.com/scholar?q=Anthropic%2C+%E2%80%9CAnthropic%3A+Claude%2C%E2%80%9D+2023%2E)
53. Google, “Google: Bard,” 2023. [find](https://scholar.google.com/scholar?q=Google%2C+%E2%80%9CGoogle%3A+Bard%2C%E2%80%9D+2023%2E)
54. N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. L. Scao, M. S. Bari, S. Shen, Z.-X. Yong, H. Schoelkopf, et al., “[Crosslingual generalization through multitask finetuning](https://md.research.elamur.ai/free/en/papers/muennighoff-2022-crosslingual-generalization-through-multitask-finetuning/),” arXiv preprint arXiv:2211.01786, 2022.
55. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al., “Scaling instruction-finetuned language models,” arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+S%2E+Longpre%2C+B%2E+Zoph%2C+Y%2E+Tay%2C+W%2E+Fedus%2C+E%2E+Li%2C+X%2E+Wang%2C+M%2E+Dehghani%2C+S%2E+Brahma%2C+et+al%2E%2C+%E2%80%9CScaling+instruction%2Dfinetuned+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2210%2E11416%2C+2022%2E)
56. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y. Xue, J. Zhai, W. Chen, Z. Liu, P. Zhang, Y. Dong, and J. Tang, “[GLM-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/),” in The Eleventh International Conference on Learning Representations, 2023.
57. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “[Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/),” arXiv preprint arXiv:2001.08361, 2020.
58. LMSYS, “Lmsys org: Chatbot arena,” 2023. [find](https://scholar.google.com/scholar?q=LMSYS%2C+%E2%80%9CLmsys+org%3A+Chatbot+arena%2C%E2%80%9D+2023%2E)
59. OpenAI, “Openai: New and improved embedding model,” 2022. [find](https://scholar.google.com/scholar?q=OpenAI%2C+%E2%80%9COpenai%3A+New+and+improved+embedding+model%2C%E2%80%9D+2022%2E)
60. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+I%2E+Sutskever%2C+et+al%2E%2C+%E2%80%9CLanguage+models+are+unsupervised+multitask+learners%2C%E2%80%9D)
61. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language model with self generated instructions,” arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Kordi%2C+S%2E+Mishra%2C+A%2E+Liu%2C+N%2E+A%2E+Smith%2C+D%2E+Khashabi%2C+and+H%2E+Hajishirzi%2C+%E2%80%9CSelf%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2C%E2%80%9D+arXiv+preprint+arXiv%3A2212%2E10560%2C+2022%2E)
62. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model.” [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
63. S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, et al., “[The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/),” arXiv preprint arXiv:2301.13688, 2023.
64. Y. Xiao, Y. Jin, Y. Bai, Y. Wu, X. Yang, X. Luo, W. Yu, X. Zhao, Y. Liu, H. Chen, et al., “Large language models can be good privacy protection learners,” arXiv preprint arXiv:2310.02469, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Xiao%2C+Y%2E+Jin%2C+Y%2E+Bai%2C+Y%2E+Wu%2C+X%2E+Yang%2C+X%2E+Luo%2C+W%2E+Yu%2C+X%2E+Zhao%2C+Y%2E+Liu%2C+H%2E+Chen%2C+et+al%2E%2C+%E2%80%9CLarge+language+models+can+be+good+privacy+protection+learners%2C%E2%80%9D+arXiv+preprint+arXiv%3A2310%2E02469%2C+2023%2E)

*Updated 2 November 2026*