--- title: "Prometheus: Inducing Fine-grained Evaluation — Kim 2023" description: "Evaluating long-form LLM outputs is now often handed to GPT-4." authors: ["Seungone Kim", "Jamin Shin", "Yejin Cho", "Joel Jang", "Shayne Longpre", "Hwaran Lee", "Sangdoo Yun", "Seongjin Shin", "Sungdong Kim", "James Thorne", "Minjoon Seo"] year: 2023 venue: "arXiv:2310.08491" source: "https://arxiv.org/abs/2310.08491" doi: "10.48550/arxiv.2310.08491" arxiv: "2310.08491" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kim-2023-prometheus-inducing-fine-grained-evaluation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Prometheus: Inducing Fine-grained Evaluation Capability in Language Models # Prometheus: Inducing Fine-grained Evaluation Capability in Language Models **Authors:** [Seungone Kim](https://md.research.elamur.ai/free/en/authors/seungone-kim/), [Jamin Shin](https://md.research.elamur.ai/free/en/authors/jamin-shin/), Yejin Cho, [Joel Jang](https://md.research.elamur.ai/free/en/authors/joel-jang/), [Shayne Longpre](https://md.research.elamur.ai/free/en/authors/shayne-longpre/), [Hwaran Lee](https://md.research.elamur.ai/free/en/authors/hwaran-lee/), [Sangdoo Yun](https://md.research.elamur.ai/free/en/authors/sangdoo-yun/), Seongjin Shin, [Sungdong Kim](https://md.research.elamur.ai/free/en/authors/sungdong-kim/), [James Thorne](https://md.research.elamur.ai/free/en/authors/james-thorne/), [Minjoon Seo](https://md.research.elamur.ai/free/en/authors/minjoon-seo/) · **Year:** 2023 · **Venue:** arXiv:2310.08491 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2310.08491) · [PDF](https://arxiv.org/pdf/2310.08491) · [DOI 10.48550/arxiv.2310.08491](https://doi.org/10.48550/arxiv.2310.08491) · [arXiv 2310.08491](https://arxiv.org/abs/2310.08491) · [OpenAlex](https://openalex.org/W4387635896) **Citations:** 13 citations (openalex-snapshot · 2026-10-14) · [254](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Evaluating long-form LLM outputs is now often handed to GPT-4. Relying on a closed commercial judge causes problems: its internals are opaque, silent version changes hurt reproducibility, and large evaluation runs are expensive. Existing automatic judges also tend to score along a single generic axis such as helpfulness. Real users often care about custom criteria, such as humor or cultural sensitivity. The authors build the Feedback Collection, a GPT-4-generated training set. Each example combines an instruction and a candidate response with a user-style score rubric, a reference answer, written feedback and a 1–5 score. Fine-tuning Llama-2-Chat-13B on this data produces Prometheus, an open evaluator that writes a critique and then assigns a score for any rubric supplied with the input. When given the rubric and a reference answer, Prometheus agrees with human raters as well as GPT-4 does. It also outperforms dedicated reward models on human preference benchmarks. This makes it a cheap, transparent and reproducible option for customized evaluation. ## Key points - The Feedback Collection contains about 1K fine-grained score rubrics, 20K instructions and 100K responses with GPT-4-written feedback. Each example also includes reference answers, which the authors find important for inducing evaluation ability. - On 45 custom rubrics drawn from MT Bench, Vicuna Bench and Feedback Bench, Prometheus (13B) reaches a Pearson correlation of 0.897 with human scores. GPT-4 reaches 0.882 and GPT-3.5-Turbo 0.392. - In pairwise comparisons, human annotators preferred Prometheus's feedback over GPT-4's in 58.67% of cases. Against GPT-3.5-Turbo, its win rate was 79.57%. - Across 1222 rubrics on MT Bench, Vicuna Bench, Feedback Bench and Flask Eval, Prometheus correlates with GPT-4 judgments better than GPT-3.5-Turbo and Llama-2-Chat 70B do. - On HHH Alignment and MT Bench Human Judgment, Prometheus achieves the best accuracy among the compared open reward models and also beats GPT-3.5-Turbo. The code, dataset and model are publicly released. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) ## Cited by Cited by 254 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 240 not yet. - [Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth](https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-frontier-llms-arabic-cultural/) - [Aya 23: Open Weight Releases to Further Multilingual Progress](https://md.research.elamur.ai/free/en/papers/aryabumi-2024-aya-23-open-weight-releases/) - [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://md.research.elamur.ai/free/en/papers/dang-2024-aya-expanse-combining-research-breakthroughs/) - [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/) - [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://md.research.elamur.ai/free/en/papers/li-2024-llms-judges-comprehensive-survey-llm/) - [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/) - [EDRAC: Benchmarking Arabic Dialect Reading Comprehension](https://md.research.elamur.ai/free/en/papers/mokh-2026-edrac-benchmarking-arabic-dialect-reading/) - [Judging Instruction Responses in a Low-Resource Language: A Case Study on Basque](https://md.research.elamur.ai/free/en/papers/ponce-2026-judging-instruction-responses-low-resource/) - [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-assisting-writing-wikipedia-like-articles/) - [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/) - [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/) - [MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering](https://md.research.elamur.ai/free/en/papers/yim-2026-morqa-benchmarking-evaluation-metrics-medical/) - [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://md.research.elamur.ai/free/en/papers/zhang-2024-generative-verifiers-reward-modeling-as/) - [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{kim2023prometheus, title = {{Prometheus: Inducing Fine-grained Evaluation Capability in Language Models}}, author = {Kim, Seungone and Shin, Jamin and Cho, Yejin and Jang, Joel and Longpre, Shayne and Lee, Hwaran and Yun, Sangdoo and Shin, Seongjin and Kim, Sungdong and Thorne, James and Seo, Minjoon}, year = {2023}, doi = {10.48550/arxiv.2310.08491}, eprint = {2310.08491}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2310.08491} } ``` - [← previous Progress measures for grokking via mechanistic interpretability](https://md.research.elamur.ai/free/en/papers/nanda-2023-progress-measures-grokking-mechanistic-interpretability/) - [next → Prompt Injection attack against LLM-integrated Applications](https://md.research.elamur.ai/free/en/papers/liu-2023-prompt-injection-attack-against-llm/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) ## References 33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+Nelson+Elhage%2C+Zac+Hatfield%2DDodds%2C+Danny+Hernandez%2C+Jackson+Kernion%2C) 2. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109, 2023. [find](https://scholar.google.com/scholar?q=Yupeng+Chang%2C+Xu+Wang%2C+Jindong+Wang%2C+Yuan+Wu%2C+Kaijie+Zhu%2C+Hao+Chen%2C+Linyi+Yang%2C+Xiaoyuan+Yi%2C+Cunxiang+Wang%2C+Yidong+Wang%2C+et+al%2E+A+survey+on+evaluation+of+large+language+models%2E+arXiv+preprint+arXiv%3A23) [our card](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) 3. Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arXiv:2306.04757, 2023. [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Pengfei+Hong%2C+Lidong+Bing%2C+and+Soujanya+Poria%2E+Instructeval%3A+Towards+holistic+evaluation+of+instruction%2Dtuned+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E04757%2C+2023%2E) 4. Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evaluations?, 2023. [find](https://scholar.google.com/scholar?q=Cheng%2DHan+Chiang+and+Hung+yi+Lee%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F%2C+2023%2E) 5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…) 6. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Xuechen+Li%2C+Rohan+Taori%2C+Tianyi+Zhang%2C+Ishaan+Gulrajani%2C+Jimmy+Ba%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Alpacafarm%3A+A+simulation+framework+for+methods+that+learn+from+) [our card](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/) 7. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023. [find](https://scholar.google.com/scholar?q=Fabrizio+Gilardi%2C+Meysam+Alizadeh%2C+and+Mae%CC%88l+Kubli%2E+Chatgpt+outperforms+crowd%2Dworkers+for+text%2Dannotation+tasks%2E+arXiv+preprint+arXiv%3A2303%2E15056%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/) 8. Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022. [find](https://scholar.google.com/scholar?q=Namgyu+Ho%2C+Laura+Schmid%2C+and+Se%2DYoung+Yun%2E+Large+language+models+are+reasoning+teachers%2E+arXiv+preprint+arXiv%3A2212%2E10071%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/ho-2023-large-language-models-are-reasoning/) 9. Ari Holtzman, Peter West, and Luke Zettlemoyer. Generative models as a complex systems science: How can we make sense of large language model behavior? arXiv preprint arXiv:2308.00189, 2023. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Peter+West%2C+and+Luke+Zettlemoyer%2E+Generative+models+as+a+complex+systems+science%3A+How+can+we+make+sense+of+large+language+model+behavior%3F+arXiv+preprint+arXiv%3A2308%2E00189%2C+2023%2E) 10. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022. [find](https://scholar.google.com/scholar?q=Or+Honovich%2C+Thomas+Scialom%2C+Omer+Levy%2C+and+Timo+Schick%2E+Unnatural+instructions%3A+Tuning+language+models+with+%28almost%29+no+human+labor%2E+arXiv+preprint+arXiv%3A2212%2E09689%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/) 11. Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045, 2023a. [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Se+June+Joo%2C+Doyoung+Kim%2C+Joel+Jang%2C+Seonghyeon+Ye%2C+Jamin+Shin%2C+and+Minjoon+Seo%2E+The+cot+collection%3A+Improving+zero%2Dshot+and+few%2Dshot+learning+of+language+models+via+chain%2Dof%2Dthought+fin) 12. Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023b. [find](https://scholar.google.com/scholar?q=Sungdong+Kim%2C+Sanghwan+Bae%2C+Jamin+Shin%2C+Soyoung+Kang%2C+Donghyun+Kwak%2C+Kang+Min+Yoo%2C+and+Minjoon+Seo%2E+Aligning+large+language+models+through+synthetic+feedback%2E+arXiv+preprint+arXiv%3A2305%2E13735%2C+2023b%2E) 13. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332, 2021. [find](https://scholar.google.com/scholar?q=Kalpesh+Krishna%2C+Aurko+Roy%2C+and+Mohit+Iyyer%2E+Hurdles+to+progress+in+long%2Dform+question+answering%2E+arXiv+preprint+arXiv%3A2103%2E06332%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/) 14. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023. [↗](https://github.com/tatsu-lab/alpaca_eval) 15. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+Holistic+evaluation+of+language+models%2E+arXiv+) 16. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004a. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. [↗](https://aclanthology.org/W04-1013) 17. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74–81, 2004b. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pp%2E+74%E2%80%9381%2C+2004b%2E) 18. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+Gpteval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2E+arXiv+preprint+arXiv%3A2303%2E16634%2C+2023%2E) 19. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Kalpesh+Krishna%2C+Xinxi+Lyu%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Pang+Wei+Koh%2C+Mohit+Iyyer%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+Factscore%3A+Fine%2Dgrained+atomic+evaluation+of+factual+precision+in+) [our card](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/) 20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311–318, 2002. [find](https://scholar.google.com/scholar?q=Kishore+Papineni%2C+Salim+Roukos%2C+Todd+Ward%2C+and+Wei%2DJing+Zhu%2E+Bleu%3A+a+method+for+automatic+evaluation+of+machine+translation%2E+In+Proceedings+of+the+40th+annual+meeting+of+the+Association+for+Computatio) [our card](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/) 21. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. [find](https://scholar.google.com/scholar?q=Baolin+Peng%2C+Chunyuan+Li%2C+Pengcheng+He%2C+Michel+Galley%2C+and+Jianfeng+Gao%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03277%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/) 22. Luiza Pozzobon, Beyza Ermis, Patrick Lewis, and Sara Hooker. On the challenges of using black-box apis for toxicity evaluation in research, 2023. [find](https://scholar.google.com/scholar?q=Luiza+Pozzobon%2C+Beyza+Ermis%2C+Patrick+Lewis%2C+and+Sara+Hooker%2E+On+the+challenges+of+using+black%2Dbox+apis+for+toxicity+evaluation+in+research%2C+2023%2E) 23. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics, 2020. URL [https://api.semanticscholar.org/CorpusID:215548699](https://api.semanticscholar.org/CorpusID:215548699). [↗](https://api.semanticscholar.org/CorpusID:215548699) 24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe) 25. Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592, 2023a. [find](https://scholar.google.com/scholar?q=Tianlu+Wang%2C+Ping+Yu%2C+Xiaoqing+Ellen+Tan%2C+Sean+O%E2%80%99Brien%2C+Ramakanth+Pasunuru%2C+Jane+Dwivedi%2DYu%2C+Olga+Golovneva%2C+Luke+Zettlemoyer%2C+Maryam+Fazel%2DZarandi%2C+and+Asli+Celikyilmaz%2E+Shepherd%3A+A+critic+for+langua) 26. Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023b. [find](https://scholar.google.com/scholar?q=Yidong+Wang%2C+Zhuohao+Yu%2C+Zhengran+Zeng%2C+Linyi+Yang%2C+Cunxiang+Wang%2C+Hao+Chen%2C+Chaoya+Jiang%2C+Rui+Xie%2C+Jindong+Wang%2C+Xing+Xie%2C+et+al%2E+Pandalm%3A+An+automatic+evaluation+benchmark+for+llm+instruction+tuning) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-pandalm-automatic-evaluation-benchmark-llm/) 27. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+) 28. Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Minjoon Seo. Selfee: Iterative self-revising llm empowered by self-feedback generation. Blog post, May 2023a. URL [https://kaistai.github.io/SelFee/](https://kaistai.github.io/SelFee/). [↗](https://kaistai.github.io/SelFee/) 29. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928, 2023b. [find](https://scholar.google.com/scholar?q=Seonghyeon+Ye%2C+Doyoung+Kim%2C+Sungdong+Kim%2C+Hyeonbin+Hwang%2C+Seungone+Kim%2C+Yongrae+Jo%2C+James+Thorne%2C+Juho+Kim%2C+and+Minjoon+Seo%2E+Flask%3A+Fine%2Dgrained+language+model+evaluation+based+on+alignment+skill+sets) 30. Weizhe Yuan, Graham Neubig, and Pengfei Liu. BARTScore: Evaluating generated text as text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=5Ya8PbvpZ9. [↗](https://openreview.net/forum?) [our card](https://md.research.elamur.ai/free/en/papers/yuan-2021-bartscore-evaluating-generated-text-text/) 31. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2C+2019%2E) 32. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+arXiv+p) 33. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023. [find](https://scholar.google.com/scholar?q=Wanjun+Zhong%2C+Ruixiang+Cui%2C+Yiduo+Guo%2C+Yaobo+Liang%2C+Shuai+Lu%2C+Yanlin+Wang%2C+Amin+Saied%2C+Weizhu+Chen%2C+and+Nan+Duan%2E+Agieval%3A+A+human%2Dcentric+benchmark+for+evaluating+foundation+models%2E+arXiv+preprint+ar) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)