---
title: "AlpaGasus: Training A Better Alpaca with Fewer — Chen 2023"
description: "Instruction-tuning datasets produced by a teacher model often contain flawed examples."
authors: ["Lichang Chen", "Shiyang Li", "Jun Yan", "Hai Wang", "Kalpa Gunaratna", "Vikas Yadav", "Zheng Tang", "Vijay Srinivasan", "Tianyi Zhou", "Heng Huang", "Hongxia Jin"]
year: 2023
venue: "arXiv:2307.08701"
source: "https://arxiv.org/abs/2307.08701"
doi: "10.48550/arxiv.2307.08701"
arxiv: "2307.08701"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/) / AlpaGasus: Training A Better Alpaca with Fewer Data

# AlpaGasus: Training A Better Alpaca with Fewer Data

**Authors:** [Lichang Chen](https://md.research.elamur.ai/free/en/authors/lichang-chen/), [Shiyang Li](https://md.research.elamur.ai/free/en/authors/shiyang-li/), [Jun Yan](https://md.research.elamur.ai/free/en/authors/jun-yan/), Hai Wang, Kalpa Gunaratna, [Vikas Yadav](https://md.research.elamur.ai/free/en/authors/vikas-yadav/), Zheng Tang, Vijay Srinivasan, [Tianyi Zhou](https://md.research.elamur.ai/free/en/authors/tianyi-zhou/), Heng Huang, Hongxia Jin · **Year:** 2023 · **Venue:** arXiv:2307.08701 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.08701) · [PDF](https://arxiv.org/pdf/2307.08701) · [DOI 10.48550/arxiv.2307.08701](https://doi.org/10.48550/arxiv.2307.08701) · [arXiv 2307.08701](https://arxiv.org/abs/2307.08701) · [OpenAlex](https://openalex.org/W4384648625)

**Citations:** 14 citations (openalex-snapshot · 2026-10-14) · [200](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Instruction-tuning datasets produced by a teacher model often contain flawed examples. The 52k Alpaca set, generated by Text-Davinci-003, includes many responses that are wrong or off-topic. Such examples can mislead the fine-tuned model and make training cost more. Earlier cleanup efforts like Alpaca-cleaned depended on manual review. This paper replaces that manual work with an automatic filter. ChatGPT is prompted to give every (instruction, input, response) triplet an accuracy score from 0 to 5, and only triplets scoring 4.5 or higher are kept. The authors then fine-tune LLaMA on this subset with the original Alpaca recipe, unchanged, and call the result AlpaGasus. The judges were GPT-4 and human annotators, on four instruction test sets. In both evaluations, the models trained on the smaller filtered data beat Alpaca at 7B and at 13B. Training also became much shorter. The approach transfers to other datasets, base models and grader LLMs. The main lesson is that the quality of instruction data can count for more than its volume, and checking quality with an LLM is cheap and scales well.

## Key points
- Filtering reduced Alpaca's 52,002 samples to 9,229, keeping about 18% of the data.
- Fine-tuning a 7B model took 14 minutes on 4× NVIDIA A100 (80GB) GPUs, compared with 80 minutes for Alpaca, about 5.7x faster.
- The test sets were Vicuna, Koala, WizardLM and Self-Instruct. GPT-4 served as the main judge, and a human study and benchmark evaluations backed up its verdicts.
- On WizardLM skills, AlpaGasus-13B averages roughly 91% of the performance of its teacher, Text-Davinci-003, and about 78% of ChatGPT and Claude. It still trails on coding and math.
- The method also worked on Dolly and GPT4LLM data, with LLaMA-1 and LLaMA-2 as base models, and with Claude-2 as an alternative grader.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

200 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 182 not yet.

- [SmolKalam: Ensemble Quality-Filtered Translation at Scale for High Quality Arabic Post-Training Data](https://md.research.elamur.ai/free/en/papers/alrashed-2025-smolkalam-ensemble-quality-filtered-translation/)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://md.research.elamur.ai/free/en/papers/guan-2023-hallusionbench-advanced-diagnostic-suite-entangled/)
- [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://md.research.elamur.ai/free/en/papers/li-2023-quantity-quality-boosting-llm-performance/)
- [Self-Alignment with Instruction Backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://md.research.elamur.ai/free/en/papers/long-2024-llms-driven-synthetic-data-generation/)
- [Parallel Corpus Filtering Based on Semantic Similarity and Surface Dissimilarity for Japanese Text Simplification with LLMs](https://md.research.elamur.ai/free/en/papers/maekawa-2026-parallel-corpus-filtering-based-semantic/)
- [Children's English Reading Story Generation via Supervised Fine-Tuning of Compact LLMs with Controllable Difficulty and Safety](https://md.research.elamur.ai/free/en/papers/shen-2026-children-s-english-reading-story/)
- [Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/singh-2024-aya-dataset-open-access-collection/)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/)
- [Aligning Large Language Models with Human: A Survey](https://md.research.elamur.ai/free/en/papers/wang-2023-aligning-large-language-models-human/)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://md.research.elamur.ai/free/en/papers/wang-2023-openchat-advancing-open-source-language/)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://md.research.elamur.ai/free/en/papers/xia-2024-less-selecting-influential-data-targeted/)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-siren-s-song-ai-ocean/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2023alpagasus,
  title = {{AlpaGasus: Training A Better Alpaca with Fewer Data}},
  author = {Chen, Lichang and Li, Shiyang and Yan, Jun and Wang, Hai and Gunaratna, Kalpa and Yadav, Vikas and Tang, Zheng and Srinivasan, Vijay and Zhou, Tianyi and Huang, Heng and Jin, Hongxia},
  year = {2023},
  doi = {10.48550/arxiv.2307.08701},
  eprint = {2307.08701},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.08701}
}
```

- [← previous AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
- [next → AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/)

## References

48 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M+Dai%2C+Orhan+Firat%2C+Melvin+Johnson%2C+Dmitry+Lepikhin%2C+Alexandre+Passos%2C+Siamak+Shakeri%2C+Emanuel+Taropa%2C+Paige+Bailey%2C+Zhifeng+Chen%2C+et+al%2E+Palm+2+technical+report%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Saurav+Kadavath%2C+Sandipan+Kundu%2C+Amanda+Askell%2C+Jackson+Kernion%2C+Andy+Jones%2C+Anna+Chen%2C+Anna+Goldie%2C+Azalia+Mirhoseini%2C+Cameron+McKinnon%2C+et+al%2E+Constitutional+ai%3A+Harmlessness+from+ai+fee) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
4. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, 2020. [find](https://scholar.google.com/scholar?q=Yonatan+Bisk%2C+Rowan+Zellers%2C+Jianfeng+Gao%2C+Yejin+Choi%2C+et+al%2E+Piqa%3A+Reasoning+about+physical+commonsense+in+natural+language%2E+In+Proceedings+of+the+AAAI+conference+on+artificial+intelligence%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/)
5. Jiuhai Chen, Lichang Chen, and Tianyi Zhou. It takes one to tango but more make trouble? in-context training with different number of demonstrations. arXiv preprint arXiv:2303.08119, 2023a. — [arXiv:2303.08119](https://arxiv.org/abs/2303.08119) [find](https://scholar.google.com/scholar?q=Jiuhai+Chen%2C+Lichang+Chen%2C+and+Tianyi+Zhou%2E+It+takes+one+to+tango+but+more+make+trouble%3F+in%2Dcontext+training+with+different+number+of+demonstrations%2E+arXiv+preprint+arXiv%3A2303%2E08119%2C+2023a%2E+%E2%80%94+%5BarXiv%3A2)
6. Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03082, 2023b. — [arXiv:2306.03082](https://arxiv.org/abs/2306.03082) [find](https://scholar.google.com/scholar?q=Lichang+Chen%2C+Jiuhai+Chen%2C+Tom+Goldstein%2C+Heng+Huang%2C+and+Tianyi+Zhou%2E+Instructzero%3A+Efficient+instruction+optimization+for+black%2Dbox+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E03082%2C+2023b%2E+%E2%80%94+%5B)
7. Lichang Chen, Minhao Cheng, and Heng Huang. Backdoor learning on sequence to sequence models. arXiv preprint arXiv:2305.02424, 2023c. — [arXiv:2305.02424](https://arxiv.org/abs/2305.02424) [find](https://scholar.google.com/scholar?q=Lichang+Chen%2C+Minhao+Cheng%2C+and+Heng+Huang%2E+Backdoor+learning+on+sequence+to+sequence+models%2E+arXiv+preprint+arXiv%3A2305%2E02424%2C+2023c%2E+%E2%80%94+%5BarXiv%3A2305%2E02424%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E02424%29)
8. Lichang Chen, Heng Huang, and Minhao Cheng. Ptp: Boosting stability and performance of prompt tuning with perturbation-based regularizer. arXiv preprint arXiv:2305.02423, 2023d. [find](https://scholar.google.com/scholar?q=Lichang+Chen%2C+Heng+Huang%2C+and+Minhao+Cheng%2E+Ptp%3A+Boosting+stability+and+performance+of+prompt+tuning+with+perturbation%2Dbased+regularizer%2E+arXiv+preprint+arXiv%3A2305%2E02423%2C+2023d%2E)
9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harri+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+et+al%2E+Evaluating+large+language+models+tra) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
10. Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arXiv:2306.04757, 2023. [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Pengfei+Hong%2C+Lidong+Bing%2C+and+Soujanya+Poria%2E+Instructeval%3A+Towards+holistic+evaluation+of+instruction%2Dtuned+large+language+models%2E+arXiv+preprint+arXiv%3A2306%2E04757%2C+2023%2E)
11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
12. Xu Chu, Ihab F Ilyas, Sanjay Krishnan, and Jiannan Wang. Data cleaning: Overview and emerging challenges. In Proceedings of the 2016 international conference on management of data, pp. 2201–2206, 2016. [find](https://scholar.google.com/scholar?q=Xu+Chu%2C+Ihab+F+Ilyas%2C+Sanjay+Krishnan%2C+and+Jiannan+Wang%2E+Data+cleaning%3A+Overview+and+emerging+challenges%2E+In+Proceedings+of+the+2016+international+conference+on+management+of+data%2C+pp%2E+2201%E2%80%932206%2C+2016)
13. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
14. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023. [find](https://scholar.google.com/scholar?q=Ning+Ding%2C+Yulin+Chen%2C+Bokai+Xu%2C+Yujia+Qin%2C+Zhi+Zheng%2C+Shengding+Hu%2C+Zhiyuan+Liu%2C+Maosong+Sun%2C+and+Bowen+Zhou%2E+Enhancing+chat+language+models+by+scaling+high%2Dquality+instructional+conversations%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/)
15. Dolly. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Blog Post, 2023. URL [https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm). [↗](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm)
16. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+Proc%2E+of+NAACL%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
17. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Xuechen+Li%2C+Rohan+Taori%2C+Tianyi+Zhang%2C+Ishaan+Gulrajani%2C+Jimmy+Ba%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Alpacafarm%3A+A+simulation+framework+for+methods+that+learn+from+) [our card](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
18. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
19. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+John+Schulman%2C+and+Jacob+Hilton%2E+Scaling+laws+for+reward+model+overoptimization%2E+arXiv+preprint+arXiv%3A2210%2E10760%2C+2022%2E)
20. Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/blog/2023/04/03/koala/](https://bair.berkeley.edu/blog/2023/04/03/koala/). [↗](https://bair.berkeley.edu/blog/2023/04/03/koala/)
21. Mustafa Hajij, Ghada Zamzmi, Karthikeyan Natesan Ramamurthy, and Aldo Guzman Saenz. Data-centric ai requires rethinking data notion. arXiv preprint arXiv:2110.02491, 2021. — [arXiv:2110.02491](https://arxiv.org/abs/2110.02491) [find](https://scholar.google.com/scholar?q=Mustafa+Hajij%2C+Ghada+Zamzmi%2C+Karthikeyan+Natesan+Ramamurthy%2C+and+Aldo+Guzman+Saenz%2E+Data%2Dcentric+ai+requires+rethinking+data+notion%2E+arXiv+preprint+arXiv%3A2110%2E02491%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E02491%5D%28https%3A%2F)
22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
23. Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. Look at the first sentence: Position bias in question answering. arXiv preprint arXiv:2004.14602, 2020. [find](https://scholar.google.com/scholar?q=Miyoung+Ko%2C+Jinhyuk+Lee%2C+Hyunjae+Kim%2C+Gangwoo+Kim%2C+and+Jaewoo+Kang%2E+Look+at+the+first+sentence%3A+Position+bias+in+question+answering%2E+arXiv+preprint+arXiv%3A2004%2E14602%2C+2020%2E)
24. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations–democratizing large language model alignment. arXiv preprint arXiv:2304.07327, 2023. [find](https://scholar.google.com/scholar?q=Andreas+K%C3%B6pf%2C+Yannic+Kilcher%2C+Dimitri+von+R%C3%BCtte%2C+Sotiris+Anagnostidis%2C+Zhi%2DRui+Tam%2C+Keith+Stevens%2C+Abdullah+Barhoum%2C+Nguyen+Minh+Duc%2C+Oliver+Stanley%2C+Rich%C3%A1rd+Nagyfi%2C+et+al%2E+Openassistant+conversations) [our card](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
25. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023. [↗](https://github.com/tatsu-lab/alpaca_eval)
26. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+et+al%2E+Holistic+evaluation+of+language+models%2E+arXiv+)
27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p)
28. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. [find](https://scholar.google.com/scholar?q=Shayne+Longpre%2C+Le+Hou%2C+Tu+Vu%2C+Albert+Webson%2C+Hyung+Won+Chung%2C+Yi+Tay%2C+Denny+Zhou%2C+Quoc+V+Le%2C+Barret+Zoph%2C+Jason+Wei%2C+et+al%2E+The+flan+collection%3A+Designing+data+and+methods+for+effective+instruction+t) [our card](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/)
29. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2C+2023%2E)
30. Mohammad Motamedi, Nikolay Sakharnykh, and Tim Kaldewey. A data-centric approach for training deep neural networks with less data. arXiv preprint arXiv:2110.03613, 2021. — [arXiv:2110.03613](https://arxiv.org/abs/2110.03613) [find](https://scholar.google.com/scholar?q=Mohammad+Motamedi%2C+Nikolay+Sakharnykh%2C+and+Tim+Kaldewey%2E+A+data%2Dcentric+approach+for+training+deep+neural+networks+with+less+data%2E+arXiv+preprint+arXiv%3A2110%2E03613%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E03613%5D%28https%3A%2F%2Fa)
31. OpenAI. Chatgpt. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2023a. [↗](https://openai.com/blog/chatgpt)
32. OpenAI. Gpt-4 technical report. arXiv, 2023b. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2E+arXiv%2C+2023b%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
33. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022a. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
34. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022b. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
35. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. [find](https://scholar.google.com/scholar?q=Baolin+Peng%2C+Chunyuan+Li%2C+Pengcheng+He%2C+Michel+Galley%2C+and+Jianfeng+Gao%2E+Instruction+tuning+with+gpt%2D4%2E+arXiv+preprint+arXiv%3A2304%2E03277%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/)
36. ShareGPT. Sharegpt. 2023. URL sharegpt.com. [find](https://scholar.google.com/scholar?q=ShareGPT%2E+Sharegpt%2E+2023%2E+URL+sharegpt%2Ecom%2E)
37. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Nathan+Scales%2C+Nathanael+Sch%C3%A4rli%2C+Sebastian+Gehrmann%2C+Yi+Tay%2C+Hyung+Won+Chung%2C+Aakanksha+Chowdhery%2C+Quoc+V+Le%2C+Ed+H+Chi%2C+Denny+Zhou%2C+%2C+and+Jason+Wei%2E+Challenging+big%2Dbench+tasks+and+whet) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
38. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
39. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edoua)
40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sy… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+Von+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wall)
41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017b. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+Advances+in+neural+information+processing+system)
42. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023%2E)
43. Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the eleventh ACM international conference on web search and data mining, pp. 610–618, 2018. [find](https://scholar.google.com/scholar?q=Xuanhui+Wang%2C+Nadav+Golbandi%2C+Michael+Bendersky%2C+Donald+Metzler%2C+and+Marc+Najork%2E+Position+bias+estimation+for+unbiased+learning+to+rank+in+personal+search%2E+In+Proceedings+of+the+eleventh+ACM+internat)
44. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
45. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. [find](https://scholar.google.com/scholar?q=Can+Xu%2C+Qingfeng+Sun%2C+Kai+Zheng%2C+Xiubo+Geng%2C+Pu+Zhao%2C+Jiazhan+Feng%2C+Chongyang+Tao%2C+and+Daxin+Jiang%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2C+2023%2E)
46. Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158, 2023. — [arXiv:2303.10158](https://arxiv.org/abs/2303.10158) [find](https://scholar.google.com/scholar?q=Daochen+Zha%2C+Zaid+Pervaiz+Bhat%2C+Kwei%2DHerng+Lai%2C+Fan+Yang%2C+Zhimeng+Jiang%2C+Shaochen+Zhong%2C+and+Xia+Hu%2E+Data%2Dcentric+artificial+intelligence%3A+A+survey%2E+arXiv+preprint+arXiv%3A2303%2E10158%2C+2023%2E+%E2%80%94+%5BarXiv%3A230)
47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+arXiv+p)
48. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023%2E)