---
title: "Platypus: Quick, Cheap, and Powerful Refinement of LLMs (2023)"
description: "Lee, Hunter and Ruiz present Platypus, a family of LLaMa-2-based models refined with LoRA… 50 references, cited by 100 papers here."
authors: ["Ariel N. Lee", "Cole J. Hunter", "Nataniel Ruiz"]
year: 2023
venue: "arXiv:2308.07317"
source: "https://arxiv.org/abs/2308.07317"
doi: "10.48550/arxiv.2308.07317"
arxiv: "2308.07317"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / Platypus: Quick, Cheap, and Powerful Refinement of LLMs

# Platypus: Quick, Cheap, and Powerful Refinement of LLMs

**Authors:** Ariel N. Lee, Cole J. Hunter and Nataniel Ruiz · **Year:** 2023 · **Venue:** arXiv:2308.07317 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2308.07317) · [PDF](https://arxiv.org/pdf/2308.07317) · [DOI 10.48550/arxiv.2308.07317](https://doi.org/10.48550/arxiv.2308.07317) · [arXiv 2308.07317](https://arxiv.org/abs/2308.07317) · [OpenAlex](https://openalex.org/W4385849424)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [100](https://md.research.elamur.ai/free/en/papers/lee-2023-platypus-quick-cheap-powerful-refinement/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Adapting large language models to specialised skills usually requires large amounts of data and compute. Lee, Hunter and Ruiz present Platypus, a family of LLaMa-2-based models refined with LoRA adapters on Open-Platypus, a compact instruction set they curated. Open-Platypus draws on 11 open sources focused on STEM and logic. Its core is a modified MATH set enriched with step-by-step PRM800K solutions, and all items use the Alpaca format. The authors remove exact repeats and near-duplicates, then screen the data against benchmark test sets. They manually sort suspicious items into duplicate, gray-area, and similar-but-different cases and discard all three groups. The trained adapters are merged with other fine-tuned models, chosen only if their training data appeared clean. At the time of writing, Platypus models ranked first on Hugging Face's Open LLM Leaderboard while using far less data and compute than competing fine-tunes. The work suggests that careful data selection, leak filtering and adapter merging can partly replace sheer scale. However, the benefit of merging depends heavily on the domain.

## Key points
- Open-Platypus combines 11 public datasets, including PRM800K, MATH, ScienceQA, ReClor, TheoremQA, ARB, airoboros and Openassistant-Guanaco. Roughly 10% of its questions were generated by an LLM, and the dataset is publicly released. Because of source licences, the models are restricted to non-commercial use.
- A 13B Platypus model can be fine-tuned on a single A100 GPU with about 25k questions in roughly 5 hours.
- Near-duplicates were identified using SentenceTransformers embeddings with an 80% cosine similarity threshold. When two items matched, the pair with the longer, more detailed answer was kept. The same 80% threshold flagged training items too close to benchmark questions.
- Only near-exact copies of test items are counted as "true" contamination. Merge candidates were also checked for leaks, including SciQ and dolphin, which were not added to the training set.
- Merging effects vary by task. For example, adding the Platypus adapter to Camel-Platypus2-70B lowered college-physics performance by 18.42%. The models are English-centric and have not been red-teamed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Boston University](https://md.research.elamur.ai/free/en/universities/boston-university/) — University, Boston, United States · 30 works · 4,343 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Ariel N. Lee, Cole J. Hunter and Nataniel Ruiz. 2023. Platypus: Quick, Cheap, and Powerful Refinement of LLMs. arXiv:2308.07317.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lee2023platypus,
  title = {{Platypus: Quick, Cheap, and Powerful Refinement of LLMs}},
  author = {Ariel N. Lee, Cole J. Hunter and Nataniel Ruiz},
  year = {2023},
  doi = {10.48550/arxiv.2308.07317},
  eprint = {2308.07317},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.07317}
}
```

- [← previous How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN](https://md.research.elamur.ai/free/en/papers/mccoy-2023-how-much-do-language-models/)
- [next → Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://md.research.elamur.ai/free/en/papers/nanda-2023-emergent-linear-representations-world-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/)

## References

50 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert gate: Lifelong learning with a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7120–7129, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=R%2E+Aljundi%2C+P%2E+Chakravarty%2C+and+T%2E+Tuytelaars%2E+Expert+gate%3A+Lifelong+learning+with+a+network+of+experts%2E+In+2017+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C+pages+7120%E2%80%937129%2C+Los)
2. E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40b: an open large language model with state-of-the-art performance, 2023. [find](https://scholar.google.com/scholar?q=E%2E+Almazrouei%2C+H%2E+Alobeidli%2C+A%2E+Alshamsi%2C+A%2E+Cappelli%2C+R%2E+Cojocaru%2C+M%2E+Debbah%2C+E%2E+Goffinet%2C+D%2E+Heslow%2C+J%2E+Launay%2C+Q%2E+Malartic%2C+B%2E+Noune%2C+B%2E+Pannier%2C+and+G%2E+Penedo%2E+Falcon%2D40b%3A+an+open+large+language+m)
3. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. CalderaAI. 30b-lazarus. [https://huggingface.co/CalderaAI/30B-Lazarus](https://huggingface.co/CalderaAI/30B-Lazarus), 2023. [↗](https://huggingface.co/CalderaAI/30B-Lazarus)
5. W. Chen, M. Yin, M. Ku, E. Wan, X. Ma, J. Xu, T. Xia, X. Wang, and P. Lu. [Theoremqa: A theorem-driven question answering dataset](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/). arXiv preprint arXiv: 2305.12524, 2023.
6. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
7. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J.… [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+S%2E+Longpre%2C+B%2E+Zoph%2C+Y%2E+Tay%2C+W%2E+Fedus%2C+Y%2E+Li%2C+X%2E+Wang%2C+M%2E+Dehghani%2C+S%2E+Brahma%2C+A%2E+Webson%2C+S%2E+S%2E+Gu%2C+Z%2E+Dai%2C+M%2E+Suzgun%2C+X%2E+Chen%2C+A%2E+Chowdhery%2C+A%2E+Castro%2DRos%2C+M%2E+Pellat%2C+K%2E+Robinson)
8. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. [Qlora: Efficient finetuning of quantized llms](https://md.research.elamur.ai/free/en/papers/dettmers-2023-qlora-efficient-finetuning-quantized-llms/). arXiv preprint arXiv: 2305.14314, 2023. airoboros-gpt4-1.4.1. [https://huggingface.co/jondurbin/](https://huggingface.co/jondurbin/) [↗](https://huggingface.co/jondurbin/)
9. J. Durbin. airoboros-gpt4-1.4.1, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Durbin%2E+airoboros%2Dgpt4%2D1%2E4%2E1%2C+2023%2E)
10. ehartford. dolphin. [https://huggingface.co/datasets/ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin), 2023. [↗](https://huggingface.co/datasets/ehartford/dolphin)
11. W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. [find](https://scholar.google.com/scholar?q=W%2E+Fedus%2C+B%2E+Zoph%2C+and+N%2E+Shazeer%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2C+2022%2E)
12. S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. D. Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y. T. Lee, and Y. Li. Textbooks are all you need. arXiv: 2306.11644, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Gunasekar%2C+Y%2E+Zhang%2C+J%2E+Aneja%2C+C%2E+C%2E+T%2E+Mendes%2C+A%2E+D%2E+Giorno%2C+S%2E+Gopi%2C+M%2E+Javaheripi%2C+P%2E+Kauffmann%2C+G%2E+de+Rosa%2C+O%2E+Saarikivi%2C+A%2E+Salim%2C+S%2E+Shah%2C+H%2E+S%2E+Behl%2C+X%2E+Wang%2C+S%2E+Bubeck%2C+R%2E+Eldan%2C+A%2E+T%2E+Kala)
13. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig. [Towards a unified view of parameter-efficient transfer learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/). In International Conference on Learning Representations, 2022.
14. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS, 2021.
15. G. Hinton, O. Vinyals, and J. Dean. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/), 2015.
16. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-opt… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
17. C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C.-Y. Lee, and T. Pfister. [Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/), 2023.
18. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations, 2022.
19. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/), 2020.
20. N. U. P. R. Lab. leetcode-solutions-python-testgen-gpt4. [https://huggingface.co/datasets/nuprl/leetcode-solutions-python-testgen-gpt4](https://huggingface.co/datasets/nuprl/leetcode-solutions-python-testgen-gpt4), 2023. [↗](https://huggingface.co/datasets/nuprl/leetcode-solutions-python-testgen-gpt4)
21. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better, 2022. [find](https://scholar.google.com/scholar?q=K%2E+Lee%2C+D%2E+Ippolito%2C+A%2E+Nystrom%2C+C%2E+Zhang%2C+D%2E+Eck%2C+C%2E+Callison%2DBurch%2C+and+N%2E+Carlini%2E+Deduplicating+training+data+makes+language+models+better%2C+2022%2E)
22. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/), 2022.
23. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. arXiv preprint arXiv: 2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
24. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022. [find](https://scholar.google.com/scholar?q=P%2E+Lu%2C+S%2E+Mishra%2C+T%2E+Xia%2C+L%2E+Qiu%2C+K%2E%2DW%2E+Chang%2C+S%2E%2DC%2E+Zhu%2C+O%2E+Tafjord%2C+P%2E+Clark%2C+and+A%2E+Kalyan%2E+Learn+to+explain%3A+Multimodal+reasoning+via+thought+chains+for+science+question+answering%2C+2022%2E)
25. Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Luo%2C+C%2E+Xu%2C+P%2E+Zhao%2C+Q%2E+Sun%2C+X%2E+Geng%2C+W%2E+Hu%2C+C%2E+Tao%2C+J%2E+Ma%2C+Q%2E+Lin%2C+and+D%2E+Jiang%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2C+2023%2E)
26. D. Mahan, R. Carlow, L. Castricato, N. Cooper, and C. Laforte. Stable beluga models, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Mahan%2C+R%2E+Carlow%2C+L%2E+Castricato%2C+N%2E+Cooper%2C+and+C%2E+Laforte%2E+Stable+beluga+models%2C+2023%2E)
27. S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, and S. Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingface/peft](https://github.com/huggingface/peft), 2022. [↗](https://github.com/huggingface/peft)
28. S. Mukherjee, A. Mitra, G. Jawahar, S. Agarwal, H. Palangi, and A. Awadallah. [Orca: Progressive learning from complex explanation traces of gpt-4](https://md.research.elamur.ai/free/en/papers/mukherjee-2023-orca-progressive-learning-complex-explanation/). arXiv: 2306.02707v1, 2023.
29. C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang. [Is chatgpt a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/), 2023.
30. N. Reimers and I. Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
31. T. Research. tigerbot-kaggle-leetcodesolutions-en-2k. [https://huggingface.co/datasets/TigerResearch/tigerbot-kaggle-leetcodesolutions-en-2k](https://huggingface.co/datasets/TigerResearch/tigerbot-kaggle-leetcodesolutions-en-2k), 2023. [↗](https://huggingface.co/datasets/TigerResearch/tigerbot-kaggle-leetcodesolutions-en-2k)
32. T. Sawada, D. Paleka, A. Havrilla, P. Tadepalli, P. Vidas, A. P. Kranias, J. J. Nay, K. Gupta, and A. Komatsuzaki. [Arb: Advanced reasoning benchmark for large language models](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/), 2023.
33. T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model, 2022. [find](https://scholar.google.com/scholar?q=T%2E+L%2E+Scao%2C+A%2E+Fan%2C+C%2E+Akiki%2C+E%2E+Pavlick%2C+S%2E+Ilic%CC%81%2C+D%2E+Hesslow%2C+R%2E+Castagn%C3%A9%2C+A%2E+S%2E+Luccioni%2C+F%2E+Yvon%2C+M%2E+Gall%C3%A9%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2C+2022%2E)
34. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+A%2E+Mirhoseini%2C+K%2E+Maziarz%2C+A%2E+Davis%2C+Q%2E+Le%2C+G%2E+Hinton%2C+and+J%2E+Dean%2E+Outrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2C+2017%2E)
35. S. Shen, L. Hou, Y. Zhou, N. Du, S. Longpre, J. Wei, H. W. Chung, B. Zoph, W. Fedus, X. Chen, T. Vu, Y. Wu, W. Chen, A. Webson, Y. Li, V. Zhao, H. Yu, K. Keutzer, T. Darrell, and D. Zhou. [Mixture-of-experts meets instruction tuning:a winning combination for large language models](https://md.research.elamur.ai/free/en/papers/shen-2023-mixture-experts-meets-instruction-tuning/), 2023.
36. K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole-Lewis, D. Neal, M. Schaekermann, A. Wang, M. Amin, S. Lachgar, P. Mansfield, S. Prakash, B. Green, E. Dominowska, B. A. y Arcas, N. Tomasev, Y. Liu, R. Wong, C. Semturs, S. S. Mahdavi, J. Barral, D. Webster, G… — [Towards Expert-Level Medical Question Answering with Large Language Models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/)
37. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. GitHub repository, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Taori%2C+I%2E+Gulrajani%2C+T%2E+Zhang%2C+Y%2E+Dubois%2C+X%2E+Li%2C+C%2E+Guestrin%2C+P%2E+Liang%2C+and+T%2E+B%2E+Hashimoto%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E+GitHub+repository%2C+2023%2E)
38. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/), 2022.
39. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
40. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
41. X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv: 2307.10635, 2023.
42. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560v2, 2023. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Kordi%2C+S%2E+Mishra%2C+A%2E+Liu%2C+N%2E+A%2E+Smith%2C+D%2E+Khashabi%2C+and+H%2E+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+arXiv%3A2212%2E10560v2%2C+2023%2E)
43. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/), 2022.
44. J. Welbl, N. F. Liu, and M. Gardner. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). arXiv: 1707.06209, 2017.
45. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/).…
46. M. Wu, A. Waheed, C. Zhang, M. Abdul-Mageed, and A. F. Aji. [Lamini-lm: A diverse herd of distilled models from large-scale instructions](https://md.research.elamur.ai/free/en/papers/wu-2023-lamini-lm-diverse-herd-distilled/). CoRR, abs/2304.14402, 2023.
47. C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Xu%2C+Q%2E+Sun%2C+K%2E+Zheng%2C+X%2E+Geng%2C+P%2E+Zhao%2C+J%2E+Feng%2C+C%2E+Tao%2C+and+D%2E+Jiang%2E+Wizardlm%3A+Empowering+large+language+models+to+follow+complex+instructions%2C+2023%2E)
48. R. Yu, S. Liu, and X. Wang. Dataset distillation: A comprehensive review, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Yu%2C+S%2E+Liu%2C+and+X%2E+Wang%2E+Dataset+distillation%3A+A+comprehensive+review%2C+2023%2E)
49. W. Yu, Z. Jiang, Y. Dong, and J. Feng. [Reclor: A reading comprehension dataset requiring logical reasoning](https://md.research.elamur.ai/free/en/papers/yu-2020-reclor-reading-comprehension-dataset-requiring/). International Conference on Learning Representations (ICLR), April 2020.
50. C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. Lima: Less is more for alignment. arXiv: 2305.11206, 2023. [find](https://scholar.google.com/scholar?q=C%2E+Zhou%2C+P%2E+Liu%2C+P%2E+Xu%2C+S%2E+Iyer%2C+J%2E+Sun%2C+Y%2E+Mao%2C+X%2E+Ma%2C+A%2E+Efrat%2C+P%2E+Yu%2C+L%2E+Yu%2C+S%2E+Zhang%2C+G%2E+Ghosh%2C+M%2E+Lewis%2C+L%2E+Zettlemoyer%2C+and+O%2E+Levy%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv%3A+2305%2E11206%2C+2023%2E)

*Updated 2 November 2026*