---
title: "QuRating — paper by Wettig et al. (2024) — summary and ideas"
description: "Which documents a language model is pre-trained on strongly shapes its abilities. 97 references, cited by 68 papers here."
authors: ["Alexander Wettig", "Aatmik Gupta", "Saumya Malik", "Danqi Chen"]
year: 2024
venue: "arXiv:2402.09739"
source: "https://arxiv.org/abs/2402.09739"
doi: "10.48550/arxiv.2402.09739"
arxiv: "2402.09739"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wettig-2024-qurating-selecting-high-quality-data/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wettig-2024-qurating-selecting-high-quality-data/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/) / QuRating: Selecting High-Quality Data for Training Language Models

# QuRating: Selecting High-Quality Data for Training Language Models

**Authors:** [Alexander Wettig](https://md.research.elamur.ai/free/en/authors/alexander-wettig/), Aatmik Gupta, [Saumya Malik](https://md.research.elamur.ai/free/en/authors/saumya-malik/) and [Danqi Chen](https://md.research.elamur.ai/free/en/authors/danqi-chen/) · **Year:** 2024 · **Venue:** arXiv:2402.09739 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2402.09739) · [PDF](https://arxiv.org/pdf/2402.09739) · [DOI 10.48550/arxiv.2402.09739](https://doi.org/10.48550/arxiv.2402.09739) · [arXiv 2402.09739](https://arxiv.org/abs/2402.09739) · [OpenAlex](https://openalex.org/W4391899422)

**Citations:** 6 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/wettig-2024-qurating-selecting-high-quality-data/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Which documents a language model is pre-trained on strongly shapes its abilities. Yet common filters, such as C4/Gopher-style rules, perplexity filtering, or resemblance to reference domains like Wikipedia, capture quality only roughly. QuRating turns intuitive human notions of quality into scores along four axes: writing style, required expertise, facts & trivia, and educational value.

GPT-3.5-turbo compares pairs of texts on each criterion. The authors found that pairwise comparisons give steadier verdicts than rating texts one at a time. A Bradley-Terry objective, similar to reward modeling in RLHF, then converts these comparisons into a fine-tuned Sheared-Llama-1.3B "QuRater" that outputs a scalar score per document. This model scores 260B tokens of SlimPajama, producing QuRatedPajama.

The authors draw 30B-token subsets by treating ratings as sampling logits with a temperature, and train 1.3B models from scratch. Pure top-k selection harms diversity, while moderate temperature sampling works better. Educational value yields the strongest in-context learning, matching a uniform baseline given 50% more training. The released judgments, model and data make this a reusable, interpretable selection tool, although its biases need attention.

## Key points
- About 250K GPT-3.5-turbo pairwise judgments were collected per criterion. The QuRater is Sheared-Llama-1.3B fine-tuned on them.
- QuRatedPajama annotates 260B SlimPajama tokens with four ratings. Models (1.3B parameters, 30B selected tokens) are evaluated on 10 in-context learning tasks against uniform sampling, perplexity filtering and DSIR.
- With temperature τ = 2.0, educational value improves average ICL accuracy by 1.8% over uniform selection. Writing style gives the lowest perplexity but little downstream benefit.
- Ordering the same training data as a curriculum by required expertise beats a random ordering.
- Limitations:
  - The ratings show social, linguistic and geographic biases, examined in part with the AboutMe dataset.
  - Experiments are limited to the 1.3B scale.
  - Each training run costs roughly 200 H100 GPU hours.
  - Resources are released at github.com/princeton-nlp/QuRating.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Princeton University](https://md.research.elamur.ai/free/en/universities/princeton-university/) — University, Princeton, United States · 113 works · 48,612 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Alexander Wettig, Aatmik Gupta, Saumya Malik and Danqi Chen. 2024. QuRating: Selecting High-Quality Data for Training Language Models. arXiv:2402.09739.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wettig2024qurating,
  title = {{QuRating: Selecting High-Quality Data for Training Language Models}},
  author = {Alexander Wettig, Aatmik Gupta, Saumya Malik and Danqi Chen},
  year = {2024},
  doi = {10.48550/arxiv.2402.09739},
  eprint = {2402.09739},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.09739}
}
```

- [← previous Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://md.research.elamur.ai/free/en/papers/gou-2024-navigating-digital-world-humans-universal/)
- [next → ReFT: Representation Finetuning for Language Models](https://md.research.elamur.ai/free/en/papers/wu-2024-reft-representation-finetuning-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/6/)

## References

97 references, 47 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abbas, A., Tirumala, K., Simig, D., Ganguli, S., and Morcos, A. S. [SemDeDup: Data-efficient learning at web-scale through semantic deduplication](https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/). arXiv preprint arXiv:2303.09540, 2023.
2. Abid, A., Farooqi, M., and Zou, J. [Persistent anti-muslim bias in large language models](https://md.research.elamur.ai/free/en/papers/abid-2021-persistent-anti-muslim-bias-large/). In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, pp. 298–306, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450384735. doi: 10.1145/3461702.3462624. U…
3. Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. [On-policy distillation of language models: Learning from self-generated mistakes](https://md.research.elamur.ai/free/en/papers/agarwal-2023-policy-distillation-language-models-learning/). In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=3zKtaqxLhW](https://openreview.net/forum?id=3zKtaqxLhW). [↗](https://openreview.net/forum?id=3zKtaqxLhW)
4. Almeida, T. and Hidalgo, J. SMS Spam Collection. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C5CC84. — [doi:10.24432/c5cc84](https://doi.org/10.24432/c5cc84) [↗](https://doi.org/10.24432/C5CC84)
5. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. [PaLM 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
6. Arora, S. and Goyal, A. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307.15936, 2023. — [arXiv:2307.15936](https://arxiv.org/abs/2307.15936) [find](https://scholar.google.com/scholar?q=Arora%2C+S%2E+and+Goyal%2C+A%2E+A+theory+for+emergence+of+complex+skills+in+language+models%2E+arXiv+preprint+arXiv%3A2307%2E15936%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E15936%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E15936%29)
7. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E%2C+Askell%2C+A%2E%2C+Chen%2C+A%2E%2C+Das%2DSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Fort%2C+S%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+)
8. Barocas, S., Crawford, K., Shapiro, A., and Wallach, H. The problem with bias: Allocative versus representational harms in machine learning. In 9th Annual conference of the special interest group for computing, information and society, 2017. [find](https://scholar.google.com/scholar?q=Barocas%2C+S%2E%2C+Crawford%2C+K%2E%2C+Shapiro%2C+A%2E%2C+and+Wallach%2C+H%2E+The+problem+with+bias%3A+Allocative+versus+representational+harms+in+machine+learning%2E+In+9th+Annual+conference+of+the+special+interest+group+for+)
9. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Association for Computing M… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) [find](https://scholar.google.com/scholar?q=Bender%2C+E%2E+M%2E%2C+Gebru%2C+T%2E%2C+McMillan%2DMajor%2C+A%2E%2C+and+Shmitchell%2C+S%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Accountabi)
10. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020.
11. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454[–5476, Online, 2020](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). Association for Computational Linguistics. doi: 1…
12. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Bradley%2C+R%2E+A%2E+and+Terry%2C+M%2E+E%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
13. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
14. Curran Associates, Inc., 2020. URL [https://proceedings.neurips.cc/paper](https://proceedings.neurips.cc/paper) files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. [↗](https://proceedings.neurips.cc/paper)
15. Caliskan, A., Bryson, J. J., and Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334): 183–186, 2017. doi: 10.1126/science.aal4230. URL [https://www.science.org/doi/abs/10.1126/science.aal4230](https://www.science.org/doi/abs/10.1126/science.aal4230). [↗](https://www.science.org/doi/abs/10.1126/science.aal4230)
16. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research, 24(240):1–113, 2023.
17. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
18. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? Try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
19. Dev, S., Monajatipoor, M., Ovalle, A., Subramonian, A., Phillips, J., and Chang, K.-W. [Harms of gender exclusivity and challenges in non-binary representation in language technologies](https://md.research.elamur.ai/free/en/papers/dev-2021-harms-gender-exclusivity-challenges-non/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1968–1994, Online an…
20. Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., and Gardner, M. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286–1305, On… [find](https://scholar.google.com/scholar?q=Dodge%2C+J%2E%2C+Sap%2C+M%2E%2C+Marasovic%CC%81%2C+A%2E%2C+Agnew%2C+W%2E%2C+Ilharco%2C+G%2E%2C+Groeneveld%2C+D%2E%2C+Mitchell%2C+M%2E%2C+and+Gardner%2C+M%2E+Documenting+large+webtext+corpora%3A+A+case+study+on+the+colossal+clean+crawled+corpus%2E+In+Proce)
21. Doudna, J. A. and Charpentier, E. Genome editing. the new frontier of genome engineering with crispr-cas9. Science (New York, N.Y.), 346(6213): 1258096, November 2014. ISSN 0036-8075. doi: 10.1126/science.1258096. URL https://doi.org/10.1126/science.1258096. — [doi:10.1126/science.1258096](https://doi.org/10.1126/science.1258096) [↗](https://doi.org/10.1126/science.1258096)
22. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M. P., Zhou, Z., Wang, T., Wang, E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K., Duke, T., Dixon, L., Zhang, K., Le, Q., Wu, Y., Chen, Z., and Cui, C… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
23. Du, Z., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024. — [arXiv:2403.15796](https://arxiv.org/abs/2403.15796) [find](https://scholar.google.com/scholar?q=Du%2C+Z%2E%2C+Zeng%2C+A%2E%2C+Dong%2C+Y%2E%2C+and+Tang%2C+J%2E+Understanding+emergent+abilities+of+language+models+from+the+loss+perspective%2E+arXiv+preprint+arXiv%3A2403%2E15796%2C+2024%2E+%E2%80%94+%5BarXiv%3A2403%2E15796%5D%28https%3A%2F%2Farxiv%2Eorg%2Fab)
24. Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id…](https://openreview.net/forum?id…) [↗](https://openreview.net/forum?id…)
25. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+et+al%2E+The+Pile%3A+An+800GB+dataset+of+diverse+text+for+language+modeling%2E+arXiv+prepri)
26. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenodo… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [↗](https://doi.org/10.5281/zenodo…)
27. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, November 2020. Association for Computational Linguistics. doi…
28. Gienapp, L., Fröbe, M., Hagen, M., and Potthast, M. Sparse pairwise re-ranking with pre-trained transformers. In Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’22, pp. 72–80, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 97… — [doi:10.1145/3539813.3545140](https://doi.org/10.1145/3539813.3545140) [find](https://scholar.google.com/scholar?q=Gienapp%2C+L%2E%2C+Fro%CC%88be%2C+M%2E%2C+Hagen%2C+M%2E%2C+and+Potthast%2C+M%2E+Sparse+pairwise+re%2Dranking+with+pre%2Dtrained+transformers%2E+In+Proceedings+of+the+2022+ACM+SIGIR+International+Conference+on+Theory+of+Information+Re)
29. Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f -divergence minimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11546–11583. PML… — [Aligning Language Models with Preferences through f-divergence Minimization](https://md.research.elamur.ai/free/en/papers/go-2023-aligning-language-models-preferences-through/)
30. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, 2023. [find](https://scholar.google.com/scholar?q=Gunasekar%2C+S%2E%2C+Zhang%2C+Y%2E%2C+Aneja%2C+J%2E%2C+Mendes%2C+C%2E+C%2E+T%2E%2C+Giorno%2C+A%2E+D%2E%2C+Gopi%2C+S%2E%2C+Javaheripi%2C+M%2E%2C+Kauffmann%2C+P%2E%2C+de+Rosa%2C+G%2E%2C+Saarikivi%2C+O%2E%2C+Salim%2C+A%2E%2C+Shah%2C+S%2E%2C+Behl%2C+H%2E+S%2E%2C+Wang%2C+X%2E%2C+Bubeck%2C+S%2E%2C+Eldan)
31. Gururangan, S., Li, M., Lewis, M., Shi, W., Althoff, T., Smith, N. A., and Zettlemoyer, L. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177, 2023. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%2C+Li%2C+M%2E%2C+Lewis%2C+M%2E%2C+Shi%2C+W%2E%2C+Althoff%2C+T%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Zettlemoyer%2C+L%2E+Scaling+expert+language+models+with+unsupervised+domain+discovery%2E+arXiv+preprint+arXiv%3A2303%2E14177%2C+2023%2E)
32. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2021.
33. Hinton, G. E., Vinyals, O., and Dean, J. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). ArXiv, abs/1503.02531, 2015. URL [https://api.semanticscholar.org/CorpusID:7200347](https://api.semanticscholar.org/CorpusID:7200347). [↗](https://api.semanticscholar.org/CorpusID:7200347)
34. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L.… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
35. Jiang, T., Yuan, X., Chen, Y., Cheng, K., Wang, L., Chen, X., and Ma, J. Fuzzydedup: Secure fuzzy deduplication for cloud storage. IEEE Transactions on Dependable and Secure Computing, 20(3):2466–2483, 2023. doi: 10.1109/TDSC.2022.3185313. — [doi:10.1109/tdsc.2022.3185313](https://doi.org/10.1109/tdsc.2022.3185313) [find](https://scholar.google.com/scholar?q=Jiang%2C+T%2E%2C+Yuan%2C+X%2E%2C+Chen%2C+Y%2E%2C+Cheng%2C+K%2E%2C+Wang%2C+L%2E%2C+Chen%2C+X%2E%2C+and+Ma%2C+J%2E+Fuzzydedup%3A+Secure+fuzzy+deduplication+for+cloud+storage%2E+IEEE+Transactions+on+Dependable+and+Secure+Computing%2C+20%283%29%3A2466%E2%80%932483)
36. Kim, C., Sabharwal, A., and Ermon, S. Exact sampling with integer linear programs and random perturbations. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Mar. 2016. doi: 10.1609/aaai.v30i1.10421. URL [https://ojs.aaai.org/index.php/AAAI/article/view/10421](https://ojs.aaai.org/index.php/AAAI/article/view/10421). [↗](https://ojs.aaai.org/index.php/AAAI/article/view/10421)
37. Kim, Y. and Rush, A. M. [Sequence-level knowledge distillation](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/). In Su, J., Duh, K., and Carreras, X. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1317–1327, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1…
38. Kingma, D. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
39. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Advances in Neural Information Processing Systems, volume 35, pp. 22199–22213. Curran Associates, Inc., 2022. URL [https://proceedings.neurips.cc/paper](https://proceedings.neurips.cc/paper) files/paper/2022/file/ 8bb0d291acd4acf… [↗](https://proceedings.neurips.cc/paper)
40. Kool, W., Van Hoof, H., and Welling, M. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3499–3508. PMLR, 09–… [find](https://scholar.google.com/scholar?q=Kool%2C+W%2E%2C+Van+Hoof%2C+H%2E%2C+and+Welling%2C+M%2E+Stochastic+beams+and+where+to+find+them%3A+The+Gumbel%2Dtop%2Dk+trick+for+sampling+sequences+without+replacement%2E+In+Proceedings+of+the+36th+International+Conference+)
41. Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In Advances in Neural Information Processing Systems, 2022a. URL [https://openreview.net/forum?id=XvI6h-s4un](https://openreview.net/forum?id=XvI6h-s4un). [↗](https://openreview.net/forum?id=XvI6h-s4un)
42. Korbak, T., Perez, E., and Buckley, C. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083–1091, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. doi: 10.18653/v1/20… [find](https://scholar.google.com/scholar?q=Korbak%2C+T%2E%2C+Perez%2C+E%2E%2C+and+Buckley%2C+C%2E+RL+with+KL+penalties+is+better+viewed+as+Bayesian+inference%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+pp%2E+1083%E2%80%931091%2C+Abu+Dhabi%2C+)
43. Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. [Pretraining language models with human preferences](https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/). In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17506–17533. P…
44. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. [Natural questions: A benchmark for question answering research](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/). T…
45. Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. [Quantifying the carbon emissions of machine learning](https://md.research.elamur.ai/free/en/papers/lacoste-2019-quantifying-carbon-emissions-machine-learning/). arXiv preprint arXiv:1910.09700, 2019.
46. Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Scao, T. L., Werra, L. V., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., allal, L. B., Toni, F. D., Pistilli, G., Nguyen, O., Nikpoor, S., M… — [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://md.research.elamur.ai/free/en/papers/lauren-ccon-2023-bigscience-roots-corpus-1-6tb/)
47. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8424–8445, Dublin, Ireland, Ma… [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Ippolito%2C+D%2E%2C+Nystrom%2C+A%2E%2C+Zhang%2C+C%2E%2C+Eck%2C+D%2E%2C+Callison%2DBurch%2C+C%2E%2C+and+Carlini%2C+N%2E+Deduplicating+training+data+makes+language+models+better%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+A)
48. Lesterhuis, M., Bouwer, R., Van Daal, T., Donche, V., and De Maeyer, S. Validity of comparative judgment scores: How assessors evaluate aspects of text quality when comparing argumentative texts. In Frontiers in Education, volume 7, pp. 216. Frontiers, 2022. [find](https://scholar.google.com/scholar?q=Lesterhuis%2C+M%2E%2C+Bouwer%2C+R%2E%2C+Van+Daal%2C+T%2E%2C+Donche%2C+V%2E%2C+and+De+Maeyer%2C+S%2E+Validity+of+comparative+judgment+scores%3A+How+assessors+evaluate+aspects+of+text+quality+when+comparing+argumentative+texts%2E+In+F)
49. Li, Y., Bubeck, S., Eldan, R., Giorno, A. D., Gunasekar, S., and Lee, Y. T. [Textbooks are all you need ii: phi-1.5 technical report](https://md.research.elamur.ai/free/en/papers/li-2023-textbooks-all-need-ii-phi/), 2023.
50. Liu, J., Cui, L., Liu, H., Huang, D., Wang, Y., and Zhang, Y. [Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3622–3628, 2020.
51. Lucy, L., Gururangan, S., Soldaini, L., Strubell, E., Bamman, D., Klein, L., and Dodge, J. [AboutMe: Using self-descriptions in webpages to document the effects of english pretraining data filters](https://md.research.elamur.ai/free/en/papers/lucy-2024-aboutme-self-descriptions-webpages-document/). arXiv preprint arXiv:2401.06408, 2024.
52. Manzini, T., Yao Chong, L., Black, A. W., and Tsvetkov, Y. [Black is to criminal as Caucasian is to police: Detecting and removing multiclass bias in word embeddings](https://md.research.elamur.ai/free/en/papers/manzini-2019-black-is-criminal-as-caucasian/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T…
53. Marion, M., Üstün, A., Pozzobon, L., Wang, A., Fadaee, M., and Hooker, S. When less is more: Investigating data pruning for pretraining LLMs at scale, 2023. [find](https://scholar.google.com/scholar?q=Marion%2C+M%2E%2C+U%CC%88stu%CC%88n%2C+A%2E%2C+Pozzobon%2C+L%2E%2C+Wang%2C+A%2E%2C+Fadaee%2C+M%2E%2C+and+Hooker%2C+S%2E+When+less+is+more%3A+Investigating+data+pruning+for+pretraining+LLMs+at+scale%2C+2023%2E)
54. Metsis, V., Androutsopoulos, I., and Paliouras, G. Spam filtering with naive bayes-which naive bayes? In CEAS, volume 17, pp. 28–69. Mountain View, CA, 2006. [find](https://scholar.google.com/scholar?q=Metsis%2C+V%2E%2C+Androutsopoulos%2C+I%2E%2C+and+Paliouras%2C+G%2E+Spam+filtering+with+naive+bayes%2Dwhich+naive+bayes%3F+In+CEAS%2C+volume+17%2C+pp%2E+28%E2%80%9369%2E+Mountain+View%2C+CA%2C+2006%2E)
55. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=j5BuTrEj35](https://openreview.net/forum?id=j5BuTrEj35). [↗](https://openreview.net/forum?id=j5BuTrEj35)
56. Nadeem, M., Bethke, A., and Reddy, S. [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),…
57. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
58. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., and Dean, J. [Carbon emissions and large neural network training](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/). arXiv preprint arXiv:2104.10350, 2021.
59. Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Alobeidli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Be… [find](https://scholar.google.com/scholar?q=Penedo%2C+G%2E%2C+Malartic%2C+Q%2E%2C+Hesslow%2C+D%2E%2C+Cojocaru%2C+R%2E%2C+Alobeidli%2C+H%2E%2C+Cappelli%2C+A%2E%2C+Pannier%2C+B%2E%2C+Almazrouei%2C+E%2E%2C+and+Launay%2C+J%2E+The+refinedweb+dataset+for+falcon+LLM%3A+Outperforming+curated+corpora+with+)
60. Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745–750, 2007. [find](https://scholar.google.com/scholar?q=Peters%2C+J%2E+and+Schaal%2C+S%2E+Reinforcement+learning+by+reward%2Dweighted+regression+for+operational+space+control%2E+In+Proceedings+of+the+24th+international+conference+on+Machine+learning%2C+pp%2E+745%E2%80%93750%2C+2007)
61. Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., and Miller, A. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC…
62. Pollitt, A. The method of adaptive comparative judgement. Assessment in Education: principles, policy & practice, 19(3):281–300, 2012. [find](https://scholar.google.com/scholar?q=Pollitt%2C+A%2E+The+method+of+adaptive+comparative+judgement%2E+Assessment+in+Education%3A+principles%2C+policy+%26+practice%2C+19%283%29%3A281%E2%80%93300%2C+2012%2E)
63. Pollitt, A. and Crisp, V. Could comparative judgements of script quality replace traditional marking and improve the validity of exam questions. In BERA annual conference, UMIST Manchester, England, 2004. [find](https://scholar.google.com/scholar?q=Pollitt%2C+A%2E+and+Crisp%2C+V%2E+Could+comparative+judgements+of+script+quality+replace+traditional+marking+and+improve+the+validity+of+exam+questions%2E+In+BERA+annual+conference%2C+UMIST+Manchester%2C+England%2C+2)
64. Qin, Z., Jagerman, R., Hui, K., Zhuang, H., Wu, J., Shen, J., Liu, T., Liu, J., Metzler, D., Wang, X., and Bendersky, M. [Large language models are effective text rankers with pairwise ranking prompting](https://md.research.elamur.ai/free/en/papers/qin-2024-large-language-models-are-effective/), 2023.
65. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E%2C+Young%2C+S%2E%2C+et+al%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+training+g)
66. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL [https://openreview.net/forum?id=HPuSIXJaa9](https://openreview.net/forum?id=HPuSIXJaa9). [↗](https://openreview.net/forum?id=HPuSIXJaa9)
67. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
68. Ramalho, M. A war of flags between guyana and venezuela. [https://www.bellingcat.com/news/2023/12/13/a-war-of-flags-between-guyana-and-venezuela/](https://www.bellingcat.com/news/2023/12/13/a-war-of-flags-between-guyana-and-venezuela/), 2023. Accessed: 2024-01-04. [↗](https://www.bellingcat.com/news/2023/12/13/a-war-of-flags-between-guyana-and-venezuela/)
69. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. [find](https://scholar.google.com/scholar?q=Sakaguchi%2C+K%2E%2C+Bras%2C+R%2E+L%2E%2C+Bhagavatula%2C+C%2E%2C+and+Choi%2C+Y%2E+Winogrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+Communications+of+the+ACM%2C+64%289%29%3A99%E2%80%93106%2C+2021%2E)
70. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. [Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter](https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/). ArXiv, abs/1910.01108, 2019. URL [https://api.semanticscholar.org/CorpusID:203626972](https://api.semanticscholar.org/CorpusID:203626972). [↗](https://api.semanticscholar.org/CorpusID:203626972)
71. Shazeer, N. M. GLU variants improve transformer. ArXiv, abs/2002.05202, 2020. [find](https://scholar.google.com/scholar?q=Shazeer%2C+N%2E+M%2E+GLU+variants+improve+transformer%2E+ArXiv%2C+abs%2F2002%2E05202%2C+2020%2E)
72. Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. [https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama](https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama), 2023. URL https://huggingface.co/datasets/ce… [↗](https://www.cerebras.net/blog/slimpajama-a-627b-token-cleaned-and-deduplicated-version-of-redpajama)
73. Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z.,… [find](https://scholar.google.com/scholar?q=Soldaini%2C+L%2E%2C+Kinney%2C+R%2E%2C+Bhagia%2C+A%2E%2C+Schwenk%2C+D%2E%2C+Atkinson%2C+D%2E%2C+Authur%2C+R%2E%2C+Bogin%2C+B%2E%2C+Chandu%2C+K%2E%2C+Dumas%2C+J%2E%2C+Elazar%2C+Y%2E%2C+Hofmann%2C+V%2E%2C+Jha%2C+A%2E+H%2E%2C+Kumar%2C+S%2E%2C+Lucy%2C+L%2E%2C+Lyu%2C+X%2E%2C+Magnusson%2C+I%2E%2C+Morriso)
74. Strubell, E., Ganesh, A., and McCallum, A. [Energy and policy considerations for deep learning in NLP](https://md.research.elamur.ai/free/en/papers/strubell-2019-energy-policy-considerations-deep-learning/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645–3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19…
75. Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Neurocomputing, 568:127063, 2024. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2023.127063. URL https://www.sciencedirect.com/science/article/pii/S0925231223011864. [↗](https://doi.org/10.1016/j.neucom.2023.127063)
76. Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., and Ren, Z. [Is ChatGPT good at search? investigating large language models as re-ranking agents](https://md.research.elamur.ai/free/en/papers/sun-2023-chatgpt-good-search-investigating-large/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14918– 14937, Singapore, December 2023. Asso…
77. Suresh, H. and Guttag, J. A framework for understanding sources of harm throughout the machine learning life cycle. In EAAMO ’21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385534. doi: 10.1145/3465416.3483305. URL https://doi.org/10.1145/3465416.3483305. — [doi:10.1145/3465416.3483305](https://doi.org/10.1145/3465416.3483305) [↗](https://doi.org/10.1145/3465416.3483305)
78. Tan, Y. C. and Celis, L. E. Assessing social and intersectional biases in contextualized word representations. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019. Curran Associates Inc. [find](https://scholar.google.com/scholar?q=Tan%2C+Y%2E+C%2E+and+Celis%2C+L%2E+E%2E+Assessing+social+and+intersectional+biases+in+contextualized+word+representations%2E+In+Proceedings+of+the+33rd+International+Conference+on+Neural+Information+Processing+Syst)
79. Thurstone, L. L. A law of comparative judgment. Psychological Review, 34(4):273–286, 1927. doi: 10.1037/h0070288. — [doi:10.1037/h0070288](https://doi.org/10.1037/h0070288) [find](https://scholar.google.com/scholar?q=Thurstone%2C+L%2E+L%2E+A+law+of+comparative+judgment%2E+Psychological+Review%2C+34%284%29%3A273%E2%80%93286%2C+1927%2E+doi%3A+10%2E1037%2Fh0070288%2E+%E2%80%94+%5Bdoi%3A10%2E1037%2Fh0070288%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1037%2Fh0070288%29)
80. Tirumala, K., Simig, D., Aghajanyan, A., and Morcos, A. [D4: Improving LLM pretraining via document de-duplication and diversification](https://md.research.elamur.ai/free/en/papers/tirumala-2023-d4-improving-llm-pretraining-document/). In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53983–53995. C…
81. TogetherAI. RedPajama: An open source recipe to reproduce llama training dataset, 2023. URL [https://github.com/togethercomputer/RedPajama-Data](https://github.com/togethercomputer/RedPajama-Data). [↗](https://github.com/togethercomputer/RedPajama-Data)
82. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023a. URL [https://arxiv.org/pdf/2302.13971.pdf](https://arxiv.org/pdf/2302.13971.pdf). [↗](https://arxiv.org/pdf/2302.13971.pdf)
83. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023b.
84. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL [https://proceedings.neurips.cc/paper](https://proceedings.neurips.cc/paper) files/paper/2017/file/ 3… [↗](https://proceedings.neurips.cc/paper)
85. Vieira, T. Gumbel-max trick and weighted reservoir sampling, 2014. URL [http://timvieira.github.io/blog/post/2014/](http://timvieira.github.io/blog/post/2014/) 08/01/gumbel-max-trick-and-weighted-reservoir-sampling/. [↗](http://timvieira.github.io/blog/post/2014/)
86. Wang, P., Li, L., Chen, L., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023. [find](https://scholar.google.com/scholar?q=Wang%2C+P%2E%2C+Li%2C+L%2E%2C+Chen%2C+L%2E%2C+Zhu%2C+D%2E%2C+Lin%2C+B%2E%2C+Cao%2C+Y%2E%2C+Liu%2C+Q%2E%2C+Liu%2C+T%2E%2C+and+Sui%2C+Z%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023%2E)
87. Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. [URL ht…](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
88. Welbl, J., Liu, N. F., and Gardner, M. [Crowdsourcing multiple choice science questions](https://md.research.elamur.ai/free/en/papers/welbl-2017-crowdsourcing-multiple-choice-science-questions/). In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94–106, 2017.
89. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. [CCNet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003–4012, Marseille, France, May 2020. European Lan…
90. Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., and Stoyanov, V. Training trajectories of language models across scales. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Xia%2C+M%2E%2C+Artetxe%2C+M%2E%2C+Zhou%2C+C%2E%2C+Lin%2C+X%2E+V%2E%2C+Pasunuru%2C+R%2E%2C+Chen%2C+D%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Stoyanov%2C+V%2E+Training+trajectories+of+language+models+across+scales%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Okazak)
91. Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=09iOdaeOzp](https://openreview.net/forum?id=09iOdaeOzp). [↗](https://openreview.net/forum?id=09iOdaeOzp)
92. Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P., Le, Q. V., Ma, T., and Yu, A. W. DoReMi: Optimizing data mixtures speeds up language model pre-training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL [https://openreview.net/forum?id=lXuByUeHhd](https://openreview.net/forum?id=lXuByUeHhd). [↗](https://openreview.net/forum?id=lXuByUeHhd)
93. Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems (NeurIPS), 2023b. [find](https://scholar.google.com/scholar?q=Xie%2C+S%2E+M%2E%2C+Santurkar%2C+S%2E%2C+Ma%2C+T%2E%2C+and+Liang%2C+P%2E+Data+selection+for+language+models+via+importance+resampling%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2023b%2E)
94. Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., and Arora, S. Skill-mix: a flexible and expandable family of evaluations for AI models. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=Jf5gplvglq](https://openreview.net/forum?id=Jf5gplvglq). [↗](https://openreview.net/forum?id=Jf5gplvglq)
95. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+HellaSwag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Ling)
96. Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024. URL [https://openreview.net/forum?id=tr0KidwPLc](https://openreview.net/forum?id=tr0KidwPLc). [↗](https://openreview.net/forum?id=tr0KidwPLc)
97. Zhang, X., Zhao, J., and LeCun, Y. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, pp. 649–657, Cambridge, MA, USA, 2015. MIT Press.

*Updated 2 November 2026*