---
title: "HelpSteer2 — paper by Wang et al. (2024) — summary and ideas"
description: "Reward models that steer LLMs toward human-preferred outputs need good preference data. 68 references, cited by 82 papers here."
authors: ["Zhilin Wang", "Yi Dong", "Olivier Delalleau", "Jiaqi Zeng", "Gerald Shen", "Daniel Egert", "Jimmy J. Zhang", "Makesh Narsimhan Sreedhar", "Oleksii Kuchaiev"]
year: 2024
venue: "arXiv:2406.08673"
source: "https://arxiv.org/abs/2406.08673"
doi: "10.48550/arxiv.2406.08673"
arxiv: "2406.08673"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2024-helpsteer2-open-source-dataset-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2024-helpsteer2-open-source-dataset-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/) / HelpSteer2: Open-source dataset for training top-performing reward models

# HelpSteer2: Open-source dataset for training top-performing reward models

**Authors:** [Zhilin Wang](https://md.research.elamur.ai/free/en/authors/zhilin-wang/), [Yi Dong](https://md.research.elamur.ai/free/en/authors/yi-dong/), [Olivier Delalleau](https://md.research.elamur.ai/free/en/authors/olivier-delalleau/), [Jiaqi Zeng](https://md.research.elamur.ai/free/en/authors/jiaqi-zeng/), [Gerald Shen](https://md.research.elamur.ai/free/en/authors/gerald-shen/), [Daniel Egert](https://md.research.elamur.ai/free/en/authors/daniel-egert/), Jimmy J. Zhang, [Makesh Narsimhan Sreedhar](https://md.research.elamur.ai/free/en/authors/makesh-narsimhan-sreedhar/) and [Oleksii Kuchaiev](https://md.research.elamur.ai/free/en/authors/oleksii-kuchaiev/) · **Year:** 2024 · **Venue:** arXiv:2406.08673 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.08673) · [PDF](https://arxiv.org/pdf/2406.08673) · [DOI 10.48550/arxiv.2406.08673](https://doi.org/10.48550/arxiv.2406.08673) · [arXiv 2406.08673](https://arxiv.org/abs/2406.08673)

**Citations:** [82](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer2-open-source-dataset-training/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reward models that steer LLMs toward human-preferred outputs need good preference data. Openly licensed collections such as HH-RLHF, Open Assistant and HelpSteer have aged, and sets distilled from GPT-4 and similar proprietary systems carry commercial-use restrictions. The NVIDIA authors address this gap with HelpSteer2, a CC-BY-4.0 dataset of about 10k prompts, mostly taken from ShareGPT user turns. Each prompt is paired with two responses from different sources: several generations of in-house Nemotron models, Mixtral-8x7B-Instruct-v0.1, and human writers. Multiple Scale AI annotators rated every response on five attributes using a Likert-5 scale. Heavy quality control and filtering of disagreeing labels raised agreement substantially. A reward model built on a strong internal base model and trained on this data reached the top RewardBench score at release. The data also proved useful for alignment. A Llama 3 70B reward model guided DPO, PPO and a new method, SteerLM 2.0, turning Llama 3 70B Base into models that match Llama 3 70B Instruct with a small fraction of its data. The work shows that a compact, carefully labeled, commercially usable dataset can compete with much larger ones.

## Key points
- **Composition:** 21,362 rated responses for 10,681 prompts.
  - Over 95% of prompts come from ShareGPT, with original assistant turns discarded.
  - About 29% of prompts are multi-turn.
  - Non-English prompts (detected with FastText) and code prompts were removed.
  - Sampling was balanced across ~1000 BERTopic clusters and across complexity levels.
- **Annotation:** each response is scored for helpfulness, correctness, coherence, complexity and verbosity.
  - Roughly 1,000 US-based Scale AI annotators took part, averaging 3.41 per response.
  - Around half of all annotations were ultimately discarded.
  - Helpfulness agreement (quadratic weighted Cohen's κ) rose from 0.465 to 0.706, then to 0.791 after post-processing.
- **Reward modeling:** the reward model scored 92.0% on RewardBench's primary dataset, ranking first as of 12 June 2024. This was achieved with only ~10k pairs, roughly ten times fewer than HH-RLHF.
- **Alignment:** the recipe used the Llama 3 70B Reward Model with 10k HelpSteer2 pairs plus 100k SFT samples (Daring Anteater), about 1% of Llama 3 70B Instruct's data.
  - Across four benchmarks (Arena Hard, AlpacaEval 2.0 LC, TruthfulQA, and MT Bench), the resulting models equaled or exceeded Llama 3 70B Instruct and outperformed GPT-4-0613.
  - DPO led on TruthfulQA and Arena Hard, PPO on AlpacaEval 2.0 LC, and SteerLM on MT Bench.
- **Resources:** the dataset is at huggingface.co/datasets/nvidia/HelpSteer2, and the code is in NVIDIA's NeMo-Aligner.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Cite

Cite as: Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar and Oleksii Kuchaiev. 2024. HelpSteer2: Open-source dataset for training top-performing reward models. arXiv:2406.08673.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2024helpsteer2,
  title = {{HelpSteer2: Open-source dataset for training top-performing reward models}},
  author = {Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar and Oleksii Kuchaiev},
  year = {2024},
  doi = {10.48550/arxiv.2406.08673},
  eprint = {2406.08673},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.08673}
}
```

- [← previous CodeGemma: Open Code Models Based on Gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/)
- [next → Is Self-Repair a Silver Bullet for Code Generation?](https://md.research.elamur.ai/free/en/papers/olausson-2024-self-repair-silver-bullet-code/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/5/)

## References

68 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Saurav+Kadavath%2C+Jackson+Kernion%2C+Tom+Conerly%2C+)
3. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023. [find](https://scholar.google.com/scholar?q=Lewis+Tunstall%2C+Edward+Beeching%2C+Nathan+Lambert%2C+Nazneen+Rajani%2C+Kashif+Rasul%2C+Younes+Belkada%2C+Shengyi+Huang%2C+Leandro+von+Werra%2C+Cl%C3%A9mentine+Fourrier%2C+Nathan+Habib%2C+Nathan+Sarrazin%2C+Omar+Sanseviero%2C+Al)
4. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
5. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
6. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf), 2024. [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf)
7. Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibau… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
8. Meta AI. Llama 3 model card. [https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md), 2024. [↗](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)
9. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
10. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudo… — [Yi: Open Foundation Models by 01.AI](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
11. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Opena… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
12. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. [Helpsteer: Multi-attribute helpfulness dataset for steerlm](https://md.research.elamur.ai/free/en/papers/wang-2024-helpsteer-multi-attribute-helpfulness-dataset/), 2023.
13. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/), 2024.
14. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Valentina+Pyatkin%2C+Jacob+Morrison%2C+LJ+Miranda%2C+Bill+Yuchen+Lin%2C+Khyathi+Chandu%2C+Nouha+Dziri%2C+Sachin+Kumar%2C+Tom+Zick%2C+Yejin+Choi%2C+Noah+A%2E+Smith%2C+and+Hannaneh+Hajishirzi%2E+Rewardbench%3A+Ev)
15. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
16. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving LLM helpfulness & harmlessness with RLAIF, November 2023. [find](https://scholar.google.com/scholar?q=Banghua+Zhu%2C+Evan+Frick%2C+Tianhao+Wu%2C+Hanlin+Zhu%2C+and+Jiantao+Jiao%2E+Starling%2D7b%3A+Improving+LLM+helpfulness+%26+harmlessness+with+RLAIF%2C+November+2023%2E)
17. Argilla. Distilabel capybara. [https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized), 2024. [↗](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized)
18. Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf, 2023. [find](https://scholar.google.com/scholar?q=Yi+Dong%2C+Zhilin+Wang%2C+Makesh+Narsimhan+Sreedhar%2C+Xianchao+Wu%2C+and+Oleksii+Kuchaiev%2E+Steerlm%3A+Attribute+conditioned+sft+as+an+%28user%2Dsteerable%29+alternative+to+rlhf%2C+2023%2E)
19. RyokoAI. RyokoAI/ShareGPT52K. [https://huggingface.co/datasets/RyokoAI/ShareGPT52K](https://huggingface.co/datasets/RyokoAI/ShareGPT52K), 2023. [↗](https://huggingface.co/datasets/RyokoAI/ShareGPT52K)
20. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. [Enriching word vectors with subword information](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/). Transactions of the Association for Computational Linguistics, 5:135–146, 2017.
21. Maarten Grootendorst. [Bertopic: Neural topic modeling with a class-based tf-idf procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/). arXiv preprint arXiv:2203.05794, 2022.
22. Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. \[#InsTag: Instruction tagging for analyzing supervised fine-tuning of large language models](https://md.research.elamur.ai/free/en/papers/lu-2023-instag-instruction-tagging-analyzing-supervised/), 2023.
23. Vivienne Zhang, Shashank Verma, Neal Vaidya, Abhishek Sawarkar, and Amanda Saunders. NVIDIA AI foundation models: Build custom enterprise chatbots and co-pilots with production-ready LLMs, 2023. [https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-) [↗](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-)
24. Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subramanian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, Vibhu Jawa, Jiwei Liu, Ameya Mahabaleshwarkar, Osvald Nitski, Annika Brundyn, James Maki, Miguel Martinez, Jiaxuan You, John Kamalu,… — [Nemotron-4 15B Technical Report](https://md.research.elamur.ai/free/en/papers/parmar-2024-nemotron-4-15b-technical-report/)
25. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. [find](https://scholar.google.com/scholar?q=Mohammad+Shoeybi%2C+Mostofa+Patwary%2C+Raul+Puri%2C+Patrick+LeGresley%2C+Jared+Casper%2C+and+Bryan+Catanzaro%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2C+2020%2E)
26. Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. NeMo-Aligner: Scalable toolkit for efficient model alignment, 2024. [find](https://scholar.google.com/scholar?q=Gerald+Shen%2C+Zhilin+Wang%2C+Olivier+Delalleau%2C+Jiaqi+Zeng%2C+Yi+Dong%2C+Daniel+Egert%2C+Shengyang+Sun%2C+Jimmy+Zhang%2C+Sahil+Jain%2C+Ali+Taghibakhshi%2C+Markel+Sanz+Ausin%2C+Ashwath+Aithal%2C+and+Oleksii+Kuchaiev%2E+NeMo%2D)
27. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume… [find](https://scholar.google.com/scholar?q=Paul+F+Christiano%2C+Jan+Leike%2C+Tom+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+I%2E+Guyon%2C+U%2E+Von+Luxburg%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+R%2E+Fergus%2C)
28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/), 2023.
29. Ron Artstein and Massimo Poesio. [Survey article: Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational Linguistics, 34(4):555–596, 2008.
30. Scikit-Learn. Cohen kappa score. [https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_2024](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_2024). [↗](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_2024)
31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-asa-judge with mt-bench and chatbot arena, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric%2E+P+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+llm%2Dasa%2Djudg)
32. Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+arXiv%3A1711%2E05101%2C+2017%2E)
33. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. [RLHF workflow: From reward modeling to online RLHF](https://md.research.elamur.ai/free/en/papers/dong-2024-rlhf-workflow-reward-modeling-online/), 2024.
34. Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. [Advancing LLM reasoning generalists with preference trees](https://md.research.elamur.ai/free/en/papers/yuan-2024-advancing-llm-reasoning-generalists-preference/), 2024.
35. Allen AI. Reward bench leaderboard. [https://huggingface.co/spaces/allenai/reward-bench](https://huggingface.co/spaces/allenai/reward-bench), 2024. [↗](https://huggingface.co/spaces/allenai/reward-bench)
36. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/), 2024.
37. Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback, 2023. [find](https://scholar.google.com/scholar?q=Nathan+Lambert+and+Roberto+Calandra%2E+The+alignment+ceiling%3A+Objective+mismatch+in+reinforcement+learning+from+human+feedback%2C+2023%2E)
38. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/), 2023.
39. Stephanie Lin, Jacob Hilton, and Owain Evans. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–325…
40. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2C+2024%2E)
41. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The Arena-Hard pipeline. [https://lmsys.org/blog/2024-04-19-arena-hard/](https://lmsys.org/blog/2024-04-19-arena-hard/), April 2024. [↗](https://lmsys.org/blog/2024-04-19-arena-hard/)
42. Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward, 2024. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+SimPO%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2C+2024%2E)
43. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233, 2023.
44. Meta AI. Introducing Meta Llama 3: The most capable openly available LLM to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/), 2024. [↗](https://ai.meta.com/blog/meta-llama-3/)
45. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. [Arb: Advanced reasoning benchmark for large language models](https://md.research.elamur.ai/free/en/papers/sawada-2023-arb-advanced-reasoning-benchmark-large/). arXiv preprint arXiv:2307.13692, 2023.
46. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. [Scibench: Evaluating college-level scientific problem-solving abilities of large language models](https://md.research.elamur.ai/free/en/papers/wang-2023-scibench-evaluating-college-level-scientific/). arXiv preprint arXiv:2307.10635, 2023.
47. TigerResearch. tigerbot-kaggle-leetcodesolutions-en-2k, 2023. [https://huggingface.co/datasets/TigerResearch/tigerbot-](https://huggingface.co/datasets/TigerResearch/tigerbot-) [↗](https://huggingface.co/datasets/TigerResearch/tigerbot-)
48. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.[20050, 2023](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/).
49. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. [Finqa: A dataset of numerical reasoning over financial data](https://md.research.elamur.ai/free/en/papers/chen-2021-finqa-dataset-numerical-reasoning-over/). arXiv preprint arXiv:2109.00122, 2021. Decoupled weight decay regularization. arXiv preprin…
50. Panupong Pasupat and Percy Liang. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). arXiv preprint arXiv:1508.00305, 2015.
51. Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, and Ramón Fernandez Astudillo. Brain: Bayesian reward-conditioned amortized inference for natural language generation from feedback, 2024. [find](https://scholar.google.com/scholar?q=Gaurav+Pandey%2C+Yatin+Nandwani%2C+Tahira+Naseem%2C+Mayank+Mishra%2C+Guangxuan+Xu%2C+Dinesh+Raghu%2C+Sachindra+Joshi%2C+Asim+Munawar%2C+and+Ram%C3%B3n+Fernandez+Astudillo%2E+Brain%3A+Bayesian+reward%2Dconditioned+amortized+infe)
52. HuggingFace. Open llm leaderboard. [https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard), 2024. [↗](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
53. Sai Vinay G. Scores of GPT3.5 and GPT4 for comparison. [https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/30\#6474fd3b82907acd](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/30\#6474fd3b82907acd) 2023. [↗](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/30\#6474fd3b82907acd)
54. Tatsu-Lab. Alpacaeval leaderboard. [https://tatsu-lab.github.io/alpaca_eval/](https://tatsu-lab.github.io/alpaca_eval/), 2023. [↗](https://tatsu-lab.github.io/alpaca_eval/)
55. LMSys. Arena-hard-auto leaderboard. [https://github.com/lm-sys/arena-hard-auto](https://github.com/lm-sys/arena-hard-auto), 2024. [↗](https://github.com/lm-sys/arena-hard-auto)
56. Philipp Schmid, Omar Sanseviero, Pedro Cuenca, Younes Belkada, and Leandro von Werra. Welcome Llama 3 - Meta’s new open LLM. [https://huggingface.co/blog/llama3](https://huggingface.co/blog/llama3), 2024. [↗](https://huggingface.co/blog/llama3)
57. Aaron J. Li, Satyapriya Krishna, and Himabindu Lakkaraju. More rlhf, more trust? on the impact of human preference alignment on language model trustworthiness, 2024. [find](https://scholar.google.com/scholar?q=Aaron+J%2E+Li%2C+Satyapriya+Krishna%2C+and+Himabindu+Lakkaraju%2E+More+rlhf%2C+more+trust%3F+on+the+impact+of+human+preference+alignment+on+language+model+trustworthiness%2C+2024%2E)
58. Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023. [find](https://scholar.google.com/scholar?q=Traian+Rebedea%2C+Razvan+Dinu%2C+Makesh+Sreedhar%2C+Christopher+Parisien%2C+and+Jonathan+Cohen%2E+Nemo+guardrails%3A+A+toolkit+for+controllable+and+safe+llm+applications+with+programmable+rails%2C+2023%2E)
59. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted qu… — [WebGPT: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
60. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2C+2022%2E)
61. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Yejin+Choi%2C+and+Swabha+Swayamdipta%2E+Understanding+dataset+difficulty+with+V%2Dusable+information%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Szepesvari%2C+Gang+Niu%2C+and+Siva)
62. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023. [find](https://scholar.google.com/scholar?q=Harrison+Lee%2C+Samrat+Phatale%2C+Hassan+Mansoor%2C+Thomas+Mesnard%2C+Johan+Ferret%2C+Kellie+Lu%2C+Colton+Bishop%2C+Ethan+Hall%2C+Victor+Carbune%2C+Abhinav+Rastogi%2C+and+Sushant+Prakash%2E+Rlaif%3A+Scaling+reinforcement+lea)
63. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. [find](https://scholar.google.com/scholar?q=Paul+Christiano%2C+Jan+Leike%2C+Tom+B%2E+Brown%2C+Miljan+Martic%2C+Shane+Legg%2C+and+Dario+Amodei%2E+Deep+reinforcement+learning+from+human+preferences%2C+2023%2E)
64. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
65. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeff+Wu%2C+Daniel+M%2E+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+Christiano%2E+Learning+to+summarize+from+human+feedback%2E+In+NeurIPS%2C+2020%2E)
66. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162… [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Yejin+Choi%2C+and+Swabha+Swayamdipta%2E+Understanding+dataset+difficulty+with+V%2Dusable+information%2E+In+Kamalika+Chaudhuri%2C+Stefanie+Jegelka%2C+Le+Song%2C+Csaba+Szepesvari%2C+Gang+Niu%2C+and+Siva)
67. Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and… — [Secrets of RLHF in Large Language Models Part II: Reward Modeling](https://md.research.elamur.ai/free/en/papers/wang-2024-secrets-rlhf-large-language-models/)
68. Tenyx. Tenyxchat: Language model alignment using tenyx fine-tuning, 2024. [find](https://scholar.google.com/scholar?q=Tenyx%2E+Tenyxchat%3A+Language+model+alignment+using+tenyx+fine%2Dtuning%2C+2024%2E)

*Updated 2 November 2026*