---
title: "Skywork-Reward — Liu 2024"
description: "Reward models turn human preferences into a training signal for LLM alignment."
authors: ["Chris Yuhao Liu", "Liang Zeng", "Jiacai Liu", "Rui Yan", "Jujie He", "Chaojie Wang", "Shuicheng Yan", "Yang Liu", "Yahui Zhou"]
year: 2024
venue: "arXiv:2410.18451"
source: "https://arxiv.org/abs/2410.18451"
doi: "10.48550/arxiv.2410.18451"
arxiv: "2410.18451"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2024-skywork-reward-bag-tricks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

# Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs

**Authors:** Chris Yuhao Liu, Liang Zeng, Jiacai Liu, [Rui Yan](https://md.research.elamur.ai/free/en/authors/rui-yan/), Jujie He, Chaojie Wang, Shuicheng Yan, [Yang Liu](https://md.research.elamur.ai/free/en/authors/yang-liu/), Yahui Zhou · **Year:** 2024 · **Venue:** arXiv:2410.18451 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.18451) · [PDF](https://arxiv.org/pdf/2410.18451) · [DOI 10.48550/arxiv.2410.18451](https://doi.org/10.48550/arxiv.2410.18451) · [arXiv 2410.18451](https://arxiv.org/abs/2410.18451) · [OpenAlex](https://openalex.org/W4404306507)

**Citations:** 1 citations (openalex-snapshot · 2026-10-14) · [103](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Reward models turn human preferences into a training signal for LLM alignment. Their quality depends heavily on the preference data, and open-source preference datasets tend to be noisy or inconsistently labelled. The authors take a data-first approach. Using only public sources (HelpSteer2, OffsetBias, WildGuardMix and several Magpie subsets), they reduce a pool of about 378K preference pairs to a compact set of about 80K through selection and filtering. They then train discriminative reward models on Gemma-2-27B and Llama-3.1-8B and compare several loss functions. The plain Bradley-Terry objective works best. As of October 2024, Skywork-Reward-Gemma-2-27B ranked first on the RewardBench leaderboard. The team also found that some training prompts overlapped with RewardBench prompts. They released a cleaned v0.2 of the dataset, and retraining on it raised scores slightly. The work shows that a small, carefully chosen dataset can beat much larger collections. The released data has already been used to build other leading reward models.

## Key points
- **Data:** Skywork Reward Preference 80K is built from public datasets only: HelpSteer2, OffsetBias, WildGuardMix, Magpie Ultra, Magpie Pro and Magpie Air. It is far smaller than earlier mixtures such as Preference 700K or the roughly 1.5M public pairs used for Llama 2.
- **Models:** Skywork-Reward-Gemma-2-27B and Skywork-Reward-Llama-3.1-8B held 1st and 7th place on RewardBench as of October 2024. Both models and the datasets are on Hugging Face.
- **Loss ablation:** In the authors' tests, the vanilla Bradley-Terry loss beat alternatives that optimise the margin between chosen and rejected responses.
- **Contamination:** The RewardBench n-gram decontamination script (n from 7 to 13) found 5,402 contaminated prompts in v0.1, mainly from Magpie Ultra, and 445 in v0.2. By comparison, it found 15,349 in Preference 700K and 2,394 in Nectar.
- **Decontaminated retraining:** Training on about 77K pairs raised the average RewardBench score from 92.5 to 93.1 for the 8B model and from 93.8 to 94.3 for the 27B model. Safety and Reasoning improved for both models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

103 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 102 not yet.

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{liu2024skywork,
  title = {{Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs}},
  author = {Liu, Chris Yuhao and Zeng, Liang and Liu, Jiacai and Yan, Rui and He, Jujie and Wang, Chaojie and Yan, Shuicheng and Liu, Yang and Zhou, Yahui},
  year = {2024},
  doi = {10.48550/arxiv.2410.18451},
  eprint = {2410.18451},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.18451}
}
```

- [← previous SIGA: A Naturalistic NLI Dataset of English Scalar Implicatures with Gradable Adjectives](https://md.research.elamur.ai/free/en/papers/nizamani-2024-siga-naturalistic-nli-dataset-english/)
- [next → Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://md.research.elamur.ai/free/en/papers/warner-2024-smarter-better-faster-longer-modern/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

54 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Achiam%2C+S%2E+Adler%2C+S%2E+Agarwal%2C+L%2E+Ahmad%2C+I%2E+Akkaya%2C+F%2E+L%2E+Aleman%2C+D%2E+Almeida%2C+J%2E+Altenschmidt%2C+S%2E+Altman%2C+S%2E+Anadkat%2C+et+al%2E+Gpt%2D4+technical+report%2E+arXiv+preprint+arXiv%3A2303%2E08774%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhattacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11704, 2024. [find](https://scholar.google.com/scholar?q=B%2E+Adler%2C+N%2E+Agarwal%2C+A%2E+Aithal%2C+D%2E+H%2E+Anh%2C+P%2E+Bhattacharya%2C+A%2E+Brundyn%2C+J%2E+Casper%2C+B%2E+Catanzaro%2C+S%2E+Clay%2C+J%2E+Cohen%2C+et+al%2E+Nemotron%2D4+340b+technical+report%2E+arXiv+preprint+arXiv%3A2406%2E11704%2C+2024%2E)
3. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+A%2E+Askell%2C+A%2E+Chen%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+Fort%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+human+feedb)
4. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable lm 2 1.6 b technical report. arXiv preprint arXiv:2402.17834, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Bellagente%2C+J%2E+Tow%2C+D%2E+Mahan%2C+D%2E+Phung%2C+M%2E+Zhuravinskyi%2C+R%2E+Adithyan%2C+J%2E+Baicoianu%2C+B%2E+Brooks%2C+N%2E+Cooper%2C+A%2E+Datta%2C+et+al%2E+Stable+lm+2+1%2E6+b+technical+report%2E+arXiv+preprint+arXiv%3A2402%2E17834%2C+2024%2E)
5. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=R%2E+A%2E+Bradley+and+M%2E+E%2E+Terry%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
6. Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Cai%2C+M%2E+Cao%2C+H%2E+Chen%2C+K%2E+Chen%2C+K%2E+Chen%2C+X%2E+Chen%2C+X%2E+Chen%2C+Z%2E+Chen%2C+Z%2E+Chen%2C+P%2E+Chu%2C+et+al%2E+Internlm2+technical+report%2E+arXiv+preprint+arXiv%3A2403%2E17297%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/cai-2024-internlm2-technical-report/)
7. C. R. Carvalho, A. D. Polson, and J. G. Scott. The dangers of inference using the bradley-terry model. The Annals of Statistics, 38(3):1491–1514, 2010. [find](https://scholar.google.com/scholar?q=C%2E+R%2E+Carvalho%2C+A%2E+D%2E+Polson%2C+and+J%2E+G%2E+Scott%2E+The+dangers+of+inference+using+the+bradley%2Dterry+model%2E+The+Annals+of+Statistics%2C+38%283%29%3A1491%E2%80%931514%2C+2010%2E)
8. S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. [find](https://scholar.google.com/scholar?q=S%2E+Casper%2C+X%2E+Davies%2C+C%2E+Shi%2C+T%2E+K%2E+Gilbert%2C+J%2E+Scheurer%2C+J%2E+Rando%2C+R%2E+Freedman%2C+T%2E+Korbak%2C+D%2E+Lindner%2C+P%2E+Freire%2C+et+al%2E+Open+problems+and+fundamental+limitations+of+reinforcement+learning+from+human)
9. G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Cui%2C+L%2E+Yuan%2C+N%2E+Ding%2C+G%2E+Yao%2C+W%2E+Zhu%2C+Y%2E+Ni%2C+G%2E+Xie%2C+Z%2E+Liu%2C+and+M%2E+Sun%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+arXiv%3A2310%2E01377%2C+2023%2E)
10. L. Daniele and Suphavadeeprasit. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(coming soon), 2023. URL [https://huggingface.co/datasets/LDJnr/Capybara](https://huggingface.co/datasets/LDJnr/Capybara) . [↗](https://huggingface.co/datasets/LDJnr/Capybara)
11. H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024. [find](https://scholar.google.com/scholar?q=H%2E+Dong%2C+W%2E+Xiong%2C+B%2E+Pang%2C+H%2E+Wang%2C+H%2E+Zhao%2C+Y%2E+Zhou%2C+N%2E+Jiang%2C+D%2E+Sahoo%2C+C%2E+Xiong%2C+and+T%2E+Zhang%2E+Rlhf+workflow%3A+From+reward+modeling+to+online+rlhf%2E+arXiv+preprint+arXiv%3A2405%2E07863%2C+2024%2E)
12. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Dubey%2C+A%2E+Jauhri%2C+A%2E+Pandey%2C+A%2E+Kadian%2C+A%2E+Al%2DDahle%2C+A%2E+Letman%2C+A%2E+Mathur%2C+A%2E+Schelten%2C+A%2E+Yang%2C+A%2E+Fan%2C+et+al%2E+The+llama+3+herd+of+models%2E+arXiv+preprint+arXiv%3A2407%2E21783%2C+2024%2E)
13. K. Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pages 5988–6008. PMLR, 2022. [find](https://scholar.google.com/scholar?q=K%2E+Ethayarajh%2C+Y%2E+Choi%2C+and+S%2E+Swayamdipta%2E+Understanding+dataset+difficulty+with+V%2Dusable+information%2E+In+International+Conference+on+Machine+Learning%2C+pages+5988%E2%80%936008%2E+PMLR%2C+2022%2E)
14. J. H. Friedman, T. Hastie, and R. Tibshirani. The Elements of Statistical Learning. Springer, 2001. [find](https://scholar.google.com/scholar?q=J%2E+H%2E+Friedman%2C+T%2E+Hastie%2C+and+R%2E+Tibshirani%2E+The+Elements+of+Statistical+Learning%2E+Springer%2C+2001%2E)
15. L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Gao%2C+J%2E+Schulman%2C+and+J%2E+Hilton%2E+Scaling+laws+for+reward+model+overoptimization%2E+In+International+Conference+on+Machine+Learning%2C+pages+10835%E2%80%9310866%2E+PMLR%2C+2023%2E)
16. I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning. MIT press, 2016. [find](https://scholar.google.com/scholar?q=I%2E+Goodfellow%2C+Y%2E+Bengio%2C+and+A%2E+Courville%2E+Deep+Learning%2E+MIT+press%2C+2016%2E)
17. S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y. Lin, N. Lambert, Y. Choi, and N. Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. arXiv preprint arXiv:2406.18495, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Han%2C+K%2E+Rao%2C+A%2E+Ettinger%2C+L%2E+Jiang%2C+B%2E+Y%2E+Lin%2C+N%2E+Lambert%2C+Y%2E+Choi%2C+and+N%2E+Dziri%2E+Wildguard%3A+Open+one%2Dstop+moderation+tools+for+safety+risks%2C+jailbreaks%2C+and+refusals+of+llms%2E+arXiv+preprint+arXiv%3A) [our card](https://md.research.elamur.ai/free/en/papers/han-2024-wildguard-open-one-stop-moderation/)
18. H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi. Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Ivison%2C+Y%2E+Wang%2C+V%2E+Pyatkin%2C+N%2E+Lambert%2C+M%2E+Peters%2C+P%2E+Dasigi%2C+J%2E+Jang%2C+D%2E+Wadden%2C+N%2E+A%2E+Smith%2C+I%2E+Beltagy%2C+and+H%2E+Hajishirzi%2E+Camels+in+a+changing+climate%3A+Enhancing+lm+adaptation+with+tulu+2%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/ivison-2023-camels-changing-climate-enhancing-lm/)
19. J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Ji%2C+M%2E+Liu%2C+J%2E+Dai%2C+X%2E+Pan%2C+C%2E+Zhang%2C+C%2E+Bian%2C+B%2E+Chen%2C+R%2E+Sun%2C+Y%2E+Wang%2C+and+Y%2E+Yang%2E+Beavertails%3A+Towards+improved+safety+alignment+of+llm+via+a+human%2Dpreference+dataset%2E+Advances+in+Neural+Inform) [our card](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/)
20. D. Jiang, X. Ren, and B. Y. Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023. [find](https://scholar.google.com/scholar?q=D%2E+Jiang%2C+X%2E+Ren%2C+and+B%2E+Y%2E+Lin%2E+Llm%2Dblender%3A+Ensembling+large+language+models+with+pairwise+ranking+and+generative+fusion%2E+arXiv+preprint+arXiv%3A2306%2E02561%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/)
21. L. Jiang, K. Rao, S. Han, A. Ettinger, F. Brahman, S. Kumar, N. Mireshghallah, X. Lu, M. Sap, Y. Choi, et al. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. arXiv preprint arXiv:2406.18510, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Jiang%2C+K%2E+Rao%2C+S%2E+Han%2C+A%2E+Ettinger%2C+F%2E+Brahman%2C+S%2E+Kumar%2C+N%2E+Mireshghallah%2C+X%2E+Lu%2C+M%2E+Sap%2C+Y%2E+Choi%2C+et+al%2E+Wildteaming+at+scale%3A+From+in%2Dthe%2Dwild+jailbreaks+to+%28adversarially%29+safer+language+models)
22. P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661–18673, 2020. [find](https://scholar.google.com/scholar?q=P%2E+Khosla%2C+P%2E+Teterwak%2C+C%2E+Wang%2C+A%2E+Sarna%2C+Y%2E+Tian%2C+P%2E+Isola%2C+A%2E+Maschinot%2C+C%2E+Liu%2C+and+D%2E+Krishnan%2E+Supervised+contrastive+learning%2E+Advances+in+neural+information+processing+systems%2C+33%3A+18661%E2%80%9318673)
23. N. Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Lambert%2C+V%2E+Pyatkin%2C+J%2E+Morrison%2C+L%2E+Miranda%2C+B%2E+Y%2E+Lin%2C+K%2E+Chandu%2C+N%2E+Dziri%2C+S%2E+Kumar%2C+T%2E+Zick%2C+Y%2E+Choi%2C+et+al%2E+Rewardbench%3A+Evaluating+reward+models+for+language+modeling%2E+arXiv+preprint+arXiv%3A24)
24. W. Lian, B. Goodson, E. Pentland, A. Cook, C. Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orc a/OpenOrca , 2023. [find](https://scholar.google.com/scholar?q=W%2E+Lian%2C+B%2E+Goodson%2C+E%2E+Pentland%2C+A%2E+Cook%2C+C%2E+Vong%2C+and+%22Teknium%22%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrc+a%2FOpenOrca+%2C+2023%2E)
25. T. Lin. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. [find](https://scholar.google.com/scholar?q=T%2E+Lin%2E+Focal+loss+for+dense+object+detection%2E+arXiv+preprint+arXiv%3A1708%2E02002%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2017-focal-loss-dense-object-detection/)
26. X. Lou, D. Yan, W. Shen, Y. Yan, J. Xie, and J. Zhang. Uncertainty-aware reward model: Teaching reward models to know what is unknown. arXiv preprint arXiv:2410.00847, 2024. — [arXiv:2410.00847](https://arxiv.org/abs/2410.00847) [find](https://scholar.google.com/scholar?q=X%2E+Lou%2C+D%2E+Yan%2C+W%2E+Shen%2C+Y%2E+Yan%2C+J%2E+Xie%2C+and+J%2E+Zhang%2E+Uncertainty%2Daware+reward+model%3A+Teaching+reward+models+to+know+what+is+unknown%2E+arXiv+preprint+arXiv%3A2410%2E00847%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E00847%5D%28https)
27. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+Advances+in+neura)
28. J. Park, S. Jwa, M. Ren, D. Kim, and S. Choi. Offsetbias: Leveraging debiased data for tuning evaluators. arXiv preprint arXiv:2407.06551, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Park%2C+S%2E+Jwa%2C+M%2E+Ren%2C+D%2E+Kim%2C+and+S%2E+Choi%2E+Offsetbias%3A+Leveraging+debiased+data+for+tuning+evaluators%2E+arXiv+preprint+arXiv%3A2407%2E06551%2C+2024%2E)
29. R. Rafailov, J. Hejna, R. Park, and C. Finn. From 𝑟 to 𝑞 ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358, 2024a. [find](https://scholar.google.com/scholar?q=R%2E+Rafailov%2C+J%2E+Hejna%2C+R%2E+Park%2C+and+C%2E+Finn%2E+From+%F0%9D%91%9F+to+%F0%9D%91%9E+%E2%88%97+%3A+Your+language+model+is+secretly+a+q%2Dfunction%2E+arXiv+preprint+arXiv%3A2404%2E12358%2C+2024a%2E)
30. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b. [find](https://scholar.google.com/scholar?q=R%2E+Rafailov%2C+A%2E+Sharma%2C+E%2E+Mitchell%2C+C%2E+D%2E+Manning%2C+S%2E+Ermon%2C+and+C%2E+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+Advances+in+Neural+Information+Processing+Sys)
31. RyokoAI. ShareGPT52K Dataset. [https://huggingface.co/datasets/RyokoAI/ShareG](https://huggingface.co/datasets/RyokoAI/ShareG) PT52K , 2023. [↗](https://huggingface.co/datasets/RyokoAI/ShareG)
32. M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 555–562, 2010. [find](https://scholar.google.com/scholar?q=M%2E+Sanderson%2C+M%2E+L%2E+Paramita%2C+P%2E+Clough%2C+and+E%2E+Kanoulas%2E+Do+user+preferences+and+evaluation+measures+line+up%3F+In+Proceedings+of+the+33rd+international+ACM+SIGIR+conference+on+Research+and+development)
33. B. Schölkopf, A. J. Smola, K. R. Müller, P. J. Bartlett, W. S. Davidson, D. C. P. J. M. Williamson, and R. C. R. Schölkopf. Kernel methods for pattern analysis. In Proceedings of the IEEE, volume 12, pages 406–417. IEEE, 2001. [find](https://scholar.google.com/scholar?q=B%2E+Sch%C3%B6lkopf%2C+A%2E+J%2E+Smola%2C+K%2E+R%2E+M%C3%BCller%2C+P%2E+J%2E+Bartlett%2C+W%2E+S%2E+Davidson%2C+D%2E+C%2E+P%2E+J%2E+M%2E+Williamson%2C+and+R%2E+C%2E+R%2E+Sch%C3%B6lkopf%2E+Kernel+methods+for+pattern+analysis%2E+In+Proceedings+of+the+IEEE%2C+volume+12%2C+)
34. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
35. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca) , 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
36. G. Team. Gemma. 2024. doi: 10.34740/KAGGLE/M/3301. URL [https://www.kaggle.com/m/3301](https://www.kaggle.com/m/3301) . — [doi:10.34740/kaggle/m/3301](https://doi.org/10.34740/kaggle/m/3301) [↗](https://www.kaggle.com/m/3301)
37. G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Team%2C+R%2E+Anil%2C+S%2E+Borgeaud%2C+Y%2E+Wu%2C+J%2E%2DB%2E+Alayrac%2C+J%2E+Yu%2C+R%2E+Soricut%2C+J%2E+Schalkwyk%2C+A%2E+M%2E+Dai%2C+A%2E+Hauth%2C+et+al%2E+Gemini%3A+a+family+of+highly+capable+multimodal+models%2E+arXiv+preprint+arXiv%3A2312%2E11805%2C) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2023-team-gemini-family-highly-capable-multimodal/)
38. G. Team, M. Reid, N. Savinov, D. Teplyashin, L. Dmitry, T. Lillicrap, J. Alayrac, R. Soricut, A. Lazaridou, O. Firat, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. in arxiv [cs. cl]. arxiv, 2024a. [find](https://scholar.google.com/scholar?q=G%2E+Team%2C+M%2E+Reid%2C+N%2E+Savinov%2C+D%2E+Teplyashin%2C+L%2E+Dmitry%2C+T%2E+Lillicrap%2C+J%2E+Alayrac%2C+R%2E+Soricut%2C+A%2E+Lazaridou%2C+O%2E+Firat%2C+et+al%2E+Gemini+1%2E5%3A+Unlocking+multimodal+understanding+across+millions+of+tokens+of) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
39. G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024b. [find](https://scholar.google.com/scholar?q=G%2E+Team%2C+M%2E+Riviere%2C+S%2E+Pathak%2C+P%2E+G%2E+Sessa%2C+C%2E+Hardin%2C+S%2E+Bhupatiraju%2C+L%2E+Hussenot%2C+T%2E+Mesnard%2C+B%2E+Shahriari%2C+A%2E+Ram%C3%A9%2C+et+al%2E+Gemma+2%3A+Improving+open+language+models+at+a+practical+size%2E+arXiv+prepri)
40. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtuned+chat+models%2E+arXiv+preprint+arXiv%3A2)
41. H. Wang, Y. Lin, W. Xiong, R. Yang, S. Diao, S. Qiu, H. Zhao, and T. Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024a. [find](https://scholar.google.com/scholar?q=H%2E+Wang%2C+Y%2E+Lin%2C+W%2E+Xiong%2C+R%2E+Yang%2C+S%2E+Diao%2C+S%2E+Qiu%2C+H%2E+Zhao%2C+and+T%2E+Zhang%2E+Arithmetic+control+of+llms+for+diverse+user+preferences%3A+Directional+preference+alignment+with+multi%2Dobjective+rewards%2E+In+A)
42. H. Wang, W. Xiong, T. Xie, H. Zhao, and T. Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024b. [find](https://scholar.google.com/scholar?q=H%2E+Wang%2C+W%2E+Xiong%2C+T%2E+Xie%2C+H%2E+Zhao%2C+and+T%2E+Zhang%2E+Interpretable+preferences+via+multi%2Dobjective+reward+modeling+and+mixture%2Dof%2Dexperts%2E+In+EMNLP%2C+2024b%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-interpretable-preferences-multi-objective-reward/)
43. P. Wang, A. Xu, Y. Zhou, C. Xiong, and S. Joty. Direct judgement preference optimization. arXiv preprint arXiv:2409.14664, 2024c. [find](https://scholar.google.com/scholar?q=P%2E+Wang%2C+A%2E+Xu%2C+Y%2E+Zhou%2C+C%2E+Xiong%2C+and+S%2E+Joty%2E+Direct+judgement+preference+optimization%2E+arXiv+preprint+arXiv%3A2409%2E14664%2C+2024c%2E)
44. T. Wang, I. Kulikov, O. Golovneva, P. Yu, W. Yuan, J. Dwivedi-Yu, R. Y. Pang, M. Fazel-Zarandi, J. Weston, and X. Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 2024d. [find](https://scholar.google.com/scholar?q=T%2E+Wang%2C+I%2E+Kulikov%2C+O%2E+Golovneva%2C+P%2E+Yu%2C+W%2E+Yuan%2C+J%2E+Dwivedi%2DYu%2C+R%2E+Y%2E+Pang%2C+M%2E+Fazel%2DZarandi%2C+J%2E+Weston%2C+and+X%2E+Li%2E+Self%2Dtaught+evaluators%2E+arXiv+preprint+arXiv%3A2408%2E02666%2C+2024d%2E)
45. Z. Wang, Y. Dong, J. Zeng, V. Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint arXiv:2311.09528, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+Y%2E+Dong%2C+J%2E+Zeng%2C+V%2E+Adams%2C+M%2E+N%2E+Sreedhar%2C+D%2E+Egert%2C+O%2E+Delalleau%2C+J%2E+P%2E+Scowcroft%2C+N%2E+Kant%2C+A%2E+Swope%2C+et+al%2E+Helpsteer%3A+Multi%2Dattribute+helpfulness+dataset+for+steerlm%2E+arXiv+preprint+arXiv)
46. Z. Wang, Y. Dong, O. Delalleau, J. Zeng, G. Shen, D. Egert, J. J. Zhang, M. N. Sreedhar, and O. Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024e. [find](https://scholar.google.com/scholar?q=Z%2E+Wang%2C+Y%2E+Dong%2C+O%2E+Delalleau%2C+J%2E+Zeng%2C+G%2E+Shen%2C+D%2E+Egert%2C+J%2E+J%2E+Zhang%2C+M%2E+N%2E+Sreedhar%2C+and+O%2E+Kuchaiev%2E+Helpsteer2%3A+Open%2Dsource+dataset+for+training+top%2Dperforming+reward+models%2E+arXiv+preprint+arXi)
47. G. I. Winata, D. Anugraha, L. Susanto, G. Kuwanto, and D. T. Wijaya. Metametrics: Calibrating metrics for generation tasks using human preferences. arXiv preprint arXiv:2410.02381, 2024. [find](https://scholar.google.com/scholar?q=G%2E+I%2E+Winata%2C+D%2E+Anugraha%2C+L%2E+Susanto%2C+G%2E+Kuwanto%2C+and+D%2E+T%2E+Wijaya%2E+Metametrics%3A+Calibrating+metrics+for+generation+tasks+using+human+preferences%2E+arXiv+preprint+arXiv%3A2410%2E02381%2C+2024%2E)
48. Z. Xu, F. Jiang, L. Niu, Y. Deng, R. Poovendran, Y. Choi, and B. Y. Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024. [find](https://scholar.google.com/scholar?q=Z%2E+Xu%2C+F%2E+Jiang%2C+L%2E+Niu%2C+Y%2E+Deng%2C+R%2E+Poovendran%2C+Y%2E+Choi%2C+and+B%2E+Y%2E+Lin%2E+Magpie%3A+Alignment+data+synthesis+from+scratch+by+prompting+aligned+llms+with+nothing%2E+arXiv+preprint+arXiv%3A2406%2E08464%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/xu-2024-magpie-alignment-data-synthesis-scratch/)
49. R. Yang, R. Ding, Y. Lin, H. Zhang, and T. Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216, 2024. — [arXiv:2406.10216](https://arxiv.org/abs/2406.10216) [find](https://scholar.google.com/scholar?q=R%2E+Yang%2C+R%2E+Ding%2C+Y%2E+Lin%2C+H%2E+Zhang%2C+and+T%2E+Zhang%2E+Regularizing+hidden+states+enables+learning+generalizable+reward+model+for+llms%2E+arXiv+preprint+arXiv%3A2406%2E10216%2C+2024%2E+%E2%80%94+%5BarXiv%3A2406%2E10216%5D%28https%3A%2F%2Fa)
50. L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y. Lin, et al. Advancing llm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024. [find](https://scholar.google.com/scholar?q=L%2E+Yuan%2C+G%2E+Cui%2C+H%2E+Wang%2C+N%2E+Ding%2C+X%2E+Wang%2C+J%2E+Deng%2C+B%2E+Shan%2C+H%2E+Chen%2C+R%2E+Xie%2C+Y%2E+Lin%2C+et+al%2E+Advancing+llm+reasoning+generalists+with+preference+trees%2E+arXiv+preprint+arXiv%3A2404%2E02078%2C+2024%2E)
51. Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023. [find](https://scholar.google.com/scholar?q=Z%2E+Zeng%2C+J%2E+Yu%2C+T%2E+Gao%2C+Y%2E+Meng%2C+T%2E+Goyal%2C+and+D%2E+Chen%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+arXiv+preprint+arXiv%3A2310%2E07641%2C+2023%2E)
52. Y. Zhang, G. Zhang, Y. Wu, K. Xu, and Q. Gu. General preference modeling with preference representations for aligning language models. arXiv preprint arXiv:2410.02197, 2024. — [arXiv:2410.02197](https://arxiv.org/abs/2410.02197) [find](https://scholar.google.com/scholar?q=Y%2E+Zhang%2C+G%2E+Zhang%2C+Y%2E+Wu%2C+K%2E+Xu%2C+and+Q%2E+Gu%2E+General+preference+modeling+with+preference+representations+for+aligning+language+models%2E+arXiv+preprint+arXiv%3A2410%2E02197%2C+2024%2E+%E2%80%94+%5BarXiv%3A2410%2E02197%5D%28https)
53. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. [find](https://scholar.google.com/scholar?q=L%2E+Zheng%2C+W%2E%2DL%2E+Chiang%2C+Y%2E+Sheng%2C+S%2E+Zhuang%2C+Z%2E+Wu%2C+Y%2E+Zhuang%2C+Z%2E+Lin%2C+Z%2E+Li%2C+D%2E+Li%2C+E%2E+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advances+in+Neural+Information+Processing+S)
54. B. Zhu, E. Frick, T. Wu, H. Zhu, and J. Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023. [find](https://scholar.google.com/scholar?q=B%2E+Zhu%2C+E%2E+Frick%2C+T%2E+Wu%2C+H%2E+Zhu%2C+and+J%2E+Jiao%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2C+November+2023%2E)