---
title: "ChatGLM-RLHF — paper by Hou et al. (2024)"
description: "This paper from Zhipu AI and Tsinghua University describes how the team built and ran the RLHF… 42 references, cited by 14 papers here."
authors: ["Zhenyu Hou", "Yilin Niu", "Zhengxiao Du", "Xiaohan Zhang", "Xiao Liu", "Aohan Zeng", "Qinkai Zheng", "Minlie Huang", "Hongning Wang", "Jie Tang", "Yuxiao Dong"]
year: 2024
venue: "arXiv:2404.00934"
source: "https://arxiv.org/abs/2404.00934"
doi: "10.48550/arxiv.2404.00934"
arxiv: "2404.00934"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hou-2024-chatglm-rlhf-practices-aligning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hou-2024-chatglm-rlhf-practices-aligning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/) / ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

# ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

**Authors:** [Zhenyu Hou](https://md.research.elamur.ai/free/en/authors/zhenyu-hou/), [Yilin Niu](https://md.research.elamur.ai/free/en/authors/yilin-niu/), [Zhengxiao Du](https://md.research.elamur.ai/free/en/authors/zhengxiao-du/), [Xiaohan Zhang](https://md.research.elamur.ai/free/en/authors/xiaohan-zhang/), [Xiao Liu](https://md.research.elamur.ai/free/en/authors/xiao-liu/), [Aohan Zeng](https://md.research.elamur.ai/free/en/authors/aohan-zeng/), [Qinkai Zheng](https://md.research.elamur.ai/free/en/authors/qinkai-zheng/), [Minlie Huang](https://md.research.elamur.ai/free/en/authors/minlie-huang/), [Hongning Wang](https://md.research.elamur.ai/free/en/authors/hongning-wang/), [Jie Tang](https://md.research.elamur.ai/free/en/authors/jie-tang/) and [Yuxiao Dong](https://md.research.elamur.ai/free/en/authors/yuxiao-dong/) · **Year:** 2024 · **Venue:** arXiv:2404.00934 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2404.00934) · [PDF](https://arxiv.org/pdf/2404.00934) · [DOI 10.48550/arxiv.2404.00934](https://doi.org/10.48550/arxiv.2404.00934) · [arXiv 2404.00934](https://arxiv.org/abs/2404.00934) · [OpenAlex](https://openalex.org/W4393729321)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/hou-2024-chatglm-rlhf-practices-aligning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper from Zhipu AI and Tsinghua University describes how the team built and ran the RLHF pipeline used in production for the ChatGLM assistant, with particular attention to Chinese. The pipeline has three stages. First, annotators make pairwise comparisons of SFT-model outputs, following guidelines on helpfulness, harmlessness and fluency, and cyclic or tied judgments are filtered out. Second, a reward model is trained on these comparisons, with measures to keep it from learning shortcuts such as favoring longer answers. Third, the policy is optimized with PPO (online) and DPO (offline). To make training work at scale, the authors subtract a baseline reward to reduce variance, use model parallelism with fused gradient descent, and add an SFT next-token-prediction loss to limit catastrophic forgetting. Tested on ChatGLM-6B and ChatGLM-32B, the RLHF models beat ChatGLM-SFT on Chinese alignment tasks by about 15% more wins on average. The work is a practical, industry-scale account of RLHF beyond small models and parameter-efficient tuning.

## Key points
- Running PPO on a 32B-parameter model takes roughly 8 or 16 A100 (8×80G) servers for 2–3 days. This scale is rarely covered by prior open RLHF studies.
- The ChatGLM-32B reward model agrees with human preferences about 65% of the time on an internal test set. This is similar to the accuracy reported for Llama2-70B, though the two were measured on different data.
- Both DPO and PPO make responses longer, by about 110 and 75 tokens respectively. PPO still scores higher than DPO, so the authors argue its outputs carry less redundancy.
- In human evaluation, writing and practical programming questions improved most. Math reasoning barely changed, and about 55% of comparisons were ties.
- Ablations show that subtracting a reference reward starts training rewards near zero and stabilizes PPO. The tuned models serve chatglm.cn and its iOS and Android apps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang and Yuxiao Dong. 2024. ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback. arXiv:2404.00934.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hou2024chatglm,
  title = {{ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback}},
  author = {Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang and Yuxiao Dong},
  year = {2024},
  doi = {10.48550/arxiv.2404.00934},
  eprint = {2404.00934},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.00934}
}
```

- [← previous Assessing the Cross-linguistic Utility of Abstract Meaning Representation](https://md.research.elamur.ai/free/en/papers/wein-2024-assessing-cross-linguistic-utility-abstract/)
- [next → Comparing Humans and Large Language Models on an Experimental Protocol Inventory for Theory of Mind Evaluation (EPITOME)](https://md.research.elamur.ai/free/en/papers/jones-2024-comparing-humans-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/8/)

## References

42 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774 (2023).
2. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609 (2023).
3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 (2022). [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+2022a%2E+Training+a+helpful+and+harmless+assistant+with+rei)
4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073 (2022).
5. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623. [find](https://scholar.google.com/scholar?q=Emily+M+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F%2E+In+Proceedings+of+the+2021+ACM+conference)
6. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint arXiv:2401.02954 (2024). [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+et+al%2E+2024%2E+DeepSeek+LLM%3A+Scaling+Open%2DSource+Language+Models+with+Longtermism%2E+)
7. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. [On the opportunities and risks of foundation models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/). arXiv preprint arXiv:2108.07258 (2021).
8. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika 39, 3/4 (1952). [find](https://scholar.google.com/scholar?q=Ralph+Allan+Bradley+and+Milton+E+Terry%2E+1952%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+The+method+of+paired+comparisons%2E+Biometrika+39%2C+3%2F4+%281952%29%2E)
9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/).
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374 (2021).
11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). Journal of Machine Learning Research 24, 240 (2023), 1–113.
12. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2022%2E+Scaling+instruction%2Dfinetuned+language+models%2E+a)
13. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377 (2023). [find](https://scholar.google.com/scholar?q=Ganqu+Cui%2C+Lifan+Yuan%2C+Ning+Ding%2C+Guanming+Yao%2C+Wei+Zhu%2C+Yuan+Ni%2C+Guotong+Xie%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2E+arXiv+preprint+ar)
14. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. [Safe rlhf: Safe reinforcement learning from human feedback](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/). arXiv preprint arXiv:2310.12773 (2023).
15. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/). In Annual Meeting of the Association for Computational Linguistics. [https://api.semanticscholar.org/](https://api.semanticscholar.org/) CorpusID:247519241 [↗](https://api.semanticscholar.org/)
16. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. [The capacity for moral self-correction in large language models](https://md.research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/). arXiv preprint arXiv:2302.07459 (2023).
17. Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. 2023. trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pr… [find](https://scholar.google.com/scholar?q=Alexander+Havrilla%2C+Maksym+Zhuravinskyi%2C+Duy+Phung%2C+Aman+Tiwari%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Quentin+Anthony%2C+and+Louis+Castricato%2E+2023%2E+trlX%3A+A+Framework+for+Large+Scale+Reinforcement+Learning+fr)
18. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. arXiv preprint arXiv:2103.14659 (2021). [find](https://scholar.google.com/scholar?q=Zachary+Kenton%2C+Tom+Everitt%2C+Laura+Weidinger%2C+Iason+Gabriel%2C+Vladimir+Mikulik%2C+and+Geoffrey+Irving%2E+2021%2E+Alignment+of+language+agents%2E+arXiv+preprint+arXiv%3A2103%2E14659+%282021%29%2E)
19. Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023. [Alignbench: Benchmarking chinese alignment of large language models](https://md.research.elamur.ai/free/en/papers/liu-2023-alignbench-benchmarking-chinese-alignment/). arXiv preprint arXiv:2311.18743 (2023).
20. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/).
21. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the Interna… [find](https://scholar.google.com/scholar?q=Deepak+Narayanan%2C+Mohammad+Shoeybi%2C+Jared+Casper%2C+Patrick+LeGresley%2C+Mostofa+Patwary%2C+Vijay+Korthikanti%2C+Dmitri+Vainbrand%2C+Prethvi+Kashinkunti%2C+Julie+Bernauer%2C+Bryan+Catanzaro%2C+et+al%2E+2021%2E+Efficient+)
22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/).
23. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). arXiv preprint arXiv:2305.18290 (2023).
24. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506. [find](https://scholar.google.com/scholar?q=Jeff+Rasley%2C+Samyam+Rajbhandari%2C+Olatunji+Ruwase%2C+and+Yuxiong+He%2E+2020%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26t)
25. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347+%282017%29%2E)
26. Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2310.05199 (2023). [find](https://scholar.google.com/scholar?q=Wei+Shen%2C+Rui+Zheng%2C+Wenyu+Zhan%2C+Jun+Zhao%2C+Shihan+Dou%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2023%2E+Loose+lips+sink+ships%3A+Mitigating+Length+Bias+in+Reinforcement+Learning+from+Human+Feedback%2E+arXiv+p)
27. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. [A long way to go: Investigating length correlations in rlhf](https://md.research.elamur.ai/free/en/papers/singhal-2023-long-way-go-investigating-length/). arXiv preprint arXiv:2310.03716 (2023).
28. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023. [Preference ranking optimization for human alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/). arXiv preprint arXiv:2306.17492 (2023).
29. Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2023. [Salmon: Self-alignment with principle-following reward models](https://md.research.elamur.ai/free/en/papers/sun-2023-salmon-self-alignment-instructable-reward/). arXiv preprint arXiv:2310.05910 (2023).
30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971 (2023).
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288 (2023).
32. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/).
33. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022). [find](https://scholar.google.com/scholar?q=BigScience+Workshop%2C+Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagn%C3%A9%2C+Alexandra+Sasha+Luccioni%2C+Fran%C3%A7ois+Yvon%2C+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dpara)
34. Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. 2023. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment. arXiv preprint arXiv:2310.00212 (2023). — [arXiv:2310.00212](https://arxiv.org/abs/2310.00212) [find](https://scholar.google.com/scholar?q=Tianhao+Wu%2C+Banghua+Zhu%2C+Ruoyu+Zhang%2C+Zhaojin+Wen%2C+Kannan+Ramchandran%2C+and+Jiantao+Jiao%2E+2023%2E+Pairwise+proximal+policy+optimization%3A+Harnessing+relative+feedback+for+llm+alignment%2E+arXiv+preprint+arX)
35. Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. 2023. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320 (20… — [arXiv:2308.01320](https://arxiv.org/abs/2308.01320) [find](https://scholar.google.com/scholar?q=Zhewei+Yao%2C+Reza+Yazdani+Aminabadi%2C+Olatunji+Ruwase%2C+Samyam+Rajbhandari%2C+Xiaoxia+Wu%2C+Ammar+Ahmad+Awan%2C+Jeff+Rasley%2C+Minjia+Zhang%2C+Conglong+Li%2C+Connor+Holmes%2C+et+al%2E+2023%2E+Deepspeed%2Dchat%3A+Easy%2C+fast+an)
36. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. [Glm-130b: An open bilingual pre-trained model](https://md.research.elamur.ai/free/en/papers/zeng-2022-glm-130b-open-bilingual-pre/).
37. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv preprint arXiv:2205.01068 (2022).
38. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. 2023. [Slic-hf: Sequence likelihood calibration with human feedback](https://md.research.elamur.ai/free/en/papers/zhao-2023-slic-hf-sequence-likelihood-calibration/). arXiv preprint arXiv:2305.10425 (2023).
39. Chenyu Zheng, Guoqiang Wu, Fan Bao, Yue Cao, Chongxuan Li, and Jun Zhu. 2023c. Revisiting discriminative vs. generative classifiers: Theory and implications. arXiv preprint arXiv:2302.02334 (2023). — [arXiv:2302.02334](https://arxiv.org/abs/2302.02334) [find](https://scholar.google.com/scholar?q=Chenyu+Zheng%2C+Guoqiang+Wu%2C+Fan+Bao%2C+Yue+Cao%2C+Chongxuan+Li%2C+and+Jun+Zhu%2E+2023c%2E+Revisiting+discriminative+vs%2E+generative+classifiers%3A+Theory+and+implications%2E+arXiv+preprint+arXiv%3A2302%2E02334+%282023%29%2E+%E2%80%94+)
40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023a. [Judging LLM-as-a-judge with MT-Bench and Chatbot Arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685 (2023).
41. Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. 2023b. [Secrets of rlhf in large language models part i: Ppo](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/). arXiv preprint arXiv:2307.04964 (2023).
42. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 (2023). [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206+%282023%29%2E)

*Updated 30 October 2026*