---
title: "BeaverTails: Towards Improved Safety Alignment of — Ji 2023"
description: "Aligning LLMs for safety depends on large amounts of human feedback."
authors: ["Jiaming Ji", "Mickel Liu", "Juntao Dai", "Xuehai Pan", "Chi Zhang", "Ce Bian", "Chi Zhang", "Ruiyang Sun", "Yizhou Wang", "Yaodong Yang"]
year: 2023
venue: "arXiv:2307.04657"
source: "https://arxiv.org/abs/2307.04657"
doi: "10.48550/arxiv.2307.04657"
arxiv: "2307.04657"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/) / BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

# BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

**Authors:** [Jiaming Ji](https://md.research.elamur.ai/free/en/authors/jiaming-ji/), [Mickel Liu](https://md.research.elamur.ai/free/en/authors/mickel-liu/), Juntao Dai, [Xuehai Pan](https://md.research.elamur.ai/free/en/authors/xuehai-pan/), [Chi Zhang](https://md.research.elamur.ai/free/en/authors/chi-zhang/), [Ce Bian](https://md.research.elamur.ai/free/en/authors/ce-bian/), [Chi Zhang](https://md.research.elamur.ai/free/en/authors/chi-zhang/), [Ruiyang Sun](https://md.research.elamur.ai/free/en/authors/ruiyang-sun/), [Yizhou Wang](https://md.research.elamur.ai/free/en/authors/yizhou-wang/), [Yaodong Yang](https://md.research.elamur.ai/free/en/authors/yaodong-yang/) · **Year:** 2023 · **Venue:** arXiv:2307.04657 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.04657) · [PDF](https://arxiv.org/pdf/2307.04657) · [DOI 10.48550/arxiv.2307.04657](https://doi.org/10.48550/arxiv.2307.04657) · [arXiv 2307.04657](https://arxiv.org/abs/2307.04657) · [OpenAlex](https://openalex.org/W7133189861)

**Citations:** 70 citations (OpenAlex, 2026-10-14) · [281](https://md.research.elamur.ai/free/en/papers/ji-2023-beavertails-towards-improved-safety-alignment/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Aligning LLMs for safety depends on large amounts of human feedback. Existing preference data usually merge helpfulness and harmlessness into a single judgement. BeaverTails, from the PKU-Beaver group at Peking University (NeurIPS 2023 Datasets and Benchmarks track), addresses this with a question-answering dataset that labels the two attributes separately. The authors took opening prompts from Anthropic's HH Red-Team data and had Alpaca-7B write several answers to each one. Crowdworkers then annotated in two stages. First, they decided whether each QA pair as a whole was safe, checking it against 14 harm categories. Second, they ranked the answers to the same prompt twice, once for helpfulness and once for harmlessness. The paper shows three uses for the data. It trains a QA-moderation classifier and compares it with prompted GPT-4. It trains separate reward and cost models from the two rankings. It then uses those models for Safe-RLHF fine-tuning of Alpaca-7B, which gives much safer answers to red-team prompts. Separate preference signals let researchers balance safety against usefulness openly instead of leaving the trade-off hidden in one score.

## Key points
- The dataset has two releases. BeaverTails-30k has 30,207 QA pairs from 7,774 prompts, with 42.68% labelled safe. BeaverTails-330k has 333,963 QA pairs from 16,851 unique prompts, with 44.64% labelled safe and about 3.34 annotations per pair.
- It includes 361,903 comparison pairs, each annotated for helpfulness and for harmlessness. The authors say it is the first dataset to separate these two preference signals.
- Agreement between annotators was 81.68% on the safety meta-label, 62.39% on helpfulness rankings and 60.91% on harmlessness rankings. Each label also has a confidence score from 0 to 3.
- 70 English-proficient crowdworkers did the labelling. The authors acknowledge that the annotators' demographic diversity is limited and that the harm categories overlap and are unevenly represented: "Child Abuse" and "Animal Abuse" are rare compared with violence-related categories.
- The data is released under CC BY-NC 4.0. The next stated goal is one million preference rankings, with answers generated by many public LLMs.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · [Bias, fairness & safety](https://md.research.elamur.ai/free/en/topics/bias-fairness-safety/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ji2023beavertails,
  title = {{BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset}},
  author = {Ji, Jiaming and Liu, Mickel and Dai, Juntao and Pan, Xuehai and Zhang, Chi and Bian, Ce and Zhang, Chi and Sun, Ruiyang and Wang, Yizhou and Yang, Yaodong},
  year = {2023},
  doi = {10.48550/arxiv.2307.04657},
  eprint = {2307.04657},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.04657}
}
```

- [← previous ViperGPT: Visual Inference via Python Execution for Reasoning](https://md.research.elamur.ai/free/en/papers/suris-2023-vipergpt-visual-inference-python-execution/)
- [next → ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/2/)

## References

62 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. OpenAI. Gpt-4 technical report, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Gpt%2D4+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
2. Rohan Anil, Andrew M. Dai, and et. al. Palm 2 technical report, 2023. [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Andrew+M%2E+Dai%2C+and+et%2E+al%2E+Palm+2+technical+report%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
3. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timoth%C3%A9e+Lacroix%2C+Baptiste+Rozi%C3%A8re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aurelien+Rodriguez%2C+Armand+Joulin%2C+Edoua)
4. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
5. Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. [find](https://scholar.google.com/scholar?q=Aiyuan+Yang%2C+Bin+Xiao%2C+Bingning+Wang%2C+Borong+Zhang%2C+Chao+Yin%2C+Chenxu+Lv%2C+Da+Pan%2C+Dian+Wang%2C+Dong+Yan%2C+Fan+Yang%2C+et+al%2E+Baichuan+2%3A+Open+large%2Dscale+language+models%2E+arXiv+preprint+arXiv%3A2309%2E10305%2C+20) [our card](https://md.research.elamur.ai/free/en/papers/yang-2023-baichuan-2-open-large-scale/)
6. Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B Costa, Mona G Flores, et al. A large language model for electronic health records. npj Digital Medicine, 5(1):194, 2022. [find](https://scholar.google.com/scholar?q=Xi+Yang%2C+Aokun+Chen%2C+Nima+PourNejatian%2C+Hoo+Chang+Shin%2C+Kaleb+E+Smith%2C+Christopher+Parisien%2C+Colin+Compas%2C+Cheryl+Martin%2C+Anthony+B+Costa%2C+Mona+G+Flores%2C+et+al%2E+A+large+language+model+for+electronic+h)
7. Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023. [find](https://scholar.google.com/scholar?q=Michael+Moor%2C+Oishi+Banerjee%2C+Zahra+Shakeri+Hossein+Abad%2C+Harlan+M+Krumholz%2C+Jure+Leskovec%2C+Eric+J+Topol%2C+and+Pranav+Rajpurkar%2E+Foundation+models+for+generalist+medical+artificial+intelligence%2E+Nature)
8. Anmol Arora and Ananya Arora. The promise of large language models in health care. The Lancet, 401(10377):641, 2023. [find](https://scholar.google.com/scholar?q=Anmol+Arora+and+Ananya+Arora%2E+The+promise+of+large+language+models+in+health+care%2E+The+Lancet%2C+401%2810377%29%3A641%2C+2023%2E)
9. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models. PLoS digital health,… [find](https://scholar.google.com/scholar?q=Tiffany+H+Kung%2C+Morgan+Cheatham%2C+Arielle+Medenilla%2C+Czarina+Sillos%2C+Lorie+De+Leon%2C+Camille+Elepa%C3%B1o%2C+Maria+Madriaga%2C+Rimel+Aggabao%2C+Giezel+Diaz%2DCandido%2C+James+Maningo%2C+et+al%2E+Performance+of+chatgpt+on+)
10. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103… [find](https://scholar.google.com/scholar?q=Enkelejda+Kasneci%2C+Kathrin+Se%C3%9Fler%2C+Stefan+K%C3%BCchemann%2C+Maria+Bannert%2C+Daryna+Dementieva%2C+Frank+Fischer%2C+Urs+Gasser%2C+Georg+Groh%2C+Stephan+G%C3%BCnnemann%2C+Eyke+H%C3%BCllermeier%2C+et+al%2E+Chatgpt+for+good%3F+on+opportuni) [our card](https://md.research.elamur.ai/free/en/papers/kasneci-2023-chatgpt-good-opportunities-challenges-large/)
11. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Available at SSRN 4389233, 2023. [find](https://scholar.google.com/scholar?q=Daniel+Martin+Katz%2C+Michael+James+Bommarito%2C+Shang+Gao%2C+and+Pablo+Arredondo%2E+Gpt%2D4+passes+the+bar+exam%2E+Available+at+SSRN+4389233%2C+2023%2E)
12. Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023. [find](https://scholar.google.com/scholar?q=Sai+Vemprala%2C+Rogerio+Bonatti%2C+Arthur+Bucker%2C+and+Ashish+Kapoor%2E+Chatgpt+for+robotics%3A+Design+principles+and+model+abilities%2E+Microsoft+Auton%2E+Syst%2E+Robot%2E+Res%2C+2%3A20%2C+2023%2E)
13. Dhruv Shah, Błażej Osiński, Sergey Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pages 492–504. PMLR, 2023. [find](https://scholar.google.com/scholar?q=Dhruv+Shah%2C+B%C5%82az%CC%87ej+Osin%CC%81ski%2C+Sergey+Levine%2C+et+al%2E+Lm%2Dnav%3A+Robotic+navigation+with+large+pre%2Dtrained+models+of+language%2C+vision%2C+and+action%2E+In+Conference+on+Robot+Learning%2C+pages+492%E2%80%93504%2E+PMLR%2C+2023)
14. Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXiv:2305.05658, 2023. — [arXiv:2305.05658](https://arxiv.org/abs/2305.05658) [find](https://scholar.google.com/scholar?q=Jimmy+Wu%2C+Rika+Antonova%2C+Adam+Kan%2C+Marion+Lepert%2C+Andy+Zeng%2C+Shuran+Song%2C+Jeannette+Bohg%2C+Szymon+Rusinkiewicz%2C+and+Thomas+Funkhouser%2E+Tidybot%3A+Personalized+robot+assistance+with+large+language+models%2E)
15. ChatGPT plugins. [https://openai.com/blog/chatgpt-plugins](https://openai.com/blog/chatgpt-plugins). Accessed: 2023-6-7. [↗](https://openai.com/blog/chatgpt-plugins)
16. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, November 2020. Association for Computational L… [find](https://scholar.google.com/scholar?q=Samuel+Gehman%2C+Suchin+Gururangan%2C+Maarten+Sap%2C+Yejin+Choi%2C+and+Noah+A+Smith%2E+RealToxicityPrompts%3A+Evaluating+neural+toxic+degeneration+in+language+models%2E+In+Findings+of+the+Association+for+Computatio) [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
17. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+John+Mellor%2C+Maribeth+Rauh%2C+Conor+Griffin%2C+Jonathan+Uesato%2C+Po%2DSen+Huang%2C+Myra+Cheng%2C+Mia+Glaese%2C+Borja+Balle%2C+Atoosa+Kasirzadeh%2C+et+al%2E+Ethical+and+social+risks+of+harm+from+language) [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
18. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. [find](https://scholar.google.com/scholar?q=Deep+Ganguli%2C+Liane+Lovitt%2C+Jackson+Kernion%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Saurav+Kadavath%2C+Ben+Mann%2C+Ethan+Perez%2C+Nicholas+Schiefer%2C+Kamal+Ndousse%2C+et+al%2E+Red+teaming+language+models+to+reduce+harms%3A+Me) [our card](https://md.research.elamur.ai/free/en/papers/ganguli-2022-red-teaming-language-models-reduce/)
19. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335, 2023. [find](https://scholar.google.com/scholar?q=Ameet+Deshpande%2C+Vishvak+Murahari%2C+Tanmay+Rajpurohit%2C+Ashwin+Kalyan%2C+and+Karthik+Narasimhan%2E+Toxicity+in+chatgpt%3A+Analyzing+persona%2Dassigned+language+models%2E+arXiv+preprint+arXiv%3A2304%2E05335%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/deshpande-2023-toxicity-chatgpt-analyzing-persona-assigned/)
20. Jon Christian. Amazing “jailbreak” bypasses ChatGPT’s ethics safeguards. [https://futurism.com/amazing-jailbreak-chatgpt](https://futurism.com/amazing-jailbreak-chatgpt), February 2023. Accessed: 2023-6-7. [↗](https://futurism.com/amazing-jailbreak-chatgpt)
21. Jim Chilton. The new risks ChatGPT poses to cybersecurity. Harvard Business Review, April 2023. [find](https://scholar.google.com/scholar?q=Jim+Chilton%2E+The+new+risks+ChatGPT+poses+to+cybersecurity%2E+Harvard+Business+Review%2C+April+2023%2E)
22. Lily Hay Newman. ChatGPT scams are infiltrating the app store and google play. Wired, May 2023. [find](https://scholar.google.com/scholar?q=Lily+Hay+Newman%2E+ChatGPT+scams+are+infiltrating+the+app+store+and+google+play%2E+Wired%2C+May+2023%2E)
23. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. [find](https://scholar.google.com/scholar?q=Ethan+Perez%2C+Saffron+Huang%2C+Francis+Song%2C+Trevor+Cai%2C+Roman+Ring%2C+John+Aslanides%2C+Amelia+Glaese%2C+Nat+McAleese%2C+and+Geoffrey+Irving%2E+Red+teaming+language+models+with+language+models%2E+arXiv+preprint+arX) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeffrey+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+wi)
25. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
26. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023. [find](https://scholar.google.com/scholar?q=Josef+Dai%2C+Xuehai+Pan%2C+Ruiyang+Sun%2C+Jiaming+Ji%2C+Xinbo+Xu%2C+Mickel+Liu%2C+Yizhou+Wang%2C+and+Yaodong+Yang%2E+Safe+rlhf%3A+Safe+reinforcement+learning+from+human+feedback%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/dai-2023-safe-rlhf-safe-reinforcement-learning/)
27. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
28. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Tom+Henighan%2C+Andy+Jones%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Nova+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+a)
29. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-Adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950–2968, Onli… [find](https://scholar.google.com/scholar?q=Jing+Xu%2C+Da+Ju%2C+Margaret+Li%2C+Y%2DLan+Boureau%2C+Jason+Weston%2C+and+Emily+Dinan%2E+Bot%2DAdversarial+dialogue+for+safe+conversational+agents%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+)
30. Google Jigsaw. Perspective API. [https://www.perspectiveapi.com/](https://www.perspectiveapi.com/), 2017. Accessed: 2023-06-05. [↗](https://www.perspectiveapi.com/)
31. Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A new generation of perspective api: Efficient multilingual character-level transformers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197–3207, 2022. [find](https://scholar.google.com/scholar?q=Alyssa+Lees%2C+Vinh+Q+Tran%2C+Yi+Tay%2C+Jeffrey+Sorensen%2C+Jai+Gupta%2C+Donald+Metzler%2C+and+Lucy+Vasserman%2E+A+new+generation+of+perspective+api%3A+Efficient+multilingual+character%2Dlevel+transformers%2E+In+Proceedi)
32. Kawin Ethayarajh, Heidi Zhang, Yizhong Wang, and Dan Jurafsky. Stanford human preferences dataset, 2023. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2C+Heidi+Zhang%2C+Yizhong+Wang%2C+and+Dan+Jurafsky%2E+Stanford+human+preferences+dataset%2C+2023%2E)
33. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064, 2019. [find](https://scholar.google.com/scholar?q=Po%2DSen+Huang%2C+Huan+Zhang%2C+Ray+Jiang%2C+Robert+Stanforth%2C+Johannes+Welbl%2C+Jack+Rae%2C+Vishal+Maini%2C+Dani+Yogatama%2C+and+Pushmeet+Kohli%2E+Reducing+sentiment+bias+in+language+models+via+counterfactual+evaluati)
34. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Tom+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+D+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+)
35. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, and et. al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022. [find](https://scholar.google.com/scholar?q=Aarohi+Srivastava%2C+Abhinav+Rastogi%2C+Abhishek+Rao%2C+and+et%2E+al%2E+Beyond+the+imitation+game%3A+Quantifying+and+extrapolating+the+capabilities+of+language+models%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
36. Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proces… [find](https://scholar.google.com/scholar?q=Nedjma+Ousidhoum%2C+Xinran+Zhao%2C+Tianqing+Fang%2C+Yangqiu+Song%2C+and+Dit%2DYan+Yeung%2E+Probing+toxic+content+in+large+pre%2Dtrained+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+)
37. Maribeth Rauh, John Mellor, Jonathan Uesato, Po-Sen Huang, Johannes Welbl, Laura Weidinger, Sumanth Dathathri, Amelia Glaese, Geoffrey Irving, Iason Gabriel, et al. Characteristics of harmful text: Towards rigorous benchmarking of language models. Advances in Neural Information Processing Systems, 3… [find](https://scholar.google.com/scholar?q=Maribeth+Rauh%2C+John+Mellor%2C+Jonathan+Uesato%2C+Po%2DSen+Huang%2C+Johannes+Welbl%2C+Laura+Weidinger%2C+Sumanth+Dathathri%2C+Amelia+Glaese%2C+Geoffrey+Irving%2C+Iason+Gabriel%2C+et+al%2E+Characteristics+of+harmful+text%3A+To)
38. Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, et al. Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324, 2023. [find](https://scholar.google.com/scholar?q=Toby+Shevlane%2C+Sebastian+Farquhar%2C+Ben+Garfinkel%2C+Mary+Phuong%2C+Jess+Whittlestone%2C+Jade+Leung%2C+Daniel+Kokotajlo%2C+Nahema+Marchal%2C+Markus+Anderljung%2C+Noam+Kolt%2C+et+al%2E+Model+evaluation+for+extreme+risks%2E)
39. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+TruthfulQA%3A+Measuring+how+models+mimic+human+falsehoods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
40. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193, 2021. [find](https://scholar.google.com/scholar?q=Alicia+Parrish%2C+Angelica+Chen%2C+Nikita+Nangia%2C+Vishakh+Padmakumar%2C+Jason+Phang%2C+Jana+Thompson%2C+Phu+Mon+Htut%2C+and+Samuel+R+Bowman%2E+Bbq%3A+A+hand%2Dbuilt+bias+benchmark+for+question+answering%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/)
41. OpenAI. Moderation API. [https://platform.openai.com/docs/guides/moderation/overview](https://platform.openai.com/docs/guides/moderation/overview), 2023. Accessed: 2023-6-5. [↗](https://platform.openai.com/docs/guides/moderation/overview)
42. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345, 2019. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+Text+summarization+with+pretrained+encoders%2E+arXiv+preprint+arXiv%3A1908%2E08345%2C+2019%2E)
43. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. [find](https://scholar.google.com/scholar?q=Nisan+Stiennon%2C+Long+Ouyang%2C+Jeffrey+Wu%2C+Daniel+Ziegler%2C+Ryan+Lowe%2C+Chelsea+Voss%2C+Alec+Radford%2C+Dario+Amodei%2C+and+Paul+F+Christiano%2E+Learning+to+summarize+with+human+feedback%2E+Advances+in+Neural+Infor)
44. Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. How ready are pre-trained abstractive models and llms for legal case judgement summarization?, 2023. [find](https://scholar.google.com/scholar?q=Aniket+Deroy%2C+Kripabandhu+Ghosh%2C+and+Saptarshi+Ghosh%2E+How+ready+are+pre%2Dtrained+abstractive+models+and+llms+for+legal+case+judgement+summarization%3F%2C+2023%2E)
45. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
46. Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.05206, 2023. — [arXiv:2302.05206](https://arxiv.org/abs/2302.05206) [find](https://scholar.google.com/scholar?q=Tianjun+Zhang%2C+Fangchen+Liu%2C+Justin+Wong%2C+Pieter+Abbeel%2C+and+Joseph+E+Gonzalez%2E+The+wisdom+of+hindsight+makes+language+models+better+instruction+followers%2E+arXiv+preprint+arXiv%3A2302%2E05206%2C+2023%2E+%E2%80%94+%5Bar)
47. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Saurav+Kadavath%2C+Sandipan+Kundu%2C+Amanda+Askell%2C+Jackson+Kernion%2C+Andy+Jones%2C+Anna+Chen%2C+Anna+Goldie%2C+Azalia+Mirhoseini%2C+Cameron+McKinnon%2C+et+al%2E+Constitutional+ai%3A+Harmlessness+from+ai+fee) [our card](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/)
48. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2C+2017%2E)
49. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training, 2023. [find](https://scholar.google.com/scholar?q=Zeqiu+Wu%2C+Yushi+Hu%2C+Weijia+Shi%2C+Nouha+Dziri%2C+Alane+Suhr%2C+Prithviraj+Ammanabrolu%2C+Noah+A%2E+Smith%2C+Mari+Ostendorf%2C+and+Hannaneh+Hajishirzi%2E+Fine%2Dgrained+human+feedback+gives+better+rewards+for+language+m) [our card](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/)
50. Hai Ci, Mickel Liu, Xuehai Pan, Yizhou Wang, et al. Proactive multi-camera collaboration for 3d human pose estimation. In The Eleventh International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Hai+Ci%2C+Mickel+Liu%2C+Xuehai+Pan%2C+Yizhou+Wang%2C+et+al%2E+Proactive+multi%2Dcamera+collaboration+for+3d+human+pose+estimation%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2C+2022%2E)
51. Xuehai Pan, Mickel Liu, Fangwei Zhong, Yaodong Yang, Song-Chun Zhu, and Yizhou Wang. Mate: Benchmarking multi-agent reinforcement learning in distributed target coverage control. Advances in Neural Information Processing Systems, 35:27862–27879, 2022. [find](https://scholar.google.com/scholar?q=Xuehai+Pan%2C+Mickel+Liu%2C+Fangwei+Zhong%2C+Yaodong+Yang%2C+Song%2DChun+Zhu%2C+and+Yizhou+Wang%2E+Mate%3A+Benchmarking+multi%2Dagent+reinforcement+learning+in+distributed+target+coverage+control%2E+Advances+in+Neural+In)
52. Rahul Tallamraju, Nitin Saini, Elia Bonetto, Michael Pabst, Yu Tang Liu, Michael J Black, and Aamir Ahmad. AirCapRL: Autonomous aerial human motion capture using deep reinforcement learning. IEEE Robotics and Automation Letters, 5(4):6678–6685, October 2020. [find](https://scholar.google.com/scholar?q=Rahul+Tallamraju%2C+Nitin+Saini%2C+Elia+Bonetto%2C+Michael+Pabst%2C+Yu+Tang+Liu%2C+Michael+J+Black%2C+and+Aamir+Ahmad%2E+AirCapRL%3A+Autonomous+aerial+human+motion+capture+using+deep+reinforcement+learning%2E+IEEE+Robo)
53. OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik’s cu… [find](https://scholar.google.com/scholar?q=OpenAI%2C+Ilge+Akkaya%2C+Marcin+Andrychowicz%2C+Maciek+Chociej%2C+Mateusz+Litwin%2C+Bob+McGrew%2C+Arthur+Petron%2C+Alex+Paino%2C+Matthias+Plappert%2C+Glenn+Powell%2C+Raphael+Ribas%2C+Jonas+Schneider%2C+Nikolas+Tezak%2C+Jerry+T)
54. Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):eabk2822, 2022. [find](https://scholar.google.com/scholar?q=Takahiro+Miki%2C+Joonho+Lee%2C+Jemin+Hwangbo%2C+Lorenz+Wellhausen%2C+Vladlen+Koltun%2C+and+Marco+Hutter%2E+Learning+robust+perceptive+locomotion+for+quadrupedal+robots+in+the+wild%2E+Science+Robotics%2C+7%2862%29%3Aeabk282)
55. OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc-Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020. [find](https://scholar.google.com/scholar?q=OpenAI%3A+Marcin+Andrychowicz%2C+Bowen+Baker%2C+Maciek+Chociej%2C+Rafal+Jozefowicz%2C+Bob+Mc%2DGrew%2C+Jakub+Pachocki%2C+Arthur+Petron%2C+Matthias+Plappert%2C+Glenn+Powell%2C+Alex+Ray%2C+et+al%2E+Learning+dexterous+in%2Dhand+man)
56. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436, 2023. [find](https://scholar.google.com/scholar?q=Hao+Sun%2C+Zhexin+Zhang%2C+Jiawen+Deng%2C+Jiale+Cheng%2C+and+Minlie+Huang%2E+Safety+assessment+of+chinese+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E10436%2C+2023%2E)
57. Eitan Altman. Constrained Markov decision processes. Routledge, 2021. [find](https://scholar.google.com/scholar?q=Eitan+Altman%2E+Constrained+Markov+decision+processes%2E+Routledge%2C+2021%2E)
58. Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research, 2023. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Jiayi+Zhou%2C+Borong+Zhang%2C+Juntao+Dai%2C+Xuehai+Pan%2C+Ruiyang+Sun%2C+Weidong+Huang%2C+Yiran+Geng%2C+Mickel+Liu%2C+and+Yaodong+Yang%2E+Omnisafe%3A+An+infrastructure+for+accelerating+safe+reinforcement+lear)
59. Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019. — [arXiv:1910.01708](https://arxiv.org/abs/1910.01708) [find](https://scholar.google.com/scholar?q=Alex+Ray%2C+Joshua+Achiam%2C+and+Dario+Amodei%2E+Benchmarking+safe+exploration+in+deep+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1910%2E01708%2C+7%281%29%3A2%2C+2019%2E+%E2%80%94+%5BarXiv%3A1910%2E01708%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1910%2E)
60. Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Juntao Dai, and Yaodong Yang. Safety-gymnasium: A unified safe reinforcement learning benchmark, 2023. [find](https://scholar.google.com/scholar?q=Jiaming+Ji%2C+Borong+Zhang%2C+Jiayi+Zhou%2C+Xuehai+Pan%2C+Weidong+Huang%2C+Ruiyang+Sun%2C+Yiran+Geng%2C+Yifan+Zhong%2C+Juntao+Dai%2C+and+Yaodong+Yang%2E+Safety%2Dgymnasium%3A+A+unified+safe+reinforcement+learning+benchmark%2C+)
61. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. [find](https://scholar.google.com/scholar?q=Amelia+Glaese%2C+Nat+McAleese%2C+Maja+Tr%CB%9Bebacz%2C+John+Aslanides%2C+Vlad+Firoiu%2C+Timo+Ewalds%2C+Maribeth+Rauh%2C+Laura+Weidinger%2C+Martin+Chadwick%2C+Phoebe+Thacker%2C+et+al%2E+Improving+alignment+of+dialogue+agents+via) [our card](https://md.research.elamur.ai/free/en/papers/glaese-2022-improving-alignment-dialogue-agents-targeted/)
62. China: hourly minimum wage by region 2023. [https://www.statista.com/statistics/233886/minimum-wage-per-hour-in-china-by-city-and-province/](https://www.statista.com/statistics/233886/minimum-wage-per-hour-in-china-by-city-and-province/). Accessed: 2023-6-7. [↗](https://www.statista.com/statistics/233886/minimum-wage-per-hour-in-china-by-city-and-province/)