--- title: "The Capacity for Moral Self-Correction in — Ganguli 2023" description: "The paper asks whether large dialogue models trained with RLHF can reduce their own biased or discriminatory outputs when simply asked to in plain…" authors: ["Deep Ganguli", "Amanda Askell", "Nicholas Schiefer", "Thomas I. Liao", "Kamile Luko\\vsiute", "Anna Chen", "Anna Goldie", "Azalia Mirhoseini", "Catherine Olsson", "Danny Hernandez", "Dawn Drain", "Dustin Li", "Eli Tran-Johnson", "Ethan Perez", "Jackson Kernion", "Jamie Kerr", "Jared Mueller", "Joshua Landau", "Kamal Ndousse", "Karina Nguyen", "Liane Lovitt", "Michael Sellitto", "Nelson Elhage", "Noemi Mercado", "Nova DasSarma", "Oliver Rausch", "Robert Lasenby", "Robin Larson", "Sam Ringer", "Sandipan Kundu", "Saurav Kadavath", "Scott Johnston", "Shauna Kravec", "Sheer El Showk", "Tamera Lanham", "Timothy Telleen-Lawton", "Tom Henighan", "Tristan Hume", "Yuntao Bai", "Zac Hatfield-Dodds", "Ben Mann", "Dario Amodei", "Nicholas Joseph", "Sam McCandlish", "Tom Brown", "Christopher Olah", "Jack Clark", "Samuel R. Bowman", "Jared Kaplan"] year: 2023 venue: "arXiv:2302.07459" source: "https://arxiv.org/abs/2302.07459" doi: "10.48550/arxiv.2302.07459" arxiv: "2302.07459" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ganguli-2023-capacity-moral-self-correction-large/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) / The Capacity for Moral Self-Correction in Large Language Models # The Capacity for Moral Self-Correction in Large Language Models **Authors:** [Deep Ganguli](https://md.research.elamur.ai/free/en/authors/deep-ganguli/), [Amanda Askell](https://md.research.elamur.ai/free/en/authors/amanda-askell/), [Nicholas Schiefer](https://md.research.elamur.ai/free/en/authors/nicholas-schiefer/), [Thomas I. Liao](https://md.research.elamur.ai/free/en/authors/thomas-i-liao/), [Kamile Luko\\vsiute](https://md.research.elamur.ai/free/en/authors/kamile-luko-vsiute/), [Anna Chen](https://md.research.elamur.ai/free/en/authors/anna-chen/), [Anna Goldie](https://md.research.elamur.ai/free/en/authors/anna-goldie/), [Azalia Mirhoseini](https://md.research.elamur.ai/free/en/authors/azalia-mirhoseini/), [Catherine Olsson](https://md.research.elamur.ai/free/en/authors/catherine-olsson/), [Danny Hernandez](https://md.research.elamur.ai/free/en/authors/danny-hernandez/), Dawn Drain, [Dustin Li](https://md.research.elamur.ai/free/en/authors/dustin-li/) et al. (49 authors) · **Year:** 2023 · **Venue:** arXiv:2302.07459 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2302.07459) · [PDF](https://arxiv.org/pdf/2302.07459) · [DOI 10.48550/arxiv.2302.07459](https://doi.org/10.48550/arxiv.2302.07459) · [arXiv 2302.07459](https://arxiv.org/abs/2302.07459) · [OpenAlex](https://openalex.org/W4321162379) **Citations:** 51 citations (openalex-snapshot · 2026-10-14) · [116](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary The paper asks whether large dialogue models trained with RLHF can reduce their own biased or discriminatory outputs when simply asked to in plain language. The authors evaluate Anthropic decoder-only models ranging from 810M to 175B parameters, at 50 to 1000 RLHF steps. They use three test sets: BBQ for stereotypes across nine social dimensions, Winogender for occupational gender bias, and a new admissions-style benchmark for racial discrimination. The new benchmark is adapted from a dataset previously used in counterfactual fairness research. Each test runs under three prompting conditions that build on one another: a bare question (Q), the question plus an instruction to be unbiased (Q+IF), and the instruction plus chain-of-thought reasoning (Q+IF+CoT). The ability to self-correct appears at around 22B parameters and generally grows with both model size and RLHF training. Importantly, the prompts never define bias or reveal the evaluation metrics, so the models rely on concepts they learned during training. This suggests that natural-language guidance could be a cheap tool for fairness, but the authors stress that the results justify only cautious optimism. ## Key points - **BBQ results:** With Q+IF+CoT, the 175B model's bias score fell by 84% compared with the plain Q condition. Bias in the baseline grows with scale, but the prompted conditions reverse that trend. More RLHF training lowered bias in every condition. - **Winogender results:** Pronoun choices could be pushed either way relative to U.S. Bureau of Labor Statistics occupational data. With instructions, the correlation fell to roughly 0, and with a "Match Stats" prompt it rose to nearly 1. Without instructions it stayed around 0.5. RLHF had little effect on this metric. - **Discrimination results:** At 175B, the model favored white over Black applicants by about 3% under Q. Under Q+IF+CoT it favored Black applicants by about 7%, an over-correction. Demographic parity occurred at roughly 600 RLHF steps with Q+IF and roughly 200 steps with Q+IF+CoT. - **Limitations:** - The benchmarks are tied to American English and U.S. cultural norms. - The discrimination test compares only two races. - Prompt sensitivity was not tested systematically. - The same techniques could be reversed to elicit harmful outputs. - **Related work:** The findings echo Constitutional AI. They also contrast with earlier negative results for RoBERTa-large on the UnQover benchmark. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) ## Cited by Cited by 116 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 100 not yet. - [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://md.research.elamur.ai/free/en/papers/baek-2024-researchagent-iterative-research-idea/) - [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://md.research.elamur.ai/free/en/papers/durmus-2023-towards-measuring-representation-subjective-global/) - [Large Language Models Cannot Self-Correct Reasoning Yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/) - [Language Models can Solve Computer Tasks](https://md.research.elamur.ai/free/en/papers/kim-2023-language-models-solve-computer-tasks/) - [Measuring Faithfulness in Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/lanham-2023-measuring-faithfulness-chain-thought-reasoning/) - [Self-Alignment with Instruction Backtranslation](https://md.research.elamur.ai/free/en/papers/li-2023-self-alignment-instruction-backtranslation/) - [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/) - [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/) - [Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models](https://md.research.elamur.ai/free/en/papers/sengupta-2023-jais-jais-chat-arabic-centric/) - [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://md.research.elamur.ai/free/en/papers/shaikh-2023-second-thought-let-s-not/) - [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://md.research.elamur.ai/free/en/papers/sun-2023-principle-driven-self-alignment-language/) - [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://md.research.elamur.ai/free/en/papers/turpin-2023-language-models-don-t-always/) - [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://md.research.elamur.ai/free/en/papers/wang-2023-decodingtrust-comprehensive-assessment-trustworthiness-gpt/) - [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/) - [Large Language Models as Optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/) - [Low-Resource Languages Jailbreak GPT-4](https://md.research.elamur.ai/free/en/papers/yong-2023-low-resource-languages-jailbreak-gpt/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{ganguli2023capacity, title = {{The Capacity for Moral Self-Correction in Large Language Models}}, author = {Ganguli, Deep and Askell, Amanda and Schiefer, Nicholas and Liao, Thomas I. and Lukovsiute, Kamile and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and Olsson, Catherine and Hernandez, Danny and Drain, Dawn and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan and Kernion, Jackson and Kerr, Jamie and Mueller, Jared and Landau, Joshua and Ndousse, Kamal and Nguyen, Karina and Lovitt, Liane and Sellitto, Michael and Elhage, Nelson and Mercado, Noemi and DasSarma, Nova and Rausch, Oliver and Lasenby, Robert and Larson, Robin and Ringer, Sam and Kundu, Sandipan and Kadavath, Saurav and Johnston, Scott and Kravec, Shauna and Showk, Sheer El and Lanham, Tamera and Telleen-Lawton, Timothy and Henighan, Tom and Hume, Tristan and Bai, Yuntao and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Olah, Christopher and Clark, Jack and Bowman, Samuel R. and Kaplan, Jared}, year = {2023}, doi = {10.48550/arxiv.2302.07459}, eprint = {2302.07459}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2302.07459} } ``` - [← previous The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://md.research.elamur.ai/free/en/papers/lauren-2023-ccon-bigscience-roots-corpus-1-6tb/) - [next → The Curse of Recursion: Training on Generated Data Makes Models Forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/) - [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/7/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. [1] A. Abid, M. Farooqi, and J. Zou. Large language models associate Muslims with violence. Nature Machine Intelligence, 3(6):461–463, June 2021. Number: 6 Publisher: Nature Publishing Group. [find](https://scholar.google.com/scholar?q=%5B1%5D+A%2E+Abid%2C+M%2E+Farooqi%2C+and+J%2E+Zou%2E+Large+language+models+associate+Muslims+with+violence%2E+Nature+Machine+Intelligence%2C+3%286%29%3A461%E2%80%93463%2C+June+2021%2E+Number%3A+6+Publisher%3A+Nature+Publishing+Group%2E)