---
title: "Training language models to follow — Ouyang 2022"
description: "Making language models larger does not by itself make them more helpful: they make up facts, generate toxic text, or disregard the user's request."
authors: ["Long Ouyang", "Jeff Wu", "Xu Jiang", "Diogo Almeida", "Carroll L. Wainwright", "Pamela Mishkin", "Chong Zhang", "Sandhini Agarwal", "Katarina Slama", "Alex Ray", "John Schulman", "Jacob Hilton", "Fraser Kelton", "Luke Miller", "Maddie Simens", "Amanda Askell", "Peter Welinder", "Paul Christiano", "Jan Leike", "Ryan Lowe"]
year: 2022
venue: "arXiv:2203.02155"
source: "https://arxiv.org/abs/2203.02155"
doi: "10.48550/arxiv.2203.02155"
arxiv: "2203.02155"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/) / Training language models to follow instructions with human feedback

# Training language models to follow instructions with human feedback

**Authors:** [Long Ouyang](https://md.research.elamur.ai/free/en/authors/long-ouyang/), [Jeff Wu](https://md.research.elamur.ai/free/en/authors/jeff-wu/), [Xu Jiang](https://md.research.elamur.ai/free/en/authors/xu-jiang/), [Diogo Almeida](https://md.research.elamur.ai/free/en/authors/diogo-almeida/), Carroll L. Wainwright, [Pamela Mishkin](https://md.research.elamur.ai/free/en/authors/pamela-mishkin/), [Chong Zhang](https://md.research.elamur.ai/free/en/authors/chong-zhang/), [Sandhini Agarwal](https://md.research.elamur.ai/free/en/authors/sandhini-agarwal/), Katarina Slama, [Alex Ray](https://md.research.elamur.ai/free/en/authors/alex-ray/), [John Schulman](https://md.research.elamur.ai/free/en/authors/john-schulman/), [Jacob Hilton](https://md.research.elamur.ai/free/en/authors/jacob-hilton/) et al. (20 authors) · **Year:** 2022 · **Venue:** arXiv:2203.02155 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2203.02155) · [PDF](https://arxiv.org/pdf/2203.02155) · [DOI 10.48550/arxiv.2203.02155](https://doi.org/10.48550/arxiv.2203.02155) · [arXiv 2203.02155](https://arxiv.org/abs/2203.02155)

**Citations:** [5770](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Making language models larger does not by itself make them more helpful: they make up facts, generate toxic text, or disregard the user's request. The authors from OpenAI propose aligning GPT-3 with human intent by fine-tuning on human feedback. First, 40 hired labelers wrote demonstration responses to prompts from the OpenAI API, mostly in English, as well as to their own prompts. The model was fine-tuned on these examples with supervised learning (SFT). The labelers then ranked several candidate responses from best to worst. A reward model (RM) was trained on these comparisons, and the model was then fine-tuned against it with the PPO algorithm, i.e., by the RLHF method. The resulting models were named InstructGPT. Evaluators preferred the responses of the 1.3B-parameter InstructGPT to those of the 175B GPT-3, even though it is roughly one hundred times smaller. InstructGPT is also more truthful and slightly less toxic. The work showed that human feedback can yield more than simply scaling up the model, and it laid the methodological foundation for modern instruction-following assistants.

## Key points
- The models of three sizes (1.3B, 6B, 175B) use the GPT-3 architecture. Evaluators preferred the responses of the 175B InstructGPT to those of the standard 175B GPT-3 in 85 ± 3% of cases. Against GPT-3 with a few-shot prompt, this figure was 71 ± 4%.
- On TruthfulQA, InstructGPT gives truthful and informative answers about twice as often as GPT-3. On closed-domain tasks such as summarization, the model fabricates information absent from the input text less often: 21% of cases versus 41%.
- On RealToxicityPrompts, there are about 25% fewer toxic outputs when the model is prompted to respond respectfully. On Winogender and CrowS-Pairs, there is no noticeable reduction in bias.
- The cost of alignment (the "alignment tax") is a drop in performance on SQuAD, DROP, HellaSwag, and WMT 2015 fr→en. The PPO-ptx variant, in which pretraining data are mixed into training, largely eliminates these losses without lowering labeler ratings.
- GPT-3 fine-tuned on the FLAN and T0++ datasets falls noticeably short of InstructGPT: its win rate against the baseline SFT model is 29.8% and 26.8%, respectively, whereas for InstructGPT it is 73.4%. Labelers whose data were not used in training prefer InstructGPT about as often as those whose data were used.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ouyang2022training,
  title = {{Training language models to follow instructions with human feedback}},
  author = {Ouyang, Long and Wu, Jeff and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll L. and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and Schulman, John and Hilton, Jacob and Kelton, Fraser and Miller, Luke and Simens, Maddie and Askell, Amanda and Welinder, Peter and Christiano, Paul and Leike, Jan and Lowe, Ryan},
  year = {2022},
  doi = {10.48550/arxiv.2203.02155},
  eprint = {2203.02155},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.02155}
}
```

- [← previous Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/)
- [next → Large Language Models Encode Clinical Knowledge](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/)

## References

91 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abramson, J., Ahuja, A., Barr, I., Brussee, A., Carnevale, F., Cassin, M., Chhaparia, R., Clark, S., Damoc, B., Dudzik, A., et al. (2020). Imitating interactive intelligence. arXiv preprint arXiv:2012.05672. — [arXiv:2012.05672](https://arxiv.org/abs/2012.05672) [find](https://scholar.google.com/scholar?q=Abramson%2C+J%2E%2C+Ahuja%2C+A%2E%2C+Barr%2C+I%2E%2C+Brussee%2C+A%2E%2C+Carnevale%2C+F%2E%2C+Cassin%2C+M%2E%2C+Chhaparia%2C+R%2E%2C+Clark%2C+S%2E%2C+Damoc%2C+B%2E%2C+Dudzik%2C+A%2E%2C+et+al%2E+%282020%29%2E+Imitating+interactive+intelligence%2E+arXiv+preprint+arXiv%3A2012)
2. Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017). Constrained policy optimization. In International Conference on Machine Learning, pages 22–31. PMLR. [find](https://scholar.google.com/scholar?q=Achiam%2C+J%2E%2C+Held%2C+D%2E%2C+Tamar%2C+A%2E%2C+and+Abbeel%2C+P%2E+%282017%29%2E+Constrained+policy+optimization%2E+In+International+Conference+on+Machine+Learning%2C+pages+22%E2%80%9331%2E+PMLR%2E)
3. Anthony, T., Tian, Z., and Barber, D. (2017). Thinking fast and slow with deep learning and tree search. arXiv preprint arXiv:1705.08439. [find](https://scholar.google.com/scholar?q=Anthony%2C+T%2E%2C+Tian%2C+Z%2E%2C+and+Barber%2C+D%2E+%282017%29%2E+Thinking+fast+and+slow+with+deep+learning+and+tree+search%2E+arXiv+preprint+arXiv%3A1705%2E08439%2E)
4. Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H. S., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021). Ext5: Towards extreme multi-task scaling for transfer learning. arXiv preprint arXiv:2111.10952. [find](https://scholar.google.com/scholar?q=Aribandi%2C+V%2E%2C+Tay%2C+Y%2E%2C+Schuster%2C+T%2E%2C+Rao%2C+J%2E%2C+Zheng%2C+H%2E+S%2E%2C+Mehta%2C+S%2E+V%2E%2C+Zhuang%2C+H%2E%2C+Tran%2C+V%2E+Q%2E%2C+Bahri%2C+D%2E%2C+Ni%2C+J%2E%2C+et+al%2E+%282021%29%2E+Ext5%3A+Towards+extreme+multi%2Dtask+scaling+for+transfer+learning%2E+arX)
5. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+DasSarma%2C+N%2E%2C+et+al%2E+%282021%29%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+pre)
6. Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A., and Bengio, Y. (2016). An actor-critic algorithm for sequence prediction. arXiv preprint arXiv:1607.07086. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Brakel%2C+P%2E%2C+Xu%2C+K%2E%2C+Goyal%2C+A%2E%2C+Lowe%2C+R%2E%2C+Pineau%2C+J%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+%282016%29%2E+An+actor%2Dcritic+algorithm+for+sequence+prediction%2E+arXiv+preprint+arXiv%3A1607%2E07086%2E) [our card](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/)
7. Bahdanau, D., Hill, F., Leike, J., Hughes, E., Hosseini, A., Kohli, P., and Grefenstette, E. (2018). Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946. — [arXiv:1806.01946](https://arxiv.org/abs/1806.01946) [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Hill%2C+F%2E%2C+Leike%2C+J%2E%2C+Hughes%2C+E%2E%2C+Hosseini%2C+A%2E%2C+Kohli%2C+P%2E%2C+and+Grefenstette%2C+E%2E+%282018%29%2E+Learning+to+understand+goal+specifications+by+modelling+reward%2E+arXiv+preprint+arXiv%3A1806%2E01946%2E+%E2%80%94+)
8. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. [find](https://scholar.google.com/scholar?q=Bender%2C+E%2E+M%2E%2C+Gebru%2C+T%2E%2C+McMillan%2DMajor%2C+A%2E%2C+and+Shmitchell%2C+S%2E+%282021%29%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference+on+Fairness%2C+Ac)
9. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. (2020). Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050. [find](https://scholar.google.com/scholar?q=Blodgett%2C+S%2E+L%2E%2C+Barocas%2C+S%2E%2C+Daum%C3%A9+III%2C+H%2E%2C+and+Wallach%2C+H%2E+%282020%29%2E+Language+%28technology%29+is+power%3A+A+critical+survey+of%22+bias%22+in+nlp%2E+arXiv+preprint+arXiv%3A2005%2E14050%2E) [our card](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/)
10. Böhm, F., Gao, Y., Meyer, C. M., Shapira, O., Dagan, I., and Gurevych, I. (2019). Better rewards yield better summaries: Learning to summarise without references. arXiv preprint arXiv:1909.01214. [find](https://scholar.google.com/scholar?q=B%C3%B6hm%2C+F%2E%2C+Gao%2C+Y%2E%2C+Meyer%2C+C%2E+M%2E%2C+Shapira%2C+O%2E%2C+Dagan%2C+I%2E%2C+and+Gurevych%2C+I%2E+%282019%29%2E+Better+rewards+yield+better+summaries%3A+Learning+to+summarise+without+references%2E+arXiv+preprint+arXiv%3A1909%2E01214%2E)
11. Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Huck, M., Hokamp, C., Koehn, P., Logacheva, V., Monz, C., Negri, M., Post, M., Scarton, C., Specia, L., and Turchi, M. (2015). Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical M… [find](https://scholar.google.com/scholar?q=Bojar%2C+O%2E%2C+Chatterjee%2C+R%2E%2C+Federmann%2C+C%2E%2C+Haddow%2C+B%2E%2C+Huck%2C+M%2E%2C+Hokamp%2C+C%2E%2C+Koehn%2C+P%2E%2C+Logacheva%2C+V%2E%2C+Monz%2C+C%2E%2C+Negri%2C+M%2E%2C+Post%2C+M%2E%2C+Scarton%2C+C%2E%2C+Specia%2C+L%2E%2C+and+Turchi%2C+M%2E+%282015%29%2E+Findings+of+the+201) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2015-findings-2015-workshop-statistical-machine/)
12. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021). On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. [find](https://scholar.google.com/scholar?q=Bommasani%2C+R%2E%2C+Hudson%2C+D%2E+A%2E%2C+Adeli%2C+E%2E%2C+Altman%2C+R%2E%2C+Arora%2C+S%2E%2C+von+Arx%2C+S%2E%2C+Bernstein%2C+M%2E+S%2E%2C+Bohg%2C+J%2E%2C+Bosselut%2C+A%2E%2C+Brunskill%2C+E%2E%2C+et+al%2E+%282021%29%2E+On+the+opportunities+and+risks+of+foundation+models) [our card](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
13. Bostrom, N. (2014). Superintelligence. Dunod. [find](https://scholar.google.com/scholar?q=Bostrom%2C+N%2E+%282014%29%2E+Superintelligence%2E+Dunod%2E)
14. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E+B%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+%282020%29%2E+Language+models+are+few%2Dshot+learners%2E+arXiv+preprint+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
15. Buchanan, B., Lohn, A., Musser, M., and Sedova, K. (2021). Truth, lies, and automation. Technical report, Center for the Study of Emerging Technology. [find](https://scholar.google.com/scholar?q=Buchanan%2C+B%2E%2C+Lohn%2C+A%2E%2C+Musser%2C+M%2E%2C+and+Sedova%2C+K%2E+%282021%29%2E+Truth%2C+lies%2C+and+automation%2E+Technical+report%2C+Center+for+the+Study+of+Emerging+Technology%2E)
16. Caliskan, A., Bryson, J. J., and Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. [find](https://scholar.google.com/scholar?q=Caliskan%2C+A%2E%2C+Bryson%2C+J%2E+J%2E%2C+and+Narayanan%2C+A%2E+%282017%29%2E+Semantics+derived+automatically+from+language+corpora+contain+human%2Dlike+biases%2E+Science%2C+356%286334%29%3A183%E2%80%93186%2E)
17. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Tramer%2C+F%2E%2C+Wallace%2C+E%2E%2C+Jagielski%2C+M%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Lee%2C+K%2E%2C+Roberts%2C+A%2E%2C+Brown%2C+T%2E%2C+Song%2C+D%2E%2C+Erlingsson%2C+U%2E%2C+et+al%2E+%282021%29%2E+Extracting+training+data+from+large+language+models%2E+In) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
18. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. [find](https://scholar.google.com/scholar?q=Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yuan%2C+Q%2E%2C+Pinto%2C+H%2E+P%2E+d%2E+O%2E%2C+Kaplan%2C+J%2E%2C+Edwards%2C+H%2E%2C+Burda%2C+Y%2E%2C+Joseph%2C+N%2E%2C+Brockman%2C+G%2E%2C+et+al%2E+%282021%29%2E+Evaluating+large+language+models+trained+on+code%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
19. Cho, W. S., Zhang, P., Zhang, Y., Li, X., Galley, M., Brockett, C., Wang, M., and Gao, J. (2018). Towards coherent and cohesive long-form text generation. arXiv preprint arXiv:1811.00511. [find](https://scholar.google.com/scholar?q=Cho%2C+W%2E+S%2E%2C+Zhang%2C+P%2E%2C+Zhang%2C+Y%2E%2C+Li%2C+X%2E%2C+Galley%2C+M%2E%2C+Brockett%2C+C%2E%2C+Wang%2C+M%2E%2C+and+Gao%2C+J%2E+%282018%29%2E+Towards+coherent+and+cohesive+long%2Dform+text+generation%2E+arXiv+preprint+arXiv%3A1811%2E00511%2E)
20. Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-t., Choi, Y., Liang, P., and Zettlemoyer, L. (2018). Quac: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. [find](https://scholar.google.com/scholar?q=Choi%2C+E%2E%2C+He%2C+H%2E%2C+Iyyer%2C+M%2E%2C+Yatskar%2C+M%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+Choi%2C+Y%2E%2C+Liang%2C+P%2E%2C+and+Zettlemoyer%2C+L%2E+%282018%29%2E+Quac%3A+Question+answering+in+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+i) [our card](https://md.research.elamur.ai/free/en/papers/choi-2018-quac-question-answering-context/)
21. Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. [https://www.alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge](https://www.alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge). [↗](https://www.alignmentforum.org/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge)
22. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E%2C+Shlegeris%2C+B%2E%2C+and+Amodei%2C+D%2E+%282018%29%2E+Supervising+strong+learners+by+amplifying+weak+experts%2E+arXiv+preprint+arXiv%3A1810%2E08575%2E)
23. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299–4307. [find](https://scholar.google.com/scholar?q=Christiano%2C+P%2E+F%2E%2C+Leike%2C+J%2E%2C+Brown%2C+T%2E%2C+Martic%2C+M%2E%2C+Legg%2C+S%2E%2C+and+Amodei%2C+D%2E+%282017%29%2E+Deep+reinforcement+learning+from+human+preferences%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+42)
24. Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. (2019). Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164. [find](https://scholar.google.com/scholar?q=Dathathri%2C+S%2E%2C+Madotto%2C+A%2E%2C+Lan%2C+J%2E%2C+Hung%2C+J%2E%2C+Frank%2C+E%2E%2C+Molino%2C+P%2E%2C+Yosinski%2C+J%2E%2C+and+Liu%2C+R%2E+%282019%29%2E+Plug+and+play+language+models%3A+A+simple+approach+to+controlled+text+generation%2E+arXiv+preprint+a) [our card](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/)
25. Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., and Gupta, R. (2021). Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872. [find](https://scholar.google.com/scholar?q=Dhamala%2C+J%2E%2C+Sun%2C+T%2E%2C+Kumar%2C+V%2E%2C+Krishna%2C+S%2E%2C+Pruksachatkun%2C+Y%2E%2C+Chang%2C+K%2E%2DW%2E%2C+and+Gupta%2C+R%2E+%282021%29%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+In+Proceedings+of)
26. Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J. (2019a). Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03842. [find](https://scholar.google.com/scholar?q=Dinan%2C+E%2E%2C+Fan%2C+A%2E%2C+Williams%2C+A%2E%2C+Urbanek%2C+J%2E%2C+Kiela%2C+D%2E%2C+and+Weston%2C+J%2E+%282019a%29%2E+Queens+are+powerful+too%3A+Mitigating+gender+bias+in+dialogue+generation%2E+arXiv+preprint+arXiv%3A1911%2E03842%2E) [our card](https://md.research.elamur.ai/free/en/papers/dinan-2020-queens-are-powerful-too-mitigating/)
27. Dinan, E., Humeau, S., Chintagunta, B., and Weston, J. (2019b). Build it break it fix it for dialogue safety: Robustness from adversarial human attack. arXiv preprint arXiv:1908.06083. [find](https://scholar.google.com/scholar?q=Dinan%2C+E%2E%2C+Humeau%2C+S%2E%2C+Chintagunta%2C+B%2E%2C+and+Weston%2C+J%2E+%282019b%29%2E+Build+it+break+it+fix+it+for+dialogue+safety%3A+Robustness+from+adversarial+human+attack%2E+arXiv+preprint+arXiv%3A1908%2E06083%2E) [our card](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/)
28. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. (2019). Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161. [find](https://scholar.google.com/scholar?q=Dua%2C+D%2E%2C+Wang%2C+Y%2E%2C+Dasigi%2C+P%2E%2C+Stanovsky%2C+G%2E%2C+Singh%2C+S%2E%2C+and+Gardner%2C+M%2E+%282019%29%2E+Drop%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+arXiv+preprint+arXiv%3A1903%2E00161%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
29. Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. [find](https://scholar.google.com/scholar?q=Fedus%2C+W%2E%2C+Zoph%2C+B%2E%2C+and+Shazeer%2C+N%2E+%282021%29%2E+Switch+transformers%3A+Scaling+to+trillion+parameter+models+with+simple+and+efficient+sparsity%2E+arXiv+preprint+arXiv%3A2101%2E03961%2E)
30. Gabriel, I. (2020). Artificial intelligence, values, and alignment. Minds and machines, 30(3):411–437. [find](https://scholar.google.com/scholar?q=Gabriel%2C+I%2E+%282020%29%2E+Artificial+intelligence%2C+values%2C+and+alignment%2E+Minds+and+machines%2C+30%283%29%3A411%E2%80%93437%2E)
31. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. [find](https://scholar.google.com/scholar?q=Gehman%2C+S%2E%2C+Gururangan%2C+S%2E%2C+Sap%2C+M%2E%2C+Choi%2C+Y%2E%2C+and+Smith%2C+N%2E+A%2E+%282020%29%2E+Realtoxicityprompts%3A+Evaluating+neural+toxic+degeneration+in+language+models%2E+arXiv+preprint+arXiv%3A2009%2E11462%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
32. Hancock, B., Bordes, A., Mazare, P.-E., and Weston, J. (2019). Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415. [find](https://scholar.google.com/scholar?q=Hancock%2C+B%2E%2C+Bordes%2C+A%2E%2C+Mazare%2C+P%2E%2DE%2E%2C+and+Weston%2C+J%2E+%282019%29%2E+Learning+from+dialogue+after+deployment%3A+Feed+yourself%2C+chatbot%21+arXiv+preprint+arXiv%3A1901%2E05415%2E)
33. Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J. (2018). Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 123–129. [find](https://scholar.google.com/scholar?q=Henderson%2C+P%2E%2C+Sinha%2C+K%2E%2C+Angelard%2DGontier%2C+N%2E%2C+Ke%2C+N%2E+R%2E%2C+Fried%2C+G%2E%2C+Lowe%2C+R%2E%2C+and+Pineau%2C+J%2E+%282018%29%2E+Ethical+challenges+in+data%2Ddriven+dialogue+systems%2E+In+Proceedings+of+the+2018+AAAI%2FACM+Conferenc)
34. Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., and Kohli, P. (2019). Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064. [find](https://scholar.google.com/scholar?q=Huang%2C+P%2E%2DS%2E%2C+Zhang%2C+H%2E%2C+Jiang%2C+R%2E%2C+Stanforth%2C+R%2E%2C+Welbl%2C+J%2E%2C+Rae%2C+J%2E%2C+Maini%2C+V%2E%2C+Yogatama%2C+D%2E%2C+and+Kohli%2C+P%2E+%282019%29%2E+Reducing+sentiment+bias+in+language+models+via+counterfactual+evaluation%2E+arXiv+pr)
35. Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. (2018). Reward learning from human preferences and demonstrations in atari. In Advances in neural information processing systems, pages 8011–8023. [find](https://scholar.google.com/scholar?q=Ibarz%2C+B%2E%2C+Leike%2C+J%2E%2C+Pohlen%2C+T%2E%2C+Irving%2C+G%2E%2C+Legg%2C+S%2E%2C+and+Amodei%2C+D%2E+%282018%29%2E+Reward+learning+from+human+preferences+and+demonstrations+in+atari%2E+In+Advances+in+neural+information+processing+systems%2C)
36. Irving, G., Christiano, P., and Amodei, D. (2018). AI safety via debate. arXiv preprint arXiv:1805.00899. [find](https://scholar.google.com/scholar?q=Irving%2C+G%2E%2C+Christiano%2C+P%2E%2C+and+Amodei%2C+D%2E+%282018%29%2E+AI+safety+via+debate%2E+arXiv+preprint+arXiv%3A1805%2E00899%2E)
37. Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. (2019). Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) [find](https://scholar.google.com/scholar?q=Jaques%2C+N%2E%2C+Ghandeharioun%2C+A%2E%2C+Shen%2C+J%2E+H%2E%2C+Ferguson%2C+C%2E%2C+Lapedriza%2C+A%2E%2C+Jones%2C+N%2E%2C+Gu%2C+S%2E%2C+and+Picard%2C+R%2E+%282019%29%2E+Way+off%2Dpolicy+batch+deep+reinforcement+learning+of+implicit+human+preferences+in+dia)
38. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving, G. (2021). Alignment of language agents. arXiv preprint arXiv:2103.14659. [find](https://scholar.google.com/scholar?q=Kenton%2C+Z%2E%2C+Everitt%2C+T%2E%2C+Weidinger%2C+L%2E%2C+Gabriel%2C+I%2E%2C+Mikulik%2C+V%2E%2C+and+Irving%2C+G%2E+%282021%29%2E+Alignment+of+language+agents%2E+arXiv+preprint+arXiv%3A2103%2E14659%2E)
39. Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. (2019). Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858. [find](https://scholar.google.com/scholar?q=Keskar%2C+N%2E+S%2E%2C+McCann%2C+B%2E%2C+Varshney%2C+L%2E+R%2E%2C+Xiong%2C+C%2E%2C+and+Socher%2C+R%2E+%282019%29%2E+Ctrl%3A+A+conditional+transformer+language+model+for+controllable+generation%2E+arXiv+preprint+arXiv%3A1909%2E05858%2E) [our card](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
40. Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. (2020). Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700. [find](https://scholar.google.com/scholar?q=Khashabi%2C+D%2E%2C+Min%2C+S%2E%2C+Khot%2C+T%2E%2C+Sabharwal%2C+A%2E%2C+Tafjord%2C+O%2E%2C+Clark%2C+P%2E%2C+and+Hajishirzi%2C+H%2E+%282020%29%2E+Unifiedqa%3A+Crossing+format+boundaries+with+a+single+qa+system%2E+arXiv+preprint+arXiv%3A2005%2E00700%2E) [our card](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/)
41. Kirk, H., Jun, Y., Iqbal, H., Benussi, E., Volpin, F., Dreyer, F. A., Shtedritski, A., and Asano, Y. M. (2021). How true is gpt-2? an empirical analysis of intersectional occupational biases. arXiv preprint arXiv:2102.04130. — [arXiv:2102.04130](https://arxiv.org/abs/2102.04130) [find](https://scholar.google.com/scholar?q=Kirk%2C+H%2E%2C+Jun%2C+Y%2E%2C+Iqbal%2C+H%2E%2C+Benussi%2C+E%2E%2C+Volpin%2C+F%2E%2C+Dreyer%2C+F%2E+A%2E%2C+Shtedritski%2C+A%2E%2C+and+Asano%2C+Y%2E+M%2E+%282021%29%2E+How+true+is+gpt%2D2%3F+an+empirical+analysis+of+intersectional+occupational+biases%2E+arXiv+pr)
42. Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. (2020). Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367. [find](https://scholar.google.com/scholar?q=Krause%2C+B%2E%2C+Gotmare%2C+A%2E+D%2E%2C+McCann%2C+B%2E%2C+Keskar%2C+N%2E+S%2E%2C+Joty%2C+S%2E%2C+Socher%2C+R%2E%2C+and+Rajani%2C+N%2E+F%2E+%282020%29%2E+Gedi%3A+Generative+discriminator+guided+sequence+generation%2E+arXiv+preprint+arXiv%3A2009%2E06367%2E) [our card](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/)
43. Kreutzer, J., Khadivi, S., Matusov, E., and Riezler, S. (2018). Can neural machine translation be improved with user feedback? arXiv preprint arXiv:1804.05958. [find](https://scholar.google.com/scholar?q=Kreutzer%2C+J%2E%2C+Khadivi%2C+S%2E%2C+Matusov%2C+E%2E%2C+and+Riezler%2C+S%2E+%282018%29%2E+Can+neural+machine+translation+be+improved+with+user+feedback%3F+arXiv+preprint+arXiv%3A1804%2E05958%2E)
44. Lawrence, C. and Riezler, S. (2018). Improving a neural semantic parser by counterfactual learning from human bandit feedback. arXiv preprint arXiv:1805.01252. [find](https://scholar.google.com/scholar?q=Lawrence%2C+C%2E+and+Riezler%2C+S%2E+%282018%29%2E+Improving+a+neural+semantic+parser+by+counterfactual+learning+from+human+bandit+feedback%2E+arXiv+preprint+arXiv%3A1805%2E01252%2E)
45. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. [find](https://scholar.google.com/scholar?q=Leike%2C+J%2E%2C+Krueger%2C+D%2E%2C+Everitt%2C+T%2E%2C+Martic%2C+M%2E%2C+Maini%2C+V%2E%2C+and+Legg%2C+S%2E+%282018%29%2E+Scalable+agent+alignment+via+reward+modeling%3A+a+research+direction%2E+arXiv+preprint+arXiv%3A1811%2E07871%2E)
46. Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. (2017). AI safety gridworlds. arXiv preprint arXiv:1711.09883. — [arXiv:1711.09883](https://arxiv.org/abs/1711.09883) [find](https://scholar.google.com/scholar?q=Leike%2C+J%2E%2C+Martic%2C+M%2E%2C+Krakovna%2C+V%2E%2C+Ortega%2C+P%2E+A%2E%2C+Everitt%2C+T%2E%2C+Lefrancq%2C+A%2E%2C+Orseau%2C+L%2E%2C+and+Legg%2C+S%2E+%282017%29%2E+AI+safety+gridworlds%2E+arXiv+preprint+arXiv%3A1711%2E09883%2E+%E2%80%94+%5BarXiv%3A1711%2E09883%5D%28https%3A%2F%2Farxi)
47. Liang, P. P., Wu, C., Morency, L.-P., and Salakhutdinov, R. (2021). Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565–6576. PMLR. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E+P%2E%2C+Wu%2C+C%2E%2C+Morency%2C+L%2E%2DP%2E%2C+and+Salakhutdinov%2C+R%2E+%282021%29%2E+Towards+understanding+and+mitigating+social+biases+in+language+models%2E+In+International+Conference+on+Machine+Learning%2C+pages+6565%E2%80%936) [our card](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/)
48. Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. [find](https://scholar.google.com/scholar?q=Lin%2C+S%2E%2C+Hilton%2C+J%2E%2C+and+Evans%2C+O%2E+%282021%29%2E+Truthfulqa%3A+Measuring+how+models+mimic+human+falsehoods%2E+arXiv+preprint+arXiv%3A2109%2E07958%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
49. Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. (2019). Does gender matter? towards fairness in dialogue systems. arXiv preprint arXiv:1910.10486. [find](https://scholar.google.com/scholar?q=Liu%2C+H%2E%2C+Dacon%2C+J%2E%2C+Fan%2C+W%2E%2C+Liu%2C+H%2E%2C+Liu%2C+Z%2E%2C+and+Tang%2C+J%2E+%282019%29%2E+Does+gender+matter%3F+towards+fairness+in+dialogue+systems%2E+arXiv+preprint+arXiv%3A1910%2E10486%2E)
50. Madaan, A., Tandon, N., Clark, P., and Yang, Y. (2022). Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009. [find](https://scholar.google.com/scholar?q=Madaan%2C+A%2E%2C+Tandon%2C+N%2E%2C+Clark%2C+P%2E%2C+and+Yang%2C+Y%2E+%282022%29%2E+Memory%2Dassisted+prompt+editing+to+improve+gpt%2D3+after+deployment%2E+arXiv+preprint+arXiv%3A2201%2E06009%2E)
51. Manela, D. d. V., Errington, D., Fisher, T., van Breugel, B., and Minervini, P. (2021). Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. arXiv preprint arXiv:2101.09688. [find](https://scholar.google.com/scholar?q=Manela%2C+D%2E+d%2E+V%2E%2C+Errington%2C+D%2E%2C+Fisher%2C+T%2E%2C+van+Breugel%2C+B%2E%2C+and+Minervini%2C+P%2E+%282021%29%2E+Stereotype+and+skew%3A+Quantifying+gender+bias+in+pre%2Dtrained+and+fine%2Dtuned+language+models%2E+arXiv+preprint+arXiv)
52. Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. (2021). Cross-task generalization via natural language crowdsourcing instructions. arXiv preprint arXiv:2104.08773. [find](https://scholar.google.com/scholar?q=Mishra%2C+S%2E%2C+Khashabi%2C+D%2E%2C+Baral%2C+C%2E%2C+and+Hajishirzi%2C+H%2E+%282021%29%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+arXiv+preprint+arXiv%3A2104%2E08773%2E)
53. Nadeem, M., Bethke, A., and Reddy, S. (2020). Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456. [find](https://scholar.google.com/scholar?q=Nadeem%2C+M%2E%2C+Bethke%2C+A%2E%2C+and+Reddy%2C+S%2E+%282020%29%2E+Stereoset%3A+Measuring+stereotypical+bias+in+pretrained+language+models%2E+arXiv+preprint+arXiv%3A2004%2E09456%2E) [our card](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/)
54. Nahian, M. S. A., Frazier, S., Harrison, B., and Riedl, M. (2021). Training value-aligned reinforcement learning agents using a normative prior. arXiv preprint arXiv:2104.09469. — [arXiv:2104.09469](https://arxiv.org/abs/2104.09469) [find](https://scholar.google.com/scholar?q=Nahian%2C+M%2E+S%2E+A%2E%2C+Frazier%2C+S%2E%2C+Harrison%2C+B%2E%2C+and+Riedl%2C+M%2E+%282021%29%2E+Training+value%2Daligned+reinforcement+learning+agents+using+a+normative+prior%2E+arXiv+preprint+arXiv%3A2104%2E09469%2E+%E2%80%94+%5BarXiv%3A2104%2E09469%5D%28h)
55. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. [find](https://scholar.google.com/scholar?q=Nakano%2C+R%2E%2C+Hilton%2C+J%2E%2C+Balaji%2C+S%2E%2C+Wu%2C+J%2E%2C+Ouyang%2C+L%2E%2C+Kim%2C+C%2E%2C+Hesse%2C+C%2E%2C+Jain%2C+S%2E%2C+Kosaraju%2C+V%2E%2C+Saunders%2C+W%2E%2C+et+al%2E+%282021%29%2E+Webgpt%3A+Browser%2Dassisted+question%2Danswering+with+human+feedback%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/)
56. Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. (2016). Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023. [find](https://scholar.google.com/scholar?q=Nallapati%2C+R%2E%2C+Zhou%2C+B%2E%2C+Gulcehre%2C+C%2E%2C+Xiang%2C+B%2E%2C+et+al%2E+%282016%29%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2E)
57. Nangia, N., Vania, C., Bhalerao, R., and Bowman, S. R. (2020). CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nangia%2C+N%2E%2C+Vania%2C+C%2E%2C+Bhalerao%2C+R%2E%2C+and+Bowman%2C+S%2E+R%2E+%282020%29%2E+CrowS%2DPairs%3A+A+Challenge+Dataset+for+Measuring+Social+Biases+in+Masked+Language+Models%2E+In+Proceedings+of+the+2020+Conference+on+Empirica) [our card](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/)
58. Ngo, H., Raterink, C., Araújo, J. G., Zhang, I., Chen, C., Morisot, A., and Frosst, N. (2021). Mitigating harm in language models with conditional-likelihood filtration. arXiv preprint arXiv:2108.07790. — [arXiv:2108.07790](https://arxiv.org/abs/2108.07790) [find](https://scholar.google.com/scholar?q=Ngo%2C+H%2E%2C+Raterink%2C+C%2E%2C+Ara%C3%BAjo%2C+J%2E+G%2E%2C+Zhang%2C+I%2E%2C+Chen%2C+C%2E%2C+Morisot%2C+A%2E%2C+and+Frosst%2C+N%2E+%282021%29%2E+Mitigating+harm+in+language+models+with+conditional%2Dlikelihood+filtration%2E+arXiv+preprint+arXiv%3A2108%2E0779)
59. Perez, E., Karamcheti, S., Fergus, R., Weston, J., Kiela, D., and Cho, K. (2019). Finding generalizable evidence by learning to convince q&a models. arXiv preprint arXiv:1909.05863. [find](https://scholar.google.com/scholar?q=Perez%2C+E%2E%2C+Karamcheti%2C+S%2E%2C+Fergus%2C+R%2E%2C+Weston%2C+J%2E%2C+Kiela%2C+D%2E%2C+and+Cho%2C+K%2E+%282019%29%2E+Finding+generalizable+evidence+by+learning+to+convince+q%26a+models%2E+arXiv+preprint+arXiv%3A1909%2E05863%2E)
60. Qian, Y., Muaz, U., Zhang, B., and Hyun, J. W. (2019). Reducing gender bias in word-level language models with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801. [find](https://scholar.google.com/scholar?q=Qian%2C+Y%2E%2C+Muaz%2C+U%2E%2C+Zhang%2C+B%2E%2C+and+Hyun%2C+J%2E+W%2E+%282019%29%2E+Reducing+gender+bias+in+word%2Dlevel+language+models+with+a+gender%2Dequalizing+loss+function%2E+arXiv+preprint+arXiv%3A1905%2E12801%2E)
61. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+%282019%29%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+Blog%2C+1%288%29%3A9%2E)
62. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021). Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E%2C+Young%2C+S%2E%2C+et+al%2E+%282021%29%2E+Scaling+language+models%3A+Methods%2C+analysis+%26+insights+from+tr)
63. Rajpurkar, P., Jia, R., and Liang, P. (2018). Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822. [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Jia%2C+R%2E%2C+and+Liang%2C+P%2E+%282018%29%2E+Know+what+you+don%E2%80%99t+know%3A+Unanswerable+questions+for+squad%2E+arXiv+preprint+arXiv%3A1806%2E03822%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
64. Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. (2018). Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana. Association for Compu… [find](https://scholar.google.com/scholar?q=Rudinger%2C+R%2E%2C+Naradowsky%2C+J%2E%2C+Leonard%2C+B%2E%2C+and+Van+Durme%2C+B%2E+%282018%29%2E+Gender+bias+in+coreference+resolution%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+for+C) [our card](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/)
65. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207. [find](https://scholar.google.com/scholar?q=Sanh%2C+V%2E%2C+Webson%2C+A%2E%2C+Raffel%2C+C%2E%2C+Bach%2C+S%2E+H%2E%2C+Sutawika%2C+L%2E%2C+Alyafeai%2C+Z%2E%2C+Chaffin%2C+A%2E%2C+Stiegler%2C+A%2E%2C+Scao%2C+T%2E+L%2E%2C+Raja%2C+A%2E%2C+et+al%2E+%282021%29%2E+Multitask+prompted+training+enables+zero%2Dshot+task+generaliz) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
66. Schick, T., Udupa, S., and Schütze, H. (2021). Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. arXiv preprint arXiv:2103.00453. [find](https://scholar.google.com/scholar?q=Schick%2C+T%2E%2C+Udupa%2C+S%2E%2C+and+Sch%C3%BCtze%2C+H%2E+%282021%29%2E+Self%2Ddiagnosis+and+self%2Ddebiasing%3A+A+proposal+for+reducing+corpus%2Dbased+bias+in+nlp%2E+arXiv+preprint+arXiv%3A2103%2E00453%2E) [our card](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
67. Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR). [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Moritz%2C+P%2E%2C+Levine%2C+S%2E%2C+Jordan%2C+M%2E%2C+and+Abbeel%2C+P%2E+%282016%29%2E+High%2Ddimensional+continuous+control+using+generalized+advantage+estimation%2E+In+Proceedings+of+the+International+Conference+on+L)
68. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=Schulman%2C+J%2E%2C+Wolski%2C+F%2E%2C+Dhariwal%2C+P%2E%2C+Radford%2C+A%2E%2C+and+Klimov%2C+O%2E+%282017%29%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
69. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. (2019). The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326. [find](https://scholar.google.com/scholar?q=Sheng%2C+E%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Natarajan%2C+P%2E%2C+and+Peng%2C+N%2E+%282019%29%2E+The+woman+worked+as+a+babysitter%3A+On+biases+in+language+generation%2E+arXiv+preprint+arXiv%3A1909%2E01326%2E) [our card](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/)
70. Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. [find](https://scholar.google.com/scholar?q=Silver%2C+D%2E%2C+Hubert%2C+T%2E%2C+Schrittwieser%2C+J%2E%2C+Antonoglou%2C+I%2E%2C+Lai%2C+M%2E%2C+Guez%2C+A%2E%2C+Lanctot%2C+M%2E%2C+Sifre%2C+L%2E%2C+Kumaran%2C+D%2E%2C+Graepel%2C+T%2E%2C+et+al%2E+%282017%29%2E+Mastering+chess+and+shogi+by+self%2Dplay+with+a+general+rei) [our card](https://md.research.elamur.ai/free/en/papers/al-2017-mastering-chess-shogi-self-play/)
71. Soares, N., Fallenstein, B., Armstrong, S., and Yudkowsky, E. (2015). Corrigibility. In Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Soares%2C+N%2E%2C+Fallenstein%2C+B%2E%2C+Armstrong%2C+S%2E%2C+and+Yudkowsky%2C+E%2E+%282015%29%2E+Corrigibility%2E+In+Workshops+at+the+Twenty%2DNinth+AAAI+Conference+on+Artificial+Intelligence%2E)
72. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E+Y%2E%2C+and+Potts%2C+C%2E+%282013%29%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Proceedings+of+the+201)
73. Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019). Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. [find](https://scholar.google.com/scholar?q=Solaiman%2C+I%2E%2C+Brundage%2C+M%2E%2C+Clark%2C+J%2E%2C+Askell%2C+A%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Wu%2C+J%2E%2C+Radford%2C+A%2E%2C+Krueger%2C+G%2E%2C+Kim%2C+J%2E+W%2E%2C+Kreps%2C+S%2E%2C+et+al%2E+%282019%29%2E+Release+strategies+and+the+social+impacts+of+language+model) [our card](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/)
74. Solaiman, I. and Dennison, C. (2021). Process for adapting language models to society (palms) with values-targeted datasets. arXiv preprint arXiv:2106.10328. [find](https://scholar.google.com/scholar?q=Solaiman%2C+I%2E+and+Dennison%2C+C%2E+%282021%29%2E+Process+for+adapting+language+models+to+society+%28palms%29+with+values%2Dtargeted+datasets%2E+arXiv+preprint+arXiv%3A2106%2E10328%2E)
75. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. (2020). Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325. [find](https://scholar.google.com/scholar?q=Stiennon%2C+N%2E%2C+Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Ziegler%2C+D%2E+M%2E%2C+Lowe%2C+R%2E%2C+Voss%2C+C%2E%2C+Radford%2C+A%2E%2C+Amodei%2C+D%2E%2C+and+Christiano%2C+P%2E+%282020%29%2E+Learning+to+summarize+from+human+feedback%2E+arXiv+preprint+arXiv%3A2009%2E01325%2E)
76. Tamkin, A., Brundage, M., Clark, J., and Ganguli, D. (2021). Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. [find](https://scholar.google.com/scholar?q=Tamkin%2C+A%2E%2C+Brundage%2C+M%2E%2C+Clark%2C+J%2E%2C+and+Ganguli%2C+D%2E+%282021%29%2E+Understanding+the+capabilities%2C+limitations%2C+and+societal+impact+of+large+language+models%2E+arXiv+preprint+arXiv%3A2102%2E02503%2E)
77. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022). Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. [find](https://scholar.google.com/scholar?q=Thoppilan%2C+R%2E%2C+De+Freitas%2C+D%2E%2C+Hall%2C+J%2E%2C+Shazeer%2C+N%2E%2C+Kulshreshtha%2C+A%2E%2C+Cheng%2C+H%2E%2DT%2E%2C+Jin%2C+A%2E%2C+Bos%2C+T%2E%2C+Baker%2C+L%2E%2C+Du%2C+Y%2E%2C+et+al%2E+%282022%29%2E+Lamda%3A+Language+models+for+dialog+applications%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/)
78. Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. M. (2020). Investigating gender bias in language models using causal mediation analysis. In NeurIPS. [find](https://scholar.google.com/scholar?q=Vig%2C+J%2E%2C+Gehrmann%2C+S%2E%2C+Belinkov%2C+Y%2E%2C+Qian%2C+S%2E%2C+Nevo%2C+D%2E%2C+Singer%2C+Y%2E%2C+and+Shieber%2C+S%2E+M%2E+%282020%29%2E+Investigating+gender+bias+in+language+models+using+causal+mediation+analysis%2E+In+NeurIPS%2E)
79. Völske, M., Potthast, M., Syed, S., and Stein, B. (2017). Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63. [find](https://scholar.google.com/scholar?q=V%C3%B6lske%2C+M%2E%2C+Potthast%2C+M%2E%2C+Syed%2C+S%2E%2C+and+Stein%2C+B%2E+%282017%29%2E+Tl%3B+dr%3A+Mining+reddit+to+learn+automatic+summarization%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C+pages+59%E2%80%9363%2E) [our card](https://md.research.elamur.ai/free/en/papers/volske-2017-tl-dr-mining-reddit-learn/)
80. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2019). Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Pruksachatkun%2C+Y%2E%2C+Nangia%2C+N%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+R%2E+%282019%29%2E+Superglue%3A+A+stickier+benchmark+for+general%2Dpurpose+language+understanding+systems%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
81. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Bosma%2C+M%2E%2C+Zhao%2C+V%2E+Y%2E%2C+Guu%2C+K%2E%2C+Yu%2C+A%2E+W%2E%2C+Lester%2C+B%2E%2C+Du%2C+N%2E%2C+Dai%2C+A%2E+M%2E%2C+and+Le%2C+Q%2E+V%2E+%282021%29%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2109%2E01652%2E)
82. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021). Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. [find](https://scholar.google.com/scholar?q=Weidinger%2C+L%2E%2C+Mellor%2C+J%2E%2C+Rauh%2C+M%2E%2C+Griffin%2C+C%2E%2C+Uesato%2C+J%2E%2C+Huang%2C+P%2E%2DS%2E%2C+Cheng%2C+M%2E%2C+Glaese%2C+M%2E%2C+Balle%2C+B%2E%2C+Kasirzadeh%2C+A%2E%2C+et+al%2E+%282021%29%2E+Ethical+and+social+risks+of+harm+from+language+models%2E+arXi) [our card](https://md.research.elamur.ai/free/en/papers/weidinger-2021-ethical-social-risks-harm-language/)
83. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. (2021). Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445. [find](https://scholar.google.com/scholar?q=Welbl%2C+J%2E%2C+Glaese%2C+A%2E%2C+Uesato%2C+J%2E%2C+Dathathri%2C+S%2E%2C+Mellor%2C+J%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Anderson%2C+K%2E%2C+Kohli%2C+P%2E%2C+Coppin%2C+B%2E%2C+and+Huang%2C+P%2E%2DS%2E+%282021%29%2E+Challenges+in+detoxifying+language+models%2E+arXiv+preprint+) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/)
84. Wu, J., Ouyang, L., Ziegler, D. M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P. (2021). Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862. [find](https://scholar.google.com/scholar?q=Wu%2C+J%2E%2C+Ouyang%2C+L%2E%2C+Ziegler%2C+D%2E+M%2E%2C+Stiennon%2C+N%2E%2C+Lowe%2C+R%2E%2C+Leike%2C+J%2E%2C+and+Christiano%2C+P%2E+%282021%29%2E+Recursively+summarizing+books+with+human+feedback%2E+arXiv+preprint+arXiv%3A2109%2E10862%2E) [our card](https://md.research.elamur.ai/free/en/papers/wu-2021-recursively-summarizing-books-human-feedback/)
85. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. (2021). Detoxifying language models risks marginalizing minority voices. arXiv preprint arXiv:2104.06390. [find](https://scholar.google.com/scholar?q=Xu%2C+A%2E%2C+Pathak%2C+E%2E%2C+Wallace%2C+E%2E%2C+Gururangan%2C+S%2E%2C+Sap%2C+M%2E%2C+and+Klein%2C+D%2E+%282021%29%2E+Detoxifying+language+models+risks+marginalizing+minority+voices%2E+arXiv+preprint+arXiv%3A2104%2E06390%2E)
86. Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. (2020). Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079. [find](https://scholar.google.com/scholar?q=Xu%2C+J%2E%2C+Ju%2C+D%2E%2C+Li%2C+M%2E%2C+Boureau%2C+Y%2E%2DL%2E%2C+Weston%2C+J%2E%2C+and+Dinan%2C+E%2E+%282020%29%2E+Recipes+for+safety+in+open%2Ddomain+chatbots%2E+arXiv+preprint+arXiv%3A2010%2E07079%2E)
87. Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., and Hakkani-Tur, D. (2019). Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015. [find](https://scholar.google.com/scholar?q=Yi%2C+S%2E%2C+Goel%2C+R%2E%2C+Khatri%2C+C%2E%2C+Cervone%2C+A%2E%2C+Chung%2C+T%2E%2C+Hedayatnia%2C+B%2E%2C+Venkatesh%2C+A%2E%2C+Gabriel%2C+R%2E%2C+and+Hakkani%2DTur%2C+D%2E+%282019%29%2E+Towards+coherent+and+engaging+spoken+dialog+response+generation+using+auto)
88. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. (2019). Hellaswag: Can a machine really finish your sentence? In Association for Computational Linguistics, pages 4791–4800. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+%282019%29%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Association+for+Computational+Linguistics%2C+pages+4791%E2%80%934800%2E)
89. Zhao, M., Anderson, P., Jain, V., Wang, S., Ku, A., Baldridge, J., and Ie, E. (2021). On the evaluation of vision-and-language navigation instructions. arXiv preprint arXiv:2101.10504. [find](https://scholar.google.com/scholar?q=Zhao%2C+M%2E%2C+Anderson%2C+P%2E%2C+Jain%2C+V%2E%2C+Wang%2C+S%2E%2C+Ku%2C+A%2E%2C+Baldridge%2C+J%2E%2C+and+Ie%2C+E%2E+%282021%29%2E+On+the+evaluation+of+vision%2Dand%2Dlanguage+navigation+instructions%2E+arXiv+preprint+arXiv%3A2101%2E10504%2E)
90. Zhou, W. and Xu, K. (2020). Learning to compare for better training and evaluation of open domain natural language generation models. arXiv preprint arXiv:2002.05058. — [arXiv:2002.05058](https://arxiv.org/abs/2002.05058) [find](https://scholar.google.com/scholar?q=Zhou%2C+W%2E+and+Xu%2C+K%2E+%282020%29%2E+Learning+to+compare+for+better+training+and+evaluation+of+open+domain+natural+language+generation+models%2E+arXiv+preprint+arXiv%3A2002%2E05058%2E+%E2%80%94+%5BarXiv%3A2002%2E05058%5D%28https%3A%2F%2Farxi)
91. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E+M%2E%2C+Stiennon%2C+N%2E%2C+Wu%2C+J%2E%2C+Brown%2C+T%2E+B%2E%2C+Radford%2C+A%2E%2C+Amodei%2C+D%2E%2C+Christiano%2C+P%2E%2C+and+Irving%2C+G%2E+%282019%29%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2E)