---
title: "Pretraining Language Models with Human — Korbak 2023"
description: "Standard pretraining teaches language models to copy web text, including toxic language, leaked personal data and messy code."
authors: ["Tomasz Korbak", "Kejian Shi", "Angelica Chen", "Rasika Bhalerao", "Christopher L. Buckley", "Jason Phang", "Samuel R. Bowman", "Ethan Perez"]
year: 2023
venue: "arXiv:2302.08582"
source: "https://arxiv.org/abs/2302.08582"
doi: "10.48550/arxiv.2302.08582"
arxiv: "2302.08582"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Pretraining Language Models with Human Preferences

# Pretraining Language Models with Human Preferences

**Authors:** [Tomasz Korbak](https://md.research.elamur.ai/free/en/authors/tomasz-korbak/), Kejian Shi, [Angelica Chen](https://md.research.elamur.ai/free/en/authors/angelica-chen/), [Rasika Bhalerao](https://md.research.elamur.ai/free/en/authors/rasika-bhalerao/), Christopher L. Buckley, [Jason Phang](https://md.research.elamur.ai/free/en/authors/jason-phang/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/), [Ethan Perez](https://md.research.elamur.ai/free/en/authors/ethan-perez/) · **Year:** 2023 · **Venue:** arXiv:2302.08582 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.08582) · [PDF](https://arxiv.org/pdf/2302.08582) · [DOI 10.48550/arxiv.2302.08582](https://doi.org/10.48550/arxiv.2302.08582) · [arXiv 2302.08582](https://arxiv.org/abs/2302.08582) · [OpenAlex](https://openalex.org/W4321392329)

**Citations:** 23 citations (openalex-snapshot · 2026-10-14) · [94](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Standard pretraining teaches language models to copy web text, including toxic language, leaked personal data and messy code. Most alignment work fixes this only after pretraining, but large models tend to retain what they learned. Removing bad data entirely also weakens the model. The authors propose pretraining with human feedback (PHF): a reward model scores each segment of the training corpus, and the training objective uses those scores. The model still sees undesirable text but is steered away from reproducing it. They compare MLE with five PHF objectives on three tasks: toxicity, PII and PEP8-style Python. Conditional training was the best option. It labels each segment with a control token and, at generation time, conditions on the "good" token. It cut undesirable outputs by up to tenfold, held up better under adversarial prompting, and kept capabilities close to MLE. It also clearly beat MLE pretraining followed by feedback finetuning. The work suggests preferences should shape models from the start of training rather than being added afterwards.

## Key points

- Six objectives were benchmarked: MLE (baseline), dataset filtering, unlikelihood training, reward-weighted regression (RWR), advantage-weighted regression (AWR) and conditional training. Conditional training used segment-level `<|good|>`/`<|bad|>` tokens with a reward threshold.
- The three tasks were toxicity avoidance, avoiding personally identifiable information (PII) and PEP8-compliant Python generation. For each task, the misalignment score is the negated reward.
- Conditional training was on the alignment–capabilities Pareto frontier for all three tasks. It stayed competitive with MLE on zero-shot benchmarks and after GLUE finetuning, and it was more robust under automated red-teaming following Perez et al. (2022).
- Feedback finetuning of MLE models covered 1.6B or 330M tokens of a 3.3B-token budget, and PHF matched or beat it in every case. On PII, PHF reached a misalignment score of 0.0013 versus 0.0018 for 1.6B-token finetuning. A PHF model needed about ten red-teaming rounds to reach the score a finetuned model hit after one.
- The paper appeared at ICML 2023. Code and datasets are at github.com/tomekkorbak/pretraining-with-human-feedback.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

## Cited by

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 16 of them have a page here, 78 not yet.

- [PaLM 2 Technical Report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://md.research.elamur.ai/free/en/papers/chao-2023-jailbreaking-black-box-large-language/)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://md.research.elamur.ai/free/en/papers/fernandes-2023-bridging-gap-survey-integrating-human/)
- [V-STaR: Training Verifiers for Self-Taught Reasoners](https://md.research.elamur.ai/free/en/papers/hosseini-2024-v-star-training-verifiers-self/)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://md.research.elamur.ai/free/en/papers/huang-2023-catastrophic-jailbreak-open-source-llms/)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://md.research.elamur.ai/free/en/papers/jain-2023-baseline-defenses-adversarial-attacks-against/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Certifying LLM Safety against Adversarial Prompting](https://md.research.elamur.ai/free/en/papers/kumar-2023-certifying-llm-safety-against-adversarial/)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://md.research.elamur.ai/free/en/papers/rottger-2023-xstest-test-suite-identifying-exaggerated/)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbreak-guard-aligned-language-models/)
- [Jailbroken: How Does LLM Safety Training Fail?](https://md.research.elamur.ai/free/en/papers/wei-2023-jailbroken-llm-safety-training-fail/)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://md.research.elamur.ai/free/en/papers/wu-2023-fine-grained-human-feedback-gives/)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://md.research.elamur.ai/free/en/papers/yuan-2023-gpt-4-too-smart-safe/)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://md.research.elamur.ai/free/en/papers/zou-2023-universal-transferable-adversarial-attacks-aligned/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{korbak2023pretraining,
  title = {{Pretraining Language Models with Human Preferences}},
  author = {Korbak, Tomasz and Shi, Kejian and Chen, Angelica and Bhalerao, Rasika and Buckley, Christopher L. and Phang, Jason and Bowman, Samuel R. and Perez, Ethan},
  year = {2023},
  doi = {10.48550/arxiv.2302.08582},
  eprint = {2302.08582},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.08582}
}
```

- [← previous Preference Ranking Optimization for Human Alignment](https://md.research.elamur.ai/free/en/papers/song-2023-preference-ranking-optimization-human-alignment/)
- [next → PreWoMe: Exploiting Presuppositions as Working Memory for Long Form Question Answering](https://md.research.elamur.ai/free/en/papers/han-2023-prewome-exploiting-presuppositions-as-working/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

86 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das-Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language assi… [find](https://scholar.google.com/scholar?q=Askell%2C+A%2E%2C+Bai%2C+Y%2E%2C+Chen%2C+A%2E%2C+Drain%2C+D%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Jones%2C+A%2E%2C+Joseph%2C+N%2E%2C+Mann%2C+B%2E%2C+Das%2DSarma%2C+N%2E%2C+Elhage%2C+N%2E%2C+Hatfield%2DDodds%2C+Z%2E%2C+Hernandez%2C+D%2E%2C+Kernion%2C+J%2E%2C+Ndousse%2C+K%2E%2C+Olsson%2C+C%2E)
2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E%2C+Askell%2C+A%2E%2C+Chen%2C+A%2E%2C+Das%2DSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Fort%2C+S%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+Joseph%2C+N%2E%2C+Kadavath%2C+S%2E%2C+Kernion%2C+J%2E%2C+Conerly%2C+T%2E%2C+El%2DShowk%2C+S%2E%2C+Elhage%2C+N%2E%2C+Hatf)
3. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., and Giampiccolo, D. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006. [find](https://scholar.google.com/scholar?q=Bar%2DHaim%2C+R%2E%2C+Dagan%2C+I%2E%2C+Dolan%2C+B%2E%2C+Ferro%2C+L%2E%2C+and+Giampiccolo%2C+D%2E+The+second+pascal+recognising+textual+entailment+challenge%2E+Proceedings+of+the+Second+PASCAL+Challenges+Workshop+on+Recognising+Textu)
4. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models, 2022. URL [https://arxiv.org/](https://arxiv.org/) abs/2202.07646. [↗](https://arxiv.org/)
5. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 1–14, Vancouver, Canada, August 2017. Associ… [find](https://scholar.google.com/scholar?q=Cer%2C+D%2E%2C+Diab%2C+M%2E%2C+Agirre%2C+E%2E%2C+Lopez%2DGazpio%2C+I%2E%2C+and+Specia%2C+L%2E+SemEval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+the+11th+Internatio)
6. Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neural probabilistic language model. J. Mach. Learn. Res., 3(null):1137–1155, mar 2003. ISSN 1532-4435. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Ducharme%2C+R%2E%2C+Vincent%2C+P%2E%2C+and+Janvin%2C+C%2E+A+neural+probabilistic+language+model%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+3%28null%29%3A1137%E2%80%931155%2C+mar+2003%2E+ISSN+1532%2D4435%2E)
7. Bentivogli, L., Magnini, B., Dagan, I., Dang, H. T., and Giampiccolo, D. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009. [find](https://scholar.google.com/scholar?q=Bentivogli%2C+L%2E%2C+Magnini%2C+B%2E%2C+Dagan%2C+I%2E%2C+Dang%2C+H%2E+T%2E%2C+and+Giampiccolo%2C+D%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+Second+Text+Analysis+Conference%2C+TAC+2009%2C+Gai)
8. NIST, 2009. URL [https://tac.nist.gov/publications/](https://tac.nist.gov/publications/) 2009/additional.papers/RTE5_overview. proceedings.pdf. [↗](https://tac.nist.gov/publications/)
9. Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E. Improving code generation by training with natural language feedback, 2023. [find](https://scholar.google.com/scholar?q=Chen%2C+A%2E%2C+Scheurer%2C+J%2E%2C+Korbak%2C+T%2E%2C+Campos%2C+J%2E+A%2E%2C+Chan%2C+J%2E+S%2E%2C+Bowman%2C+S%2E+R%2E%2C+Cho%2C+K%2E%2C+and+Perez%2C+E%2E+Improving+code+generation+by+training+with+natural+language+feedback%2C+2023%2E)
10. Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, pp. 491–500, New York, NY, USA, 2019. Association for Computing Machinery.… — [doi:10.1145/3308560](https://doi.org/10.1145/3308560) [find](https://scholar.google.com/scholar?q=Borkan%2C+D%2E%2C+Dixon%2C+L%2E%2C+Sorensen%2C+J%2E%2C+Thain%2C+N%2E%2C+and+Vasserman%2C+L%2E+Nuanced+metrics+for+measuring+unintended+bias+with+real+data+for+text+classification%2E+In+Companion+Proceedings+of+The+2019+World+Wide+)
11. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Process… [find](https://scholar.google.com/scholar?q=Chen%2C+L%2E%2C+Lu%2C+K%2E%2C+Rajeswaran%2C+A%2E%2C+Lee%2C+K%2E%2C+Grover%2C+A%2E%2C+Laskin%2C+M%2E%2C+Abbeel%2C+P%2E%2C+Srinivas%2C+A%2E%2C+and+Mordatch%2C+I%2E+Decision+transformer%3A+Reinforcement+learning+via+sequence+modeling%2E+In+Ranzato%2C+M%2E%2C+Beygel)
12. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., H. P., Kaplan, J., E… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+Chen%2C+M%2E%2C+Tworek%2C+J%2E%2C+Jun%2C+H%2E%2C+Yua)
13. Carlini, N., Liu, C., Erlingsson, U., Kos, J., and Song, D. 2021b. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., the 28th USENIX Conference on Security Symposium, Fedus, W., Li, Y., Wang,… [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Liu%2C+C%2E%2C+Erlingsson%2C+U%2E%2C+Kos%2C+J%2E%2C+and+Song%2C+D%2E+2021b%2E+The+secret+sharer%3A+Evaluating+and+testing+unintended+memorization+in+neural+networks%2E+In+Proceedings+of+Chung%2C+H%2E+W%2E%2C+Hou%2C+L%2E%2C+Longpr) [our card](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/)
14. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, pp. 177–190,… [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluating+Predictive)
15. Dai, N., Liang, J., Qiu, X., and Huang, X. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5997–6007, Florence, Italy, July 2019. Association for Computational… [find](https://scholar.google.com/scholar?q=Dai%2C+N%2E%2C+Liang%2C+J%2E%2C+Qiu%2C+X%2E%2C+and+Huang%2C+X%2E+Style+transformer%3A+Unpaired+text+style+transfer+without+disentangled+latent+representation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/dai-2019-style-transformer-unpaired-text-style/)
16. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Biderman%2C+S%2E%2C+Black%2C+S%2E%2C+Golding%2C+L%2E%2C+Hoppe%2C+T%2E%2C+Foster%2C+C%2E%2C+Phang%2C+J%2E%2C+He%2C+H%2E%2C+Thite%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+Presser%2C+S%2E%2C+and+Leahy%2C+C%2E+The+Pile%3A+An+800gb+dataset+of+diverse+text+for+language+mo)
17. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, November 2020. Association for Computational Linguistics. doi… [find](https://scholar.google.com/scholar?q=Gehman%2C+S%2E%2C+Gururangan%2C+S%2E%2C+Sap%2C+M%2E%2C+Choi%2C+Y%2E%2C+and+Smith%2C+N%2E+A%2E+RealToxicityPrompts%3A+Evaluating+neural+toxic+degeneration+in+language+models%2E+In+Findings+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
18. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1–9, Prague, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/W07… [↗](https://aclanthology.org/W07…)
19. Dettmers, T. and Zettlemoyer, L. The case for 4-bit precision: k-bit inference scaling laws, 2023. [find](https://scholar.google.com/scholar?q=Dettmers%2C+T%2E+and+Zettlemoyer%2C+L%2E+The+case+for+4%2Dbit+precision%3A+k%2Dbit+inference+scaling+laws%2C+2023%2E)
20. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Lee%2C+K%2E%2C+and+Toutanova%2C+K%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapt)
21. Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f-divergence minimization, 2023. [find](https://scholar.google.com/scholar?q=Go%2C+D%2E%2C+Korbak%2C+T%2E%2C+Kruszewski%2C+G%2E%2C+Rozen%2C+J%2E%2C+Ryu%2C+N%2E%2C+and+Dymetman%2C+M%2E+Aligning+language+models+with+preferences+through+f%2Ddivergence+minimization%2C+2023%2E)
22. Hanu, L. and Unitary team. Detoxify. [https://github.com/unitaryai/detoxify](https://github.com/unitaryai/detoxify), 2020. [↗](https://github.com/unitaryai/detoxify)
23. Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https://aclanthology.org/I05-5002. [↗](https://aclanthology.org/I05-5002) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
24. Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. URL [https://openreview.net/](https://openreview.net/) forum?id=S874XAIpkR-. [↗](https://openreview.net/)
25. Github. Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, pp. 123–129, New York, NY, USA, 2018. Association for Comput… — [doi:10.1145/3278721.3278777](https://doi.org/10.1145/3278721.3278777) [find](https://scholar.google.com/scholar?q=Github%2E+Henderson%2C+P%2E%2C+Sinha%2C+K%2E%2C+Angelard%2DGontier%2C+N%2E%2C+Ke%2C+N%2E+R%2E%2C+Fried%2C+G%2E%2C+Lowe%2C+R%2E%2C+and+Pineau%2C+J%2E+Ethical+challenges+in+data%2Ddriven+dialogue+systems%2E+In+Proceedings+of+the+2018+AAAI%2FACM+Conferenc)
26. Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems (NeurIPS), 2019. [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Mazeika%2C+M%2E%2C+Kadavath%2C+S%2E%2C+and+Song%2C+D%2E+Using+self%2Dsupervised+learning+can+improve+model+robustness+and+uncertainty%2E+Advances+in+Neural+Information+Processing+Systems+%28NeurIPS%29%2C+2019%2E)
27. Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-… [find](https://scholar.google.com/scholar?q=Fan%2C+A%2E%2C+Lewis%2C+M%2E%2C+and+Dauphin%2C+Y%2E+Hierarchical+neural+story+generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+pp%2E+889%E2%80%9389)
28. Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744–2751, Online, July 2020. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Hendrycks%2C+D%2E%2C+Liu%2C+X%2E%2C+Wallace%2C+E%2E%2C+Dziedzic%2C+A%2E%2C+Krishnan%2C+R%2E%2C+and+Song%2C+D%2E+Pretrained+transformers+improve+out%2Dof%2Ddistribution+robustness%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associati) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-pretrained-transformers-improve-out-distribution/)
29. Ficler, J. and Goldberg, Y. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4912. URL https://aclanthology.org/W… [↗](https://aclanthology.org/W…) [our card](https://md.research.elamur.ai/free/en/papers/ficler-2017-controlling-linguistic-style-aspects-neural/)
30. Hewitt, J. Initializing new word embeddings for pretrained language models, 2021. [find](https://scholar.google.com/scholar?q=Hewitt%2C+J%2E+Initializing+new+word+embeddings+for+pretrained+language+models%2C+2021%2E)
31. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L.… [find](https://scholar.google.com/scholar?q=Hoffmann%2C+J%2E%2C+Borgeaud%2C+S%2E%2C+Mensch%2C+A%2E%2C+Buchatskaya%2C+E%2E%2C+Cai%2C+T%2E%2C+Rutherford%2C+E%2E%2C+de+las+Casas%2C+D%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Welbl%2C+J%2E%2C+Clark%2C+A%2E%2C+Hennigan%2C+T%2E%2C+Noland%2C+E%2E%2C+Millican%2C+K%2E%2C+van+den+Driessche%2C+G)
32. K. (eds.), Advances in Neural Information Processing Systems, 2022a. URL [https://openreview.net/](https://openreview.net/) forum?id=XvI6h-s4un. [↗](https://openreview.net/)
33. Korbak, T., Perez, E., and Buckley, C. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083–1091, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. URL https://aclanth… [find](https://scholar.google.com/scholar?q=Korbak%2C+T%2E%2C+Perez%2C+E%2E%2C+and+Buckley%2C+C%2E+RL+with+KL+penalties+is+better+viewed+as+Bayesian+inference%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2022%2C+pp%2E+1083%E2%80%931091%2C+Abu+Dhabi%2C+)
34. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=rygGQyrFvH. [↗](https://openreview.net/forum?)
35. Kumar, A., Peng, X. B., and Levine, S. Reward-conditioned policies, 2019. URL https://arxiv.org/abs/ 1912.13465. [↗](https://arxiv.org/abs/)
36. Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. spaCy: Industrial-strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo.1212303. [find](https://scholar.google.com/scholar?q=Honnibal%2C+M%2E%2C+Montani%2C+I%2E%2C+Van+Landeghem%2C+S%2E%2C+and+Boyd%2C+A%2E+spaCy%3A+Industrial%2Dstrength+Natural+Language+Processing+in+Python%2E+2020%2E+doi%3A+10%2E5281%2Fzenodo%2E1212303%2E)
37. Jang, Y., Lee, J., and Kim, K.-E. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=qaxhBG1UUaS](https://openreview.net/forum?id=qaxhBG1UUaS). [↗](https://openreview.net/forum?id=qaxhBG1UUaS)
38. Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021. [find](https://scholar.google.com/scholar?q=Janner%2C+M%2E%2C+Li%2C+Q%2E%2C+and+Levine%2C+S%2E+Offline+reinforcement+learning+as+one+big+sequence+modeling+problem%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2021%2E)
39. Jaques, N., Shen, J. H., Ghandeharioun, A., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985–4003, Online, Novem… [find](https://scholar.google.com/scholar?q=Jaques%2C+N%2E%2C+Shen%2C+J%2E+H%2E%2C+Ghandeharioun%2C+A%2E%2C+Ferguson%2C+C%2E%2C+Lapedriza%2C+A%2E%2C+Jones%2C+N%2E%2C+Gu%2C+S%2E%2C+and+Picard%2C+R%2E+Human%2Dcentric+dialog+training+via+offline+reinforcement+learning%2E+In+Proceedings+of+the+2020+)
40. Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. Ctrl: A conditional transformer language model for controllable generation, 2019. URL https://arxiv.org/abs/1909.05858. [↗](https://arxiv.org/abs/1909.05858) [our card](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)
41. Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. [find](https://scholar.google.com/scholar?q=Kumar%2C+A%2E%2C+Zhou%2C+A%2E%2C+Tucker%2C+G%2E%2C+and+Levine%2C+S%2E+Conservative+q%2Dlearning+for+offline+reinforcement+learning%2E+In+Proceedings+of+the+34th+International+Conference+on+Neural+Information+Processing+Systems)
42. Levesque, H. J. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. AAAI, 2011. URL [http://dblp.uni-trier.de/db/conf/](http://dblp.uni-trier.de/db/conf/) aaaiss/aaaiss2011-6.html\#Levesque11. [↗](http://dblp.uni-trier.de/db/conf/)
43. Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL [https://arxiv.org/](https://arxiv.org/) abs/2005.01643. [↗](https://arxiv.org/)
44. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–119, San Diego,… [find](https://scholar.google.com/scholar?q=Li%2C+J%2E%2C+Galley%2C+M%2E%2C+Brockett%2C+C%2E%2C+Gao%2C+J%2E%2C+and+Dolan%2C+B%2E+A+diversity%2Dpromoting+objective+function+for+neural+conversation+models%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of) [our card](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/)
45. Khalifa, M., Elsahar, H., and Dymetman, M. A distributional approach to controlled text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. [find](https://scholar.google.com/scholar?q=Khalifa%2C+M%2E%2C+Elsahar%2C+H%2E%2C+and+Dymetman%2C+M%2E+A+distributional+approach+to+controlled+text+generation%2E+In+9th+International+Conference+on+Learning+Representations%2C+ICLR+2021%2C+Virtual+Event%2C+Austria%2C+May+)
46. OpenReview.net, 2021. URL [https://openreview.net/](https://openreview.net/) forum?id=jWkw45-9AbL. [↗](https://openreview.net/)
47. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214– 3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1… [find](https://scholar.google.com/scholar?q=Lin%2C+S%2E%2C+Hilton%2C+J%2E%2C+and+Evans%2C+O%2E+TruthfulQA%3A+Measuring+how+models+mimic+human+falsehoods%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+P) [our card](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/)
48. Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis,… [find](https://scholar.google.com/scholar?q=Korbak%2C+T%2E%2C+Elsahar%2C+H%2E%2C+Kruszewski%2C+G%2E%2C+and+Dymetman%2C+M%2E+On+reinforcement+learning+and+distribution+matching+for+fine%2Dtuning+language+models+with+no+catastrophic+forgetting%2E+In+Oh%2C+A%2E+H%2E%2C+Agarwal%2C+A%2E)
49. Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. QUARK: Controllable text generation with reinforced unlearning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL [https://openreview.ne…](https://openreview.ne…) [↗](https://openreview.ne…) [our card](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/)
50. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286. [↗](https://arxiv.org/abs/2202.03286) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-red-teaming-language-models-language/)
51. Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pp. 745–750, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595937933. doi: 10. 114… — [doi:10.1145/1273496.1273590](https://doi.org/10.1145/1273496.1273590) [find](https://scholar.google.com/scholar?q=Peters%2C+J%2E+and+Schaal%2C+S%2E+Reinforcement+learning+by+reward%2Dweighted+regression+for+operational+space+control%2E+In+Proceedings+of+the+24th+International+Conference+on+Machine+Learning%2C+ICML+%E2%80%9907%2C+pp%2E+745)
52. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. Teaching language models to support answers with verified quotes, 2022. URL https://arxiv.org/abs/ 2203.11147. [↗](https://arxiv.org/abs/) [our card](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/)
53. Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E+and+Narasimhan%2C+K%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
54. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp. 234–239, 2012. doi: 10.1109/SLT.2012.6424228. — [doi:10.1109/slt.2012.6424228](https://doi.org/10.1109/slt.2012.6424228) [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E+and+Zweig%2C+G%2E+Context+dependent+recurrent+neural+network+language+model%2E+In+2012+IEEE+Spoken+Language+Technology+Workshop+%28SLT%29%2C+pp%2E+234%E2%80%93239%2C+2012%2E+doi%3A+10%2E1109%2FSLT%2E2012%2E6424228%2E+%E2%80%94+%5Bdoi%3A10)
55. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
56. Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets, 2020. URL https://arxiv.org/abs/ 2006.09359. [↗](https://arxiv.org/abs/)
57. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 1… [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Zhang%2C+J%2E%2C+Lopyrev%2C+K%2E%2C+and+Liang%2C+P%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
58. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with h… [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Gray%2C+A%2E%2C+Schulman%2C+J%2E%2C+Hilton%2C+J%2E%2C+Kelton%2C+F%2E%2C+Miller%2C+L%2E%2C+Simens%2C+M%2E%2C+Askell%2C+A%2E%2C+Welinder)
59. Ramasesh, V. V., Lewkowycz, A., and Dyer, E. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?](https://openreview.net/forum?) id=GhVS8_yPeEa. [↗](https://openreview.net/forum?)
60. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume… [find](https://scholar.google.com/scholar?q=Paperno%2C+D%2E%2C+Kruszewski%2C+G%2E%2C+Lazaridou%2C+A%2E%2C+Pham%2C+N%2E+Q%2E%2C+Bernardi%2C+R%2E%2C+Pezzelle%2C+S%2E%2C+Baroni%2C+M%2E%2C+Boleda%2C+G%2E%2C+and+Ferna%CC%81ndez%2C+R%2E+The+LAMBADA+dataset%3A+Word+prediction+requiring+a+broad+discourse+context) [our card](https://md.research.elamur.ai/free/en/papers/paperno-2016-lambada-dataset-word-prediction-requiring/)
61. Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X.,… [find](https://scholar.google.com/scholar?q=Sanh%2C+V%2E%2C+Webson%2C+A%2E%2C+Raffel%2C+C%2E%2C+Bach%2C+S%2E%2C+Sutawika%2C+L%2E%2C+Alyafeai%2C+Z%2E%2C+Chaffin%2C+A%2E%2C+Stiegler%2C+A%2E%2C+Raja%2C+A%2E%2C+Dey%2C+M%2E%2C+Bari%2C+M%2E+S%2E%2C+Xu%2C+C%2E%2C+Thakker%2C+U%2E%2C+Sharma%2C+S%2E+S%2E%2C+Szczechla%2C+E%2E%2C+Kim%2C+T%2E%2C+Chhablani) [our card](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
62. Peng, N., Ghazvininejad, M., May, J., and Knight, K. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pp. 43–49, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-1505. URL https://aclanthology.org/W18-1505. [↗](https://aclanthology.org/W18-1505)
63. Sap, M., Card, D., Gabriel, S., Choi, Y., and Smith, N. A. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1668–1678, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653… [find](https://scholar.google.com/scholar?q=Sap%2C+M%2E%2C+Card%2C+D%2E%2C+Gabriel%2C+S%2E%2C+Choi%2C+Y%2E%2C+and+Smith%2C+N%2E+A%2E+The+risk+of+racial+bias+in+hate+speech+detection%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C) [our card](https://md.research.elamur.ai/free/en/papers/sap-2019-risk-racial-bias-hate-speech/)
64. Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/abs/1910.00177. [↗](https://arxiv.org/abs/1910.00177)
65. Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback, 2022. URL https://arxiv.org/abs/ 2204.14146. [↗](https://arxiv.org/abs/)
66. Vu, T., Barua, A., Lester, B., Cer, D., Iyyer, M., and Constant, N. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9279–9300, Abu Dhabi, United Arab Emirates, December 2022. Asso… [find](https://scholar.google.com/scholar?q=Vu%2C+T%2E%2C+Barua%2C+A%2E%2C+Lester%2C+B%2E%2C+Cer%2C+D%2E%2C+Iyyer%2C+M%2E%2C+and+Constant%2C+N%2E+Overcoming+catastrophic+forgetting+in+zero%2Dshot+cross%2Dlingual+generation%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods)
67. Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback at scale, 2023. [find](https://scholar.google.com/scholar?q=Scheurer%2C+J%2E%2C+Campos%2C+J%2E+A%2E%2C+Korbak%2C+T%2E%2C+Chan%2C+J%2E+S%2E%2C+Chen%2C+A%2E%2C+Cho%2C+K%2E%2C+and+Perez%2C+E%2E+Training+language+models+with+language+feedback+at+scale%2C+2023%2E)
68. Schmidhuber, J. Reinforcement learning upside down: Don’t predict rewards – just map them to actions, 2019. URL https://arxiv.org/abs/1912.02875. — [arXiv:1912.02875](https://arxiv.org/abs/1912.02875) [↗](https://arxiv.org/abs/1912.02875)
69. Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. Offline rl for natural language generation with implicit language q learning, 2022. URL https://arxiv.org/abs/2206.11871. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) [↗](https://arxiv.org/abs/2206.11871)
70. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631– 1642, Seattle, Washington, USA, Oct… [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Perelygin%2C+A%2E%2C+Wu%2C+J%2E%2C+Chuang%2C+J%2E%2C+Manning%2C+C%2E+D%2E%2C+Ng%2C+A%2E%2C+and+Potts%2C+C%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Proceedings+of+the+2013+Conferenc)
71. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353– 355, Brussels, Belgium, Nove… [find](https://scholar.google.com/scholar?q=Wang%2C+A%2E%2C+Singh%2C+A%2E%2C+Michael%2C+J%2E%2C+Hill%2C+F%2E%2C+Levy%2C+O%2E%2C+and+Bowman%2C+S%2E+GLUE%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+In+Proceedings+of+the+2018+EMNLP+Workshop+Bl) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
72. Wang, B., Ping, W., Xiao, C., Xu, P., Patwary, M., Shoeybi, M., Li, B., Anandkumar, A., and Catanzaro, B. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processi… [find](https://scholar.google.com/scholar?q=Wang%2C+B%2E%2C+Ping%2C+W%2E%2C+Xiao%2C+C%2E%2C+Xu%2C+P%2E%2C+Patwary%2C+M%2E%2C+Shoeybi%2C+M%2E%2C+Li%2C+B%2E%2C+Anandkumar%2C+A%2E%2C+and+Catanzaro%2C+B%2E+Exploring+the+limits+of+domain%2Dadaptive+training+for+detoxifying+large%2Dscale+language+models%2E+)
73. Solaiman, I. and Dennison, C. Process for adapting language models to society (PALMS) with values-targeted datasets. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL [https://openreview.net/](https://openreview.net/) forum?id=k-ghaB9VZBw. [↗](https://openreview.net/)
74. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. [find](https://scholar.google.com/scholar?q=Warstadt%2C+A%2E%2C+Singh%2C+A%2E%2C+and+Bowman%2C+S%2E+R%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2C+2018%2E)
75. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447–2469, Punta Cana, Dominican Republic, No… [find](https://scholar.google.com/scholar?q=Welbl%2C+J%2E%2C+Glaese%2C+A%2E%2C+Uesato%2C+J%2E%2C+Dathathri%2C+S%2E%2C+Mellor%2C+J%2E%2C+Hendricks%2C+L%2E+A%2E%2C+Anderson%2C+K%2E%2C+Kohli%2C+P%2E%2C+Coppin%2C+B%2E%2C+and+Huang%2C+P%2E%2DS%2E+Challenges+in+detoxifying+language+models%2E+In+Findings+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/)
76. Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. J. Distilling task-specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136, 2019. — [arXiv:1903.12136](https://arxiv.org/abs/1903.12136) [find](https://scholar.google.com/scholar?q=Tang%2C+R%2E%2C+Lu%2C+Y%2E%2C+Liu%2C+L%2E%2C+Mou%2C+L%2E%2C+Vechtomova%2C+O%2E%2C+and+Lin%2C+J%2E+J%2E+Distilling+task%2Dspecific+knowledge+from+bert+into+simple+neural+networks%2E+ArXiv%2C+abs%2F1903%2E12136%2C+2019%2E+%E2%80%94+%5BarXiv%3A1903%2E12136%5D%28https%3A%2F%2Fa)
77. Tay, Y., Wei, J., Chung, H. W., Tran, V. Q., So, D. R., Shakeri, S., Garcia, X., Zheng, H. S., Rao, J., Chowdhery, A., Zhou, D., Metzler, D., Petrov, S., Houlsby, N., Le, Q. V., and Dehghani, M. Transcending scaling laws with 0.1 URL https://arxiv.org/abs/2210.11399. Welleck, S., Kulikov, I., Roller… [↗](https://arxiv.org/abs/2210.11399)
78. Tunstall, L., von Werra, L., and Wolf, T. Natural Language Processing with Transformers: Building Language Applications with Hugging Face. O’Reilly Media, Incorporated, 2022. ISBN 1098103246. URL [https://books.google.ch/books?id=7hhyzgEACAAJ](https://books.google.ch/books?id=7hhyzgEACAAJ). van Rossum, G., Warsaw, B., and Coghlan, N. Style guide… [↗](https://books.google.ch/books?id=7hhyzgEACAAJ)
79. Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. Will we run out of data? an analysis of the limits of scaling datasets in machine learning, 2022. URL https://arxiv.org/abs/2211.04325. [↗](https://arxiv.org/abs/2211.04325)
80. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112… [find](https://scholar.google.com/scholar?q=Williams%2C+A%2E%2C+Nangia%2C+N%2E%2C+and+Bowman%2C+S%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Asso)
81. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… [find](https://scholar.google.com/scholar?q=Wolf%2C+T%2E%2C+Debut%2C+L%2E%2C+Sanh%2C+V%2E%2C+Chaumond%2C+J%2E%2C+Delangue%2C+C%2E%2C+Moi%2C+A%2E%2C+Cistac%2C+P%2E%2C+Rault%2C+T%2E%2C+Louf%2C+R%2E%2C+Funtowicz%2C+M%2E%2C+Davison%2C+J%2E%2C+Shleifer%2C+S%2E%2C+von+Platen%2C+P%2E%2C+Ma%2C+C%2E%2C+Jernite%2C+Y%2E%2C+Plu%2C+J%2E%2C+Xu%2C+C%2E%2C+Le+) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
82. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2390–2397, O… [find](https://scholar.google.com/scholar?q=Xu%2C+A%2E%2C+Pathak%2C+E%2E%2C+Wallace%2C+E%2E%2C+Gururangan%2C+S%2E%2C+Sap%2C+M%2E%2C+and+Klein%2C+D%2E+Detoxifying+language+models+risks+marginalizing+minority+voices%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Cha)
83. Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Recipes for safety in open-domain chatbots, 2020. URL https://arxiv.org/abs/2010.07079. [↗](https://arxiv.org/abs/2010.07079)
84. Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097–1100, 2018. [find](https://scholar.google.com/scholar?q=Zhu%2C+Y%2E%2C+Lu%2C+S%2E%2C+Zheng%2C+L%2E%2C+Guo%2C+J%2E%2C+Zhang%2C+W%2E%2C+Wang%2C+J%2E%2C+and+Yu%2C+Y%2E+Texygen%3A+A+benchmarking+platform+for+text+generation+models%2E+In+The+41st+International+ACM+SIGIR+Conference+on+Research+%26+Developme) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2018-texygen-benchmarking-platform-text-generation/)
85. Ziegler, D., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., de Haas, D., Shlegeris, B., and Thomas, N. Adversarial training for high-stakes reliability. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Info… [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E%2C+Nix%2C+S%2E%2C+Chan%2C+L%2E%2C+Bauman%2C+T%2E%2C+Schmidt%2DNielsen%2C+P%2E%2C+Lin%2C+T%2E%2C+Scherlis%2C+A%2E%2C+Nabeshima%2C+N%2E%2C+Weinstein%2DRaun%2C+B%2E%2C+de+Haas%2C+D%2E%2C+Shlegeris%2C+B%2E%2C+and+Thomas%2C+N%2E+Adversarial+training+for+high%2Dstak)
86. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. [find](https://scholar.google.com/scholar?q=Ziegler%2C+D%2E+M%2E%2C+Stiennon%2C+N%2E%2C+Wu%2C+J%2E%2C+Brown%2C+T%2E+B%2E%2C+Radford%2C+A%2E%2C+Amodei%2C+D%2E%2C+Christiano%2C+P%2E%2C+and+Irving%2C+G%2E+Fine%2Dtuning+language+models+from+human+preferences%2E+arXiv+preprint+arXiv%3A1909%2E08593%2C+2019%2E)