---
title: "Language Model Self-improvement by Reinforcement Learning"
description: "The authors propose SIRLC (Self-Improvement by Reinforcement Learning Contemplation), an… 28 references, cited by 20 papers here."
authors: ["Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang;", "Yang Yu"]
year: 2023
venue: "arXiv:2305.14483"
source: "https://arxiv.org/abs/2305.14483"
doi: "10.48550/arxiv.2305.14483"
arxiv: "2305.14483"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/pang-2023-language-model-self-improvement-reinforcement/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pang-2023-language-model-self-improvement-reinforcement/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/) / Language Model Self-improvement by Reinforcement Learning Contemplation

# Language Model Self-improvement by Reinforcement Learning Contemplation

**Authors:** Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and [Yang Yu](https://md.research.elamur.ai/free/en/authors/yang-yu/) · **Year:** 2023 · **Venue:** arXiv:2305.14483 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2305.14483) · [PDF](https://arxiv.org/pdf/2305.14483) · [DOI 10.48550/arxiv.2305.14483](https://doi.org/10.48550/arxiv.2305.14483) · [arXiv 2305.14483](https://arxiv.org/abs/2305.14483) · [OpenAlex](https://openalex.org/W4378469456)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/pang-2023-language-model-self-improvement-reinforcement/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Adapting large language models usually depends on labelled answers or human preference data, and collecting either is costly. The authors propose SIRLC (Self-Improvement by Reinforcement Learning Contemplation), an unsupervised fine-tuning scheme built on one observation: judging the quality of a text is easier for a model than producing it.

In SIRLC the same LLM plays two roles. It generates answers to unlabelled questions, and it also grades those answers through evaluation prompts. The grades serve as the reward for reinforcement learning, so no external labels or trained reward model are needed.

The authors first test the premise. On CommonGen, FLAN-T5 models judge outputs more accurately than they generate them, and their self-assessments correlate positively with BLEU, ROUGE and BERTScore. They then apply SIRLC to BigBench-Hard reasoning tasks, CNN summarization and IWSLT 2017 translation, comparing it against direct generation, self-consistency and LMSI. SIRLC generally performs best among label-free methods.

The work matters because it extends label-free self-improvement beyond chain-of-thought reasoning to open-ended generation tasks.

## Key points
- **Self-evaluation vs. generation:** on CommonGen, self-evaluation accuracy beats generation accuracy for every FLAN-T5 size tested. For the smaller models (FLAN-T5-Large/XL), the gap reaches about 15%.
- **Reported gains:**
  - Reasoning: answer accuracy rises by about 5.6% on BigBench-Hard tasks.
  - Translation (IWSLT 2017): BERTScore improves from 0.818 to 0.86.
  - Summarization: BERTScore improves from 0.886 to 0.899.
- **Model sizes:** SIRLC improves FLAN-T5-Small (80M), Base (250M) and Large (780M). The smallest, weakest model benefits most.
- **Generalization:** a model trained jointly on five BigBench tasks was tested on five unseen ones. It improved on three, by 0.8% on average, including +2.2% on Penguins in a Table.
- **Limitations noted by the authors:**
  - The method still needs a set of unlabelled questions.
  - It uses a separate, fixed evaluator model, which may not keep pace as the trained model improves.
  - Experiments are mostly limited to models of up to 780M parameters.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,512 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and Yang Yu. 2023. Language Model Self-improvement by Reinforcement Learning Contemplation. arXiv:2305.14483.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2023language,
  title = {{Language Model Self-improvement by Reinforcement Learning Contemplation}},
  author = {Jing-Cheng Pang; Pengyuan Wang; Kaiyuan Li; Xiong-Hui Chen; Jiacheng Xu; Zongzhang Zhang; and Yang Yu},
  year = {2023},
  doi = {10.48550/arxiv.2305.14483},
  eprint = {2305.14483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.14483}
}
```

- [← previous InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning](https://md.research.elamur.ai/free/en/papers/cahyawijaya-2023-instructalign-high-low-resource-language/)
- [next → Morphological Inflection: A Reality Check](https://md.research.elamur.ai/free/en/papers/kodner-2023-inflection-reality-check/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/12/)

## References

28 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. B. Brown, B. Mann, and N. R. et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). In NeurIPS, 2020.
2. CarperAI. Transformer reinforcement learning X. GitHub repository, GitHub, 2020. [find](https://scholar.google.com/scholar?q=CarperAI%2E+Transformer+reinforcement+learning+X%2E+GitHub+repository%2C+GitHub%2C+2020%2E)
3. A. Celikyilmaz, E. Clark, and J. Gao. [Evaluation of text generation: A survey](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2020-evaluation-text-generation-survey/). arXiv, 2006.14799, 2020.
4. M. Cettolo, M. Federico, L. Bentivogli, J. Niehues, S. Stüker, K. Sudoh, K. Yoshino, and C. Federmann. [Overview of the IWSLT 2017 evaluation campaign](https://md.research.elamur.ai/free/en/papers/cettolo-2017-overview-iwslt-2017-evaluation-campaign/). In IWSLT, 2017.
5. X. Chen, Y. Yuan, G. Zeng, and J. Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In CVPR, 2021. [find](https://scholar.google.com/scholar?q=X%2E+Chen%2C+Y%2E+Yuan%2C+G%2E+Zeng%2C+and+J%2E+Wang%2E+Semi%2Dsupervised+semantic+segmentation+with+cross+pseudo+supervision%2E+In+CVPR%2C+2021%2E)
6. W. S. Cho, P. Zhang, Y. Zhang, X. Li, M. Galley, C. Brockett, M. Wang, and J. Gao. Towards coherent and cohesive long-form text generation. In Workshop on Narrative Understanding, 2018. [find](https://scholar.google.com/scholar?q=W%2E+S%2E+Cho%2C+P%2E+Zhang%2C+Y%2E+Zhang%2C+X%2E+Li%2C+M%2E+Galley%2C+C%2E+Brockett%2C+M%2E+Wang%2C+and+J%2E+Gao%2E+Towards+coherent+and+cohesive+long%2Dform+text+generation%2E+In+Workshop+on+Narrative+Understanding%2C+2018%2E)
7. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. [PaLM: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv, 2204.02311, 2022.
8. H. W. Chung, L. Hou, and S. L. et al. Scaling instruction-finetuned language models. arXiv, 2210.11416, 2022. [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+and+S%2E+L%2E+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv%2C+2210%2E11416%2C+2022%2E)
9. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv, 2110.14168, 2021.
10. A. Gupta, D. Chugh, Anjum, and R. Katarya. Automated news summarization using transformers. arXiv, 2108.01064, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Gupta%2C+D%2E+Chugh%2C+Anjum%2C+and+R%2E+Katarya%2E+Automated+news+summarization+using+transformers%2E+arXiv%2C+2108%2E01064%2C+2021%2E)
11. J. He, J. Gu, J. Shen, and M. Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020. [find](https://scholar.google.com/scholar?q=J%2E+He%2C+J%2E+Gu%2C+J%2E+Shen%2C+and+M%2E+Ranzato%2E+Revisiting+self%2Dtraining+for+neural+sequence+generation%2E+In+ICLR%2C+2020%2E)
12. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). arXiv, 2210.11610, 2022a.
13. S. Huang, R. F. J. Dossa, A. Raffin, A. Kanervisto, and W. Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022b. [find](https://scholar.google.com/scholar?q=S%2E+Huang%2C+R%2E+F%2E+J%2E+Dossa%2C+A%2E+Raffin%2C+A%2E+Kanervisto%2C+and+W%2E+Wang%2E+The+37+implementation+details+of+proximal+policy+optimization%2E+In+ICLR+Blog+Track%2C+2022b%2E)
14. Y. Jang, J. Lee, and K.-E. Kim. Gpt-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In ICLR, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Jang%2C+J%2E+Lee%2C+and+K%2E%2DE%2E+Kim%2E+Gpt%2Dcritic%3A+Offline+reinforcement+learning+for+end%2Dto%2Dend+task%2Doriented+dialogue+systems%2E+In+ICLR%2C+2022%2E)
15. B. Y. Lin, M. Shen, W. Zhou, P. Zhou, C. Bhagavatula, Y. Choi, and X. Ren. Commongen: A constrained text generation challenge for generative commonsense reasoning. In AKBC, 2020. [find](https://scholar.google.com/scholar?q=B%2E+Y%2E+Lin%2C+M%2E+Shen%2C+W%2E+Zhou%2C+P%2E+Zhou%2C+C%2E+Bhagavatula%2C+Y%2E+Choi%2C+and+X%2E+Ren%2E+Commongen%3A+A+constrained+text+generation+challenge+for+generative+commonsense+reasoning%2E+In+AKBC%2C+2020%2E)
16. C.-Y. Lin. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In TSBO, 2004.
17. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). ACM Computing Surveys, 55(9):1–35, 2023.
18. R. Lou, K. Zhang, and W. Yin. Is prompt all you need? no. A comprehensive and broader view of instruction learning. arXiv, 2303.10475, 2023. [find](https://scholar.google.com/scholar?q=R%2E+Lou%2C+K%2E+Zhang%2C+and+W%2E+Yin%2E+Is+prompt+all+you+need%3F+no%2E+A+comprehensive+and+broader+view+of+instruction+learning%2E+arXiv%2C+2303%2E10475%2C+2023%2E)
19. R. Nallapati, B. Zhou, C. N. dos Santos, Ç. Gülçehre, and B. Xiang. [Abstractive text summarization using sequence-to-sequence rnns and beyond](https://md.research.elamur.ai/free/en/papers/nallapati-2016-abstractive-text-summarization-sequence-sequence/). In CoNLL, 2016.
20. K. Nguyen, H. Daumé III, and J. Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. arXiv, 1707.07402, 2017. [find](https://scholar.google.com/scholar?q=K%2E+Nguyen%2C+H%2E+Daum%C3%A9+III%2C+and+J%2E+Boyd%2DGraber%2E+Reinforcement+learning+for+bandit+neural+machine+translation+with+simulated+human+feedback%2E+arXiv%2C+1707%2E07402%2C+2017%2E)
21. OpenAI. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv, 2303.08774, 2023.
22. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). In NeurIPS, 2022.
23. K. Papineni, S. Roukos, T. Ward, and W. Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In ACL, 2002.
24. M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994. [find](https://scholar.google.com/scholar?q=M%2E+L%2E+Puterman%2E+Markov+Decision+Processes%3A+Discrete+Stochastic+Dynamic+Programming%2E+Wiley+Series+in+Probability+and+Statistics%2E+Wiley%2C+1994%2E)
25. S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, and S. Tellex. Planning with large language models via corrective re-prompting. arXiv, 2211.09935, 2022. [find](https://scholar.google.com/scholar?q=S%2E+S%2E+Raman%2C+V%2E+Cohen%2C+E%2E+Rosen%2C+I%2E+Idrees%2C+D%2E+Paulius%2C+and+S%2E+Tellex%2E+Planning+with+large+language+models+via+corrective+re%2Dprompting%2E+arXiv%2C+2211%2E09935%2C+2022%2E)
26. A. RoyChowdhury, P. Chakrabarty, A. Singh, S. Jin, H. Jiang, L. Cao, and E. Learned-Miller. Automatic adaptation of object detectors to new domains using self-training. In CVPR, 2019. [find](https://scholar.google.com/scholar?q=A%2E+RoyChowdhury%2C+P%2E+Chakrabarty%2C+A%2E+Singh%2C+S%2E+Jin%2C+H%2E+Jiang%2C+L%2E+Cao%2C+and+E%2E+Learned%2DMiller%2E+Automatic+adaptation+of+object+detectors+to+new+domains+using+self%2Dtraining%2E+In+CVPR%2C+2019%2E)
27. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv, 1707.06347, 2017. [find](https://scholar.google.com/scholar?q=J%2E+Schulman%2C+F%2E+Wolski%2C+P%2E+Dhariwal%2C+A%2E+Radford%2C+and+O%2E+Klimov%2E+Proximal+policy+optimization+algorithms%2E+arXiv%2C+1707%2E06347%2C+2017%2E)
28. J. Shen, Y. Yin, L. Li, L. Shang, X. Jiang, M. Zhang, and Q. Liu. Generate & rank: A multi-task framework for math word problems. In EMNLP, 2021. A. Srivastava, A. Rastogi, and A. R. et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). arXiv, 2206.0461…

*Updated 2 November 2026*