---
title: "Coder Reviewer Reranking for Code Generation — Zhang 2022"
description: "Code language models often generate many candidate programs and choose the one they consider… 38 references, cited by 44 papers here."
authors: ["Tianyi Zhang", "Tao Yu", "Tatsunori B. Hashimoto", "Mike Lewis", "Wen-tau Yih", "Daniel Fried", "Sida I. Wang"]
year: 2022
venue: "arXiv:2211.16490"
source: "https://arxiv.org/abs/2211.16490"
doi: "10.48550/arxiv.2211.16490"
arxiv: "2211.16490"
licence: "CC BY-NC-SA 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-sa/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2022-coder-reviewer-reranking-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2022-coder-reviewer-reranking-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / Coder Reviewer Reranking for Code Generation

# Coder Reviewer Reranking for Code Generation

**Authors:** [Tianyi Zhang](https://md.research.elamur.ai/free/en/authors/tianyi-zhang/), [Tao Yu](https://md.research.elamur.ai/free/en/authors/tao-yu/), [Tatsunori B. Hashimoto](https://md.research.elamur.ai/free/en/authors/tatsunori-hashimoto/), [Mike Lewis](https://md.research.elamur.ai/free/en/authors/mike-lewis/), [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/), [Daniel Fried](https://md.research.elamur.ai/free/en/authors/daniel-fried/) and [Sida I. Wang](https://md.research.elamur.ai/free/en/authors/sida-wang/) · **Year:** 2022 · **Venue:** arXiv:2211.16490 · **Licence:** [CC BY-NC-SA 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-sa/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2211.16490) · [PDF](https://arxiv.org/pdf/2211.16490) · [DOI 10.48550/arxiv.2211.16490](https://doi.org/10.48550/arxiv.2211.16490) · [arXiv 2211.16490](https://arxiv.org/abs/2211.16490) · [OpenAlex](https://openalex.org/W4310514998)

**Citations:** 8 citations (OpenAlex, 2026-10-14) · [44](https://md.research.elamur.ai/free/en/papers/zhang-2022-coder-reviewer-reranking-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code language models often generate many candidate programs and choose the one they consider most probable. This selection step tends to favour broken outputs, such as trivially short functions or highly repetitive code, and it gets worse as more candidates are sampled. Zhang et al. borrow the idea of code review from software teams. The usual "Coder" scores a program given the instruction, p(y|x). A "Reviewer" adds the reverse score, p(x|y): how well the program explains the original instruction. The Reviewer needs no extra training. The same pretrained model is simply prompted with the program placed before the instruction. Candidates are ranked by the product of both scores, which the authors show is a form of the Maximum Mutual Information objective. They tested this on six datasets in three programming languages, using models from the Codex, InCoder and CodeGen families. The method consistently beats Coder-only ranking and is cheap to implement. Unlike most competing methods, it does not require running the code.

## Key points
- **Accuracy gains:** Coder-Reviewer gains up to 17 points of absolute accuracy over ranking by p(y|x) alone. Benchmarks include HumanEval, MBPP-Sanitized and NL2Bash.
- **Model count:** The paper is inconsistent about how many models it tested, giving eight in one place and seven in another.
- **Execution-based baselines:** With executability filtering added, the method often outperforms MBR-Exec (Shi et al., 2022). With 100 samples, it is competitive with CodeT. For example, normalized Coder-Reviewer reaches 66.9 vs. 65.8 on HumanEval with Codex002. CodeT still leads on MBPP-S (67.7 vs. 64.7).
- **Robustness:** Coder-only ranking loses accuracy as the sample count grows, while Coder-Reviewer remains stable. A simple rule-based filter that discards degenerate programs helps every ranking method, and helps unnormalized Coder the most.
- **Hyperparameters and code:** The Coder/Reviewer mixing weight is fixed at α = 0.5, and tuning it usually adds less than 1%. Code is released at github.com/facebookresearch/coder_reviewer_reranking.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 567 works · 335,189 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 633 works · 195,217 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 127 works · 13,823 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 184 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 194 works

## Cite

Cite as: Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried and Sida I. Wang. 2022. Coder Reviewer Reranking for Code Generation. arXiv:2211.16490.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhang2022coder,
  title = {{Coder Reviewer Reranking for Code Generation}},
  author = {Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried and Sida I. Wang},
  year = {2022},
  doi = {10.48550/arxiv.2211.16490},
  eprint = {2211.16490},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2211.16490}
}
```

- [← previous Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review](https://md.research.elamur.ai/free/en/papers/kuznetsov-2022-revise-resubmit-intertextual-model-text/)
- [next → Transfer Learning Approaches for Building Cross-Language Dense Retrieval Models](https://md.research.elamur.ai/free/en/papers/nair-2022-transfer-learning-approaches-building/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/)

## References

38 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allamanis, M., Barr, E. T., Bird, C., and Sutton, C. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015. [find](https://scholar.google.com/scholar?q=Allamanis%2C+M%2E%2C+Barr%2C+E%2E+T%2E%2C+Bird%2C+C%2E%2C+and+Sutton%2C+C%2E+Suggesting+accurate+method+and+class+names%2E+In+Proceedings+of+the+2015+10th+Joint+Meeting+on+Foundations+of+Software+Engineering%2C+2015%2E)
2. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
3. Bahl, L. R., Brown, P. F., de Souza, P. V., and Mercer, R. L. Maximum mutual information estimation of hidden markov model parameters for speech recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing, 1986. [find](https://scholar.google.com/scholar?q=Bahl%2C+L%2E+R%2E%2C+Brown%2C+P%2E+F%2E%2C+de+Souza%2C+P%2E+V%2E%2C+and+Mercer%2C+R%2E+L%2E+Maximum+mutual+information+estimation+of+hidden+markov+model+parameters+for+speech+recognition%2E+IEEE+International+Conference+on+Acoustics)
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33: 1877–1901, 2020.
5. Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W. Codet: Code generation with generated tests. ArXiv, abs/2207.10397, 2022. [find](https://scholar.google.com/scholar?q=Chen%2C+B%2E%2C+Zhang%2C+F%2E%2C+Nguyen%2C+A%2E%2C+Zan%2C+D%2E%2C+Lin%2C+Z%2E%2C+Lou%2C+J%2E%2DG%2E%2C+and+Chen%2C+W%2E+Codet%3A+Code+generation+with+generated+tests%2E+ArXiv%2C+abs%2F2207%2E10397%2C+2022%2E)
6. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., S… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
7. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A. B., Barnes, P., Tay, Y., Shazeer, N. M., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B. C., Pope, R., Bradbury,… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
8. Fried, D., Hu, R., Cirik, V., Rohrbach, A., Andreas, J., Morency, L.-P., Berg-Kirkpatrick, T., Saenko, K., Klein, D., and Darrell, T. Speaker-follower models for vision-and-language navigation. In NeurIPS, 2018. [find](https://scholar.google.com/scholar?q=Fried%2C+D%2E%2C+Hu%2C+R%2E%2C+Cirik%2C+V%2E%2C+Rohrbach%2C+A%2E%2C+Andreas%2C+J%2E%2C+Morency%2C+L%2E%2DP%2E%2C+Berg%2DKirkpatrick%2C+T%2E%2C+Saenko%2C+K%2E%2C+Klein%2C+D%2E%2C+and+Darrell%2C+T%2E+Speaker%2Dfollower+models+for+vision%2Dand%2Dlanguage+navigation%2E+In+Neu)
9. Fried, D., Aghajanyan, A., Lin, J., Wang, S. I., Wallace, E., Shi, F., Zhong, R., Yih, W.-t., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. ArXiv, abs/2204.05999, 2022. [find](https://scholar.google.com/scholar?q=Fried%2C+D%2E%2C+Aghajanyan%2C+A%2E%2C+Lin%2C+J%2E%2C+Wang%2C+S%2E+I%2E%2C+Wallace%2C+E%2E%2C+Shi%2C+F%2E%2C+Zhong%2C+R%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E+ArXiv%2C+abs%2F22)
10. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. [Measuring coding challenge competence with APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In NeurIPS Datasets and Benchmarks Track, 2021.
11. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In ICLR, 2020.
12. Inala, J. P., Wang, C., Yang, M., Codas, A., Encarnaci’on, M., Lahiri, S. K., Musuvathi, M., and Gao, J. [Fault-aware neural code rankers](https://md.research.elamur.ai/free/en/papers/inala-2022-fault-aware-neural-code-rankers/). ArXiv, abs/2206.03865, 2022.
13. Iyer, S., Konstas, I., Cheung, A., and Zettlemoyer, L. [Mapping language to code in programmatic context](https://md.research.elamur.ai/free/en/papers/iyer-2018-mapping-language-code-programmatic-context/). In EMNLP, 2018.
14. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., tau Yih, S. W., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022. [find](https://scholar.google.com/scholar?q=Lai%2C+Y%2E%2C+Li%2C+C%2E%2C+Wang%2C+Y%2E%2C+Zhang%2C+T%2E%2C+Zhong%2C+R%2E%2C+Zettlemoyer%2C+L%2E%2C+tau+Yih%2C+S%2E+W%2E%2C+Fried%2C+D%2E%2C+Wang%2C+S%2E%2C+and+Yu%2C+T%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+code+generation%2E+ArXiv%2C+abs)
15. Lewis, M. and Fan, A. Generative question answering: Learning to answer the whole question. In ICLR, 2019. [find](https://scholar.google.com/scholar?q=Lewis%2C+M%2E+and+Fan%2C+A%2E+Generative+question+answering%3A+Learning+to+answer+the+whole+question%2E+In+ICLR%2C+2019%2E)
16. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. [A diversity-promoting objective function for neural conversation models](https://md.research.elamur.ai/free/en/papers/li-2016-diversity-promoting-objective-function-neural/). In NAACL, 2016.
17. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). arXiv preprint arXiv:2203.07814, 2022.
18. Lin, X. V., Wang, C., Zettlemoyer, L., and Ernst, M. D. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In LREC, 2018. [find](https://scholar.google.com/scholar?q=Lin%2C+X%2E+V%2E%2C+Wang%2C+C%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Ernst%2C+M%2E+D%2E+NL2Bash%3A+A+corpus+and+semantic+parser+for+natural+language+interface+to+the+linux+operating+system%2E+In+LREC%2C+2018%2E)
19. Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., and Senior, A. [Latent predictor networks for code generation](https://md.research.elamur.ai/free/en/papers/ling-2016-latent-predictor-networks-code-generation/). In ACL, 2016.
20. Min, S., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. [Noisy channel language model prompting for few-shot text classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/). ArXiv, abs/2108.04106, 2021.
21. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. 2022. [find](https://scholar.google.com/scholar?q=Nijkamp%2C+E%2E%2C+Pang%2C+B%2E%2C+Hayashi%2C+H%2E%2C+Tu%2C+L%2E%2C+Wang%2C+H%2E%2C+Zhou%2C+Y%2E%2C+Savarese%2C+S%2E%2C+and+Xiong%2C+C%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+2022%2E)
22. Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., and Wang, S. I. [Natural language to code translation with execution](https://md.research.elamur.ai/free/en/papers/shi-2022-natural-language-code-translation-execution/). In EMNLP, 2022.
23. Stahlberg, F. and Byrne, B. [On nmt search errors and model errors: Cat got your tongue?](https://md.research.elamur.ai/free/en/papers/stahlberg-2019-nmt-search-errors-model-errors/) arXiv preprint arXiv:1908.10090, 2019.
24. Yasunaga, M. and Liang, P. Graph-based, self-supervised program repair from diagnostic feedback. In ICML, 2020. [find](https://scholar.google.com/scholar?q=Yasunaga%2C+M%2E+and+Liang%2C+P%2E+Graph%2Dbased%2C+self%2Dsupervised+program+repair+from+diagnostic+feedback%2E+In+ICML%2C+2020%2E)
25. Ye, S., Kim, D., Jang, J., Shin, J., and Seo, M. Guess the instruction! flipped learning makes language models stronger zero-shot learners. ArXiv, abs/2210.02969, 2022. — [arXiv:2210.02969](https://arxiv.org/abs/2210.02969) [find](https://scholar.google.com/scholar?q=Ye%2C+S%2E%2C+Kim%2C+D%2E%2C+Jang%2C+J%2E%2C+Shin%2C+J%2E%2C+and+Seo%2C+M%2E+Guess+the+instruction%21+flipped+learning+makes+language+models+stronger+zero%2Dshot+learners%2E+ArXiv%2C+abs%2F2210%2E02969%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E02969%5D%28https%3A%2F%2Far)
26. Yin, P. and Neubig, G. [A syntactic neural model for general-purpose code generation](https://md.research.elamur.ai/free/en/papers/yin-2017-syntactic-neural-model-general-purpose/). In ACL, 2017.
27. Yin, P. and Neubig, G. Reranking for neural semantic parsing. In ACL, 2019. [find](https://scholar.google.com/scholar?q=Yin%2C+P%2E+and+Neubig%2C+G%2E+Reranking+for+neural+semantic+parsing%2E+In+ACL%2C+2019%2E)
28. Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In EMNLP, 2018.
29. Zhong, R., Snell, C., Klein, D., and Eisner, J. Active programming by example with a natural language prior. ArXiv, 2022. [find](https://scholar.google.com/scholar?q=Zhong%2C+R%2E%2C+Snell%2C+C%2E%2C+Klein%2C+D%2E%2C+and+Eisner%2C+J%2E+Active+programming+by+example+with+a+natural+language+prior%2E+ArXiv%2C+2022%2E)
30. A. Understanding the Relation between Coder-Reviewer Reranking and Maximum Mutual Information. We include the derivation from Li et al. (2016) to show that Coder-Reviewer reranking is a special instantiation of the Maximum Mutual Information objective. We can show p(y, x) p(x)p(y) α = argmax y (1 −… [find](https://scholar.google.com/scholar?q=A%2E+Understanding+the+Relation+between+Coder%2DReviewer+Reranking+and+Maximum+Mutual+Information%2E+We+include+the+derivation+from+Li+et+al%2E+%282016%29+to+show+that+Coder%2DReviewer+reranking+is+a+special+instan)
31. Alternatively, Li et al. (2016) propose another way to instantiate this objective, p(y, x) p(x)p(y) α = argmax y log p(y\|x) − α log p(y). argmax y log We show in Appendix C.4 that this alternate formulation leads to worse performance, which is a similar conclusion to the original finding in Li et al… [find](https://scholar.google.com/scholar?q=Alternatively%2C+Li+et+al%2E+%282016%29+propose+another+way+to+instantiate+this+objective%2C+p%28y%2C+x%29+p%28x%29p%28y%29+%CE%B1+%3D+argmax+y+log+p%28y%5C%7Cx%29+%E2%88%92+%CE%B1+log+p%28y%29%2E+argmax+y+log+We+show+in+Appendix+C%2E4+that+this+alternate+form)
32. B. Additional Experimental details B.1. Detailed Dataset Descriptions HumanEval (Chen et al., 2021) contains 164 hand-written Python programming questions Chen et al. (2021). We use the prompts released by Chen et al. (2022), which removes the input-output cases present in the original prompts. This… [find](https://scholar.google.com/scholar?q=B%2E+Additional+Experimental+details+B%2E1%2E+Detailed+Dataset+Descriptions+HumanEval+%28Chen+et+al%2E%2C+2021%29+contains+164+hand%2Dwritten+Python+programming+questions+Chen+et+al%2E+%282021%29%2E+We+use+the+prompts+releas)
33. MBPP-Sanitized (Austin et al., 2021) contains 427 crowdsourced Python programming questions. [find](https://scholar.google.com/scholar?q=MBPP%2DSanitized+%28Austin+et+al%2E%2C+2021%29+contains+427+crowdsourced+Python+programming+questions%2E)
34. Chen et al. (2022) adapts each problem into having a function header and relocate the natural language instructions into function docstrings, similar to the format of HumanEval. We use this prompt format for our experiment. [find](https://scholar.google.com/scholar?q=Chen+et+al%2E+%282022%29+adapts+each+problem+into+having+a+function+header+and+relocate+the+natural+language+instructions+into+function+docstrings%2C+similar+to+the+format+of+HumanEval%2E+We+use+this+prompt+for)
35. MBPP (Austin et al., 2021) original version consists of 974 python questions, with 500 of them used for testing and the rest for few-shot prompting. The prompt taken from Shi et al. (2022) uses one input-output assertion as additional context to help make the language instruction more specific. Unli… [find](https://scholar.google.com/scholar?q=MBPP+%28Austin+et+al%2E%2C+2021%29+original+version+consists+of+974+python+questions%2C+with+500+of+them+used+for+testing+and+the+rest+for+few%2Dshot+prompting%2E+The+prompt+taken+from+Shi+et+al%2E+%282022%29+uses+one+in)
36. Spider (Yu et al., 2018) is a benchmark of natural language to SQL query generation. There are 7000 examples for training/demonstration and 1034 questions for testing. The prompt taken from Shi et al. (2022) prepends the database schema as the program context. The generated SQL commands are evaluate… [find](https://scholar.google.com/scholar?q=Spider+%28Yu+et+al%2E%2C+2018%29+is+a+benchmark+of+natural+language+to+SQL+query+generation%2E+There+are+7000+examples+for+training%2Fdemonstration+and+1034+questions+for+testing%2E+The+prompt+taken+from+Shi+et+al%2E)
37. NL2Bash (Lin et al., 2018) is a benchmark of translating natural language to bash commands. Because it is difficult to obtain executable enviroments for bash commands, this dataset evaluates character-level BLEU-4 score. 0-shot Codex 002 on Plotting 0-shot Codex 002 on HumanEval 0.65 0.675 0.650 0.6… [find](https://scholar.google.com/scholar?q=NL2Bash+%28Lin+et+al%2E%2C+2018%29+is+a+benchmark+of+translating+natural+language+to+bash+commands%2E+Because+it+is+difficult+to+obtain+executable+enviroments+for+bash+commands%2C+this+dataset+evaluates+character)
38. Improvement from tuning on the best Coder-Reviewer variant 7.00% 6.00% Codex-002 Codex-001 Codex-Cushman CodeGen-16B CodeGen-6B CodeGen-2B 5.00% 4.00% 3.00% 2.00% 1.00% 0.00% MBPP-S Plotting HumanEval Figure 8. Accuracy improvement from grid searching ensemble mixing ratio α. Setting α = 0.5 (i.e.,… [find](https://scholar.google.com/scholar?q=Improvement+from+tuning+on+the+best+Coder%2DReviewer+variant+7%2E00%25+6%2E00%25+Codex%2D002+Codex%2D001+Codex%2DCushman+CodeGen%2D16B+CodeGen%2D6B+CodeGen%2D2B+5%2E00%25+4%2E00%25+3%2E00%25+2%2E00%25+1%2E00%25+0%2E00%25+MBPP%2DS+Plotting+HumanEval)

*Updated 3 November 2026*