---
title: "Don't Judge Code by Its Cover — paper by Moon et al. (2026)"
description: "They build a benchmark in five languages (C++, Python, Java, JavaScript, Go). 53 references, cited by 3 papers here."
authors: ["Jiwon Moon", "Yerin Hwang", "Dongryeol Lee", "Taegwan Kang", "Yongil Kim", "Kyomin Jung"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.70/"
doi: "10.18653/v1/2026.findings-eacl.70"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/moon-2026-don-t-judge-code-its/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/moon-2026-don-t-judge-code-its/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation

# Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation

**Authors:** Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim and [Kyomin Jung](https://md.research.elamur.ai/free/en/authors/kyomin-jung/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.70/) · [PDF](https://aclanthology.org/2026.findings-eacl.70.pdf) · [DOI 10.18653/v1/2026.findings-eacl.70](https://doi.org/10.18653/v1/2026.findings-eacl.70) · [OpenAlex](https://openalex.org/W7140108392)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/moon-2026-don-t-judge-code-its/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs are increasingly used to judge whether generated code is correct without reference solutions or tests. This paper asks whether such judges stay consistent when code keeps the same behavior but changes on the surface. The authors identify six kinds of bias: cues that appeal to authority, comments in which the code claims to be correct, changed variable names, the inverse of the authority cue, comments that misrepresent the task, and complexity that is only apparent. They build a benchmark in five languages (C++, Python, Java, JavaScript, Go). Each language has 200 task descriptions paired with correct and incorrect solutions, and the biases are injected into those solutions. Every tested judge was affected. Some cues pushed verdicts toward "correct": expert-style comments, self-affirming comments, and longer variable names. Others pushed toward "incorrect": misleading task comments and novice-author remarks. A larger model did not guarantee more robustness. Asking judges to write test cases before deciding helped only slightly. This matters because reference-free LLM judging of code can be manipulated by trivial edits. Such edits can inflate scores for wrong code or penalize correct code, so more robust evaluation methods are needed.

## Key points
- The benchmark covers 5 programming languages with 200 tasks each. Correct and incorrect solutions are perturbed with 6 bias types, and the effects are classed as positive bias (favoring a "correct" verdict) or negative bias (favoring "incorrect").
- GPT-4o's accuracy fell by as much as 26.7 percentage points under biased inputs.
- Renaming variables to random strings of only two characters already produced consistent positive bias, which grew with name length. Single-character names had the opposite, negative effect.
- Adding more meaningless dummy functions steadily strengthened positive bias. With 8 dummy functions, accuracy on correct code rose from 79.67 to 89.33, while accuracy on incorrect code dropped from 63.50 to 46.65.
- Test-case-generation prompting reduced MAD only modestly and resisted negative biases somewhat better. Positive biases stayed, and accuracy on unbiased inputs fell from 71.6% to 66.75%.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Seoul National University](https://md.research.elamur.ai/free/en/universities/seoul-national-university/) — University, Seoul, South Korea · 41 works · 45,671 citations
- [LG AI Research](https://md.research.elamur.ai/free/en/companies/lg-ai-research/) — Company, Seoul, South Korea · 7 works · 374 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim and Kyomin Jung. 2026. Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{moon2026don,
  title = {{Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation}},
  author = {Jiwon Moon, Yerin Hwang, Dongryeol Lee, Taegwan Kang, Yongil Kim and Kyomin Jung},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.70},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.70}
}
```

- [← previous Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction](https://md.research.elamur.ai/free/en/papers/zhang-2026-don-t-act-blindly-robust/)
- [next → EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios](https://md.research.elamur.ai/free/en/papers/xu-2026-edubench-comprehensive-benchmarking-dataset-evaluating/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

53 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. Codesift: An llm-based reference-less framework for automatic code validation. In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404–410. IEEE. [find](https://scholar.google.com/scholar?q=Pooja+Aggarwal%2C+Oishik+Chatterjee%2C+Ting+Dai%2C+Prateeti+Mohapatra%2C+Brent+Paulovicks%2C+Brad+Blancett%2C+and+Arthur+De+Magalhaes%2E+2024%2E+Codesift%3A+An+llm%2Dbased+reference%2Dless+framework+for+automatic+code+vali)
3. Anthropic. 2024. Claude 3.5 sonnet. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3%2E5+sonnet%2E)
4. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, and 1 others. 2024. [Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks](https://md.research.elamur.ai/free/en/papers/bavaresco-2024-llms-instead-human-judges-large/). arXiv preprint arX…
5. Riccardo Cantini, Alessio Orsino, Massimo Ruggiero, and Domenico Talia. 2025. Benchmarking adversarial robustness to bias elicitation in large language models: Scalable automated assessment with llm-asa-judge. arXiv preprint arXiv:2504.07887. — [arXiv:2504.07887](https://arxiv.org/abs/2504.07887) [find](https://scholar.google.com/scholar?q=Riccardo+Cantini%2C+Alessio+Orsino%2C+Massimo+Ruggiero%2C+and+Domenico+Talia%2E+2025%2E+Benchmarking+adversarial+robustness+to+bias+elicitation+in+large+language+models%3A+Scalable+automated+assessment+with+llm%2Da)
6. Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. [Chateval: Towards better llm-based evaluators through multi-agent debate](https://md.research.elamur.ai/free/en/papers/chan-2023-chateval-better-llm-based/). arXiv preprint arXiv:2308.07201.
7. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2022%2E+Codet%3A+Code+generation+with+generated+tests%2E+arXiv+preprint+arXiv%3A2207%2E10397%2E)
8. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024a. Mllm-asa-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Dongping+Chen%2C+Ruoxi+Chen%2C+Shilin+Zhang%2C+Yaochen+Wang%2C+Yinuo+Liu%2C+Huichi+Zhou%2C+Qihui+Zhang%2C+Yao+Wan%2C+Pan+Zhou%2C+and+Lichao+Sun%2E+2024a%2E+Mllm%2Dasa%2Djudge%3A+Assessing+multimodal+llm%2Das%2Da%2Djudge+with+vision%2Dla)
9. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024b. [Humans or llms as the judge? a study on judgement biases](https://md.research.elamur.ai/free/en/papers/chen-2024-humans-llms-judge-study-judgement/). arXiv preprint arXiv:2402.10669.
10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
11. Enrique Dehaerne, Bappaditya Dey, Sandip Halder, Stefan De Gendt, and Wannes Meert. 2022. Code generation using machine learning: A systematic review. Ieee Access, 10:82434–82455. [find](https://scholar.google.com/scholar?q=Enrique+Dehaerne%2C+Bappaditya+Dey%2C+Sandip+Halder%2C+Stefan+De+Gendt%2C+and+Wannes+Meert%2E+2022%2E+Code+generation+using+machine+learning%3A+A+systematic+review%2E+Ieee+Access%2C+10%3A82434%E2%80%9382455%2E)
12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
13. Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software, 203:111741. [find](https://scholar.google.com/scholar?q=Mikhail+Evtikhiev%2C+Egor+Bogomolov%2C+Yaroslav+Sokolov%2C+and+Timofey+Bryksin%2E+2023%2E+Out+of+the+bleu%3A+how+should+we+assess+quality+of+the+code+generation+models%3F+Journal+of+Systems+and+Software%2C+203%3A111741)
14. Google. 2025. Gemini 2.0 flash. [find](https://scholar.google.com/scholar?q=Google%2E+2025%2E+Gemini+2%2E0+flash%2E)
15. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+and+1+others%2E+2024%2E+A+survey+on+llm%2Das%2Da%2Djudge%2E+arXiv+preprint+arXiv%3A2411%2E15)
16. Johannes C Hofmeister, Janet Siegmund, and Daniel V Holt. 2019. Shorter identifier names take longer to comprehend. Empirical Software Engineering, 24:417–443. [find](https://scholar.google.com/scholar?q=Johannes+C+Hofmeister%2C+Janet+Siegmund%2C+and+Daniel+V+Holt%2E+2019%2E+Shorter+identifier+names+take+longer+to+comprehend%2E+Empirical+Software+Engineering%2C+24%3A417%E2%80%93443%2E)
17. Yerin Hwang, Yongil Kim, Jahyun Koo, Taegwan Kang, Hyunkyung Bae, and Kyomin Jung. 2025. Llms can be easily confused by instructional distractions. arXiv preprint arXiv:2502.04362. [find](https://scholar.google.com/scholar?q=Yerin+Hwang%2C+Yongil+Kim%2C+Jahyun+Koo%2C+Taegwan+Kang%2C+Hyunkyung+Bae%2C+and+Kyomin+Jung%2E+2025%2E+Llms+can+be+easily+confused+by+instructional+distractions%2E+arXiv+preprint+arXiv%3A2502%2E04362%2E)
18. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. [Dense passage retrieval for open-domain question answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/). In EMNLP (1), pages 6769–6781.
19. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. arXiv preprint arXiv:2309.17012. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E+arXiv+preprint+arXiv%3A2309%2E17012%2E)
20. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32. [find](https://scholar.google.com/scholar?q=Sumith+Kulal%2C+Panupong+Pasupat%2C+Kartik+Chandra%2C+Mina+Lee%2C+Oded+Padon%2C+Alex+Aiken%2C+and+Percy+S+Liang%2E+2019%2E+Spoc%3A+Search%2Dbased+pseudocode+to+code%2E+Advances+in+Neural+Information+Processing+Systems%2C+32%2E)
21. Dongryeol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation. arXiv preprint arXiv:2410.20774. [find](https://scholar.google.com/scholar?q=Dongryeol+Lee%2C+Yerin+Hwang%2C+Yongil+Kim%2C+Joonsuk+Park%2C+and+Kyomin+Jung%2E+2024%2E+Are+llm%2Djudges+robust+to+expressions+of+uncertainty%3F+investigating+the+effect+of+epistemic+markers+on+llm%2Dbased+evaluation%2E)
22. Kefan Li and Yuan Yuan. 2024. Large language models as test case generators: Performance evaluation and enhancement. arXiv preprint arXiv:2404.13340. — [arXiv:2404.13340](https://arxiv.org/abs/2404.13340) [find](https://scholar.google.com/scholar?q=Kefan+Li+and+Yuan+Yuan%2E+2024%2E+Large+language+models+as+test+case+generators%3A+Performance+evaluation+and+enhancement%2E+arXiv+preprint+arXiv%3A2404%2E13340%2E+%E2%80%94+%5BarXiv%3A2404%2E13340%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E13)
23. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text summarization branches out, pages 74–81.
24. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023a. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). Advances in Neural Information Processing Systems, 36:21558– 21572.
25. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. [G-eval: Nlg evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). arXiv preprint arXiv:2303.16634.
26. Delano Oliveira, Reydne Santos, Fernanda Madeiral, Hidehiko Masuhara, and Fernando Castor. 2023. A systematic literature review on the impact of formatting elements on code legibility. Journal of Systems and Software, 203:111728. [find](https://scholar.google.com/scholar?q=Delano+Oliveira%2C+Reydne+Santos%2C+Fernanda+Madeiral%2C+Hidehiko+Masuhara%2C+and+Fernando+Castor%2E+2023%2E+A+systematic+literature+review+on+the+impact+of+formatting+elements+on+code+legibility%2E+Journal+of+Syst)
27. OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024a%2E+Gpt%2D4o+mini%3A+advancing+cost%2Defficient+intelligence%2E)
28. OpenAI. 2024b. Hello gpt-4o. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2024b%2E+Hello+gpt%2D4o%2E)
29. Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1–28. [find](https://scholar.google.com/scholar?q=Shuyin+Ouyang%2C+Jie+M+Zhang%2C+Mark+Harman%2C+and+Meng+Wang%2E+2025%2E+An+empirical+study+of+the+non%2Ddeterminism+of+chatgpt+in+code+generation%2E+ACM+Transactions+on+Software+Engineering+and+Methodology%2C+34%282%29%3A1)
30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
31. Pranjal Kumar. 2024. Large language models (llms): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10):260. [find](https://scholar.google.com/scholar?q=Pranjal+Kumar%2E+2024%2E+Large+language+models+%28llms%29%3A+survey%2C+technical+frameworks%2C+and+future+challenges%2E+Artificial+Intelligence+Review%2C+57%2810%29%3A260%2E)
32. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, and 1 others. 2021. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655. [find](https://scholar.google.com/scholar?q=Ruchir+Puri%2C+David+S+Kung%2C+Geert+Janssen%2C+Wei+Zhang%2C+Giacomo+Domeniconi%2C+Vladimir+Zolotov%2C+Julian+Dolby%2C+Jie+Chen%2C+Mihir+Choudhury%2C+Lindsey+Decker%2C+and+1+others%2E+2021%2E+Codenet%3A+A+large%2Dscale+ai+for+co)
33. Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006. What’s in a name? a study of identifiers. In 14th IEEE international conference on program comprehension (ICPC’06), pages 3–12. IEEE. [find](https://scholar.google.com/scholar?q=Dawn+Lawrie%2C+Christopher+Morrell%2C+Henry+Feild%2C+and+David+Binkley%2E+2006%2E+What%E2%80%99s+in+a+name%3F+a+study+of+identifiers%2E+In+14th+IEEE+international+conference+on+program+comprehension+%28ICPC%E2%80%9906%29%2C+pages+3%E2%80%9312%2E+)
34. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. [find](https://scholar.google.com/scholar?q=Shuo+Ren%2C+Daya+Guo%2C+Shuai+Lu%2C+Long+Zhou%2C+Shujie+Liu%2C+Duyu+Tang%2C+Neel+Sundaresan%2C+Ming+Zhou%2C+Ambrosio+Blanco%2C+and+Shuai+Ma%2E+2020%2E+Codebleu%3A+a+method+for+automatic+evaluation+of+code+synthesis%2E+arXiv+pr)
35. LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeong-won Jo, and 1 others. 2024. Exaone 3.5: Series of large language models for real-world use cases. arXiv preprint arXiv:2412.04862. — [arXiv:2412.04862](https://arxiv.org/abs/2412.04862) [find](https://scholar.google.com/scholar?q=LG+Research%2C+Soyoung+An%2C+Kyunghoon+Bae%2C+Eunbi+Choi%2C+Kibong+Choi%2C+Stanley+Jungkyu+Choi%2C+Seokhee+Hong%2C+Junwon+Hwang%2C+Hyojin+Jeon%2C+Gerrard+Jeong%2Dwon+Jo%2C+and+1+others%2E+2024%2E+Exaone+3%2E5%3A+Series+of+large+la)
36. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
37. Lin Shi, Chiyu Ma, Wenhua Liang, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic investigation of position bias in pairwise comparative assessments by llms. arXiv preprint arXiv:2406.07791. — [arXiv:2406.07791](https://arxiv.org/abs/2406.07791) [find](https://scholar.google.com/scholar?q=Lin+Shi%2C+Chiyu+Ma%2C+Wenhua+Liang%2C+Weicheng+Ma%2C+and+Soroush+Vosoughi%2E+2024%2E+Judging+the+judges%3A+A+systematic+investigation+of+position+bias+in+pairwise+comparative+assessments+by+llms%2E+arXiv+preprint+ar)
38. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. [find](https://scholar.google.com/scholar?q=Sijun+Tan%2C+Siyuan+Zhuang%2C+Kyle+Montgomery%2C+William+Y+Tang%2C+Alejandro+Cuadron%2C+Chenguang+Wang%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2024%2E+Judgebench%3A+A+benchmark+for+evaluating+llm%2Dbased+judges%2E+arXiv+prep)
39. Weixi Tong and Tianyi Zhang. 2024. Codejudge: Evaluating code generation with large language models. arXiv preprint arXiv:2410.02184. [find](https://scholar.google.com/scholar?q=Weixi+Tong+and+Tianyi+Zhang%2E+2024%2E+Codejudge%3A+Evaluating+code+generation+with+large+language+models%2E+arXiv+preprint+arXiv%3A2410%2E02184%2E)
40. Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does bleu score work for code migration? In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pages 165–176. IEEE. [find](https://scholar.google.com/scholar?q=Ngoc+Tran%2C+Hieu+Tran%2C+Son+Nguyen%2C+Hoan+Nguyen%2C+and+Tien+Nguyen%2E+2019%2E+Does+bleu+score+work+for+code+migration%3F+In+2019+IEEE%2FACM+27th+International+Conference+on+Program+Comprehension+%28ICPC%29%2C+pages+165)
41. Jianxun Wang and Yixiang Chen. 2023. A review on code generation with llms: Application and evaluation. In 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI), pages 284– 289. IEEE. [find](https://scholar.google.com/scholar?q=Jianxun+Wang+and+Yixiang+Chen%2E+2023%2E+A+review+on+code+generation+with+llms%3A+Application+and+evaluation%2E+In+2023+IEEE+International+Conference+on+Medical+Artificial+Intelligence+%28MedAI%29%2C+pages+284%E2%80%93+289)
42. Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. arXiv preprint arXiv:2502.06193. — [arXiv:2502.06193](https://arxiv.org/abs/2502.06193) [find](https://scholar.google.com/scholar?q=Ruiqi+Wang%2C+Jiyu+Guo%2C+Cuiyun+Gao%2C+Guodong+Fan%2C+Chun+Yong+Chong%2C+and+Xin+Xia%2E+2025%2E+Can+llms+replace+human+evaluators%3F+an+empirical+study+of+llm%2Das%2Da%2Djudge+in+software+engineering%2E+arXiv+preprint+arXiv)
43. Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, and Yuqing Kong. 2024. Benchmarking llms’ judgments with no gold standard. arXiv preprint arXiv:2411.07127. — [arXiv:2411.07127](https://arxiv.org/abs/2411.07127) [find](https://scholar.google.com/scholar?q=Shengwei+Xu%2C+Yuxuan+Lu%2C+Grant+Schoenebeck%2C+and+Yuqing+Kong%2E+2024%2E+Benchmarking+llms%E2%80%99+judgments+with+no+gold+standard%2E+arXiv+preprint+arXiv%3A2411%2E07127%2E+%E2%80%94+%5BarXiv%3A2411%2E07127%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2411%2E0)
44. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. [Justice or prejudice?](https://md.research.elamur.ai/free/en/papers/ye-2024-justice-prejudice-quantifying-biases-llm/) quantifying biases in llm-as-ajudge. arXiv preprint arXiv:2410.02736.
45. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+arXiv+preprint+arXiv%3A1904%2E09675%2E)
46. Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. 2024. Codejudge-eval: Can large language models be good judges in code understanding? arXiv preprint arXiv:2408.10718. [find](https://scholar.google.com/scholar?q=Yuwei+Zhao%2C+Ziyang+Luo%2C+Yuchen+Tian%2C+Hongzhan+Lin%2C+Weixiang+Yan%2C+Annan+Li%2C+and+Jing+Ma%2E+2024%2E+Codejudge%2Deval%3A+Can+large+language+models+be+good+judges+in+code+understanding%3F+arXiv+preprint+arXiv%3A2408%2E)
47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623.
48. Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint arXiv:2302.05527. [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Uri+Alon%2C+Sumit+Agarwal%2C+and+Graham+Neubig%2E+2023%2E+Codebertscore%3A+Evaluating+code+generation+with+pretrained+models+of+code%2E+arXiv+preprint+arXiv%3A2302%2E05527%2E)
49. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. [Judgelm: Fine-tuned large language models are scalable judges](https://md.research.elamur.ai/free/en/papers/zhu-2023-judgelm-fine-tuned-large-language/). arXiv preprint arXiv:2310.17631.
50. Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2232–2242. [find](https://scholar.google.com/scholar?q=Terry+Yue+Zhuo%2E+2024%2E+Ice%2Dscore%3A+Instructing+large+language+models+to+evaluate+code%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EACL+2024%2C+pages+2232%E2%80%932242%2E)
51. Yanlin Wang, Tianyue Jiang, Mingwei Liu, Jiachi Chen, and Zibin Zheng. 2024. Beyond functional correctness: Investigating coding style inconsistencies in large language models. arXiv preprint arXiv:2407.00456. — [arXiv:2407.00456](https://arxiv.org/abs/2407.00456) [find](https://scholar.google.com/scholar?q=Yanlin+Wang%2C+Tianyue+Jiang%2C+Mingwei+Liu%2C+Jiachi+Chen%2C+and+Zibin+Zheng%2E+2024%2E+Beyond+functional+correctness%3A+Investigating+coding+style+inconsistencies+in+large+language+models%2E+arXiv+preprint+arXiv%3A24)
52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in neural information processing systems, 35:24824– 24837.
53. Minghao Wu and Alham Fikri Aji. 2023. [Style over substance: Evaluation biases for large language models](https://md.research.elamur.ai/free/en/papers/wu-2025-style-over-substance-evaluation-biases/). arXiv preprint arXiv:2307.03025.

*Updated 1 November 2026*