--- title: "CodeHacker — paper by Shi et al. (2026) — summary and ideas" description: "The agent only proposes inputs. 5 references, cited by 7 papers here. Among them: 0." authors: ["Jingwei Shi", "Xinxiang Yin", "Jing Huang", "Shengyu Tao", "Jinman Zhao"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2026.acl-long.108/" doi: "10.18653/v1/2026.acl-long.108" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shi-2026-codehacker-automated-test-case-generation/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2026-codehacker-automated-test-case-generation/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions # CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions **Authors:** Jingwei Shi, Xinxiang Yin, [Jing Huang](https://md.research.elamur.ai/free/en/authors/jing-huang/), Shengyu Tao and [Jinman Zhao](https://md.research.elamur.ai/free/en/authors/jinman-zhao/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.108/) · [PDF](https://aclanthology.org/2026.acl-long.108.pdf) · [DOI 10.18653/v1/2026.acl-long.108](https://doi.org/10.18653/v1/2026.acl-long.108) · [OpenAlex](https://openalex.org/W7166822288) **Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/shi-2026-codehacker-automated-test-case-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Code benchmarks for LLMs often have test suites that miss subtle corner cases, so flawed programs are wrongly marked correct. CodeHacker addresses this with an LLM agent modeled on the "hack" practice of competitive programming. For a specific submission, it searches for an input that breaks it, using stress testing, anti-hash attacks and logic-targeted cases guided by code analysis. The agent only proposes inputs. Expected outputs come from a verified standard solution, which avoids hallucinated answers. Before attacking contestant code, a Calibration Phase has the agent probe and repair its own Validator and Checker. On CodeContests+ problems that need special judges, the added hack cases raise TNR sharply while TPR stays stable. The generated cases also work well as training data for RL, improving results on LiveCodeBench. The authors further release CodeHackerBench to measure how well models reason adversarially. The work matters because it makes judging stricter and gives a harder benchmark for testing models' reasoning. ## Key points - A hack counts as successful only if three conditions hold: the input passes the Validator, the standard solution handles it correctly, and the target submission fails with WA, RE, TLE or MLE. - In the ablation study, the full agent reached a 51.40% hack success rate with GPT-5-mini. Removing the Refinement Loop caused the largest drop, to 46.60%. Without the Code Analyst it fell to 49.86%, and without Stress Test to 50.12%. - The Anti-Hash Generator was used for only about 0.4% of submissions (those relying on rolling hashes), but it broke all of them. - On 1000 Special Judge problems, TNR rose from 82.18% (CodeContests+ baseline) to 96.05%, and TPR rose from 95.34% to 96.38%. - Adding the refined validator alone slightly lowered TNR to 82.08%. This shows that invalid inputs had previously been rejecting some wrong solutions for the wrong reason. All evaluation runs offline in a local sandbox. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 96 works · 142,007 citations - [Meituan](https://md.research.elamur.ai/free/en/companies/meituan/) — Company, Beijing, China · 17 works · 3,769 citations - [Shanghai University of Finance and Economics](https://md.research.elamur.ai/free/en/universities/shanghai-university-of-finance-and-economics/) — University, Shanghai, China · 13 works · 825 citations - [Northwestern Polytechnical University](https://md.research.elamur.ai/free/en/universities/northwestern-polytechnical-university/) — University, Xi'an, China · 13 works · 67 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works ## Cite Cite as: Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao and Jinman Zhao. 2026. CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{shi2026codehacker, title = {{CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions}}, author = {Jingwei Shi, Xinxiang Yin, Jing Huang, Shengyu Tao and Jinman Zhao}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2026}, doi = {10.18653/v1/2026.acl-long.108}, url = {https://doi.org/10.18653/v1/2026.acl-long.108} } ``` - [← previous CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/su-2026-ce-gppo-coordinating-entropy-gradient/) - [next → Confidence Estimation for LLMs in Multi-turn Interactions](https://md.research.elamur.ai/free/en/papers/zhang-2026-confidence-estimation-llms-multi-turn/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) ## References 5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732. 2. Yuhan Cao, Zian Chen, Kun Quan, Ziliang Zhang, Yu Wang, Xiaoning Dong, Yeqi Feng, Guanzhong He, Jingcheng Huang, Jianhao Li, Yixuan Tan, Jiafu Tang, Yilin Tang, Junlei Wu, Qianyu Xiao, Can Zheng, Shouchen Zhou, Yuxiang Zhu, Yiming Huang, and 2 others. 2025. Can llms generate reliable test case gener… — [arXiv:2506.06821](https://arxiv.org/abs/2506.06821) [find](https://scholar.google.com/scholar?q=Yuhan+Cao%2C+Zian+Chen%2C+Kun+Quan%2C+Ziliang+Zhang%2C+Yu+Wang%2C+Xiaoning+Dong%2C+Yeqi+Feng%2C+Guanzhong+He%2C+Jingcheng+Huang%2C+Jianhao+Li%2C+Yixuan+Tan%2C+Jiafu+Tang%2C+Yilin+Tang%2C+Junlei+Wu%2C+Qianyu+Xiao%2C+Can+Zheng%2C+Shou) 3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/) 5. Codeforces. 2023. Codeforces command lines. [https://codeforces.com/blog/entry/121114](https://codeforces.com/blog/entry/121114). \#\# Limitations [↗](https://codeforces.com/blog/entry/121114) *Updated 31 October 2026*