---
title: "OctoBench — paper by Ding et al. (2026) — summary and ideas"
description: "Neither shows whether an agent respects rules that come from many sources and must be kept over… 46 references, cited by 4 papers here."
authors: ["Deming Ding", "Shichun Liu", "Enhui Yang", "Jiahang Lin", "Ziying Chen", "Shihan Dou", "Honglin Guo", "Weiyu Cheng", "Pengyu Zhao", "Chengjun Xiao", "Qunhong Zeng", "Qi Zhang", "Xuanjing Huang", "Qidi Xu", "Tao Gui"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.269/"
doi: "10.18653/v1/2026.acl-long.269"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ding-2026-octobench-benchmarking-scaffold-aware-instruction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ding-2026-octobench-benchmarking-scaffold-aware-instruction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/) / OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding

# OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding

**Authors:** Deming Ding, [Shichun Liu](https://md.research.elamur.ai/free/en/authors/shichun-liu/), Enhui Yang, Jiahang Lin, Ziying Chen, [Shihan Dou](https://md.research.elamur.ai/free/en/authors/shihan-dou/), Honglin Guo, Weiyu Cheng, Pengyu Zhao, [Chengjun Xiao](https://md.research.elamur.ai/free/en/authors/chengjun-xiao/), Qunhong Zeng, [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/), Qidi Xu and [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.269/) · [PDF](https://aclanthology.org/2026.acl-long.269.pdf) · [DOI 10.18653/v1/2026.acl-long.269](https://doi.org/10.18653/v1/2026.acl-long.269) · [OpenAlex](https://openalex.org/W7166853245)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [4](https://md.research.elamur.ai/free/en/papers/ding-2026-octobench-benchmarking-scaffold-aware-instruction/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Coding scaffolds such as Claude Code, Kilo and Droid turn LLMs into repository-level agents. Existing evaluations are either single-turn instruction-following tests or outcome-based agent benchmarks that check whether tests pass. Neither shows whether an agent respects rules that come from many sources and must be kept over long interactions. These sources include system prompts, repository policy files, skill docs, tool schemas, memory and system reminders.

OctoBench addresses this gap. It packages executable Docker environments with curated task specifications and pairs each instance with a binary, objectively decidable checklist. The checklists were built by:
- running a GPT-5.1 reference agent and recording its trajectories,
- generating and merging checks with LLM help,
- validating them through joint human–LLM review and human audits.

An observation harness logs each agent's full trajectory, and an LLM judge scores it against the checklist. This separates solving the task from obeying the rules. A companion set, OctoBench-Conflict, probes how models prioritize clashing instructions.

Across eight models, per-check compliance often fails to become full instance success. Skill constraints are the hardest category, and compliance usually drops as interactions grow longer. The benchmark supports training and evaluating coding agents that are more aware of their scaffold.

## Key points
- Scale: 34 environments, 217 tasks under three scaffolds (Claude Code, Kilo, Droid), and 7,098 checklist items (32.7 per instance on average, median 34). The code is released at github.com/MiniMax-AI/mini-vela.
- Checklist construction used 16 independent GPT-5.1 rollouts per task. Checks were then deduplicated, reviewed jointly by humans and an LLM, and given a 20% manual spot-check.
- There is a large gap between ISR (instance-level success) and CSR (per-check success). Skill constraints are a persistent bottleneck compared with memory constraints, and cross-scaffold robustness is weak for many models.
- ISR generally falls as the number of interaction turns grows. Claude-Opus-4.5 is the exception, staying robust on long interactions. Claude-Opus-4.5 and MiniMax-M2.1 consistently ranked top two.
- Judge reliability:
  - Model rankings shift by at most one place across three judges (GPT-5.1, Claude-Sonnet-4.5, Gemini-3-Pro), with no self-preference bias.
  - Agreement between the two human experts is 91.0% (κ = 0.811).
  - Average human–LLM agreement is 81.5% (κ = 0.63).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 167 works · 20,987 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

## Cite

Cite as: Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, Qunhong Zeng, Qi Zhang, Xuanjing Huang, Qidi Xu and Tao Gui. 2026. OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ding2026octobench,
  title = {{OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding}},
  author = {Deming Ding, Shichun Liu, Enhui Yang, Jiahang Lin, Ziying Chen, Shihan Dou, Honglin Guo, Weiyu Cheng, Pengyu Zhao, Chengjun Xiao, Qunhong Zeng, Qi Zhang, Xuanjing Huang, Qidi Xu and Tao Gui},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.269},
  url = {https://doi.org/10.18653/v1/2026.acl-long.269}
}
```

- [← previous NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment](https://md.research.elamur.ai/free/en/papers/wu-2026-novbench-evaluating-large-language-models/)
- [next → OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/zheng-2026-offside-benchmarking-unlearning-misinformation-multimodal/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/4/)

## References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2025a. Claude code best practices. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025a%2E+Claude+code+best+practices%2E)
2. Anthropic. 2025b. Introducing claude opus 4.5. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025b%2E+Introducing+claude+opus+4%2E5%2E)
3. Anthropic. 2025c. Introducing claude sonnet 4.5. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025c%2E+Introducing+claude+sonnet+4%2E5%2E)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732.
5. Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, and Timofey Bryksin. 2024. Long code arena: A set of benchmarks for long-context code models. Preprint, arXiv:2406.11612. — [arXiv:2406.11612](https://arxiv.org/abs/2406.11612) [find](https://scholar.google.com/scholar?q=Egor+Bogomolov%2C+Aleksandra+Eliseeva%2C+Timur+Galimzyanov%2C+Evgeniy+Glukhov%2C+Anton+Shapkin%2C+Maria+Tigina%2C+Yaroslav+Golubev%2C+Alexander+Kovrigin%2C+Arie+van+Deursen%2C+Maliheh+Izadi%2C+and+Timofey+Bryksin%2E+2024%2E+)
6. Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, JinKe, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, Noah Wang, Boyang Wang, Xianjie Wu, Bing Wang, Tongliang Li, Liqun Yang, Sufeng Duan, Zhaoxiang Zhang, and Zhoujun Li. 2024. Mceval: Massively multilingual code evaluation. In The Thir… [find](https://scholar.google.com/scholar?q=Linzheng+Chai%2C+Shukai+Liu%2C+Jian+Yang%2C+Yuwei+Yin%2C+JinKe%2C+Jiaheng+Liu%2C+Tao+Sun%2C+Ge+Zhang%2C+Changyu+Ren%2C+Hongcheng+Guo%2C+Noah+Wang%2C+Boyang+Wang%2C+Xianjie+Wu%2C+Bing+Wang%2C+Tongliang+Li%2C+Liqun+Yang%2C+Sufeng+Duan)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Thirty-Seventh Conference on Neural… [find](https://scholar.google.com/scholar?q=Yangruibo+Ding%2C+Zijian+Wang%2C+Wasi+Uddin+Ahmad%2C+Hantian+Ding%2C+Ming+Tan%2C+Nihal+Jain%2C+Murali+Krishna+Ramanathan%2C+Ramesh+Nallapati%2C+Parminder+Bhatia%2C+Dan+Roth%2C+and+Bing+Xiang%2E+2023%2E+Crosscodeeval%3A+A+diver)
9. Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, Tao Gui, Chao Xin, Wei Chengzhi, Lin Yan, Qi Zhang, and Xuanjing Huang. 2025. Evalearn: Quantifying the learning capability and efficiency of llms via sequential problem so… [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Ming+Zhang%2C+Chenhao+Huang%2C+Jiayi+Chen%2C+Feng+Chen%2C+Shichun+Liu%2C+Yan+Liu%2C+Chenxiao+Liu%2C+Cheng+Zhong%2C+Zongzhang+Zhang%2C+Tao+Gui%2C+Chao+Xin%2C+Wei+Chengzhi%2C+Lin+Yan%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+)
10. Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, Huaibing Xie, Jianglu Hu, Shaolei Wang, Weichao Wang, Yanling Xiao, Yiting Liu, Zenan Xu, Zhen Guo, Pluto Zhou, and 8 others. 2026. Cl-bench: A benchmark for context learnin… — [arXiv:2602.03587](https://arxiv.org/abs/2602.03587) [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Ming+Zhang%2C+Zhangyue+Yin%2C+Chenhao+Huang%2C+Yujiong+Shen%2C+Junzhe+Wang%2C+Jiayi+Chen%2C+Yuchen+Ni%2C+Junjie+Ye%2C+Cheng+Zhang%2C+Huaibing+Xie%2C+Jianglu+Hu%2C+Shaolei+Wang%2C+Weichao+Wang%2C+Yanling+Xiao%2C+Yitin)
11. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2025. Length-controlled alpacaeval: A simple way to debias automatic evaluators. Preprint, arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+2025%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+Preprint%2C+arXiv%3A2404%2E04475%2E)
12. Factory.ai. 2025. Droid: The \#1 software development agent on terminal-bench. [find](https://scholar.google.com/scholar?q=Factory%2Eai%2E+2025%2E+Droid%3A+The+%5C%231+software+development+agent+on+terminal%2Dbench%2E)
13. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. [find](https://scholar.google.com/scholar?q=Jiawei+Gu%2C+Xuhui+Jiang%2C+Zhichao+Shi%2C+Hexiang+Tan%2C+Xuehao+Zhai%2C+Chengjin+Xu%2C+Wei+Li%2C+Yinghan+Shen%2C+Shengjie+Ma%2C+Honghao+Liu%2C+Saizhuo+Wang%2C+Kun+Zhang%2C+Yuanzhuo+Wang%2C+Wen+Gao%2C+Lionel+Ni%2C+and+Jian+Guo%2E+20)
14. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. [Followbench: A multi-level fine-grained constraints following benchmark for large language models](https://md.research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/). In Proceedings of the 62nd Annual Meeting of the Association for Computa…
15. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) Preprint, arXiv:2310.06770.
16. Kilo. 2025. Kilo - move at kilo speed. [find](https://scholar.google.com/scholar?q=Kilo%2E+2025%2E+Kilo+%2D+move+at+kilo+speed%2E)
17. Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024a. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. Preprint, arXiv:2404.00599. — [arXiv:2404.00599](https://arxiv.org/abs/2404.00599) [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Ge+Li%2C+Xuanming+Zhang%2C+Yihong+Dong%2C+and+Zhi+Jin%2E+2024a%2E+Evocodebench%3A+An+evolving+code+generation+benchmark+aligned+with+real%2Dworld+code+repositories%2E+Preprint%2C+arXiv%3A2404%2E00599%2E+%E2%80%94+%5BarXiv%3A2404)
18. Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. 2024b. Deveval: A manually-annotated code generation benchmark al… [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Ge+Li%2C+Yunfei+Zhao%2C+Yongmin+Li%2C+Huanyu+Liu%2C+Hao+Zhu%2C+Lecheng+Wang%2C+Kaibo+Liu%2C+Zheng+Fang%2C+Lanshen+Wang%2C+Jiazheng+Ding%2C+Xuanming+Zhang%2C+Yuqi+Zhu%2C+Yihong+Dong%2C+Zhi+Jin%2C+Binhua+Li%2C+Fei+Huang%2C+Yon)
19. Tianyang Liu, Canwen Xu, and Julian McAuley. 2023a. Repobench: Benchmarking repository-level code auto-completion systems. Preprint, arXiv:2306.03091. [find](https://scholar.google.com/scholar?q=Tianyang+Liu%2C+Canwen+Xu%2C+and+Julian+McAuley%2E+2023a%2E+Repobench%3A+Benchmarking+repository%2Dlevel+code+auto%2Dcompletion+systems%2E+Preprint%2C+arXiv%3A2306%2E03091%2E)
20. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2023b. [Agentbench: Evaluating llms as agents](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/). In The Twelfth…
21. Renze Lou, Kai Zhang, and Wenpeng Yin. 2024. [Large language model instruction following: A survey of progresses and challenges](https://md.research.elamur.ai/free/en/papers/lou-2024-large-language-model-instruction-following/). Preprint, arXiv:2303.10475.
22. Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, and 66 othe… — [arXiv:2601.11868](https://arxiv.org/abs/2601.11868) [find](https://scholar.google.com/scholar?q=Mike+A%2E+Merrill%2C+Alexander+G%2E+Shaw%2C+Nicholas+Carlini%2C+Boxuan+Li%2C+Harsh+Raj%2C+Ivan+Bercovich%2C+Lin+Shi%2C+Jeong+Yeon+Shin%2C+Thomas+Walshe%2C+E%2E+Kelly+Buchanan%2C+Junhong+Shen%2C+Guanghao+Ye%2C+Haowei+Lin%2C+Jason+Pou)
23. MiniMax. 2025a. Minimax m2 & agent: Ingenious in simplicity. [find](https://scholar.google.com/scholar?q=MiniMax%2E+2025a%2E+Minimax+m2+%26+agent%3A+Ingenious+in+simplicity%2E)
24. MiniMax. 2025b. Minimax m2.1: Significantly enhanced multi-language programming, built for real-world complex tasks - minimax news. [find](https://scholar.google.com/scholar?q=MiniMax%2E+2025b%2E+Minimax+m2%2E1%3A+Significantly+enhanced+multi%2Dlanguage+programming%2C+built+for+real%2Dworld+complex+tasks+%2D+minimax+news%2E)
25. MiniMax, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, and 108 others. 2025. [Minimax-m1: Scaling test-time compute efficiently with l…](https://md.research.elamur.ai/free/en/papers/minimax-2025-minimax-m1-scaling-test/)
26. Moonshot. 2025. Introducing kimi k2 thinking. [find](https://scholar.google.com/scholar?q=Moonshot%2E+2025%2E+Introducing+kimi+k2+thinking%2E)
27. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Gpt%2D5%2E1%3A+A+smarter%2C+more+conversational+chatgpt%2E)
28. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. 2025. [Generalizing verifiable instruction following](https://md.research.elamur.ai/free/en/papers/pyatkin-2025-generalizing-verifiable-instruction-following/). Preprint, arXiv:2507.02833.
29. Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentif: Benchmarking instruction following of large language models in agentic scenarios. Preprint, arXiv:2505.16944. — [arXiv:2505.16944](https://arxiv.org/abs/2505.16944) [find](https://scholar.google.com/scholar?q=Yunjia+Qi%2C+Hao+Peng%2C+Xiaozhi+Wang%2C+Amy+Xin%2C+Youfeng+Liu%2C+Bin+Xu%2C+Lei+Hou%2C+and+Juanzi+Li%2E+2025%2E+Agentif%3A+Benchmarking+instruction+following+of+large+language+models+in+agentic+scenarios%2E+Preprint%2C+arXi)
30. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. [Infobench: Evaluating instruction following ability in large language models](https://md.research.elamur.ai/free/en/papers/qin-2024-infobench-evaluating-instruction-following-ability/). Preprint, arXiv:2401.03601.
31. Bytedance Seed. 2026. Seed1.8 model card: Towards generalized real-world agency. Preprint, arXiv:2603.20633. — [arXiv:2603.20633](https://arxiv.org/abs/2603.20633) [find](https://scholar.google.com/scholar?q=Bytedance+Seed%2E+2026%2E+Seed1%2E8+model+card%3A+Towards+generalized+real%2Dworld+agency%2E+Preprint%2C+arXiv%3A2603%2E20633%2E+%E2%80%94+%5BarXiv%3A2603%2E20633%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2603%2E20633%29)
32. Chihao Shen, Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, and Yizheng Chen. 2025. Secrepobench: Benchmarking code agents for secure code completion in real-world repositories. Preprint, arXiv:2504.21205. — [arXiv:2504.21205](https://arxiv.org/abs/2504.21205) [find](https://scholar.google.com/scholar?q=Chihao+Shen%2C+Connor+Dilgren%2C+Purva+Chiniya%2C+Luke+Griffith%2C+Yu+Ding%2C+and+Yizheng+Chen%2E+2025%2E+Secrepobench%3A+Benchmarking+code+agents+for+secure+code+completion+in+real%2Dworld+repositories%2E+Preprint%2C+arXi)
33. Manav Singhal, Tushar Aggarwal, Abhijeet Awasthi, Nagarajan Natarajan, and Aditya Kanade. 2024. No-funeval: Funny how code lms falter on requirements beyond functional correctness. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Manav+Singhal%2C+Tushar+Aggarwal%2C+Abhijeet+Awasthi%2C+Nagarajan+Natarajan%2C+and+Aditya+Kanade%2E+2024%2E+No%2Dfuneval%3A+Funny+how+code+lms+falter+on+requirements+beyond+functional+correctness%2E+In+First+Conference)
34. Sundar Pichai, Demis Hassabis, and Koray Kavukcuoglu. 2025. Gemini 3: Introducing the latest gemini ai model from google. [find](https://scholar.google.com/scholar?q=Sundar+Pichai%2C+Demis+Hassabis%2C+and+Koray+Kavukcuoglu%2E+2025%2E+Gemini+3%3A+Introducing+the+latest+gemini+ai+model+from+google%2E)
35. Xiangru Tang, Yuliang Liu, Zefan Cai, Yanjun Shao, Junjie Lu, Yichi Zhang, Zexuan Deng, Helan Hu, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Liang Chen, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yin Fang, and 5 others. 2024. [Ml-bench: Evaluating large language models and…](https://md.research.elamur.ai/free/en/papers/tang-2023-ml-bench-evaluating-large-language/)
36. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… — [Gemini: A Family of Highly Capable Multimodal Models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/)
37. GLM-4 5 Team, Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, and 152 others. 2025b. [Glm-4.5: Agentic, reasoning, and coding (arc)…](https://md.research.elamur.ai/free/en/papers/team-2025-4-glm-4-5-agentic/)
38. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, and 150 others. 2025c. [Kimi k2: Open agentic intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/). Preprint,…
39. Ming Zhang, Yujiong Shen, Jingyi Deng, Yuhui Wang, Huayu Sha, Kexin Tan, Qiyuan Peng, Yue Zhang, Junzhe Wang, Shichun Liu, Yueyuan Huang, Jingqi Tong, Changhao Jiang, Yilong Wu, Zhihao Zhang, Mingqi Wu, Mingxu Chai, Zhiheng Xi, Shihan Dou, and 3 others. 2026. [Llmeval-fair: A large-scale longitudinal…](https://md.research.elamur.ai/free/en/papers/zhang-2026-llmeval-fair-large-scale-longitudinal/)
40. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025a. Llmeval-med: A real-world clinical benchmark for medical llms with physician validation. Pr… [find](https://scholar.google.com/scholar?q=Ming+Zhang%2C+Yujiong+Shen%2C+Zelin+Li%2C+Huayu+Sha%2C+Binze+Hu%2C+Yuhui+Wang%2C+Chenhao+Huang%2C+Shichun+Liu%2C+Jingqi+Tong%2C+Changhao+Jiang%2C+Mingxu+Chai%2C+Zhiheng+Xi%2C+Shihan+Dou%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang)
41. Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Llmeval: A preliminary study on how to evaluate large language models. Preprint, arXiv:2312.07398. — [arXiv:2312.07398](https://arxiv.org/abs/2312.07398) [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Ming+Zhang%2C+Haipeng+Yuan%2C+Shichun+Liu%2C+Yongyao+Shi%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2023%2E+Llmeval%3A+A+preliminary+study+on+how+to+evaluate+large+language+models%2E+Preprint%2C+arXiv%3A2312%2E)
42. Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, and Meng Jiang. 2025b. Iheval: Evaluating language models on following the instruction hierarchy. Preprint, arXiv:2502.08745. [find](https://scholar.google.com/scholar?q=Zhihan+Zhang%2C+Shiyang+Li%2C+Zixuan+Zhang%2C+Xin+Liu%2C+Haoming+Jiang%2C+Xianfeng+Tang%2C+Yifan+Gao%2C+Zheng+Li%2C+Haodong+Wang%2C+Zhaoxuan+Tan%2C+Yichuan+Li%2C+Qingyu+Yin%2C+Bing+Yin%2C+and+Meng+Jiang%2E+2025b%2E+Iheval%3A+Evaluat)
43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Preprint, arXiv:2306.05685.
44. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). Preprint, arXiv:2311.07911.
45. Kaiwen Yan, Hongcheng Guo, Xuanqing Shi, Shaosheng Cao, Donglin Di, and Zhoujun Li. 2025. Codeif: Benchmarking the instruction-following capabilities of large language models for code generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: In… [find](https://scholar.google.com/scholar?q=Kaiwen+Yan%2C+Hongcheng+Guo%2C+Xuanqing+Shi%2C+Shaosheng+Cao%2C+Donglin+Di%2C+and+Zhoujun+Li%2E+2025%2E+Codeif%3A+Benchmarking+the+instruction%2Dfollowing+capabilities+of+large+language+models+for+code+generation%2E+In+P)
46. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, page… [find](https://scholar.google.com/scholar?q=Hao+Yu%2C+Bo+Shen%2C+Dezhi+Ran%2C+Jiaxin+Zhang%2C+Qi+Zhang%2C+Yuchi+Ma%2C+Guangtai+Liang%2C+Ying+Li%2C+Qianxiang+Wang%2C+and+Tao+Xie%2E+2024%2E+Codereval%3A+A+benchmark+of+pragmatic+code+generation+with+generative+pre%2Dtraine)

*Updated 31 October 2026*