---
title: "SciReplicate-Bench — paper by Xiang et al. (2025)"
description: "The authors also propose Sci-Reproducer, in which a Paper Agent collects algorithmic details from… 89 references, cited by 22 papers here."
authors: ["Yanzheng Xiang", "Hanqi Yan", "Shuyin Ouyang", "Lin Gui", "Yulan He"]
year: 2025
venue: "arXiv:2504.00255"
source: "https://arxiv.org/abs/2504.00255"
doi: "10.48550/arxiv.2504.00255"
arxiv: "2504.00255"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xiang-2025-scireplicate-bench-benchmarking-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xiang-2025-scireplicate-bench-benchmarking-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/) / SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers

# SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers

**Authors:** Yanzheng Xiang, [Hanqi Yan](https://md.research.elamur.ai/free/en/authors/hanqi-yan/), Shuyin Ouyang, [Lin Gui](https://md.research.elamur.ai/free/en/authors/lin-gui/) and [Yulan He](https://md.research.elamur.ai/free/en/authors/yulan-he/) · **Year:** 2025 · **Venue:** arXiv:2504.00255 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2504.00255) · [PDF](https://arxiv.org/pdf/2504.00255) · [DOI 10.48550/arxiv.2504.00255](https://doi.org/10.48550/arxiv.2504.00255) · [arXiv 2504.00255](https://arxiv.org/abs/2504.00255) · [OpenAlex](https://openalex.org/W4417224607)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [22](https://md.research.elamur.ai/free/en/papers/xiang-2025-scireplicate-bench-benchmarking-llms/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Turning an algorithm from a scientific paper into working code requires two skills: reading the paper and its cited works closely, and navigating a real codebase with its internal dependencies and external APIs. Existing ML engineering benchmarks are built on Kaggle tasks or GitHub repositories and do not test this combination. The authors created SciReplicate-Bench, a set of 100 repository-level tasks drawn from 36 NLP papers published in 2024. They chose this recent window to limit data leakage. Each task asks a model to implement one function or method and includes annotated reference code, dependencies and test cases. The authors also propose Sci-Reproducer, in which a Paper Agent collects algorithmic details from the literature and a Code Agent searches the repository and writes the implementation. Evaluation uses execution accuracy, CodeBLEU and dependency/API recall, plus a new reasoning graph accuracy metric that compares graphs built from code comments and data flow. Even the strongest model reaches only 39% execution accuracy. Models understand the algorithms well but fail at implementation, often because papers omit or misstate details. The benchmark offers an objective, execution-based test of LLM-assisted research reproducibility.

## Key points
- SciReplicate-Bench comprises 100 tasks, sourced from 36 NLP papers that appeared in 2024. Inputs are the function signature, the LaTeX algorithm description, the paper with its references, and the full repository.
- Sci-Reproducer pairs a Paper Agent (searching the paper, its sections and the cited literature) with a Code Agent (searching the repository and compiling code). No tested LLM exceeds 40% execution accuracy, and the best reaches 39%.
- Reasoning models such as o3-mini make far fewer tool calls than non-reasoning models and need more compile-debug cycles. As a result, they perform no better than GPT-4o-mini, a behaviour the authors call "overthinking."
- Adding the Code Agent cuts syntax error rates from about 80–83% to 29.4% (Code Agent only) and 24.9% (full Sci-Reproducer).
- Supplying manually identified missing or mismatched details raises execution accuracy by 0.05 for GPT-4o-mini, 0.25 for Deepseek-V3 (reaching 0.47) and 0.04 for o3-mini-low. The missing details fall into four types: hyperparameters, numerical-stability tricks, implementation logic and coding strategy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,888 works

## Cite

Cite as: Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui and Yulan He. 2025. SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers. arXiv:2504.00255.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{xiang2025scireplicate,
  title = {{SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers}},
  author = {Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui and Yulan He},
  year = {2025},
  doi = {10.48550/arxiv.2504.00255},
  eprint = {2504.00255},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.00255}
}
```

- [← previous RWKV-7 "Goose" with Expressive Dynamic State Evolution](https://md.research.elamur.ai/free/en/papers/peng-2025-rwkv-7-goose-expressive-dynamic/)
- [next → Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations](https://md.research.elamur.ai/free/en/papers/cao-2025-specializing-large-language-models-simulate/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/5/)

## References

89 references, 39 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. GPT 4o mini. [https://platform.openai.com/docs/models/gpt-4o-mini](https://platform.openai.com/docs/models/gpt-4o-mini). 2024. [↗](https://platform.openai.com/docs/models/gpt-4o-mini)
2. Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, and M. Aßenmacher. Adaptive contrastive search: Uncertainty-guided decoding for open-ended text generation. In Conference on Empirical Methods in Natural Language Processing, 2024. [find](https://scholar.google.com/scholar?q=Esteban+Garces+Arias%2C+Julian+Rodemann%2C+Meimingwei+Li%2C+Christian+Heumann%2C+and+M%2E+A%C3%9Fenmacher%2E+Adaptive+contrastive+search%3A+Uncertainty%2Dguided+decoding+for+open%2Dended+text+generation%2E+In+Conference+on+Em)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). ArXiv, abs/2108.07732, 2021.
4. Markus J. Buehler. Accelerating scientific discovery with generative knowledge extraction, graph-based representation, and multimodal intelligent graph reasoning. Machine Learning: Science and Technology, 2024. URL [http://iopscience.iop.org/article/10.1088/](http://iopscience.iop.org/article/10.1088/) 2632-2153/ad7228. [↗](http://iopscience.iop.org/article/10.1088/)
5. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal A. Patwardhan, Lilian Weng, and Aleksander Mkadry. [Mle-bench: Evaluating machine learning agents on machine learning engineering](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/). ArXiv, abs/2410.07095, 2024.
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alet… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
7. Qizhou Chen, Taolin Zhang, Dongyang Li, Longtao Huang, Hui Xue, Chengyu Wang, and Xiaofeng He. Lifelong knowledge editing for llms with retrieval-augmented continuous prompt learning. In Conference on Empirical Methods in Natural Language Processing, 2024a. [find](https://scholar.google.com/scholar?q=Qizhou+Chen%2C+Taolin+Zhang%2C+Dongyang+Li%2C+Longtao+Huang%2C+Hui+Xue%2C+Chengyu+Wang%2C+and+Xiaofeng+He%2E+Lifelong+knowledge+editing+for+llms+with+retrieval%2Daugmented+continuous+prompt+learning%2E+In+Conference+on)
8. Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Yu Zhang. Routerdc: Query-based router by dual contrastive learning for assembling large language models. ArXiv, abs/2409.19886, 2024b. — [arXiv:2409.19886](https://arxiv.org/abs/2409.19886) [find](https://scholar.google.com/scholar?q=Shuhao+Chen%2C+Weisen+Jiang%2C+Baijiong+Lin%2C+James+T%2E+Kwok%2C+and+Yu+Zhang%2E+Routerdc%3A+Query%2Dbased+router+by+dual+contrastive+learning+for+assembling+large+language+models%2E+ArXiv%2C+abs%2F2409%2E19886%2C+2024b%2E+%E2%80%94+%5Ba)
9. Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. Learning to maximize mutual information for chain-of-thought distillation. In Annual Meeting of the Association for Computational Linguistics, 2024c. [find](https://scholar.google.com/scholar?q=Xin+Chen%2C+Hanxian+Huang%2C+Yanjun+Gao%2C+Yi+Wang%2C+Jishen+Zhao%2C+and+Ke+Ding%2E+Learning+to+maximize+mutual+information+for+chain%2Dof%2Dthought+distillation%2E+In+Annual+Meeting+of+the+Association+for+Computationa)
10. Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. [When is tree search useful for llm planning? it depends on the discriminator](https://md.research.elamur.ai/free/en/papers/chen-2024-when-is-tree-search-useful/). In Annual Meeting of the Association for Computational Linguistics, 2024d.
11. Yew Ken Chia, Guizhen Chen, Weiwen Xu, Anh Tuan Luu, Soujanya Poria, and Li Bing. Reasoning paths optimization: Learning to reason and explore from diverse paths. ArXiv, abs/2410.10858, 2024. [find](https://scholar.google.com/scholar?q=Yew+Ken+Chia%2C+Guizhen+Chen%2C+Weiwen+Xu%2C+Anh+Tuan+Luu%2C+Soujanya+Poria%2C+and+Li+Bing%2E+Reasoning+paths+optimization%3A+Learning+to+reason+and+explore+from+diverse+paths%2E+ArXiv%2C+abs%2F2410%2E10858%2C+2024%2E)
12. Neil Chowdhury, Franklin Wang, Sumedh Shenoy, Douwe Kiela, Sarah Schwettmann, and Tristan Thrush. Nearest neighbor normalization improves multimodal retrieval. In Conference on Empirical Methods in Natural Language Processing, 2024. [find](https://scholar.google.com/scholar?q=Neil+Chowdhury%2C+Franklin+Wang%2C+Sumedh+Shenoy%2C+Douwe+Kiela%2C+Sarah+Schwettmann%2C+and+Tristan+Thrush%2E+Nearest+neighbor+normalization+improves+multimodal+retrieval%2E+In+Conference+on+Empirical+Methods+in+Na)
13. Claude-Sonnet-3.7. [https://www.anthropic.com/claude/sonnet](https://www.anthropic.com/claude/sonnet). 2025. [↗](https://www.anthropic.com/claude/sonnet)
14. Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-action dil… — [The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Tasks](https://md.research.elamur.ai/free/en/papers/cuadron-2025-danger-overthinking-examining-reasoning-action/)
15. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Ha… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bing%2DLi+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dong%2DLi+Ji%2C+Erhang)
16. Zhenyun Deng, M. Schlichtkrull, and Andreas Vlachos. Document-level claim extraction and decontextualisation for fact-checking. In Annual Meeting of the Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Zhenyun+Deng%2C+M%2E+Schlichtkrull%2C+and+Andreas+Vlachos%2E+Document%2Dlevel+claim+extraction+and+decontextualisation+for+fact%2Dchecking%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2024)
17. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, Chen Xing, Cheng Jiayang, Zhaowei Wang, Yi… — [LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing](https://md.research.elamur.ai/free/en/papers/du-2024-llms-assist-nlp-researchers-critique/)
18. Gemini-2.0-Flash. [https://deepmind.google/technologies/gemini/flash/](https://deepmind.google/technologies/gemini/flash/). 2024. [↗](https://deepmind.google/technologies/gemini/flash/)
19. Alireza Ghafarollahi and Markus J. Buehler. [Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning](https://md.research.elamur.ai/free/en/papers/ghafarollahi-2024-sciagents-automating-scientific-discovery/). ArXiv, abs/2409.05556, 2024.
20. GPT-4o. [https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o). 2024. [↗](https://platform.openai.com/docs/models/gpt-4o)
21. Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. 2025. [find](https://scholar.google.com/scholar?q=Mourad+Gridach%2C+Jay+Nanavati%2C+Khaldoun+Zine+El+Abidine%2C+Lenon+Mendes%2C+and+Christina+Mack%2E+Agentic+ai+for+scientific+discovery%3A+A+survey+of+progress%2C+challenges%2C+and+future+directions%2E+2025%2E)
22. Tianyang Gu, Jingjin Wang, Zhihao Zhang, and HaoHong Li. Llms can realize combinatorial creativity: generating creative ideas via llms for scientific research. ArXiv, abs/2412.14141, 2024. — [arXiv:2412.14141](https://arxiv.org/abs/2412.14141) [find](https://scholar.google.com/scholar?q=Tianyang+Gu%2C+Jingjin+Wang%2C+Zhihao+Zhang%2C+and+HaoHong+Li%2E+Llms+can+realize+combinatorial+creativity%3A+generating+creative+ideas+via+llms+for+scientific+research%2E+ArXiv%2C+abs%2F2412%2E14141%2C+2024%2E+%E2%80%94+%5BarXiv%3A24)
23. Xuemei Gu and Mario Krenn. Interesting scientific idea generation using knowledge graphs and llms: Evaluations with 100 research group leaders, 2025. URL [https://arxiv.org/](https://arxiv.org/) abs/2405.17044. — [arXiv:2405.17044](https://arxiv.org/abs/2405.17044) [↗](https://arxiv.org/)
24. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). ArXiv, abs/2105.09938, 2021.
25. Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. [Mlagentbench: Evaluating language agents on machine learning experimentation](https://md.research.elamur.ai/free/en/papers/huang-2023-mlagentbench-evaluating-language-agents/). In International Conference on Machine Learning, 2023.
26. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). ArXiv, abs/2403.07974, 2024.
27. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) ArXiv, abs/2310.06770, 2023.
28. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. Exploring concept depth: How large language models acquire knowledge at different layers? ArXiv, abs/2404.07066, 2024. — [arXiv:2404.07066](https://arxiv.org/abs/2404.07066) [find](https://scholar.google.com/scholar?q=Mingyu+Jin%2C+Qinkai+Yu%2C+Jingyuan+Huang%2C+Qingcheng+Zeng%2C+Zhenting+Wang%2C+Wenyue+Hua%2C+Haiyan+Zhao%2C+Kai+Mei%2C+Yanda+Meng%2C+Kaize+Ding%2C+Fan+Yang%2C+Mengnan+Du%2C+and+Yongfeng+Zhang%2E+Exploring+concept+depth%3A+How+l)
29. Amir Hossein Kargaran, Franccois Yvon, and Hinrich Schutze. Masklid: Code-switching language identification through iterative masking. ArXiv, abs/2406.06263, 2024. [find](https://scholar.google.com/scholar?q=Amir+Hossein+Kargaran%2C+Franccois+Yvon%2C+and+Hinrich+Schutze%2E+Masklid%3A+Code%2Dswitching+language+identification+through+iterative+masking%2E+ArXiv%2C+abs%2F2406%2E06263%2C+2024%2E)
30. Yeachan Kim, Junho Kim, and SangKeun Lee. Towards robust and generalized parameter-efficient fine-tuning for noisy label learning. ArXiv, abs/2411.00873, 2024. [find](https://scholar.google.com/scholar?q=Yeachan+Kim%2C+Junho+Kim%2C+and+SangKeun+Lee%2E+Towards+robust+and+generalized+parameter%2Defficient+fine%2Dtuning+for+noisy+label+learning%2E+ArXiv%2C+abs%2F2411%2E00873%2C+2024%2E)
31. Sandeep Kumar, Tirthankar Ghosal, Vinayak Goyal, and Asif Ekbal. [Can large language models unlock novel scientific research ideas?](https://md.research.elamur.ai/free/en/papers/kumar-2025-can-large-language-models-unlock/) ArXiv, abs/2409.06185, 2024.
32. Wen Lai, Viktor Hangya, and Alexander Fraser. Style-specific neurons for steering llms in text style transfer. ArXiv, abs/2410.00593, 2024. [find](https://scholar.google.com/scholar?q=Wen+Lai%2C+Viktor+Hangya%2C+and+Alexander+Fraser%2E+Style%2Dspecific+neurons+for+steering+llms+in+text+style+transfer%2E+ArXiv%2C+abs%2F2410%2E00593%2C+2024%2E)
33. Jia Li, Ge Li, Yunfei Zhao, Yongming Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. Deveval: A manually-annotated code generation benchmark aligned with real-world code r… [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Ge+Li%2C+Yunfei+Zhao%2C+Yongming+Li%2C+Huanyu+Liu%2C+Hao+Zhu%2C+Lecheng+Wang%2C+Kaibo+Liu%2C+Zheng+Fang%2C+Lanshen+Wang%2C+Jiazheng+Ding%2C+Xuanming+Zhang%2C+Yuqi+Zhu%2C+Yihong+Dong%2C+Zhi+Jin%2C+Binhua+Li%2C+Fei+Huang%2C+an)
34. Hongfu Liu, Yuxi Xie, Ye Wang, and Michael Shieh. Advancing adversarial suffix transfer learning on aligned large language models. In Conference on Empirical Methods in Natural Language Processing, 2024a. [find](https://scholar.google.com/scholar?q=Hongfu+Liu%2C+Yuxi+Xie%2C+Ye+Wang%2C+and+Michael+Shieh%2E+Advancing+adversarial+suffix+transfer+learning+on+aligned+large+language+models%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+20)
35. Wanlong Liu, Li Zhou, DingYi Zeng, Yichen Xiao, Shaohuan Cheng, Chen Zhang, Grandee Lee, Malu Zhang, and Wenyu Chen. Beyond single-event extraction: Towards efficient document-level multi-event argument extraction. ArXiv, abs/2405.01884, 2024b. [find](https://scholar.google.com/scholar?q=Wanlong+Liu%2C+Li+Zhou%2C+DingYi+Zeng%2C+Yichen+Xiao%2C+Shaohuan+Cheng%2C+Chen+Zhang%2C+Grandee+Lee%2C+Malu+Zhang%2C+and+Wenyu+Chen%2E+Beyond+single%2Devent+extraction%3A+Towards+efficient+document%2Dlevel+multi%2Devent+argume)
36. Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schütze. [Transmi: A framework to create strong baselines from multilingual pretrained language models for transliterated data](https://md.research.elamur.ai/free/en/papers/liu-2025-transmi-framework-create-strong-baselines/). ArXiv, abs/2405.09913, 2024c.
37. Yujie Liu, Zonglin Yang, Tong Xie, Jinjie Ni, Ben Gao, Yuqiang Li, Shixiang Tang, Wanli Ouyang, Erik Cambria, and Dongzhan Zhou. [Researchbench: Benchmarking llms in scientific discovery via inspiration-based task decomposition](https://md.research.elamur.ai/free/en/papers/liu-2025-researchbench-benchmarking-llms-scientific/). ArXiv, abs/2503.21248, 2025.
38. Yuliang Liu, Xiangru Tang, Zefan Cai, Junjie Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zheng Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao… — [ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code](https://md.research.elamur.ai/free/en/papers/tang-2023-ml-bench-evaluating-large-language/)
39. Zejie Liu, Xiaoyu Hu, Deyu Zhou, Lin Li, Xu Zhang, and Yanzheng Xiang. Code generation from flowcharts with texts: A benchmark dataset and an approach. In Conference on Empirical Methods in Natural Language Processing, 2022. [find](https://scholar.google.com/scholar?q=Zejie+Liu%2C+Xiaoyu+Hu%2C+Deyu+Zhou%2C+Lin+Li%2C+Xu+Zhang%2C+and+Yanzheng+Xiang%2E+Code+generation+from+flowcharts+with+texts%3A+A+benchmark+dataset+and+an+approach%2E+In+Conference+on+Empirical+Methods+in+Natural+La)
40. Lingli Long, Yongjin Zhu, Jun Shao, Zheng Kong, Jian Li, Yanzheng Xiang, and Xu Zhang. Nl2sql generation with noise labels based on multi-task learning. Journal of Physics: Conference Series, 2294, 2022. [find](https://scholar.google.com/scholar?q=Lingli+Long%2C+Yongjin+Zhu%2C+Jun+Shao%2C+Zheng+Kong%2C+Jian+Li%2C+Yanzheng+Xiang%2C+and+Xu+Zhang%2E+Nl2sql+generation+with+noise+labels+based+on+multi%2Dtask+learning%2E+Journal+of+Physics%3A+Conference+Series%2C+2294%2C+20)
41. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. [The ai scientist: Towards fully automated open-ended scientific discovery](https://md.research.elamur.ai/free/en/papers/lu-2024-ai-scientist-fully-automated/). ArXiv, abs/2408.06292, 2024.
42. Michele Mastromattei and Fabio Massimo Zanzotto. Less is ken: a universal and simple non-parametric pruning algorithm for large language models. ArXiv, abs/2402.03142, 2024. [find](https://scholar.google.com/scholar?q=Michele+Mastromattei+and+Fabio+Massimo+Zanzotto%2E+Less+is+ken%3A+a+universal+and+simple+non%2Dparametric+pruning+algorithm+for+large+language+models%2E+ArXiv%2C+abs%2F2402%2E03142%2C+2024%2E)
43. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Niko lay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. M… — [MLGym: A New Framework and Benchmark for Advancing AI Research Agents](https://md.research.elamur.ai/free/en/papers/nathani-2025-mlgym-new-framework-benchmark/)
44. Charles O’Neill, Tirthankar Ghosal, Roberta Raileanu, Mike Walmsley, Thang Bui, Kevin Schawinski, and Ioana Ciuca. [Sparks of science: Hypothesis generation using structured paper data](https://md.research.elamur.ai/free/en/papers/o-neill-2025-sparks-science-hypothesis-generation-using/). ArXiv, abs/2504.12976, 2025.
45. Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael E. Moseley, Curtis P. Langlotz, Akshay S. Chaudhari, and Jean-Benoit Delbrouck. Green: Generative radiology report evaluation and error notation. ArXiv, abs/2405.03595, 2024. [find](https://scholar.google.com/scholar?q=Sophie+Ostmeier%2C+Justin+Xu%2C+Zhihong+Chen%2C+Maya+Varma%2C+Louis+Blankemeier%2C+Christian+Bluethgen%2C+Arne+Edward+Michalson%2C+Michael+E%2E+Moseley%2C+Curtis+P%2E+Langlotz%2C+Akshay+S%2E+Chaudhari%2C+and+Jean%2DBenoit+Delbro)
46. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Annual Meeting of the Association for Computational Linguistics, 2002.
47. Tianyu Peng and Jiajun Zhang. [Enhancing knowledge distillation of large language models through efficient multi-modal distribution alignment](https://md.research.elamur.ai/free/en/papers/peng-2025-enhancing-knowledge-distillation-large-language/). In International Conference on Computational Linguistics, 2024.
48. Duy-Tung Pham, Thien Trang Nguyen Vu, Tung Nguyen, Linh Ngo Van, Duc Anh Nguyen, and Thien Huu Nguyen. Neuromax: Enhancing neural topic modeling via maximizing mutual information and group topic regularization. ArXiv, abs/2409.19749, 2024. [find](https://scholar.google.com/scholar?q=Duy%2DTung+Pham%2C+Thien+Trang+Nguyen+Vu%2C+Tung+Nguyen%2C+Linh+Ngo+Van%2C+Duc+Anh+Nguyen%2C+and+Thien+Huu+Nguyen%2E+Neuromax%3A+Enhancing+neural+topic+modeling+via+maximizing+mutual+information+and+group+topic+regul)
49. Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. Evaluating the text-to-sql capabilities of large language models. ArXiv, abs/2204.00498, 2022. [find](https://scholar.google.com/scholar?q=Nitarshan+Rajkumar%2C+Raymond+Li%2C+and+Dzmitry+Bahdanau%2E+Evaluating+the+text%2Dto%2Dsql+capabilities+of+large+language+models%2E+ArXiv%2C+abs%2F2204%2E00498%2C+2022%2E)
50. Tal Reiss, George Kour, Naama Zwerdling, Ateret Anaby-Tavor, and Yedid Hoshen. From zero to hero: Cold-start anomaly detection. ArXiv, abs/2405.20341, 2024. [find](https://scholar.google.com/scholar?q=Tal+Reiss%2C+George+Kour%2C+Naama+Zwerdling%2C+Ateret+Anaby%2DTavor%2C+and+Yedid+Hoshen%2E+From+zero+to+hero%3A+Cold%2Dstart+anomaly+detection%2E+ArXiv%2C+abs%2F2405%2E20341%2C+2024%2E)
51. Pu Ren, N. Benjamin Erichson, Shashank Subramanian, Omer San, Zarija Lukic, and Michael W. Mahoney. Superbench: A super-resolution benchmark dataset for scientific machine learning. ArXiv, abs/2306.14070, 2023. — [arXiv:2306.14070](https://arxiv.org/abs/2306.14070) [find](https://scholar.google.com/scholar?q=Pu+Ren%2C+N%2E+Benjamin+Erichson%2C+Shashank+Subramanian%2C+Omer+San%2C+Zarija+Lukic%2C+and+Michael+W%2E+Mahoney%2E+Superbench%3A+A+super%2Dresolution+benchmark+dataset+for+scientific+machine+learning%2E+ArXiv%2C+abs%2F2306%2E14)
52. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, M. Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. ArXiv, abs/2009.10297, 2020. [find](https://scholar.google.com/scholar?q=Shuo+Ren%2C+Daya+Guo%2C+Shuai+Lu%2C+Long+Zhou%2C+Shujie+Liu%2C+Duyu+Tang%2C+M%2E+Zhou%2C+Ambrosio+Blanco%2C+and+Shuai+Ma%2E+Codebleu%3A+a+method+for+automatic+evaluation+of+code+synthesis%2E+ArXiv%2C+abs%2F2009%2E10297%2C+2020%2E)
53. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. [Raptor: Recursive abstractive processing for tree-organized retrieval](https://md.research.elamur.ai/free/en/papers/sarthi-2024-raptor-recursive-abstractive-processing-tree/). ArXiv, abs/2401.18059, 2024.
54. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). ArXiv, abs/2302.04761, 2023.
55. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. [Agent laboratory: Using llm agents as research assistants](https://md.research.elamur.ai/free/en/papers/schmidgall-2025-agent-laboratory-using-llm/). ArXiv, abs/2501.04227, 2025.
56. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. [Paper2code: Automating code generation from scientific papers in machine learning](https://md.research.elamur.ai/free/en/papers/seo-2025-paper2code-automating-code-generation/). ArXiv, abs/2504.17192, 2025. URL [https://api.semanticscholar.org/CorpusID:278033490](https://api.semanticscholar.org/CorpusID:278033490). [↗](https://api.semanticscholar.org/CorpusID:278033490)
57. Dan Shi, Renren Jin, Tianhao Shen, Weilong Dong, Xinwei Wu, and Deyi Xiong. Ircan: Mitigating knowledge conflicts in llm generation via identifying and reweighting context-aware neurons. ArXiv, abs/2406.18406, 2024. — [arXiv:2406.18406](https://arxiv.org/abs/2406.18406) [find](https://scholar.google.com/scholar?q=Dan+Shi%2C+Renren+Jin%2C+Tianhao+Shen%2C+Weilong+Dong%2C+Xinwei+Wu%2C+and+Deyi+Xiong%2E+Ircan%3A+Mitigating+knowledge+conflicts+in+llm+generation+via+identifying+and+reweighting+context%2Daware+neurons%2E+ArXiv%2C+abs%2F24)
58. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. [Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). ArXiv, abs/2409.04109, 2024.
59. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. [Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers](https://md.research.elamur.ai/free/en/papers/si-2024-llms-generate-novel-research/). In The Thirteenth International Conference on Learning Representations, 2025. URL [https://openreview.net/forum?id=](https://openreview.net/forum?id=) M23dTGWCZy. [↗](https://openreview.net/forum?id=)
60. Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. [Core-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark](https://md.research.elamur.ai/free/en/papers/siegel-2024-core-bench-fostering-credibility/). Trans. Mach. Learn. Res., 2024, 2024.
61. Sanghyeob Song, Jaihyun Lew, Hyemi Jang, and Sungroh Yoon. Unsupervised homography estimation on multimodal image pair via alternating optimization. ArXiv, abs/2411.13036, 2024. — [arXiv:2411.13036](https://arxiv.org/abs/2411.13036) [find](https://scholar.google.com/scholar?q=Sanghyeob+Song%2C+Jaihyun+Lew%2C+Hyemi+Jang%2C+and+Sungroh+Yoon%2E+Unsupervised+homography+estimation+on+multimodal+image+pair+via+alternating+optimization%2E+ArXiv%2C+abs%2F2411%2E13036%2C+2024%2E+%E2%80%94+%5BarXiv%3A2411%2E13036%5D%28h)
62. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research. ArXiv, abs/2504.[01848, 2025](https://md.research.elamur.ai/free/en/papers/starace-2025-paperbench-evaluating-ai-s/).
63. Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. [Stop overthinking: A survey on efficient reasoning for large language models](https://md.research.elamur.ai/free/en/papers/sui-2025-stop-overthinking-survey-efficient-reasoning/). 2025.
64. Liyan Tang, Philippe Laban, and Greg Durrett. [Minicheck: Efficient fact-checking of llms on grounding documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/). In Conference on Empirical Methods in Natural Language Processing, 2024a.
65. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. [Language-specific neurons: The key to multilingual capabilities in large language models](https://md.research.elamur.ai/free/en/papers/tang-2024-language-specific-neurons-key-multilingual/). ArXiv, abs/2402.16438, 2024b.
66. Hung Quoc To, Minh Huynh Nguyen, and Nghi D. Q. Bui. Functional overlap reranking for neural code generation. In Annual Meeting of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Hung+Quoc+To%2C+Minh+Huynh+Nguyen%2C+and+Nghi+D%2E+Q%2E+Bui%2E+Functional+overlap+reranking+for+neural+code+generation%2E+In+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2023%2E)
67. Cunda Wang, Weihua Wang, Qiuyu Liang, Feilong Bao, and Guanglai Gao. Unifying dual-space embedding for entity alignment via contrastive learning. ArXiv, abs/2412.05028, 2024a. [find](https://scholar.google.com/scholar?q=Cunda+Wang%2C+Weihua+Wang%2C+Qiuyu+Liang%2C+Feilong+Bao%2C+and+Guanglai+Gao%2E+Unifying+dual%2Dspace+embedding+for+entity+alignment+via+contrastive+learning%2E+ArXiv%2C+abs%2F2412%2E05028%2C+2024a%2E)
68. Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. [Scimon: Scientific inspiration machines optimized for novelty](https://md.research.elamur.ai/free/en/papers/wang-2023-scimon-scientific-inspiration-machines/). In Annual Meeting of the Association for Computational Linguistics, 2023.
69. Xinyu Wang, Yanzheng Xiang, Lin Gui, and Yulan He. Garlic: Llm-guided dynamic progress control with hierarchical weighted graph for long document qa. ArXiv, abs/2410.04790, 2024b. — [arXiv:2410.04790](https://arxiv.org/abs/2410.04790) [find](https://scholar.google.com/scholar?q=Xinyu+Wang%2C+Yanzheng+Xiang%2C+Lin+Gui%2C+and+Yulan+He%2E+Garlic%3A+Llm%2Dguided+dynamic+progress+control+with+hierarchical+weighted+graph+for+long+document+qa%2E+ArXiv%2C+abs%2F2410%2E04790%2C+2024b%2E+%E2%80%94+%5BarXiv%3A2410%2E04790%5D)
70. Yaoke Wang, Yun Zhu, Wenqiao Zhang, Yueting Zhuang, Yunfei Li, and Siliang Tang. Bridging local details and global context in text-attributed graphs. In Conference on Empirical Methods in Natural Language Processing, 2024c. [find](https://scholar.google.com/scholar?q=Yaoke+Wang%2C+Yun+Zhu%2C+Wenqiao+Zhang%2C+Yueting+Zhuang%2C+Yunfei+Li%2C+and+Siliang+Tang%2E+Bridging+local+details+and+global+context+in+text%2Dattributed+graphs%2E+In+Conference+on+Empirical+Methods+in+Natural+Lang)
71. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. [Execution-based evaluation for open-domain code generation](https://md.research.elamur.ai/free/en/papers/wang-2023-execution-based-evaluation-open-domain/). In Conference on Empirical Methods in Natural Language Processing, 2022.
72. Yanzheng Xiang, Qian-Wen Zhang, Xu Zhang, Zejie Liu, Yunbo Cao, and Deyu Zhou. G³r: A graph-guided generate-and-rerank framework for complex and cross-domain text-to-sql generation. In Annual Meeting of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Yanzheng+Xiang%2C+Qian%2DWen+Zhang%2C+Xu+Zhang%2C+Zejie+Liu%2C+Yunbo+Cao%2C+and+Deyu+Zhou%2E+G%C2%B3r%3A+A+graph%2Dguided+generate%2Dand%2Drerank+framework+for+complex+and+cross%2Ddomain+text%2Dto%2Dsql+generation%2E+In+Annual+Meeting+)
73. Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. Addressing order sensitivity of in-context demonstration examples in causal language models. ArXiv, abs/2402.15637, 2024. [find](https://scholar.google.com/scholar?q=Yanzheng+Xiang%2C+Hanqi+Yan%2C+Lin+Gui%2C+and+Yulan+He%2E+Addressing+order+sensitivity+of+in%2Dcontext+demonstration+examples+in+causal+language+models%2E+ArXiv%2C+abs%2F2402%2E15637%2C+2024%2E)
74. Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. [Safedecoding: Defending against jailbreak attacks via safety-aware decoding](https://md.research.elamur.ai/free/en/papers/xu-2024-safedecoding-defending-against-jailbreak-attacks/). ArXiv, abs/2402.08983, 2024.
75. Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Nicolaus Foerster, Jeff Clune, and David Ha. [The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search](https://md.research.elamur.ai/free/en/papers/yamada-2025-ai-scientist-v2-workshop/). ArXiv, abs/2504.08066, 2025.
76. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). ArXiv, abs/2210.03629, 2022.
77. Rui Ying, Mengting Hu, Jianfeng Wu, Yalan Xie, Xiaoyi Liu, Zhunheng Wang, Ming Jiang, Hang Gao, Linlin Zhang, and Renhong Cheng. Simple but effective compound geometric operations for temporal knowledge graph completion. In Annual Meeting of the Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Rui+Ying%2C+Mengting+Hu%2C+Jianfeng+Wu%2C+Yalan+Xie%2C+Xiaoyi+Liu%2C+Zhunheng+Wang%2C+Ming+Jiang%2C+Hang+Gao%2C+Linlin+Zhang%2C+and+Renhong+Cheng%2E+Simple+but+effective+compound+geometric+operations+for+temporal+knowled)
78. Yao-Ching Yu, Chun-Chih Kuo, Ziqi Ye, Yu-Cheng Chang, and Yueh-Se Li. Breaking the ceiling of the llm community by treating token generation as a classification for ensembling. In Conference on Empirical Methods in Natural Language Processing, 2024. [find](https://scholar.google.com/scholar?q=Yao%2DChing+Yu%2C+Chun%2DChih+Kuo%2C+Ziqi+Ye%2C+Yu%2DCheng+Chang%2C+and+Yueh%2DSe+Li%2E+Breaking+the+ceiling+of+the+llm+community+by+treating+token+generation+as+a+classification+for+ensembling%2E+In+Conference+on+Empiri)
79. Zeping Yu and Sophia Ananiadou. Neuron-level knowledge attribution in large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. [find](https://scholar.google.com/scholar?q=Zeping+Yu+and+Sophia+Ananiadou%2E+Neuron%2Dlevel+knowledge+attribution+in+large+language+models%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+2023%2E)
80. Weizhe Yuan, Pengfei Liu, and Graham Neubig. [Can we automate scientific reviewing?](https://md.research.elamur.ai/free/en/papers/yuan-2021-automate-scientific-reviewing/) J. Artif. Int. Res., 75, December 2022. ISSN 1076-9757. doi: 10.1613/jair.1.12862. URL https://doi.org/10.1613/jair.1.12862. [↗](https://doi.org/10.1613/jair.1.12862)
81. Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. [Gliner: Generalist model for named entity recognition using bidirectional transformer](https://md.research.elamur.ai/free/en/papers/zaratiana-2024-gliner-generalist-model-named-entity/). ArXiv, abs/2311.08526, 2023.
82. Jiahao Zhang, H. Zhang, Dongmei Zhang, Yong Liu, and Sheng Huang. End-to-end beam retrieval for multi-hop question answering. In North American Chapter of the Association for Computational Linguistics, 2023a. [find](https://scholar.google.com/scholar?q=Jiahao+Zhang%2C+H%2E+Zhang%2C+Dongmei+Zhang%2C+Yong+Liu%2C+and+Sheng+Huang%2E+End%2Dto%2Dend+beam+retrieval+for+multi%2Dhop+question+answering%2E+In+North+American+Chapter+of+the+Association+for+Computational+Linguistics)
83. Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use api search tools. ArXiv, abs/2305.04032, 2023b. — [arXiv:2305.04032](https://arxiv.org/abs/2305.04032) [find](https://scholar.google.com/scholar?q=Kechi+Zhang%2C+Ge+Li%2C+Jia+Li%2C+Zhuo+Li%2C+and+Zhi+Jin%2E+Toolcoder%3A+Teach+code+generation+models+to+use+api+search+tools%2E+ArXiv%2C+abs%2F2305%2E04032%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2305%2E04032%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2305%2E04032%29)
84. Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. [Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges](https://md.research.elamur.ai/free/en/papers/zhang-2024-codeagent-enhancing-code-generation-tool/). In Annual Meeting of the Association for Computational Linguistics, 2024a.
85. Xu Zhang, Yanzheng Xiang, Zejie Liu, Xiaoyu Hu, and Deyu Zhou. I2r: Intra and inter-modal representation learning for code search. Intell. Data Anal., 28:807–823, 2023c. [find](https://scholar.google.com/scholar?q=Xu+Zhang%2C+Yanzheng+Xiang%2C+Zejie+Liu%2C+Xiaoyu+Hu%2C+and+Deyu+Zhou%2E+I2r%3A+Intra+and+inter%2Dmodal+representation+learning+for+code+search%2E+Intell%2E+Data+Anal%2E%2C+28%3A807%E2%80%93823%2C+2023c%2E)
86. Xu Zhang, Xiaoyu Hu, Zejie Liu, Yanzheng Xiang, and Deyu Zhou. Hfd: Hierarchical feature decoupling for sql generation from text. Intell. Data Anal., 28:991–1005, 2024b. [find](https://scholar.google.com/scholar?q=Xu+Zhang%2C+Xiaoyu+Hu%2C+Zejie+Liu%2C+Yanzheng+Xiang%2C+and+Deyu+Zhou%2E+Hfd%3A+Hierarchical+feature+decoupling+for+sql+generation+from+text%2E+Intell%2E+Data+Anal%2E%2C+28%3A991%E2%80%931005%2C+2024b%2E)
87. Xu Zhang, Zexu Lin, Xiaoyu Hu, Jianlei Wang, Wenpeng Lu, and Deyu Zhou. Secon: Maintaining semantic consistency in data augmentation for code search. ACM Transactions on Information Systems, 2024c. [find](https://scholar.google.com/scholar?q=Xu+Zhang%2C+Zexu+Lin%2C+Xiaoyu+Hu%2C+Jianlei+Wang%2C+Wenpeng+Lu%2C+and+Deyu+Zhou%2E+Secon%3A+Maintaining+semantic+consistency+in+data+augmentation+for+code+search%2E+ACM+Transactions+on+Information+Systems%2C+2024c%2E)
88. W. Zhao, C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Ratescore: A metric for radiology report generation. In Conference on Empirical Methods in Natural Language Processing, 2024. [find](https://scholar.google.com/scholar?q=W%2E+Zhao%2C+C%2E+Wu%2C+X%2E+Zhang%2C+Y%2E+Zhang%2C+Y%2E+Wang%2C+and+W%2E+Xie%2E+Ratescore%3A+A+metric+for+radiology+report+generation%2E+In+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+2024%2E)
89. Yuxiang Zhou, Jiazheng Li, Yanzheng Xiang, Hanqi Yan, Lin Gui, and Yulan He. The mystery of in-context learning: A comprehensive survey on interpretation and analysis. In Conference on Empirical Methods in Natural Language Processing, 2023. URL [https://api.semanticscholar.org/CorpusID:264832783](https://api.semanticscholar.org/CorpusID:264832783). [↗](https://api.semanticscholar.org/CorpusID:264832783)

*Updated 3 November 2026*