---
title: "No More Stale Feedback — paper by Li et al. (2026)"
description: "They propose ECHO (Evolving Critic for Hindsight-Guided Optimization), which trains the policy and… 36 references, cited by 3 papers here."
authors: ["Zhicong Li", "Lingjie Jiang", "Yulan Hu", "Xingchen Zeng", "Yixia Li", "Xiangwen Zhang", "Guanhua Chen", "Zheng Pan", "Xin Li", "Yong Liu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.576/"
doi: "10.18653/v1/2026.acl-long.576"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2026-no-more-stale-feedback-co/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2026-no-more-stale-feedback-co/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning

# No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning

**Authors:** Zhicong Li, [Lingjie Jiang](https://md.research.elamur.ai/free/en/authors/lingjie-jiang/), [Yulan Hu](https://md.research.elamur.ai/free/en/authors/yulan-hu/), Xingchen Zeng, [Yixia Li](https://md.research.elamur.ai/free/en/authors/yixia-li/), Xiangwen Zhang, [Guanhua Chen](https://md.research.elamur.ai/free/en/authors/guanhua-chen/), [Zheng Pan](https://md.research.elamur.ai/free/en/authors/zheng-pan/), [Xin Li](https://md.research.elamur.ai/free/en/authors/xin-li/) and [Yong Liu](https://md.research.elamur.ai/free/en/authors/yong-liu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.576/) · [PDF](https://aclanthology.org/2026.acl-long.576.pdf) · [DOI 10.18653/v1/2026.acl-long.576](https://doi.org/10.18653/v1/2026.acl-long.576) · [OpenAlex](https://openalex.org/W7166880888)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [3](https://md.research.elamur.ai/free/en/papers/li-2026-no-more-stale-feedback-co/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Critique-guided reinforcement learning supplements sparse outcome rewards for LLM agents with natural-language feedback. The authors argue that most existing critics are fixed templates or separately trained models that stay frozen during training. As the on-policy agent improves, its typical mistakes shift from coarse to subtle, so a frozen critic grows "stale": its advice becomes redundant, poorly calibrated, or even harmful.

They propose ECHO (Evolving Critic for Hindsight-Guided Optimization), which trains the policy and critic together. In a cascaded rollout, the policy first produces a trajectory, which a reward model scores. The critic then writes N diagnoses of it, and the policy produces one refined attempt per diagnosis. These groups feed GRPO-style updates for both models (dual-track GRPO). The critic is rewarded by the improvement its feedback causes. A saturation-aware gain gives extra credit for small gains on trajectories that already score near the maximum.

Across four open-world environments, ECHO trains more stably and reaches higher long-horizon success than GRPO and a frozen-critic variant. This shows that keeping critics synchronized with the policy matters.

## Key points
- The paper identifies "critic staleness" and shows it empirically: freezing the critic makes its feedback less useful as the policy gets better.
- ECHO is evaluated on WebShop, ALFWorld, SciWorld and DeepSearch, with analyses run on Qwen3-4B.
- The training curves differ by environment:
  - **WebShop:** the frozen-critic variant leads early but is overtaken later.
  - **ALFWorld and SciWorld:** ECHO pulls ahead of GRPO mainly in mid-to-late training.
  - **DeepSearch:** ECHO gains fastest early, likely by fixing format and protocol errors.
- Replacing the saturation-aware reward with a linear reward (s_r − s_o) lowers results on both WebShop and SciWorld, with the larger drop on WebShop. Density plots show more reward-improving refinements near the score ceiling under the saturation-aware reward.
- A stated limitation is that both models rely on one external reward model, so noisy or biased rewards can mislead the critic and policy. Merging the scorer and critic into one model is left for future work.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 168 works · 15,386 citations
- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 114 works · 16,118 citations
- [The Hong Kong University of Science and Technology (Guangzhou)](https://md.research.elamur.ai/free/en/universities/the-hong-kong-university-of-science-and-technology-guangzhou/) — University, Guangzhou, China · 53 works · 1,367 citations
- [Southern University of Science and Technology](https://md.research.elamur.ai/free/en/universities/southern-university-of-science-and-technology/) — University, Shenzhen, China · 19 works · 1,337 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li and Yong Liu. 2026. No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2026no,
  title = {{No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning}},
  author = {Zhicong Li, Lingjie Jiang, Yulan Hu, Xingchen Zeng, Yixia Li, Xiangwen Zhang, Guanhua Chen, Zheng Pan, Xin Li and Yong Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.576},
  url = {https://doi.org/10.18653/v1/2026.acl-long.576}
}
```

- [← previous NL2Logic: AST-Guided Translation of Natural Language into First-Order Logic with Large Language Models](https://md.research.elamur.ai/free/en/papers/putra-2026-nl2logic-ast-guided-translation-natural/)
- [next → Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation](https://md.research.elamur.ai/free/en/papers/xiao-2026-not-all-directions-matter-structured/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/)

## References

36 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. AI Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. claude-3 model card. In Conference on Natural Language Processing, volume 1. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+2024%2E+The+claude+3+model+family%3A+Opus%2C+sonnet%2C+haiku%2E+claude%2D3+model+card%2E+In+Conference+on+Natural+Language+Processing%2C+volume+1%2E)
3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
5. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2024. [Chain-of-verification reduces hallucination in large language models](https://md.research.elamur.ai/free/en/papers/dhuliawala-2023-chain-verification-reduces-hallucination-large/). In Findings of the association for computational linguistics: ACL 2024, pages 3563–3578.
6. Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, and 1 others. 2025. Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. In Findings of the Association for Computational Lin… [find](https://scholar.google.com/scholar?q=Bofei+Gao%2C+Zefan+Cai%2C+Runxin+Xu%2C+Peiyi+Wang%2C+Ce+Zheng%2C+Runji+Lin%2C+Keming+Lu%2C+Dayiheng+Liu%2C+Chang+Zhou%2C+Wen+Xiao%2C+and+1+others%2E+2025%2E+Llm+critics+help+catch+bugs+in+mathematics%3A+Towards+a+better+mathem)
7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
8. Qihan Huang, Weilong Dai, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, Jingyuan Chen, Chang Yao, and Jie Song. 2025. Boosting mllm reasoning with text-debiased hint-grpo. arXiv preprint arXiv:2503.23905. — [arXiv:2503.23905](https://arxiv.org/abs/2503.23905) [find](https://scholar.google.com/scholar?q=Qihan+Huang%2C+Weilong+Dai%2C+Jinlong+Liu%2C+Wanggui+He%2C+Hao+Jiang%2C+Mingli+Song%2C+Jingyuan+Chen%2C+Chang+Yao%2C+and+Jie+Song%2E+2025%2E+Boosting+mllm+reasoning+with+text%2Ddebiased+hint%2Dgrpo%2E+arXiv+preprint+arXiv%3A2503)
9. Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, and 1 others. 2024. [Critiquellm: Towards an informative critique generation model for evaluation of large language model generation](https://md.research.elamur.ai/free/en/papers/ke-2023-critiquellm-informative-critique-generation/). In Proceedings of the 62nd Annual Meeting of…
10. Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, and Dandan Tu. 2025. GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning. arXiv preprint. ArXiv:2507.10628 [cs]. — [arXiv:2507.10628](https://arxiv.org/abs/2507.10628) [find](https://scholar.google.com/scholar?q=Ziru+Liu%2C+Cheng+Gong%2C+Xinyu+Fu%2C+Yaofang+Liu%2C+Ran+Chen%2C+Shoubo+Hu%2C+Suiyun+Zhang%2C+Rui+Liu%2C+Qingfu+Zhang%2C+and+Dandan+Tu%2E+2025%2E+GHPO%3A+Adaptive+Guidance+for+Stable+and+Efficient+LLM+Reinforcement+Learning%2E)
11. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
12. Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215. — [arXiv:2407.00215](https://arxiv.org/abs/2407.00215) [find](https://scholar.google.com/scholar?q=Nat+McAleese%2C+Rai+Michael+Pokorny%2C+Juan+Felipe+Ceron+Uribe%2C+Evgenia+Nitishinskaya%2C+Maja+Trebacz%2C+and+Jan+Leike%2E+2024%2E+Llm+critics+help+catch+llm+bugs%2E+arXiv+preprint+arXiv%3A2407%2E00215%2E+%E2%80%94+%5BarXiv%3A2407%2E00)
13. Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287. Citeseer. [find](https://scholar.google.com/scholar?q=Andrew+Y+Ng%2C+Daishi+Harada%2C+and+Stuart+Russell%2E+1999%2E+Policy+invariance+under+reward+transformations%3A+Theory+and+application+to+reward+shaping%2E+In+Icml%2C+volume+99%2C+pages+278%E2%80%93287%2E+Citeseer%2E)
14. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. [Self-critiquing models for assisting human evaluators](https://md.research.elamur.ai/free/en/papers/saunders-2022-self-critiquing-models-assisting-human/). arXiv preprint arXiv:2206.05802.
15. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.
16. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. [Alfworld: Aligning text and embodied environments for interactive learning](https://md.research.elamur.ai/free/en/papers/shridhar-2020-alfworld-aligning-text-embodied-environments/). arXiv preprint arXiv:2010.03768.
17. Richard S Sutton, Andrew G Barto, and 1 others. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. [find](https://scholar.google.com/scholar?q=Richard+S+Sutton%2C+Andrew+G+Barto%2C+and+1+others%2E+1998%2E+Reinforcement+learning%3A+An+introduction%2C+volume+1%2E+MIT+press+Cambridge%2E)
18. Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Saiyong Yang, and Yunfang Wu. 2025. [Do not step into the same river twice: Learning to reason from trial and error](https://md.research.elamur.ai/free/en/papers/tang-2026-do-not-step-into-same/). arXiv preprint arXiv:2510.26109.
19. Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, and 1 others. Self-evolving critique abilities in large language models. In Second Conference on Language Modeling. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen,… — [Kimi K2: Open Agentic Intelligence](https://md.research.elamur.ai/free/en/papers/kimi-team-2025-kimi-k2-open-agentic/)
20. Qwen Team and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). [find](https://scholar.google.com/scholar?q=Qwen+Team+and+1+others%2E+2024%2E+Qwen2+technical+report%2E+arXiv+preprint+arXiv%3A2407%2E10671%2C+2%283%29%2E)
21. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
22. Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, and 1 others. 2025. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base llms. arXiv preprint arXiv:2506.14245. [find](https://scholar.google.com/scholar?q=Xumeng+Wen%2C+Zihan+Liu%2C+Shun+Zheng%2C+Shengyu+Ye%2C+Zhirong+Wu%2C+Yang+Wang%2C+Zhijian+Xu%2C+Xiao+Liang%2C+Junjie+Li%2C+Ziming+Miao%2C+and+1+others%2E+2025%2E+Reinforcement+learning+with+verifiable+rewards+implicitly+ince)
23. Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Jiazheng Zhang, Wenxiang Chen, and 1 others. 2025. Agentgym-rl: Training llm agents for long-horizon decision making through multi-turn reinforcement learning. arXiv preprint arXiv:2509.08755. — [arXiv:2509.08755](https://arxiv.org/abs/2509.08755) [find](https://scholar.google.com/scholar?q=Zhiheng+Xi%2C+Jixuan+Huang%2C+Chenyang+Liao%2C+Baodai+Huang%2C+Honglin+Guo%2C+Jiaqi+Liu%2C+Rui+Zheng%2C+Junjie+Ye%2C+Jiazheng+Zhang%2C+Wenxiang+Chen%2C+and+1+others%2E+2025%2E+Agentgym%2Drl%3A+Training+llm+agents+for+long%2Dhorizo)
24. Zhiheng Xi, Dingwen Yang, Jixuan Huang, Jiafu Tang, Guanyu Li, Yiwen Ding, Wei He, Boyang Hong, Shihan Do, Wenyu Zhan, and 1 others. 2024. Enhancing llm reasoning via critique models with test-time and training-time supervision. arXiv preprint arXiv:2411.16579. — [arXiv:2411.16579](https://arxiv.org/abs/2411.16579) [find](https://scholar.google.com/scholar?q=Zhiheng+Xi%2C+Dingwen+Yang%2C+Jixuan+Huang%2C+Jiafu+Tang%2C+Guanyu+Li%2C+Yiwen+Ding%2C+Wei+He%2C+Boyang+Hong%2C+Shihan+Do%2C+Wenyu+Zhan%2C+and+1+others%2E+2024%2E+Enhancing+llm+reasoning+via+critique+models+with+test%2Dtime+an)
25. Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. 2025. Teaching language models to critique via reinforcement learning. arXiv preprint arXiv:2502.03492. — [arXiv:2502.03492](https://arxiv.org/abs/2502.03492) [find](https://scholar.google.com/scholar?q=Zhihui+Xie%2C+Jie+Chen%2C+Liyu+Chen%2C+Weichao+Mao%2C+Jingjing+Xu%2C+and+Lingpeng+Kong%2E+2025%2E+Teaching+language+models+to+critique+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2502%2E03492%2E+%E2%80%94+%5BarXiv%3A2502%2E0349)
26. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. [find](https://scholar.google.com/scholar?q=Jianhao+Yan%2C+Yafu+Li%2C+Zican+Hu%2C+Zhi+Wang%2C+Ganqu+Cui%2C+Xiaoye+Qu%2C+Yu+Cheng%2C+and+Yue+Zhang%2E+2025%2E+Learning+to+reason+under+off%2Dpolicy+guidance%2E+arXiv+preprint+arXiv%3A2504%2E14945%2E)
27. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
28. Ruihan Yang, Fanghua Ye, Jian Li, Siyu Yuan, Yikai Zhang, Zhaopeng Tu, Xiaolong Li, and Deqing Yang. 2025b. The lighthouse of language: Enhancing llm agents via critique-guided improvement. arXiv preprint arXiv:2503.16024. — [arXiv:2503.16024](https://arxiv.org/abs/2503.16024) [find](https://scholar.google.com/scholar?q=Ruihan+Yang%2C+Fanghua+Ye%2C+Jian+Li%2C+Siyu+Yuan%2C+Yikai+Zhang%2C+Zhaopeng+Tu%2C+Xiaolong+Li%2C+and+Deqing+Yang%2E+2025b%2E+The+lighthouse+of+language%3A+Enhancing+llm+agents+via+critique%2Dguided+improvement%2E+arXiv+prep)
29. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. [Webshop: Towards scalable real-world web interaction with grounded language agents](https://md.research.elamur.ai/free/en/papers/yao-2022-webshop-towards-scalable-real-world/). Advances in Neural Information Processing Systems, 35:20744–20757.
30. Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022a. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540. [find](https://scholar.google.com/scholar?q=Ruoyao+Wang%2C+Peter+Jansen%2C+Marc%2DAlexandre+C%C3%B4t%C3%A9%2C+and+Prithviraj+Ammanabrolu%2E+2022a%2E+Scienceworld%3A+Is+your+agent+smarter+than+a+5th+grader%3F+arXiv+preprint+arXiv%3A2203%2E07540%2E)
31. Tianshu Yu, Chao Xiang, Mingchuan Yang, Pei Ke, Bosi Wen, Cunxiang Wang, Jiale Cheng, Li Zhang, Xinyu Mu, Chuxiong Sun, and 1 others. 2025. Training language model to critique for better refinement. arXiv preprint arXiv:2506.22157. [find](https://scholar.google.com/scholar?q=Tianshu+Yu%2C+Chao+Xiang%2C+Mingchuan+Yang%2C+Pei+Ke%2C+Bosi+Wen%2C+Cunxiang+Wang%2C+Jiale+Cheng%2C+Li+Zhang%2C+Xinyu+Mu%2C+Chuxiong+Sun%2C+and+1+others%2E+2025%2E+Training+language+model+to+critique+for+better+refinement%2E+a)
32. Xinyi Wang, Jinyi Han, Zishang Jiang, Tingyun Li, Jiaqing Liang, Sihang Jiang, Zhaoqian Dai, Shuguang Ma, Fei Yu, and Yanghua Xiao. 2025. Hint: Helping ineffective rollouts navigate towards effectiveness. arXiv preprint arXiv:2510.09388. — [arXiv:2510.09388](https://arxiv.org/abs/2510.09388) [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Jinyi+Han%2C+Zishang+Jiang%2C+Tingyun+Li%2C+Jiaqing+Liang%2C+Sihang+Jiang%2C+Zhaoqian+Dai%2C+Shuguang+Ma%2C+Fei+Yu%2C+and+Yanghua+Xiao%2E+2025%2E+Hint%3A+Helping+ineffective+rollouts+navigate+towards+effectiven)
33. Feng Zhang, Zezhong Tan, Xinhong Ma, Ziqiang Dong, Xi Leng, Jianfei Zhao, Xin Sun, and Yang Yang. 2025a. Adhint: Adaptive hints with difficulty priors for reinforcement learning. arXiv preprint arXiv:2512.13095. — [arXiv:2512.13095](https://arxiv.org/abs/2512.13095) [find](https://scholar.google.com/scholar?q=Feng+Zhang%2C+Zezhong+Tan%2C+Xinhong+Ma%2C+Ziqiang+Dong%2C+Xi+Leng%2C+Jianfei+Zhao%2C+Xin+Sun%2C+and+Yang+Yang%2E+2025a%2E+Adhint%3A+Adaptive+hints+with+difficulty+priors+for+reinforcement+learning%2E+arXiv+preprint+arXiv%3A)
34. Kaiyi Zhang, Ang Lv, Jinpeng Li, Yongbo Wang, Feng Wang, Haoyuan Hu, and Rui Yan. 2025b. [Stephint: Multi-level stepwise hints enhance reinforcement learning to reason](https://md.research.elamur.ai/free/en/papers/zhang-2026-stephint-multi-level-stepwise-hints/). arXiv preprint arXiv:2507.02841.
35. Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, and Jiaya Jia. 2025c. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning. arXiv preprint arXiv:2510.19807. — [arXiv:2510.19807](https://arxiv.org/abs/2510.19807) [find](https://scholar.google.com/scholar?q=Xichen+Zhang%2C+Sitong+Wu%2C+Yinghao+Zhu%2C+Haoru+Tan%2C+Shaozuo+Yu%2C+Ziyi+He%2C+and+Jiaya+Jia%2E+2025c%2E+Scaf%2Dgrpo%3A+Scaffolded+group+relative+policy+optimization+for+enhancing+llm+reasoning%2E+arXiv+preprint+arXiv%3A2)
36. Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, Yuehao Huang, Yaxuan Guo, Kexin Zhang, Hao Wang, Liang Liu, and 1 others. 2025. [Mas-bench: A unified benchmark for shortcut-augmented hybrid mobile gui agents](https://md.research.elamur.ai/free/en/papers/zhao-2026-mas-bench-unified-benchmark-shortcut/). arXiv preprint arXiv:2509.06477.

*Updated 1 November 2026*