---
title: "TinyJudge — paper by Zeng et al. (2026) — summary and ideas"
description: "Based on this, they propose TinyJudge, which distills frontier-model judgments into separate ~0.6B specialist models, one per… 23 references."
authors: ["Yirong Zeng", "Yufei Liu", "Xiao Ding", "Yutai Hou", "Yuxian Wang", "Wu Ning", "Haonan Song", "Dandan Tu", "Qixun Zhang", "Yuxiang He", "Bibo Cai", "Ting Liu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1204/"
doi: "10.18653/v1/2026.acl-long.1204"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2026-tinyjudge-unverifiable-constraint-alignment-lightweight/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2026-tinyjudge-unverifiable-constraint-alignment-lightweight/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/) / TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles

# TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles

**Authors:** Yirong Zeng, Yufei Liu, [Xiao Ding](https://md.research.elamur.ai/free/en/authors/xiao-ding/), [Yutai Hou](https://md.research.elamur.ai/free/en/authors/yutai-hou/), Yuxian Wang, Wu Ning, Haonan Song, [Dandan Tu](https://md.research.elamur.ai/free/en/authors/dandan-tu/), Qixun Zhang, Yuxiang He, [Bibo Cai](https://md.research.elamur.ai/free/en/authors/bibo-cai/) and [Ting Liu](https://md.research.elamur.ai/free/en/authors/ting-liu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1204/) · [PDF](https://aclanthology.org/2026.acl-long.1204.pdf) · [DOI 10.18653/v1/2026.acl-long.1204](https://doi.org/10.18653/v1/2026.acl-long.1204) · [OpenAlex](https://openalex.org/W7166818199)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Reinforcement learning with verifiable rewards (RLVR) for instruction following usually checks hard constraints, such as length or format, with code rules. Soft constraints, such as tone or style, are handed to a large LLM judge. The authors show that this setup is flawed. Judges like Qwen3-32B often miss violations when several constraints are checked together, and policies learn to exploit this: reward scores rise during training while test results fall. The judge also makes training much slower. Their analysis finds that some soft constraint types generalize far better than others. Based on this, they propose TinyJudge, which distills frontier-model judgments into separate ~0.6B specialist models, one per high-generalization soft constraint type, and keeps code rules for hard constraints. Training uses GRPO on VerInstruct across five benchmarks, including IFEval and CFBench. TinyJudge gains about 10% over the base model on average, gives 12% more precise rewards, and cuts training time roughly threefold. This makes alignment to unverifiable instructions cheaper and more reliable.

## Key points
- Pilot study: Qwen2.5-7B-Instruct was trained on VerInstruct (22,000 instances; 77.7% soft, 22.3% hard constraints). The hard-only model beat the soft-only model (judged by Qwen3-32B) by 3.0% on IFEval and the mixed model by 2.4%. This pattern held across Qwen2.5, Qwen3 and LLama3.2 models from 3B to 32B.
- Cost of the LLM judge: using Qwen3-32B as judge raised total training time by about 339% and per-response reward latency by 11×. Rule-based checks took roughly 30 ms per instance.
- Ablation (Table 3), with three specialists such as Qwen3-0.6B: the average over IFEval and CFBench rose from 58.23 to 68.41, and CFBench went from 44.00 to 54.00. With Qwen3-4B-Inst, the "3-for-3" design beat a single shared judge by 2% ("1-for-all") and 5% ("1-for-3").
- Efficiency: the tiny judges are about 6× faster per response and about 3× faster in total training time than Qwen3-32B. Their overhead is close to that of rule-only training.
- Reward precision was measured against Gemini-3.0-Pro labels (an average of five runs) on a 5% held-out set. The 4B specialist ensemble was 12% more precise than Qwen3-32B.
- Limitations noted by the authors:
  - There is no principled way to choose how many experts to use.
  - Each specialist inherits the biases of its teacher model.
  - Every new constraint type requires a new distilled model.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 315 works · 43,267 citations
- [Harbin Institute of Technology](https://md.research.elamur.ai/free/en/universities/harbin-institute-of-technology/) — University, Harbin, China · 185 works · 29,140 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 96 works · 15,432 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai and Ting Liu. 2026. TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeng2026tinyjudge,
  title = {{TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles}},
  author = {Yirong Zeng, Yufei Liu, Xiao Ding, Yutai Hou, Yuxian Wang, Wu Ning, Haonan Song, Dandan Tu, Qixun Zhang, Yuxiang He, Bibo Cai and Ting Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1204},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1204}
}
```

- [← previous TingIS: Real-time Risk Event Discovery from Noisy Customer Incidents at Enterprise Scale](https://md.research.elamur.ai/free/en/papers/wang-2026-tingis-real-time-risk-event/)
- [next → To make someone do something: mining alert-style directives in Bulgarian social media for low-resource language modelling](https://md.research.elamur.ai/free/en/papers/margova-2026-make-someone-do-something-mining/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/53/)

## References

23 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Yunxuan+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+2024%2E+Scaling+instruction%2Dfinetuned+language+models%2E+J)
2. Muzhi Dai, Chenxu Yang, and Qingyi Si. 2025. S-grpo: Early exit via reinforcement learning in reasoning models. arXiv preprint arXiv:2505.07686. — [arXiv:2505.07686](https://arxiv.org/abs/2505.07686) [find](https://scholar.google.com/scholar?q=Muzhi+Dai%2C+Chenxu+Yang%2C+and+Qingyi+Si%2E+2025%2E+S%2Dgrpo%3A+Early+exit+via+reinforcement+learning+in+reasoning+models%2E+arXiv+preprint+arXiv%3A2505%2E07686%2E+%E2%80%94+%5BarXiv%3A2505%2E07686%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E07686%29)
3. Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. 2024. Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542. [find](https://scholar.google.com/scholar?q=Guanting+Dong%2C+Keming+Lu%2C+Chengpeng+Li%2C+Tingyu+Xia%2C+Bowen+Yu%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2024%2E+Self%2Dplay+with+execution+feedback%3A+Improving+instruction%2Dfollowing+capabilities+of+large+language+mode)
4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
5. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Andrew+Ilyas%2C+Shibani+Santurkar%2C+Dimitris+Tsipras%2C+Firdaus+Janoos%2C+Larry+Rudolph%2C+and+Aleksander+Madry%2E+2020%2E+Implementation+matters+in+deep+policy+gradients%3A+A+case+study+on+ppo+and+t)
6. Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. 2025. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05185. — [arXiv:2504.05185](https://arxiv.org/abs/2504.05185) [find](https://scholar.google.com/scholar?q=Mehdi+Fatemi%2C+Banafsheh+Rafiee%2C+Mingjie+Tang%2C+and+Kartik+Talamadupula%2E+2025%2E+Concise+reasoning+via+reinforcement+learning%2E+arXiv+preprint+arXiv%3A2504%2E05185%2E+%E2%80%94+%5BarXiv%3A2504%2E05185%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025a. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
8. Zhengkang Guo, Wenhao Liu, Mingchen Xie, Jingwen Xu, Zisu Huang, Muzhao Tian, Jianhan Xu, Yuanzhe Shen, Qi Qian, Muling Wu, Xiaohua Wang, Changze Lv, He-Da Wang, Hu Yao, Xiaoqing Zheng, and Xuanjing Huang. 2025b. Recast: Expanding the boundaries of llms’ complex instruction following with multi-cons… — [arXiv:2505.19030](https://arxiv.org/abs/2505.19030) [find](https://scholar.google.com/scholar?q=Zhengkang+Guo%2C+Wenhao+Liu%2C+Mingchen+Xie%2C+Jingwen+Xu%2C+Zisu+Huang%2C+Muzhao+Tian%2C+Jianhan+Xu%2C+Yuanzhe+Shen%2C+Qi+Qian%2C+Muling+Wu%2C+Xiaohua+Wang%2C+Changze+Lv%2C+He%2DDa+Wang%2C+Hu+Yao%2C+Xiaoqing+Zheng%2C+and+Xuanjing+H)
9. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. 2024. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553. — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) [find](https://scholar.google.com/scholar?q=Yun+He%2C+Di+Jin%2C+Chaoqi+Wang%2C+Chloe+Bi%2C+Karishma+Mandyam%2C+Hejia+Zhang%2C+Chen+Zhu%2C+Ning+Li%2C+Tengyu+Xu%2C+Hongjiang+Lv%2C+et+al%2E+2024%2E+Multi%2Dif%3A+Benchmarking+llms+on+multi%2Dturn+and+multilingual+instructions+f)
10. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. [Followbench: A multi-level fine-grained constraints following benchmark for large language models](https://md.research.elamur.ai/free/en/papers/jiang-2024-followbench-multi-level-fine-grained/). In Proceedings of the 62nd Annual Meeting of the Association for Computa…
11. Joongwon Kim, Anirudh Goyal, Aston Zhang, Bo Xiong, Rui Hou, Melanie Kambadur, Dhruv Mahajan, Hannaneh Hajishirzi, and Liang Tan. 2025. A systematic examination of preference learning through the lens of instruction-following. In Proceedings of the 2025 Conference of the Nations of the Americas Chap… [find](https://scholar.google.com/scholar?q=Joongwon+Kim%2C+Anirudh+Goyal%2C+Aston+Zhang%2C+Bo+Xiong%2C+Rui+Hou%2C+Melanie+Kambadur%2C+Dhruv+Mahajan%2C+Hannaneh+Hajishirzi%2C+and+Liang+Tan%2E+2025%2E+A+systematic+examination+of+preference+learning+through+the+lens)
12. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. 2024. [Tulu 3: Pushing frontiers in open language model post-training](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/). arXiv preprint arXiv:2411.15124.
13. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al. 2025. [Deepseek-v3](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-v3-2-pushing/). 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
14. OpenAI. 2025. Introducing GPT-5. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+GPT%2D5%2E)
15. Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2025. [Verif: Verification engineering for reinforcement learning in instruction following](https://md.research.elamur.ai/free/en/papers/peng-2025-verif-verification-engineering-reinforcement-learning/). arXiv preprint arXiv:2506.09942.
16. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. 2025. [Generalizing verifiable instruction following](https://md.research.elamur.ai/free/en/papers/pyatkin-2025-generalizing-verifiable-instruction-following/). arXiv preprint arXiv:2507.02833.
17. Qwen. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2E+2025%2E+Qwen2%2E5+technical+report%2E+Preprint%2C+arXiv%3A2412%2E15115%2E)
18. Qingyu Ren, Qianyu He, Bowei Zhang, Jie Zeng, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. 2025. [Instructions are all you need: Self-supervised reinforcement learning for instruction following](https://md.research.elamur.ai/free/en/papers/ren-2026-instructions-are-all-you-need/). arXiv preprint arXiv:2510.14420.
19. Tao Zhang, Chenglin Zhu, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, et al. 2025. [Cfbench: A comprehensive constraints-following benchmark for llms](https://md.research.elamur.ai/free/en/papers/zhang-2025-cfbench-comprehensive-constraints-following-benchmark/). In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume…
20. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. [Instruction-following evaluation for large language models](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). CoRR.
21. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+2017%2E+Proximal+policy+optimization+algorithms%2E+arXiv+preprint+arXiv%3A1707%2E06347%2E)
22. Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. 2025. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint arXiv:2506.01347. [find](https://scholar.google.com/scholar?q=Xinyu+Zhu%2C+Mengzhou+Xia%2C+Zhepei+Wei%2C+Wei%2DLin+Chen%2C+Danqi+Chen%2C+and+Yu+Meng%2E+2025%2E+The+surprising+effectiveness+of+negative+reinforcement+in+llm+reasoning%2E+arXiv+preprint+arXiv%3A2506%2E01347%2E)
23. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). arXiv preprint arXiv:2402.03300.

*Updated 2 November 2026*