---
title: "AutoTaskEval — paper by Lyu et al. (2026) — summary and ideas"
description: "AutoTaskEval instead builds domain-specific benchmarks from raw, unstructured corpora, using only open-source models. 24 references."
authors: ["Qingqing Lyu", "Linjuan Wu", "Yongliang Shen", "Hengwei Liu", "Hao Li", "Shengpei Jiang", "Yin Zhang", "Weiming Lu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.280/"
doi: "10.18653/v1/2026.acl-long.280"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lyu-2026-autotaskeval-domain-specific-fine-grained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lyu-2026-autotaskeval-domain-specific-fine-grained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs

# AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs

**Authors:** [Qingqing Lyu](https://md.research.elamur.ai/free/en/authors/qingqing-lyu/), [Linjuan Wu](https://md.research.elamur.ai/free/en/authors/linjuan-wu/), [Yongliang Shen](https://md.research.elamur.ai/free/en/authors/yongliang-shen/), [Hengwei Liu](https://md.research.elamur.ai/free/en/authors/hengwei-liu/), [Hao Li](https://md.research.elamur.ai/free/en/authors/hao-li/), Shengpei Jiang, [Yin Zhang](https://md.research.elamur.ai/free/en/authors/yin-zhang/) and [Weiming Lu](https://md.research.elamur.ai/free/en/authors/weiming-lu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.280/) · [PDF](https://aclanthology.org/2026.acl-long.280.pdf) · [DOI 10.18653/v1/2026.acl-long.280](https://doi.org/10.18653/v1/2026.acl-long.280) · [OpenAlex](https://openalex.org/W7166839861)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Static, manually built benchmarks are expensive, cover few tasks, and lose discriminative power as LLMs improve. Earlier automatic generators also only fill in task types defined in advance. AutoTaskEval instead builds domain-specific benchmarks from raw, unstructured corpora, using only open-source models. The pipeline has three stages:

1. **Task discovery.** It samples diverse passages with K-Center-Greedy and derives scenarios from them. Each scenario is paired with two meta-capabilities from an extended Bloom's Taxonomy to define a task.
2. **Context enrichment.** Teacher and student agents question each other in a Socratic loop to surface key concepts. These concepts are then used to retrieve supporting text from the corpus.
3. **Instance synthesis.** The teacher writes questions, the student answers, and a reviewer judges quality. The loop repeats, pushing each new item to be harder than the last.

The authors tested the framework mainly on Chinese law, using LawBench as the reference. Model rankings on the generated items closely tracked rankings on the human-curated benchmark. The items were also harder than those from competing automatic generators. A second study on noisy e-commerce reviews shows the approach also works where no domain ontology exists. This matters for any team that needs evaluation sets for specialized fields, since few benchmarks cover them.

## Key points
- **Capability taxonomy:** two levels, with the six Bloom dimensions (Remember, Understand, Apply, Analyze, Evaluate, Create) split into 27 fine-grained sub-skills.
- **Legal validation:** four LawBench tasks were regenerated and nine models were scored on them. Model rankings on the generated items correlated with LawBench rankings at Spearman 0.9.
- **Difficulty:** the generated items were harder than those from Zero-shot Benchmarking (ZSB) and BenchAgents, and roughly as hard as LawBench. The "Ours-Hard" subset was harder than LawBench.
- **Capability profiles:** larger Qwen2.5 models gained most at the Create level. Qwen2.5-7B-Instruct was strong in Analyze but weak in C1 (Ideation) and C5 (Prototyping & Refinement).
- **AutoCommerceEval:**
  - Task diversity, measured as Shannon entropy over capabilities, reached 4.36, higher than the structured legal benchmarks.
  - Models were tested on three tasks with 60 instances each, across the Qwen2.5, gemma-3 and llama-3 families.
  - Within each family, scores generally rose with model size.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 224 works · 13,587 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Qingqing Lyu, Linjuan Wu, Yongliang Shen, Hengwei Liu, Hao Li, Shengpei Jiang, Yin Zhang and Weiming Lu. 2026. AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{lyu2026autotaskeval,
  title = {{AutoTaskEval: Towards Domain-Specific and Fine-Grained Evaluation for LLMs}},
  author = {Qingqing Lyu, Linjuan Wu, Yongliang Shen, Hengwei Liu, Hao Li, Shengpei Jiang, Yin Zhang and Weiming Lu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.280},
  url = {https://doi.org/10.18653/v1/2026.acl-long.280}
}
```

- [← previous AutoSUIT Bench - Automated Security UnIt Test Benchmark for LLM Coding](https://md.research.elamur.ai/free/en/papers/osebe-2026-autosuit-bench-automated-security-unit/)
- [next → AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems](https://md.research.elamur.ai/free/en/papers/yin-2026-autoue-automated-generation-3d-games/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2025. Claude sonnet 4. [https://huggingface.co/CometAPI/Claude_Sonnet4](https://huggingface.co/CometAPI/Claude_Sonnet4). Accessed: 2025-10-06. [↗](https://huggingface.co/CometAPI/Claude_Sonnet4)
2. Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, and Vidhisha Balachandran. 2024. Benchagents: Automated benchmark creation with agent interaction. Preprint, arXiv:2410.22584. — [arXiv:2410.22584](https://arxiv.org/abs/2410.22584) [find](https://scholar.google.com/scholar?q=Natasha+Butt%2C+Varun+Chandrasekaran%2C+Neel+Joshi%2C+Besmira+Nushi%2C+and+Vidhisha+Balachandran%2E+2024%2E+Benchagents%3A+Automated+benchmark+creation+with+agent+interaction%2E+Preprint%2C+arXiv%3A2410%2E22584%2E+%E2%80%94+%5BarXiv%3A2)
3. Yulong Chen, Yang Liu, Jianhao Yan, Xuefeng Bai, Ming Zhong, Yinghao Yang, Ziyi Yang, Chenguang Zhu, and Yue Zhang. 2024. See what llms cannot answer: A self-challenge framework for uncovering llm weaknesses. Preprint, arXiv:2408.08978. — [arXiv:2408.08978](https://arxiv.org/abs/2408.08978) [find](https://scholar.google.com/scholar?q=Yulong+Chen%2C+Yang+Liu%2C+Jianhao+Yan%2C+Xuefeng+Bai%2C+Ming+Zhong%2C+Yinghao+Yang%2C+Ziyi+Yang%2C+Chenguang+Zhu%2C+and+Yue+Zhang%2E+2024%2E+See+what+llms+cannot+answer%3A+A+self%2Dchallenge+framework+for+uncovering+llm+wea)
4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). Preprint, arXiv:2403.04132.
5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arX…
6. Google DeepMind. 2025. [Gemini 2.5 pro: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). https://arxiv.org/abs/2507.06261. Accessed: 2025-10-06. [↗](https://arxiv.org/abs/2507.06261)
7. Hu Ding, Haikuo Yu, and Zixiu Wang. 2019. Greedy strategy works for k-center clustering with outliers and coreset construction. arXiv preprint arXiv:1901.08219. — [arXiv:1901.08219](https://arxiv.org/abs/1901.08219) [find](https://scholar.google.com/scholar?q=Hu+Ding%2C+Haikuo+Yu%2C+and+Zixiu+Wang%2E+2019%2E+Greedy+strategy+works+for+k%2Dcenter+clustering+with+outliers+and+coreset+construction%2E+arXiv+preprint+arXiv%3A1901%2E08219%2E+%E2%80%94+%5BarXiv%3A1901%2E08219%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
8. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. [Lawbench: Benchmarking legal knowledge of large language models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/). Preprint, arXiv:2309.16289.
9. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21… — [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
10. Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. 2023. The art of SOCRATIC QUESTIONING: Recursive thinking with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4177–4199, Singapore. Associatio… [find](https://scholar.google.com/scholar?q=Jingyuan+Qi%2C+Zhiyang+Xu%2C+Ying+Shen%2C+Minqian+Liu%2C+Di+Jin%2C+Qifan+Wang%2C+and+Lifu+Huang%2E+2023%2E+The+art+of+SOCRATIC+QUESTIONING%3A+Recursive+thinking+with+large+language+models%2E+In+Proceedings+of+the+2023+Co)
11. Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Manning, Dan Jurafsky, and Daniel E. Ho. 2022. [Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset](https://md.research.elamur.ai/free/en/papers/henderson-2022-pile-law-learning-responsible-data/). Preprint, arXiv:2207.00220.
12. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
13. Kunze Li and Yu Zhang. 2024. Planning first, question second: An LLM-guided method for controllable question generation. In Findings of the Asso-Roucher. 2023. ciation for Computational Linguistics: ACL 2024, Aymeric amazon p roduct r eviews d ataf initi. . pages 4715–4729, Bangkok, Thailand. Associ… [find](https://scholar.google.com/scholar?q=Kunze+Li+and+Yu+Zhang%2E+2024%2E+Planning+first%2C+question+second%3A+An+LLM%2Dguided+method+for+controllable+question+generation%2E+In+Findings+of+the+Asso%2DRoucher%2E+2023%2E+ciation+for+Computational+Linguistics%3A+A)
14. Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. 2025. Autobencher: Towards declarative benchmark construction. Preprint, arXiv:2407.08351. [find](https://scholar.google.com/scholar?q=Vedant+Shah%2C+Dingli+Yu%2C+Kaifeng+Lyu%2C+Simon+Xiang+Lisa+Li%2C+Farzaan+Kaiyom%2C+Evan+Zheran+Liu%2C+Yifan+Mai%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2025%2E+Autobencher%3A+Towards+declarative+benchmark+constructio)
15. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
16. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
17. AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. [https://ai.meta](https://ai.meta). com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. [↗](https://ai.meta)
18. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
19. David Owen. 2024. How predictable is language model benchmark performance? Preprint, arXiv:2401.04757. [find](https://scholar.google.com/scholar?q=David+Owen%2E+2024%2E+How+predictable+is+language+model+benchmark+performance%3F+Preprint%2C+arXiv%3A2401%2E04757%2E)
20. Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. 2025. Ai-assisted generation of difficult math questions. Preprint, arXiv:2407.21009. [find](https://scholar.google.com/scholar?q=Park%2C+Jiatong+Yu%2C+Yinghui+He%2C+Nan+Rosemary+Ke%2C+Michael+Mozer%2C+Yoshua+Bengio%2C+Sanjeev+Arora%2C+and+Anirudh+Goyal%2E+2025%2E+Ai%2Dassisted+generation+of+difficult+math+questions%2E+Preprint%2C+arXiv%3A2407%2E21009%2E)
21. Qwen Team. 2024. Qwen2.5: A party of foundation models. Fang-Fang Zhao, Han-Jie He, Jia-Jian Liang, Jingyun Cen, Yun Wang, Hongjie Lin, Feifei Chen, Tai-Ping Li, Jian-Feng Yang, Lan Chen, and 1 others. 2025a. Benchmarking the performance of large language models in uveitis: a comparative analysis of… [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2E+Fang%2DFang+Zhao%2C+Han%2DJie+He%2C+Jia%2DJian+Liang%2C+Jingyun+Cen%2C+Yun+Wang%2C+Hongjie+Lin%2C+Feifei+Chen%2C+Tai%2DPing+Li%2C+Jian%2DFeng+Yang%2C+Lan+Chen%2C+and+1+others)
22. Yilun Zhao, Kaiyan Zhang, Tiansheng Hu, Sihong Wu, Ronan Le Bras, Taira Anderson, Jonathan Bragg, Joseph Chee Chang, Jesse Dodge, Matt Latzke, Yixin Liu, Charles McGrady, Xiangru Tang, Zihang Wang, Chen Zhao, Hannaneh Hajishirzi, Doug Downey, and Arman Cohan. 2025b. [Sciarena: An open evaluation plat…](https://md.research.elamur.ai/free/en/papers/zhao-2025-sciarena-open-evaluation-platform-non/)
23. Arkil Patel, Siva Reddy, and Dzmitry Bahdanau. 2025. How to get your llm to generate challenging problems for evaluation. Preprint, arXiv:2502.14678. [find](https://scholar.google.com/scholar?q=Arkil+Patel%2C+Siva+Reddy%2C+and+Dzmitry+Bahdanau%2E+2025%2E+How+to+get+your+llm+to+generate+challenging+problems+for+evaluation%2E+Preprint%2C+arXiv%3A2502%2E14678%2E)
24. José Pombal, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2025. Zero-shot benchmarking: A framework for flexible and scalable automatic evaluation of language models. Preprint, arXiv:2504.01001. — [arXiv:2504.01001](https://arxiv.org/abs/2504.01001) [find](https://scholar.google.com/scholar?q=Jos%C3%A9+Pombal%2C+Nuno+M%2E+Guerreiro%2C+Ricardo+Rei%2C+and+Andr%C3%A9+F%2E+T%2E+Martins%2E+2025%2E+Zero%2Dshot+benchmarking%3A+A+framework+for+flexible+and+scalable+automatic+evaluation+of+language+models%2E+Preprint%2C+arXiv%3A2504%2E)

*Updated 2 November 2026*