---
title: "Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents"
description: "The authors present Agentic CLEAR, an open-source framework that produces textual diagnostics at three levels: system… 23 references."
authors: ["Asaf Yehudai", "Lilach Eden", "Michal Shmueli-Scheuer"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
source: "https://aclanthology.org/2026.acl-demo.74/"
doi: "10.18653/v1/2026.acl-demo.74"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yehudai-2026-agentic-clear-automating-multi-level/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yehudai-2026-agentic-clear-automating-multi-level/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/) / Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

# Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

**Authors:** [Asaf Yehudai](https://md.research.elamur.ai/free/en/authors/asaf-yehudai/), [Lilach Eden](https://md.research.elamur.ai/free/en/authors/lilach-eden/) and [Michal Shmueli-Scheuer](https://md.research.elamur.ai/free/en/authors/michal-shmueli-scheuer/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-demo.74/) · [PDF](https://aclanthology.org/2026.acl-demo.74.pdf) · [DOI 10.18653/v1/2026.acl-demo.74](https://doi.org/10.18653/v1/2026.acl-demo.74) · [OpenAlex](https://openalex.org/W7166875709)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
LLM agents often fail in subtle ways, such as loops, misbehaving sub-agents, or errors that cascade across steps. These problems are hard to spot from final outputs alone. Observability tools like LangSmith and LangFuse record traces but provide only shallow evaluation, and published error taxonomies are fixed and cannot adapt to new tasks. The authors present Agentic CLEAR, an open-source framework that produces textual diagnostics at three levels: system, node, and trace.

The pipeline has two stages. First, an LLM judge scores and critiques each LLM call, critiques each trace as a whole, and checks the trace against rubrics it generates for the task. Second, the earlier CLEAR method clusters these critiques into recurring issues for each node and for the whole system. Results are explored through an interactive dashboard.

Across four benchmarks and seven agent configurations, the discovered issues correspond reasonably well to human-annotated TRAIL error categories, and the judge scores help predict whether a task succeeded. This gives developers a way to diagnose agents without writing tests or rubrics by hand.

## Key points
- Distributed as a PyPI package (`pip install clear-eval`, Apache 2.0). It reads OpenTelemetry/LangFuse-formatted traces, is configured through a YAML file, and runs with a single CLI command.
- Evaluation covered AppWorld (CUGA agent), GAIA, SWE-bench Verified Mini, and τ²-Bench, using GPT-5 and OSS-120B as judges.
- TRAIL alignment used 12 of 20 categories (the non-execution ones, which account for 94% of labels) across 117 traces:
  - GPT-5 reached macro-F1 0.459 and micro-F1 0.497.
  - A frequency-weighted random baseline reached micro-F1 0.459 but only 0.199 macro-F1.
  - GPT-5's 15 issues covered all 12 categories; OSS-120B's 12 issues covered 10.
- For predicting success (AUC), the trace-level score was generally the strongest of three methods (trace, rubric, step-wise).
  - On AppWorld every method exceeded 0.75, and GPT-5 reached at least 0.82.
  - On τ²-Bench no method went above 0.62.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 100 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 52 works

## Cite

Cite as: Asaf Yehudai, Lilach Eden and Michal Shmueli-Scheuer. 2026. Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yehudai2026agentic,
  title = {{Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents}},
  author = {Asaf Yehudai, Lilach Eden and Michal Shmueli-Scheuer},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-demo.74},
  url = {https://doi.org/10.18653/v1/2026.acl-demo.74}
}
```

- [← previous Agentic AI for Human Resources: LLM-Driven Candidate Assessment](https://md.research.elamur.ai/free/en/papers/yuksel-2026-agentic-ai-human-resources-llm/)
- [next → Agentic Context Strategies for Multi-Format Document Understanding: When Should Language Models Use Tools?](https://md.research.elamur.ai/free/en/papers/uniyal-2026-agentic-context-strategies-multi-format/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/21/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, and 12 others. 2026. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. In The Fourteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Liu%2C+Botao+Yu%2C+Amit+Arora%2C+Dongyoon+Hahm%2C+Harsh+Trivedi%2C+Huan+Sun%2C+and+12+others%2E+2026%2E+Holistic+agent+leaderboard%3A+The+missing+infrastructure+for+AI+agent+evaluation%2E+In+The+Fourteenth+International+)
2. Anthropic. 2025. Claude-code. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+Claude%2Dcode%2E)
3. Elron Bandel, Asaf Yehudai, Lilach Eden, Yehoshua Sagron, Yotam Perlitz, Elad Venezian, Natalia Razinkov, Natan Ergas, Shlomit Shachor Ifergan, Segev Shlomov, Michal Jacovi, Leshem Choshen, Liat Ein-Dor, Yoav Katz, and Michal Shmueli-Scheuer. 2026a. General agent evaluation. Preprint, arXiv:2602.229… — [arXiv:2602.22953](https://arxiv.org/abs/2602.22953) [find](https://scholar.google.com/scholar?q=Elron+Bandel%2C+Asaf+Yehudai%2C+Lilach+Eden%2C+Yehoshua+Sagron%2C+Yotam+Perlitz%2C+Elad+Venezian%2C+Natalia+Razinkov%2C+Natan+Ergas%2C+Shlomit+Shachor+Ifergan%2C+Segev+Shlomov%2C+Michal+Jacovi%2C+Leshem+Choshen%2C+Liat+Ein%2DD)
4. Elron Bandel, Asaf Yehudai, Alexandre Lacoste, Avijit Ghosh, Graham Neubig, Margaret Mitchell, Michal Shmueli-Scheuer, and Leshem Choshen. 2026b. Agentic systems should be general. SSRN Electronic Journal. [find](https://scholar.google.com/scholar?q=Elron+Bandel%2C+Asaf+Yehudai%2C+Alexandre+Lacoste%2C+Avijit+Ghosh%2C+Graham+Neubig%2C+Margaret+Mitchell%2C+Michal+Shmueli%2DScheuer%2C+and+Leshem+Choshen%2E+2026b%2E+Agentic+systems+should+be+general%2E+SSRN+Electronic+Jou)
5. Elron Bandel, Asaf Yehudai, and Michal Shmueli-Scheuer. 2026c. Ready for general agents? let’s test it. In ICLR Blogposts 2026. Https://iclr-blogposts.github.io/2026/blog/2026/general-agent-evaluation/. [find](https://scholar.google.com/scholar?q=Elron+Bandel%2C+Asaf+Yehudai%2C+and+Michal+Shmueli%2DScheuer%2E+2026c%2E+Ready+for+general+agents%3F+let%E2%80%99s+test+it%2E+In+ICLR+Blogposts+2026%2E+Https%3A%2F%2Ficlr%2Dblogposts%2Egithub%2Eio%2F2026%2Fblog%2F2026%2Fgeneral%2Dagent%2Devaluation)
6. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. [τ 2 -bench: Evaluating conversational agents in a dual-control environment](https://md.research.elamur.ai/free/en/papers/barres-2025-tau-2-bench-evaluating-conversational/). Preprint, arXiv:2506.07982.
7. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2026. [Why do multi-agent LLM systems fail?](https://md.research.elamur.ai/free/en/papers/cemri-2025-why-multi-agent-llm/) In The Thirty-ninth Annual Conference on Neural Inf…
8. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. 2025. Trail: Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638. [find](https://scholar.google.com/scholar?q=Darshan+Deshpande%2C+Varun+Gangal%2C+Hersh+Mehta%2C+Jitin+Krishnan%2C+Anand+Kannappan%2C+and+Rebecca+Qian%2E+2025%2E+Trail%3A+Trace+reasoning+and+agentic+issue+localization%2E+arXiv+preprint+arXiv%3A2505%2E08638%2E)
9. Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, and Asaf Yehudai. 2025. [Justrank: Benchmarking llm judges for system ranking](https://md.research.elamur.ai/free/en/papers/gera-2025-justrank-benchmarking-llm-judges-system/). Preprint, arXiv:2412.09569.
10. Alireza Ghafarollahi and Markus J Buehler. 2025. Sciagents: automating scientific discovery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials, 37(22):2413523. [find](https://scholar.google.com/scholar?q=Alireza+Ghafarollahi+and+Markus+J+Buehler%2E+2025%2E+Sciagents%3A+automating+scientific+discovery+through+bioinspired+multi%2Dagent+intelligent+graph+reasoning%2E+Advanced+Materials%2C+37%2822%29%3A2413523%2E)
11. Harbor Framework Team. 2026. Harbor: A framework for evaluating and optimizing agents and models in container environments. [find](https://scholar.google.com/scholar?q=Harbor+Framework+Team%2E+2026%2E+Harbor%3A+A+framework+for+evaluating+and+optimizing+agents+and+models+in+container+environments%2E)
12. Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko, Aman Jaiswal, Kusha Sareen, Shailesh Nanisetty, Joan Cabezas, Manuel Del Verme, Omar G. Younis, Simone Baratta, Matteo Avalle, Imene Kerboua, Xing Han Lù, Elron Bandel, Michal Shmueli-Scheuer, Asaf Yehudai, Leshem Choshen, Jonathan Lebensold, Sean… — [arXiv:2603.15798](https://arxiv.org/abs/2603.15798) [find](https://scholar.google.com/scholar?q=Alexandre+Lacoste%2C+Nicolas+Gontier%2C+Oleh+Shliazhko%2C+Aman+Jaiswal%2C+Kusha+Sareen%2C+Shailesh+Nanisetty%2C+Joan+Cabezas%2C+Manuel+Del+Verme%2C+Omar+G%2E+Younis%2C+Simone+Baratta%2C+Matteo+Avalle%2C+Imene+Kerboua%2C+Xing+H)
13. LangFuse. 2023. Langfuse: Observability for ai applications. [find](https://scholar.google.com/scholar?q=LangFuse%2E+2023%2E+Langfuse%3A+Observability+for+ai+applications%2E)
14. LangSmith. 2023. Langsmith: Evaluation framework for ai applications. [find](https://scholar.google.com/scholar?q=LangSmith%2E+2023%2E+Langsmith%3A+Evaluation+framework+for+ai+applications%2E)
15. Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stanczak, Peter Shaw, Christopher Pal, and Siva Reddy. 2025. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. In Second Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Xing+Han+L%C3%B9%2C+Amirhossein+Kazemnejad%2C+Nicholas+Meade%2C+Arkil+Patel%2C+Dongchan+Shin%2C+Alejandra+Zambrano%2C+Karolina+Stanczak%2C+Peter+Shaw%2C+Christopher+Pal%2C+and+Siva+Reddy%2E+2025%2E+Agentrewardbench%3A+Evaluating+)
16. Sami Marreed, Alon Oved, Avi Yaeli, Segev Shlomov, Ido Levy, Offer Akrabi, Aviad Sela, Asaf Adi, and Nir Mashkif. 2025. Towards enterprise-ready computer using generalist agent. Preprint, arXiv:2503.01861. — [arXiv:2503.01861](https://arxiv.org/abs/2503.01861) [find](https://scholar.google.com/scholar?q=Sami+Marreed%2C+Alon+Oved%2C+Avi+Yaeli%2C+Segev+Shlomov%2C+Ido+Levy%2C+Offer+Akrabi%2C+Aviad+Sela%2C+Asaf+Adi%2C+and+Nir+Mashkif%2E+2025%2E+Towards+enterprise%2Dready+computer+using+generalist+agent%2E+Preprint%2C+arXiv%3A2503%2E0)
17. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. [Gaia: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). Preprint, arXiv:2311.12983.
18. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, and more. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/).…
19. OpenAI. 2025. Chatgpt agent: bridging research and action. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Chatgpt+agent%3A+bridging+research+and+action%2E)
20. Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. 2025. Open-source DeepResearch – Freeing our search agents. Accessed: 2025-02-04. [find](https://scholar.google.com/scholar?q=Aymeric+Roucher%2C+Albert+Villanova+del+Moral%2C+Merve+Noyan%2C+Thomas+Wolf%2C+and+Cl%C3%A9mentine+Fourrier%2E+2025%2E+Open%2Dsource+DeepResearch+%E2%80%93+Freeing+our+search+agents%2E+Accessed%3A+2025%2D02%2D04%2E)
21. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) Preprint, arXiv:2310.06770.
22. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in neural information processing systems, 36:68539–68551.
23. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aide… — [OpenAI GPT-5 System Card](https://md.research.elamur.ai/free/en/papers/singh-2025-openai-gpt-5-system-card/)

*Updated 2 November 2026*