---
title: "Towards Knowledge Graph-Grounded Evaluation of Agentic"
description: "Agentic LLMs for offensive security are usually scored on whether they capture the flag, or against ad hoc checkpoints defined… 31 references."
authors: ["Daniel Schlör", "Marius Bohn", "Maximilian Wolf", "Kevin Bergner", "Christian Goldschmied", "Andreas Hotho"]
year: 2026
venue: "Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26"
source: "https://aclanthology.org/2026.kallm-1.15/"
doi: "10.63317/4ne74wm45iti"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schlor-2026-knowledge-graph-grounded-evaluation-agentic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schlor-2026-knowledge-graph-grounded-evaluation-agentic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges

# Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges

**Authors:** Daniel Schlör, Marius Bohn, Maximilian Wolf, Kevin Bergner, Christian Goldschmied and [Andreas Hotho](https://md.research.elamur.ai/free/en/authors/andreas-hotho/) · **Year:** 2026 · **Venue:** [Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26](https://md.research.elamur.ai/free/en/venues/kallm/) ([Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26 2026](https://md.research.elamur.ai/free/en/venues/kallm/2026/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.kallm-1.15/) · [PDF](https://aclanthology.org/2026.kallm-1.15.pdf) · [DOI 10.63317/4ne74wm45iti](https://doi.org/10.63317/4ne74wm45iti) · [OpenAlex](https://openalex.org/W7166575371)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Agentic LLMs for offensive security are usually scored on whether they capture the flag, or against ad hoc checkpoints defined per challenge. These methods say little about partial progress or about valid alternative solutions. The authors address this with BraceGreen, which turns human-written CTF writeups into knowledge graphs that serve as gold-standard rubrics.

Each attack step is a node tagged with:
- a MITRE ATT&CK tactic
- a natural-language goal
- a reference command and its expected output
- a semantic result

Edges capture attack order, equivalent alternative commands or multi-command workflows, data-flow prerequisites, and conditions that make a command a poor choice.

A LangGraph workflow uses an LLM-as-judge with chain-of-thought reasoning. The judge checks an agent's prediction against every alternative the graph encodes, not just one reference answer.

The released resource covers seven VulnHub machines and connects to live targets through virtual machines and an MCP server. The excerpt contains no quantitative evaluation results. Its main value is a standardized, reproducible schema for fine-grained, step-level assessment of security agents, independent of the agent framework used.

## Key points
- **Benchmark:** 7 VulnHub CTF machines with full knowledge graph annotations, released at github.com/LSX-UniWue/brace-ctf-data.
- **Graph schema:** RDF-compatible, with three node types (Step, Command, Command group).
  - Structural edges: `next_step`, `or` and `has_subcmd`.
  - Relations: `has_tactic`, `has_goal`, `has_cmd`, `has_output`, `has_result`, `requires`, `contraindicated_by` and `is_optional`.
- **Tactic coverage:** steps map to 10 MITRE ATT&CK tactic categories, from Reconnaissance and Discovery through Privilege Escalation, Lateral Movement and Command and Control.
- **Evaluation modes:** three modes (command prediction, goal inference, anticipated result), each scored by an LLM-as-judge inside a LangGraph pipeline.
- **Positioning:** unlike binary benchmarks (NYU CTF Bench), milestone-based ones (AutoPenBench, Cybench) or rubric trees (PentestJudge), BraceGreen encodes alternative solution paths and prerequisites explicitly. It is limited to technical attack steps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Würzburg](https://md.research.elamur.ai/free/en/universities/university-of-wurzburg/) — University, Würzburg, Germany · 8 works · 60 citations

Also on the byline: [Coburg University of Applied Sciences](https://ror.org/02p5hsv84).

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 64 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) · 192 works

## Cite

Cite as: Daniel Schlör, Marius Bohn, Maximilian Wolf, Kevin Bergner, Christian Goldschmied and Andreas Hotho. 2026. Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges. Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schlor2026knowledge,
  title = {{Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges}},
  author = {Daniel Schlör, Marius Bohn, Maximilian Wolf, Kevin Bergner, Christian Goldschmied and Andreas Hotho},
  booktitle = {Proceedings of the Knowledge Graphs and Large Language Models Workshop (KG-LLM) @ LREC26},
  year = {2026},
  doi = {10.63317/4ne74wm45iti},
  url = {https://doi.org/10.63317/4ne74wm45iti}
}
```

- [← previous Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice](https://md.research.elamur.ai/free/en/papers/wang-2026-ip-intelligence-benchmarking-large-language/)
- [next → Towards Linguistically-informed Representations for English as a Second or Foreign Language: Review, Construction and Application](https://md.research.elamur.ai/free/en/papers/li-2026-towards-linguistically-informed-representations-english/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/)

## References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. 2025. Enigma: Interactive tool… [find](https://scholar.google.com/scholar?q=Talor+Abramovich%2C+Meet+Udeshi%2C+Minghao+Shao%2C+Kilian+Lieret%2C+Haoran+Xi%2C+Kimberly+Milner%2C+Sofija+Jancheska%2C+John+Yang%2C+Carlos+E%2E+Jimenez%2C+Farshad+Khorrami%2C+Prashanth+Krishnamurthy%2C+Brendan+Dolan%2DGavitt%2C)
2. Razvan Beuran. 2025. Capture the flag platforms. In Cybersecurity Education and Training, pages 193–219. Springer. [find](https://scholar.google.com/scholar?q=Razvan+Beuran%2E+2025%2E+Capture+the+flag+platforms%2E+In+Cybersecurity+Education+and+Training%2C+pages+193%E2%80%93219%2E+Springer%2E)
3. Shane Caldwell, Max Harley, Michael Kouremetis, Vincent Abruzzo, and Will Pearce. 2025. PentestJudge: Judging Agent Behavior Against Operational Requirements. ArXiv:2508.02921 [cs]. — [arXiv:2508.02921](https://arxiv.org/abs/2508.02921) [find](https://scholar.google.com/scholar?q=Shane+Caldwell%2C+Max+Harley%2C+Michael+Kouremetis%2C+Vincent+Abruzzo%2C+and+Will+Pearce%2E+2025%2E+PentestJudge%3A+Judging+Agent+Behavior+Against+Operational+Requirements%2E+ArXiv%3A2508%2E02921+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2508%2E0292)
4. Brian Challita and Pierre Parrend. 2025. RedTeam-LLM: an Agentic AI framework for offensive security. ArXiv:2505.06913 [cs]. — [arXiv:2505.06913](https://arxiv.org/abs/2505.06913) [find](https://scholar.google.com/scholar?q=Brian+Challita+and+Pierre+Parrend%2E+2025%2E+RedTeam%2DLLM%3A+an+Agentic+AI+framework+for+offensive+security%2E+ArXiv%3A2505%2E06913+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2505%2E06913%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2505%2E06913%29)
5. Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. 2024. PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24), pages 847–… [find](https://scholar.google.com/scholar?q=Gelei+Deng%2C+Yi+Liu%2C+V%C3%ADctor+Mayoral%2DVilches%2C+Peng+Liu%2C+Yuekang+Li%2C+Yuan+Xu%2C+Tianwei+Zhang%2C+Yang+Liu%2C+Martin+Pinzger%2C+and+Stefan+Rass%2E+2024%2E+PentestGPT%3A+Evaluating+and+harnessing+large+language+models+f)
6. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. 2024. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144. — [arXiv:2404.08144](https://arxiv.org/abs/2404.08144) [find](https://scholar.google.com/scholar?q=Richard+Fang%2C+Rohan+Bindu%2C+Akul+Gupta%2C+and+Daniel+Kang%2E+2024%2E+Llm+agents+can+autonomously+exploit+one%2Dday+vulnerabilities%2E+arXiv+preprint+arXiv%3A2404%2E08144%2E+%E2%80%94+%5BarXiv%3A2404%2E08144%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
7. Rikhiya Ghosh, Hans-Martin von Stockhausen, Martin Schmitt, George Marica Vasile, Sanjeev Kumar Karn, and Oladimeji Farri. 2025. Cve-llm: Ontology-assisted automatic vulnerability evaluation using large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pag… [find](https://scholar.google.com/scholar?q=Rikhiya+Ghosh%2C+Hans%2DMartin+von+Stockhausen%2C+Martin+Schmitt%2C+George+Marica+Vasile%2C+Sanjeev+Kumar+Karn%2C+and+Oladimeji+Farri%2E+2025%2E+Cve%2Dllm%3A+Ontology%2Dassisted+automatic+vulnerability+evaluation+using+lar)
8. Yasod Ginige, Akila Niroshan, Sajal Jain, and Suranga Seneviratne. 2025. AutoPentester: An LLM Agent-based Framework for Automated Pentesting. In 2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pages 163–174, Guiyang, China. IEEE. [find](https://scholar.google.com/scholar?q=Yasod+Ginige%2C+Akila+Niroshan%2C+Sajal+Jain%2C+and+Suranga+Seneviratne%2E+2025%2E+AutoPentester%3A+An+LLM+Agent%2Dbased+Framework+for+Automated+Pentesting%2E+In+2025+IEEE+24th+International+Conference+on+Trust%2C+Secu)
9. Luca Gioacchini, Marco Mellia, Idilio Drago, Alexander Delsanto, Giuseppe Siracusano, and Roberto Bifulco. 2024. Autopenbench: Benchmarking generative agents for penetration testing. arXiv preprint arXiv:2410.03225. — [arXiv:2410.03225](https://arxiv.org/abs/2410.03225) [find](https://scholar.google.com/scholar?q=Luca+Gioacchini%2C+Marco+Mellia%2C+Idilio+Drago%2C+Alexander+Delsanto%2C+Giuseppe+Siracusano%2C+and+Roberto+Bifulco%2E+2024%2E+Autopenbench%3A+Benchmarking+generative+agents+for+penetration+testing%2E+arXiv+preprint+ar)
10. Zimo Ji, Daoyuan Wu, Wenyuan Jiang, Pingchuan Ma, Zongjie Li, and Shuai Wang. 2025. Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, pages 603–617, Taipei Taiwan. ACM. [find](https://scholar.google.com/scholar?q=Zimo+Ji%2C+Daoyuan+Wu%2C+Wenyuan+Jiang%2C+Pingchuan+Ma%2C+Zongjie+Li%2C+and+Shuai+Wang%2E+2025%2E+Measuring+and+Augmenting+Large+Language+Models+for+Solving+Capture%2Dthe%2DFlag+Challenges%2E+In+Proceedings+of+the+2025+A)
11. Mudita Khurana and Raunak Jain. 2025. SoK: Measuring What Matters for Closed-Loop Security Agents. ArXiv:2510.01654 [cs]. — [arXiv:2510.01654](https://arxiv.org/abs/2510.01654) [find](https://scholar.google.com/scholar?q=Mudita+Khurana+and+Raunak+Jain%2E+2025%2E+SoK%3A+Measuring+What+Matters+for+Closed%2DLoop+Security+Agents%2E+ArXiv%3A2510%2E01654+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2510%2E01654%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2510%2E01654%29)
12. He Kong, Die Hu, Jingguo Ge, Liangxiong Li, Tong Li, and Bingzhen Wu. 2025. Vulnbot: Autonomous penetration testing for a multi-agent collaborative framework. arXiv preprint arXiv:2501.13411. — [arXiv:2501.13411](https://arxiv.org/abs/2501.13411) [find](https://scholar.google.com/scholar?q=He+Kong%2C+Die+Hu%2C+Jingguo+Ge%2C+Liangxiong+Li%2C+Tong+Li%2C+and+Bingzhen+Wu%2E+2025%2E+Vulnbot%3A+Autonomous+penetration+testing+for+a+multi%2Dagent+collaborative+framework%2E+arXiv+preprint+arXiv%3A2501%2E13411%2E+%E2%80%94+%5BarXiv)
13. Knowledge Graph from Cyber Threat Intelligence Reports. ArXiv:2111.07093 [cs]. — [arXiv:2111.07093](https://arxiv.org/abs/2111.07093) [find](https://scholar.google.com/scholar?q=Knowledge+Graph+from+Cyber+Threat+Intelligence+Reports%2E+ArXiv%3A2111%2E07093+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2111%2E07093%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2111%2E07093%29)
14. Bruno Lourenço, Pedro Adão, João F. Ferreira, Mario Monteiro Marques, and Cátia Vaz. 2025. Structuring Security: A Survey of Cybersecurity Ontologies, Semantic Log Processing, and LLMs Application. ArXiv:2510.16610 [cs]. — [arXiv:2510.16610](https://arxiv.org/abs/2510.16610) [find](https://scholar.google.com/scholar?q=Bruno+Louren%C3%A7o%2C+Pedro+Ad%C3%A3o%2C+Jo%C3%A3o+F%2E+Ferreira%2C+Mario+Monteiro+Marques%2C+and+C%C3%A1tia+Vaz%2E+2025%2E+Structuring+Security%3A+A+Survey+of+Cybersecurity+Ontologies%2C+Semantic+Log+Processing%2C+and+LLMs+Application%2E+Ar)
15. Haokai Ma, Javier Yong, Yunshan Ma, Kuei Chen, Anis Yusof, Zhenkai Liang, and Ee-Chien Chang. 2025. AttackSeqBench: Benchmarking Large Language Models in Analyzing Attack Sequences within Cyber Threat Intelligence. ArXiv:2503.03170 [cs]. — [arXiv:2503.03170](https://arxiv.org/abs/2503.03170) [find](https://scholar.google.com/scholar?q=Haokai+Ma%2C+Javier+Yong%2C+Yunshan+Ma%2C+Kuei+Chen%2C+Anis+Yusof%2C+Zhenkai+Liang%2C+and+Ee%2DChien+Chang%2E+2025%2E+AttackSeqBench%3A+Benchmarking+Large+Language+Models+in+Analyzing+Attack+Sequences+within+Cyber+Threat)
16. MITRE Corporation. 2023. MITRE ATT&CK Framework. [https://attack.mitre.org/](https://attack.mitre.org/). Accessed: 2026-02-27. [↗](https://attack.mitre.org/)
17. Alessandro Palma and Marco Angelini. 2024. It is time to steer: A scalable framework for analysis-driven attack graph generation. In European Symposium on Research in Computer Security, pages 229–250. Springer. [find](https://scholar.google.com/scholar?q=Alessandro+Palma+and+Marco+Angelini%2E+2024%2E+It+is+time+to+steer%3A+A+scalable+framework+for+analysis%2Ddriven+attack+graph+generation%2E+In+European+Symposium+on+Research+in+Computer+Security%2C+pages+229%E2%80%93250%2E)
18. RDI Foundation. 2024. AgentBeats: Agent-to-Agent Protocol for Multi-Agent Systems. [https://github.com/RDI-Foundation/](https://github.com/RDI-Foundation/) AgentBeats. [↗](https://github.com/RDI-Foundation/)
19. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, et al. 2024. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. Advances in Neural Info… [find](https://scholar.google.com/scholar?q=Minghao+Shao%2C+Sofija+Jancheska%2C+Meet+Udeshi%2C+Brendan+Dolan%2DGavitt%2C+Kimberly+Milner%2C+Boyuan+Chen%2C+Max+Yin%2C+Siddharth+Garg%2C+Prashanth+Krishnamurthy%2C+Farshad+Khorrami%2C+et+al%2E+2024%2E+Nyu+ctf+bench%3A+A+scala)
20. Minghao Shao, Nanda Rani, Kimberly Milner, Haoran Xi, Meet Udeshi, Saksham Aggarwal, Venkata Sai Charan Putrevu, Sandeep K Shukla, Prashanth Krishnamurthy, Farshad Khorrami, et al. 2026. Towards effective offensive security llm agents: Hyperparameter tuning, llm as a judge, and a lightweight ctf ben… [find](https://scholar.google.com/scholar?q=Minghao+Shao%2C+Nanda+Rani%2C+Kimberly+Milner%2C+Haoran+Xi%2C+Meet+Udeshi%2C+Saksham+Aggarwal%2C+Venkata+Sai+Charan+Putrevu%2C+Sandeep+K+Shukla%2C+Prashanth+Krishnamurthy%2C+Farshad+Khorrami%2C+et+al%2E+2026%2E+Towards+effec)
21. Alyzia-Maria Konsta, Alberto Lluch Lafuente, Beatrice Spiga, and Nicola Dragoni. 2024. Survey: Automatic generation of attack trees and attack graphs. Computers & Security, 137:103602. [find](https://scholar.google.com/scholar?q=Alyzia%2DMaria+Konsta%2C+Alberto+Lluch+Lafuente%2C+Beatrice+Spiga%2C+and+Nicola+Dragoni%2E+2024%2E+Survey%3A+Automatic+generation+of+attack+trees+and+attack+graphs%2E+Computers+%26+Security%2C+137%3A103602%2E)
22. Minghao Shao, Nanda Rani, Kimberly Milner, Haoran Xi, Meet Udeshi, Saksham Aggarwal, Venkata Sai Charan Putrevu, Sandeep Kumar Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. 2025. Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as… — [arXiv:2508.05674](https://arxiv.org/abs/2508.05674) [find](https://scholar.google.com/scholar?q=Minghao+Shao%2C+Nanda+Rani%2C+Kimberly+Milner%2C+Haoran+Xi%2C+Meet+Udeshi%2C+Saksham+Aggarwal%2C+Venkata+Sai+Charan+Putrevu%2C+Sandeep+Kumar+Shukla%2C+Prashanth+Krishnamurthy%2C+Farshad+Khorrami%2C+Ramesh+Karri%2C+and+Muha)
23. Zhenyuan Li, Jun Zeng, Yan Chen, and Zhenkai Liang. 2022. [find](https://scholar.google.com/scholar?q=Zhenyuan+Li%2C+Jun+Zeng%2C+Yan+Chen%2C+and+Zhenkai+Liang%2E+2022%2E)
24. AttacKG: Constructing Technique Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. 2025. Pentestagent: Incorporating llm agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Secur… [find](https://scholar.google.com/scholar?q=AttacKG%3A+Constructing+Technique+Xiangmin+Shen%2C+Lingzhi+Wang%2C+Zhenyuan+Li%2C+Yan+Chen%2C+Wencheng+Zhao%2C+Dawei+Sun%2C+Jiashui+Wang%2C+and+Wei+Ruan%2E+2025%2E+Pentestagent%3A+Incorporating+llm+agents+to+automated+pene)
25. Leslie F. Sikos. 2023. Cybersecurity knowledge graphs. Knowledge and Information Systems, 65(9):3511–3531. [find](https://scholar.google.com/scholar?q=Leslie+F%2E+Sikos%2E+2023%2E+Cybersecurity+knowledge+graphs%2E+Knowledge+and+Information+Systems%2C+65%289%29%3A3511%E2%80%933531%2E)
26. VulnHub. 2026. VulnHub: Vulnerable By Design. [https://www.vulnhub.com/](https://www.vulnhub.com/). Accessed: 2026-02-27. [↗](https://www.vulnhub.com/)
27. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. 2024. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv preprint arXiv:2408.08926. [find](https://scholar.google.com/scholar?q=Andy+K+Zhang%2C+Neil+Perry%2C+Riya+Dulepet%2C+Joey+Ji%2C+Celeste+Menders%2C+Justin+W+Lin%2C+Eliot+Jones%2C+Gashon+Hussein%2C+Samantha+Liu%2C+Donovan+Jasper%2C+et+al%2E+2024%2E+Cybench%3A+A+framework+for+evaluating+cybersecurit)
28. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/).
29. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. 2025. Cve-bench: a benchmark for ai agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332. — [arXiv:2503.17332](https://arxiv.org/abs/2503.17332) [find](https://scholar.google.com/scholar?q=Yuxuan+Zhu%2C+Antony+Kellermann%2C+Dylan+Bowman%2C+Philip+Li%2C+Akul+Gupta%2C+Adarsh+Danda%2C+Richard+Fang%2C+Conner+Jensen%2C+Eric+Ihli%2C+Jason+Benn%2C+et+al%2E+2025%2E+Cve%2Dbench%3A+a+benchmark+for+ai+agents%E2%80%99+ability+to+expl)
30. Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. 2024. [Teams of llm agents can exploit zero-day vulnerabilities](https://md.research.elamur.ai/free/en/papers/zhu-2026-teams-llm-agents-can-exploit/). arXiv preprint arXiv:2406.01637.
31. Yuwen Zou, Jia Liu, and Wenjun Fan. 2026. Ctfa-gent: An llm-powered agent for ctf challenge solving. Journal of Information Security and Applications, 96:104305. [find](https://scholar.google.com/scholar?q=Yuwen+Zou%2C+Jia+Liu%2C+and+Wenjun+Fan%2E+2026%2E+Ctfa%2Dgent%3A+An+llm%2Dpowered+agent+for+ctf+challenge+solving%2E+Journal+of+Information+Security+and+Applications%2C+96%3A104305%2E)

*Updated 1 November 2026*