---
title: "Towards IP Intelligence — paper by Wang et al. (2026)"
description: "To fill this gap, the authors build a task taxonomy for the whole IP field based on Webb's Depth of Knowledge… 10.8% accuracy. 32 references."
authors: ["Qiyao Wang", "Guhong Chen", "Hongbo Wang", "Huaren Liu", "Minghui Zhu", "Zhifei Qin", "Li Linwei", "Yilin Yue", "Shiqiang Wang", "Jiayan Li", "Wu Yihang", "Ziqiang Liu", "Longze Chen", "Run Luo", "Liyang Fan", "Jiaming Li", "Lei Zhang", "Kan Xu", "Hamid Alinejad-Rokny", "Chengming Li", "Shiwen Ni", "Yuan Lin", "Min Yang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1000/"
doi: "10.18653/v1/2026.findings-acl.1000"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-ip-intelligence-benchmarking-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-ip-intelligence-benchmarking-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/) / Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice

# Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice

**Authors:** Qiyao Wang, Guhong Chen, [Hongbo Wang](https://md.research.elamur.ai/free/en/authors/hongbo-wang/), Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, [Run Luo](https://md.research.elamur.ai/free/en/authors/run-luo/), Liyang Fan, Jiaming Li, [Lei Zhang](https://md.research.elamur.ai/free/en/authors/lei-zhang/), Kan Xu, Hamid Alinejad-Rokny, [Chengming Li](https://md.research.elamur.ai/free/en/authors/chengming-li/), [Shiwen Ni](https://md.research.elamur.ai/free/en/authors/shiwen-ni/), Yuan Lin and [Min Yang](https://md.research.elamur.ai/free/en/authors/min-yang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1000/) · [PDF](https://aclanthology.org/2026.findings-acl.1000.pdf) · [DOI 10.18653/v1/2026.findings-acl.1000](https://doi.org/10.18653/v1/2026.findings-acl.1000) · [OpenAlex](https://openalex.org/W7166848692)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
Most existing resources for evaluating LLMs on intellectual property cover only patents or a few narrow tasks. They rarely test the legal reasoning and decision-making that real IP work requires. To fill this gap, the authors build a task taxonomy for the whole IP field based on Webb's Depth of Knowledge (DOK) theory. The taxonomy is organized into four cognitive levels, with Information Processing as the first, Logical Reasoning as the second, Discriminant Evaluation as the third, and Creative Generation as the fourth. On top of the taxonomy they release IPBench, a bilingual English–Chinese benchmark grounded in US and mainland Chinese law. It includes trademarks, copyrights and trade secrets alongside patents, and its task formats range from classification and multiple-choice questions to open-ended generation. The authors test 19 general-purpose, law-oriented and IP-specific chat and reasoning models in zero-shot, few-shot and chain-of-thought settings. Even the strongest model, DeepSeek-V3, reaches only 75.8%. Closed-source general models outperform open-source domain-specialized ones. The work offers the first broad, cross-jurisdictional testbed for IP-oriented LLMs and shows that current systems remain unreliable for practical IP work.

## Key points
- IPBench has 10,374 data points spanning 20 tasks and 8 IP mechanisms. The data, collected from public sources, and the code are released on GitHub (IPBench/IPBench).
- Fine-grained patent classification remains very hard. DeepSeek-R1 leads on IPC with 10.8% accuracy, and DeepSeek-V3 leads on CPC with 9.5%.
- The authors propose LLMScore, an LLM-as-a-judge metric for generation tasks scored on five dimensions. It correlates better with human judgments than BLEU, ROUGE-L or BERTScore (Spearman 0.47 vs. 0.35 for BLEU). Inter-annotator agreement among the human experts was high (Krippendorff's alpha 0.828).
- More few-shot examples usually raise scores, though Llama3.1-8B is an exception. Chain-of-thought prompting slightly lowers accuracy, by about 0.4–0.6%.
- A manual review of 300 wrong GPT-4o-mini answers sorted the mistakes into 7 error types. Reasoning errors are the most common (33%), followed by hallucinations (22%) and priority errors (18%).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/government/chinese-academy-of-sciences/) — Government body, Beijing, China · 194 works · 23,906 citations
- [University of Chinese Academy of Sciences](https://md.research.elamur.ai/free/en/universities/university-of-chinese-academy-of-sciences/) — University, Beijing, China · 147 works · 8,480 citations
- [UNSW Sydney](https://md.research.elamur.ai/free/en/universities/unsw-sydney/) — University, Sydney, Australia · 34 works · 1,509 citations
- [Shenzhen Institutes of Advanced Technology](https://md.research.elamur.ai/free/en/labs/shenzhen-institutes-of-advanced-technology/) — Research institute, Shenzhen, China · 17 works · 2,599 citations
- [Dalian University of Technology](https://md.research.elamur.ai/free/en/universities/dalian-university-of-technology/) — University, Dalian, China · 12 works · 46 citations
- [Shenzhen MSU-BIT University](https://md.research.elamur.ai/free/en/universities/shenzhen-msu-bit-university/) — University, Shenzhen, China · 3 works · 38 citations
- [Shenzhen University of Advanced Technology](https://md.research.elamur.ai/free/en/universities/shenzhen-university-of-advanced-technology/) — University, Shenzhen, China · 2 works · 1 citation

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [💼 Business, Management and Accounting](https://md.research.elamur.ai/free/en/shelves/business-management-and-accounting/) · 32 works · [💼 Management of Technology and Innovation](https://md.research.elamur.ai/free/en/shelves/business-management-and-accounting/management-of-technology-and-innovation/) · 6 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin and Min Yang. 2026. Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026ip,
  title = {{Towards IP Intelligence: Benchmarking Large Language Models on Intellectual Property Knowledge and Practice}},
  author = {Qiyao Wang, Guhong Chen, Hongbo Wang, Huaren Liu, Minghui Zhu, Zhifei Qin, Li Linwei, Yilin Yue, Shiqiang Wang, Jiayan Li, Wu Yihang, Ziqiang Liu, Longze Chen, Run Luo, Liyang Fan, Jiaming Li, Lei Zhang, Kan Xu, Hamid Alinejad-Rokny, Chengming Li, Shiwen Ni, Yuan Lin and Min Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1000},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1000}
}
```

- [← previous Towards Integrating Pose Estimation with Neuroimaging for the Analysis of Signed Language Video Stimuli](https://md.research.elamur.ai/free/en/papers/vandenitte-2026-integrating-pose-estimation-neuroimaging-analysis/)
- [next → Towards Knowledge Graph-Grounded Evaluation of Agentic LLMs on Cybersecurity Capture-the-Flag Challenges](https://md.research.elamur.ai/free/en/papers/schlor-2026-knowledge-graph-grounded-evaluation-agentic/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/46/)

## References

32 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zilong Bai, Ruiji Zhang, Linqing Chen, Qijun Cai, Yuan Zhong, Cong Wang, Yan Fang, Jie Fang, Jing Sun, Weikuan Wang, and 1 others. 2024. Patentgpt: A large language model for intellectual property. arXiv preprint arXiv:2404.18255. — [arXiv:2404.18255](https://arxiv.org/abs/2404.18255) [find](https://scholar.google.com/scholar?q=Zilong+Bai%2C+Ruiji+Zhang%2C+Linqing+Chen%2C+Qijun+Cai%2C+Yuan+Zhong%2C+Cong+Wang%2C+Yan+Fang%2C+Jie+Fang%2C+Jing+Sun%2C+Weikuan+Wang%2C+and+1+others%2E+2024%2E+Patentgpt%3A+A+large+language+model+for+intellectual+property%2E+ar)
2. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. [M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/). In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang…
3. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. [What disease does this patient have? a large-scale open domain question answering dataset from medical exams](https://md.research.elamur.ai/free/en/papers/jin-2020-disease-this-patient-have-large/). Applied Sciences, 11(14):6421.
4. DeepSeek-AI, Daya Guo, Dejian Yang, and Haowei Zhang et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). ArXiv, abs/2501.12948.
5. Valentin Knappich, Simon Razniewski, Anna Hätty, and Annemarie Friedrich. 2024. Pap2pat: Towards automated paper-to-patent drafting using chunk-based outline-guided generation. arXiv preprint arXiv:2410.07009. — [arXiv:2410.07009](https://arxiv.org/abs/2410.07009) [find](https://scholar.google.com/scholar?q=Valentin+Knappich%2C+Simon+Razniewski%2C+Anna+H%C3%A4tty%2C+and+Annemarie+Friedrich%2E+2024%2E+Pap2pat%3A+Towards+automated+paper%2Dto%2Dpatent+drafting+using+chunk%2Dbased+outline%2Dguided+generation%2E+arXiv+preprint+arXiv%3A24)
6. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, and Bingxuan Wang et al. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+and+Bingxuan+Wang+et+al%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+CoRR%2C+abs%2F2412%2E19437%2E)
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Abhishek Kadian et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). ArXiv, abs/2407.21783.
8. Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. 2025. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill? Preprint, arXiv:2504.06514. — [arXiv:2504.06514](https://arxiv.org/abs/2504.06514) [find](https://scholar.google.com/scholar?q=Chenrui+Fan%2C+Ming+Li%2C+Lichao+Sun%2C+and+Tianyi+Zhou%2E+2025%2E+Missing+premise+exacerbates+overthinking%3A+Are+reasoning+models+losing+critical+thinking+skill%3F+Preprint%2C+arXiv%3A2504%2E06514%2E+%E2%80%94+%5BarXiv%3A2504%2E06514%5D)
9. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2023. [Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/). Advances in Neural Informati…
10. Wanwei He, Jiabao Wen, Lei Zhang, Hao Cheng, Bowen Qin, Yunshui Li, Feng Jiang, Junying Chen, Benyou Wang, and Min Yang. 2023. Hanfei-1.0. [https://github.com/siat-nlp/HanFei](https://github.com/siat-nlp/HanFei). [↗](https://github.com/siat-nlp/HanFei)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations.
12. Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and Min Yang. 2024a. [E-EVAL: A comprehensive Chinese k-12 education evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/hou-2024-e-eval-comprehensive-chinese/). In Findings of the Association for Computational Linguist…
13. Jinchang Hou, Chang Ao, Haihong Wu, Xiangtao Kong, Zhigang Zheng, Daijia Tang, Chengming Li, Xiping Hu, Ruifeng Xu, Shiwen Ni, and 1 others. 2024b. [E-eval: A comprehensive chinese k-12 education evaluation benchmark for large language models](https://md.research.elamur.ai/free/en/papers/hou-2024-e-eval-comprehensive-chinese/). In ACL (Findings).
14. OpenAI Aaron Hurst, Adam Lerer, and Adam P. Goucher et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). ArXiv, abs/2410.21276.
15. Fengqing Jiang. 2024. Identifying and mitigating vulnerabilities in llm-integrated applications. Master’s thesis, University of Washington. [find](https://scholar.google.com/scholar?q=Fengqing+Jiang%2E+2024%2E+Identifying+and+mitigating+vulnerabilities+in+llm%2Dintegrated+applications%2E+Master%E2%80%99s+thesis%2C+University+of+Washington%2E)
16. Haitao Li, You Chen, Qingyao Ai, WU Yueyue, Ruizhe Zhang, and LIU Yiqun. 2024. Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. [find](https://scholar.google.com/scholar?q=Haitao+Li%2C+You+Chen%2C+Qingyao+Ai%2C+WU+Yueyue%2C+Ruizhe+Zhang%2C+and+LIU+Yiqun%2E+2024%2E+Lexeval%3A+A+comprehensive+chinese+legal+benchmark+for+evaluating+large+language+models%2E+In+The+Thirty%2Deight+Conference+on+)
17. Haitao Li, Jiaying Ye, Yiran Hu, Jia Chen, Qingyao Ai, Yueyue Wu, Junjie Chen, Yifan Chen, Cheng Luo, Quan Zhou, and 1 others. 2025. Casegen: A benchmark for multi-stage legal case documents generation. arXiv preprint arXiv:2502.17943. [find](https://scholar.google.com/scholar?q=Haitao+Li%2C+Jiaying+Ye%2C+Yiran+Hu%2C+Jia+Chen%2C+Qingyao+Ai%2C+Yueyue+Wu%2C+Junjie+Chen%2C+Yifan+Chen%2C+Cheng+Luo%2C+Quan+Zhou%2C+and+1+others%2E+2025%2E+Casegen%3A+A+benchmark+for+multi%2Dstage+legal+case+documents+generatio)
18. Percy Liang, Rishi Bommasani, and Tony Lee et al. 2022. Holistic evaluation of language models. CoRR, abs/2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+and+Tony+Lee+et+al%2E+2022%2E+Holistic+evaluation+of+language+models%2E+CoRR%2C+abs%2F2211%2E09110%2E)
19. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
20. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. [G-eval: NLG evaluation using gpt-4 with better human alignment](https://md.research.elamur.ai/free/en/papers/liu-2023-g-eval-nlg-evaluation-gpt/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
21. Shiwen Ni, Minghuan Tan, Yuelin Bai, Fuqiang Niu, Min Yang, Bowen Zhang, Ruifeng Xu, Xiaojun Chen, Chengming Li, and Xiping Hu. 2024. Mozip: A multilingual benchmark to evaluate large language models in intellectual property. In Proceedings of the 2024 Joint International Conference on Computational… [find](https://scholar.google.com/scholar?q=Shiwen+Ni%2C+Minghuan+Tan%2C+Yuelin+Bai%2C+Fuqiang+Niu%2C+Min+Yang%2C+Bowen+Zhang%2C+Ruifeng+Xu%2C+Xiaojun+Chen%2C+Chengming+Li%2C+and+Xiping+Hu%2E+2024%2E+Mozip%3A+A+multilingual+benchmark+to+evaluate+large+language+models+)
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
23. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
24. Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and Cassidy Hardin et al. 2024. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.00118. [find](https://scholar.google.com/scholar?q=Gemma+Team+Morgane+Riviere%2C+Shreya+Pathak%2C+Pier+Giuseppe+Sessa%2C+and+Cassidy+Hardin+et+al%2E+2024%2E+Gemma+2%3A+Improving+open+language+models+at+a+practical+size%2E+ArXiv%2C+abs%2F2408%2E00118%2E)
25. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2019%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+ArXiv%2C+abs%2F1904%2E09675%2E)
26. Mirac Suzgun, Luke Melas-Kyriazi, Suproteem Sarkar, Scott D Kominers, and Stuart Shieber. 2023. The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications. Advances in neural information processing systems, 36:57908–57946. [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Luke+Melas%2DKyriazi%2C+Suproteem+Sarkar%2C+Scott+D+Kominers%2C+and+Stuart+Shieber%2E+2023%2E+The+harvard+uspto+patent+dataset%3A+A+large%2Dscale%2C+well%2Dstructured%2C+and+multi%2Dpurpose+corpus+of+patent+app)
27. Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Y. Wong, and Simon See. 2025. The curse of cot: On the limitations of chain-of-thought in in-context learning. Preprint, arXiv:2504.05081. — [arXiv:2504.05081](https://arxiv.org/abs/2504.05081) [find](https://scholar.google.com/scholar?q=Tianshi+Zheng%2C+Yixiang+Chen%2C+Chengxi+Li%2C+Chunyang+Li%2C+Qing+Zong%2C+Haochen+Shi%2C+Baixuan+Xu%2C+Yangqiu+Song%2C+Ginny+Y%2E+Wong%2C+and+Simon+See%2E+2025%2E+The+curse+of+cot%3A+On+the+limitations+of+chain%2Dof%2Dthought+in+)
28. M-A-P Team, Xinrun Du, Yifan Yao, and Kaijing Ma et al. 2025. Supergpqa: Scaling llm evaluation across 285 graduate disciplines. Preprint, arXiv:2502.14739. [find](https://scholar.google.com/scholar?q=M%2DA%2DP+Team%2C+Xinrun+Du%2C+Yifan+Yao%2C+and+Kaijing+Ma+et+al%2E+2025%2E+Supergpqa%3A+Scaling+llm+evaluation+across+285+graduate+disciplines%2E+Preprint%2C+arXiv%3A2502%2E14739%2E)
29. You Zuo, Kim Gerdes, Éric Clergerie, and Benoît Sagot. 2024. PatentEval: Understanding errors in patent generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2687–27… [find](https://scholar.google.com/scholar?q=You+Zuo%2C+Kim+Gerdes%2C+%C3%89ric+Clergerie%2C+and+Beno%C3%AEt+Sagot%2E+2024%2E+PatentEval%3A+Understanding+errors+in+patent+generation%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapter+of+the+Associati)
30. Qwen Team. 2025b. Qwq-32b: Embracing the power of reinforcement learning. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2025b%2E+Qwq%2D32b%3A+Embracing+the+power+of+reinforcement+learning%2E)
31. Juanyan Wang, Sai Krishna Reddy Mudhiganti, and Manali Sharma. 2024a. Patentformer: A novel method to automate the generation of patent applications. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1361–1380. [find](https://scholar.google.com/scholar?q=Juanyan+Wang%2C+Sai+Krishna+Reddy+Mudhiganti%2C+and+Manali+Sharma%2E+2024a%2E+Patentformer%3A+A+novel+method+to+automate+the+generation+of+patent+applications%2E+In+Proceedings+of+the+2024+Conference+on+Empirical)
32. Qiyao Wang, Jianguo Huang, Shule Lu, Yuan Lin, Kan Xu, Liang Yang, and Hongfei Lin. 2024b. Ipeval: A bilingual intellectual property agency consultation evaluation benchmark for large language models. arXiv preprint arXiv:2406.12386. — [arXiv:2406.12386](https://arxiv.org/abs/2406.12386) [find](https://scholar.google.com/scholar?q=Qiyao+Wang%2C+Jianguo+Huang%2C+Shule+Lu%2C+Yuan+Lin%2C+Kan+Xu%2C+Liang+Yang%2C+and+Hongfei+Lin%2E+2024b%2E+Ipeval%3A+A+bilingual+intellectual+property+agency+consultation+evaluation+benchmark+for+large+language+models%2E)

*Updated 1 November 2026*