---
title: "Qwen3 Technical Report — Yang 2025"
description: "Qwen3 is the Qwen Team's open-weight family of large language models."
authors: ["An Yang", "Anfeng Li", "Baosong Yang", "Beichen Zhang", "Binyuan Hui", "Bo Zheng", "Bowen Yu", "Chang Gao", "Chengen Huang", "Chenxu Lv", "Chujie Zheng", "Dayiheng Liu", "Fan Zhou", "Fei Huang", "Feng Hu", "Hao Ge", "Haoran Wei", "Huan Lin", "Jialong Tang", "Jian Yang", "Jianhong Tu", "Jianwei Zhang", "Jianxin Yang", "Jiaxi Yang", "Jing Zhou", "Jingren Zhou", "Junyang Lin", "Kai Dang", "Keqin Bao", "Kexin Yang", "Le Yu", "Lianghao Deng", "Mei Li", "Mingfeng Xue", "Mingze Li", "Pei Zhang", "Peng Wang", "Qin Zhu", "Rui Men", "Ruize Gao", "Shixuan Liu", "Shuang Luo", "Tianhao Li", "Tianyi Tang", "Wenbiao Yin", "Xingzhang Ren", "Xinyu Wang", "Xinyu Zhang", "Xuancheng Ren", "Yang Fan", "Yang Su", "Yichang Zhang", "Yinger Zhang", "Yu Wan", "Yuqiong Liu", "Zekun Wang", "Zeyu Cui", "Zhenru Zhang", "Zhipeng Zhou", "Zihan Qiu"]
year: 2025
venue: "arXiv:2505.09388"
source: "https://arxiv.org/abs/2505.09388"
doi: "10.48550/arxiv.2505.09388"
arxiv: "2505.09388"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Qwen3 Technical Report

# Qwen3 Technical Report

**Authors:** [An Yang](https://md.research.elamur.ai/free/en/authors/an-yang/), Anfeng Li, [Baosong Yang](https://md.research.elamur.ai/free/en/authors/baosong-yang/), [Beichen Zhang](https://md.research.elamur.ai/free/en/authors/beichen-zhang/), [Binyuan Hui](https://md.research.elamur.ai/free/en/authors/binyuan-hui/), [Bo Zheng](https://md.research.elamur.ai/free/en/authors/bo-zheng/), [Bowen Yu](https://md.research.elamur.ai/free/en/authors/bowen-yu/), [Chang Gao](https://md.research.elamur.ai/free/en/authors/chang-gao/), [Chengen Huang](https://md.research.elamur.ai/free/en/authors/chengen-huang/), [Chenxu Lv](https://md.research.elamur.ai/free/en/authors/chenxu-lv/), [Chujie Zheng](https://md.research.elamur.ai/free/en/authors/chujie-zheng/), [Dayiheng Liu](https://md.research.elamur.ai/free/en/authors/dayiheng-liu/) et al. (60 authors) · **Year:** 2025 · **Venue:** arXiv:2505.09388 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2505.09388) · [PDF](https://arxiv.org/pdf/2505.09388) · [DOI 10.48550/arxiv.2505.09388](https://doi.org/10.48550/arxiv.2505.09388) · [arXiv 2505.09388](https://arxiv.org/abs/2505.09388) · [OpenAlex](https://openalex.org/W4414939355)

**Citations:** 146 citations (openalex-snapshot · 2026-10-14) · [3126](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Qwen3 is the Qwen Team's open-weight family of large language models. It aims to combine strong reasoning, efficient inference and wide language coverage in one release. The family has six dense models (0.6B–32B) and two Mixture-of-Experts models. The flagship, Qwen3-235B-A22B, uses 22B of its 235B parameters for each token. The main design choice is that one model can either reason step by step (thinking mode) or answer directly (non-thinking mode). The user picks the mode through the chat template, so separate chat and reasoning models are no longer needed. A thinking budget sets how many reasoning tokens the model may spend. Pre-training used about 36 trillion tokens in 119 languages and dialects, ran in three stages and ended with long-context training. The data was expanded with text extracted from PDFs by Qwen2.5-VL and with synthetic data from Qwen2.5-Math and Qwen2.5-Coder. Post-training has four steps: long-CoT cold start, reasoning RL, fusing the two modes, and general RL. The smaller models are built more cheaply by distilling from the larger ones. Qwen3 competes with much larger and proprietary systems on math, coding and agent benchmarks. Its Apache 2.0 license makes it a practical base for work in less-resourced languages such as Ukrainian.

## Key points
- **Models:** dense Qwen3-0.6B, 1.7B, 4B, 8B, 14B and 32B, plus MoE Qwen3-30B-A3B and Qwen3-235B-A22B. The MoE models have 128 experts with 8 active per token and no shared experts. QK-Norm replaces the QKV-bias used in Qwen2. The tokenizer is byte-level BPE with 151,669 entries, and context length reaches 128K.
- **Data and pre-training:** about 36T tokens covering 119 languages, up from 29 in Qwen2.5. An annotation system labelled over 30T tokens so the data mix could be tuned for individual documents rather than whole sources. Stage S1 used about 30T tokens at sequence length 4,096. Stage S2 added more STEM, code and reasoning data. Stage S3 extended the context length to 32,768.
- **Headline scores (Qwen3-235B-A22B):** AIME'24 85.7, AIME'25 81.5, LiveCodeBench v5 70.7, CodeForces rating 2,056 and BFCL v3 70.8. Larger thinking budgets steadily improved scores across tasks.
- **Small models:** strong-to-weak distillation (off-policy and on-policy) beat direct RL for the small models, giving better results for less compute.
- **Multilingual evaluation:** the models were tested on MMMLU, INCLUDE, MT-AIME24, PolyMath and MLogiQA. They were also tested on Belebele, whose supported languages include Ukrainian (ukr Cyrl). On the Indo-European group, Qwen3-32B in thinking mode scored 90.7, against 89.2 for Gemma-3-27B-IT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

3126 papers of our library cite it (of the 140673 whose bibliographies we read); 78 of them have a page here, 3048 not yet.

- [Do Diacritics Matter? Evaluating the Impact of Arabic Diacritics on Tokenization and LLM Benchmarks](https://md.research.elamur.ai/free/en/papers/aclbulk-2026-inoue-do-diacritics-matter-evaluating-impact/)
- [Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation](https://md.research.elamur.ai/free/en/papers/alqadi-2026-arabic-benchmarks-reliable-qimma-s/)
- [Babies Learn to Look Ahead: Multi-Token Prediction in Small LMs](https://md.research.elamur.ai/free/en/papers/aynetdinov-2025-babies-learn-look-ahead-multi/)
- [MAWARITH: A Dataset and Benchmark for Legal Inheritance Reasoning with LLMs](https://md.research.elamur.ai/free/en/papers/bouchekif-2026-mawarith-dataset-benchmark-legal-inheritance/)
- [Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://md.research.elamur.ai/free/en/papers/cui-2023-chatlaw-multi-agent-legal-assistant/)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-v3-2-pushing/)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/)
- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://md.research.elamur.ai/free/en/papers/glm-2025-4-glm-4-5-agentic/)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://md.research.elamur.ai/free/en/papers/glm-2026-5-glm-5-vibe-coding/)
- [AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic](https://md.research.elamur.ai/free/en/papers/iacobacci-2026-aradynfact-dynamic-evaluation-factual-knowledge/)
- [Arabic Morphosyntactic Tagging and Dependency Parsing with Large Language Models](https://md.research.elamur.ai/free/en/papers/langstream-2026-adel-arabic-morphosyntactic-tagging-dependency-parsing/)
- [When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates](https://md.research.elamur.ai/free/en/papers/langstream-2026-liang-similar-means-different-evaluating-llms/)
- [Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs](https://md.research.elamur.ai/free/en/papers/langstream-2026-shmidman-dicta-lm-3-0-advancing/)
- [The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs Using Indian Riddles](https://md.research.elamur.ai/free/en/papers/m-2026-riddle-reflection-evaluating-reasoning-self/)
- [HistoryBankQA: Multilingual Temporal Question Answering on Historical Events](https://md.research.elamur.ai/free/en/papers/mandal-2026-historybankqa-multilingual-temporal-question-answering/)
- [Rethinking Gender Annotation for Bias Evaluation in Machine Translation: Can LLMs Improve Reliability?](https://md.research.elamur.ai/free/en/papers/manna-2026-rethinking-gender-annotation-bias-evaluation/)
- [Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus](https://md.research.elamur.ai/free/en/papers/mannion-2026-is-biomedical-specialization-still-worth/)
- [Breaking the Benchmark: Revealing LLM Bias via Minimal Contextual Augmentation](https://md.research.elamur.ai/free/en/papers/miandoab-2026-breaking-benchmark-revealing-llm-bias/)
- [Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?](https://md.research.elamur.ai/free/en/papers/modica-2026-do-factual-recall-mechanisms-carry/)
- [Structured Legal Document Generation in India: A Model-Agnostic Wrapper Approach with VidhikDastaavej](https://md.research.elamur.ai/free/en/papers/nigam-2026-structured-legal-document-generation-india/)
- [InstructSum: A Benchmark to Evaluate Instruction-Following Capability of Large Language Models in Summarization](https://md.research.elamur.ai/free/en/papers/nishida-2026-instructsum-benchmark-evaluate-instruction-following/)
- [EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions](https://md.research.elamur.ai/free/en/papers/noel-2026-eiffel-novel-benchmark-measure-bias/)
- [Push and Pull: Training Sentence Encoders with Contrastive Losses for Distance-Based Multi-Label Text Classification](https://md.research.elamur.ai/free/en/papers/nooten-2026-push-pull-training-sentence-encoders/)
- [Data-Centric Approach at the LoResMT 2026 Turkic Translation Challenge: Russian-Kyrgyz](https://md.research.elamur.ai/free/en/papers/novokshanov-2026-data-centric-approach-loresmt-2026/)
- [From Bones to Rocks: A Systematic Evaluation of Specialized Definition Generation for Portuguese](https://md.research.elamur.ai/free/en/papers/nunes-2026-bones-rocks-systematic-evaluation-specialized/)
- [Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation](https://md.research.elamur.ai/free/en/papers/ojastu-2026-estonian-winogrande-dataset-comparative-analysis/)
- [MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments](https://md.research.elamur.ai/free/en/papers/olsen-2026-muc-4-revisited-document-level/)
- [A Pocket Offline Model for Simultaneous Speech Translation as CUNI Submission to IWSLT 2026](https://md.research.elamur.ai/free/en/papers/ortega-2026-pocket-offline-model-simultaneous-speech/)
- [Interlinear Glosses as a Multilingual Pivot for Machine Translation: An Updated Study on Turkish with Restricted Resources](https://md.research.elamur.ai/free/en/papers/ozer-2026-interlinear-glosses-as-multilingual-pivot/)
- [S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature](https://md.research.elamur.ai/free/en/papers/pardo-2026-s-vocal-dataset-evaluation-framework/)
- [Toward Equitable Machine Translation for Atypical Speech: An LLM Post-Correction Approach](https://md.research.elamur.ai/free/en/papers/pasion-2026-equitable-machine-translation-atypical-speech/)
- [Parallel Corpora of Scholarly Documents for English-French Machine Translation](https://md.research.elamur.ai/free/en/papers/peng-2026-parallel-corpora-scholarly-documents-english/)
- [SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series](https://md.research.elamur.ai/free/en/papers/pennec-2026-sagaqa-multi-hop-reasoning-benchmark/)
- [Judging Instruction Responses in a Low-Resource Language: A Case Study on Basque](https://md.research.elamur.ai/free/en/papers/ponce-2026-judging-instruction-responses-low-resource/)
- [LST at MWE-2026 AdMIRe 2: Advancing Multimodal Idiomaticity Representation](https://md.research.elamur.ai/free/en/papers/qiu-2026-lst-mwe-2026-admire-2/)
- [Detecting Hallucinations in Authentic LLM–Human Interactions](https://md.research.elamur.ai/free/en/papers/ren-2026-detecting-hallucinations-authentic-llm-human/)
- [From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence](https://md.research.elamur.ai/free/en/papers/sahitaj-2026-articles-premises-building-primefacts-extraction/)
- [LegalSim-PT: Building a Dataset for Legal Document Simplification in Portuguese Leveraging Linguistic Metrics](https://md.research.elamur.ai/free/en/papers/scalercio-2026-legalsim-pt-building-dataset-legal/)
- [Multi-Agent Debate for Machine Translation: A Case Study on English-Japanese Translation](https://md.research.elamur.ai/free/en/papers/shen-2026-multi-agent-debate-machine-translation/)
- [STAR-IL: A Dataset for Style-Aware Machine Translation of Product Reviews in Indian Languages](https://md.research.elamur.ai/free/en/papers/shetye-2026-star-il-dataset-style-aware/)
- [Śmigiel Dataset: Laying Foundations for Investigating Machine-Generated Text Detection in Polish](https://md.research.elamur.ai/free/en/papers/strebeyko-2026-smigiel-dataset-laying-foundations-investigating/)
- [Evaluating Small Language Models for English-to-Portuguese Translation: Impact of Model Scale and Quantization](https://md.research.elamur.ai/free/en/papers/tamiosso-2026-evaluating-small-language-models-english/)
- [大模型参数高效微调方法综述: 技术、趋势与挑战](https://md.research.elamur.ai/free/en/papers/tang-2026-parameter-efficient-fine/)
- [AmDi - Ambiguous Words Diachronic Dataset](https://md.research.elamur.ai/free/en/papers/thielen-2026-amdi-ambiguous-words-diachronic-dataset/)
- [Simulating Student Interactions for Virtual Pretesting with In-Context Learning](https://md.research.elamur.ai/free/en/papers/thuy-2026-simulating-student-interactions-virtual-pretesting/)
- [Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation](https://md.research.elamur.ai/free/en/papers/tian-2026-beyond-literal-mapping-benchmarking-improving/)
- [The Moral Foundations Reddit Corpus](https://md.research.elamur.ai/free/en/papers/trager-2026-moral-foundations-reddit-corpus/)
- [Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models](https://md.research.elamur.ai/free/en/papers/tuck-2026-orthographic-constraint-satisfaction-human-difficulty/)
- [Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning](https://md.research.elamur.ai/free/en/papers/vaishnav-2026-symbolic-grounding-reveals-representational-bottlenecks/)
- [Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights](https://md.research.elamur.ai/free/en/papers/valero-2026-merge-conquer-instructing-multilingual-models/)
- [Sema System for the DMR 2026 Shared Task: Multistage UMR Parsing with Qwen3-4B](https://md.research.elamur.ai/free/en/papers/vergnette-2026-sema-system-dmr-2026-shared/)
- [Semantic Parsing for Evaluating Large Language Models: Separating Linguistic Abilities with YARN](https://md.research.elamur.ai/free/en/papers/vergnette-2026-semantic-parsing-evaluating-large-language/)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://md.research.elamur.ai/free/en/papers/wang-2025-80-20-rule-high-entropy/)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://md.research.elamur.ai/free/en/papers/wang-2025-internvl3-5-advancing-open/)
- [高效提升多模态大语言模型推理能力的级联强化学习策略](https://md.research.elamur.ai/free/en/papers/wang-2026-cascade-reinforcement-learning/)
- [On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation](https://md.research.elamur.ai/free/en/papers/wang-2026-temperature-constrained-non-deterministic-machine/)
- [NegNLI-BR: A Brazilian Portuguese Benchmark for Negation in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/westhelle-2026-negnli-br-brazilian-portuguese-benchmark/)
- [The Surge of Anti-Semitism in German Social Media following the October 7 Attacks](https://md.research.elamur.ai/free/en/papers/wiedemann-2026-surge-anti-semitism-german-social/)
- [Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark](https://md.research.elamur.ai/free/en/papers/wu-2026-detecting-latin-historical-books-large/)
- [Test-Time Adaptation of an Offline Multimodal Foundation Model for Simultaneous Speech Translation](https://md.research.elamur.ai/free/en/papers/xing-2026-test-time-adaptation-offline-multimodal/)
- [Qwen3-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen3-omni-technical-report/)
- [Paraphrasing as Zero-shot Translation with Feature-guided Diversity Enhancement](https://md.research.elamur.ai/free/en/papers/yan-2026-paraphrasing-as-zero-shot-translation/)
- [Rhetorical Questions in LLM Representations: A Linear Probing Study](https://md.research.elamur.ai/free/en/papers/yao-2026-rhetorical-questions-llm-representations-linear/)
- [Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation](https://md.research.elamur.ai/free/en/papers/yuan-2026-culture-aware-machine-translation-large/)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/)
- [ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification](https://md.research.elamur.ai/free/en/papers/zaghouani-2026-arabicdialectsafety-dialect-aware-benchmark-arabic/)
- [PEI at #SMM4H-HeaRD 2026: Enhancing Patient Metadata Detection via Hypothesis-Conditioned Classification and Paraphrase-Based Data Augmentation](https://md.research.elamur.ai/free/en/papers/zeidi-2026-pei-smm4h-heard-2026-enhancing/)
- [Can You Be More Explicit? A Task and Dataset on Explicitations of Implicit Meaning](https://md.research.elamur.ai/free/en/papers/zeidler-2026-can-you-be-more-explicit/)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-qwen3-embedding-advancing-text/)
- [Dynamic Meta-Metrics: Source-Sentence Conditioned Weighting for MT Evaluation](https://md.research.elamur.ai/free/en/papers/zhang-2026-dynamic-meta-metrics-source-sentence/)
- [Redefining Machine Simultaneous Interpretation: From Incremental Translation to Human-Like Strategies](https://md.research.elamur.ai/free/en/papers/zhang-2026-redefining-machine-simultaneous-interpretation-incremental/)
- [The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks](https://md.research.elamur.ai/free/en/papers/zhao-2026-mark-fades-adaptive-evolutionary-paraphrase/)
- [Group Sequence Policy Optimization](https://md.research.elamur.ai/free/en/papers/zheng-2025-group-sequence-policy-optimization/)
- [Simplify-Pro: A Two-level and Progressive LLM-based Framework for Auto Long Text Simplification](https://md.research.elamur.ai/free/en/papers/zhou-2026-simplify-pro-two-level-progressive/)
- [Encoding Logical Relations of Chinese Complex Sentences within the Universal Dependencies Framework](https://md.research.elamur.ai/free/en/papers/zhu-2026-encoding-logical-relations-chinese-complex/)
- [SPARTA: Evaluating Reasoning Segmentation Robustness through Black-Box Adversarial Paraphrasing in Text Autoencoder Latent Space](https://md.research.elamur.ai/free/en/papers/zinkovich-2026-sparta-evaluating-reasoning-segmentation-robustness/)
- [SemBench: A Universal Semantic Framework for LLM Evaluation](https://md.research.elamur.ai/free/en/papers/zubillaga-2026-sembench-universal-semantic-framework-llm/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2025qwen3,
  title = {{Qwen3 Technical Report}},
  author = {Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and Zheng, Chujie and Liu, Dayiheng and Zhou, Fan and Huang, Fei and Hu, Feng and Ge, Hao and Wei, Haoran and Lin, Huan and Tang, Jialong and Yang, Jian and Tu, Jianhong and Zhang, Jianwei and Yang, Jianxin and Yang, Jiaxi and Zhou, Jing and Zhou, Jingren and Lin, Junyang and Dang, Kai and Bao, Keqin and Yang, Kexin and Yu, Le and Deng, Lianghao and Li, Mei and Xue, Mingfeng and Li, Mingze and Zhang, Pei and Wang, Peng and Zhu, Qin and Men, Rui and Gao, Ruize and Liu, Shixuan and Luo, Shuang and Li, Tianhao and Tang, Tianyi and Yin, Wenbiao and Ren, Xingzhang and Wang, Xinyu and Zhang, Xinyu and Ren, Xuancheng and Fan, Yang and Su, Yang and Zhang, Yichang and Zhang, Yinger and Wan, Yu and Liu, Yuqiong and Wang, Zekun and Cui, Zeyu and Zhang, Zhenru and Zhou, Zhipeng and Qiu, Zihan},
  year = {2025},
  doi = {10.48550/arxiv.2505.09388},
  eprint = {2505.09388},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.09388}
}
```

- [← previous Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://md.research.elamur.ai/free/en/papers/zhang-2025-qwen3-embedding-advancing-text/)
- [next → Qwen3-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen3-omni-technical-report/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

73 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. [find](https://scholar.google.com/scholar?q=Marah+Abdin%2C+Jyoti+Aneja%2C+Harkirat+Behl%2C+Se%CC%81bastien+Bubeck%2C+Ronen+Eldan%2C+Suriya+Gunasekar%2C+Michael+Harrison%2C+Russell+J+Hewett%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+et+al%2E+Phi%2D4+technical+report%2E+arXiv+p) [our card](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/)
2. AIME. AIME problems and solutions, 2025. URL [https://artofproblemsolving.com/wiki/index.p](https://artofproblemsolving.com/wiki/index.p) hp/AIME Problems and Solutions. [↗](https://artofproblemsolving.com/wiki/index.p)
3. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query Transformer models from multi-head checkpoints. In EMNLP, pp. 4895–4901. Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Joshua+Ainslie%2C+James+Lee%2DThorp%2C+Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Federico+Lebro%CC%81n%2C+and+Sumit+Sanghai%2E+GQA%3A+Training+generalized+multi%2Dquery+Transformer+models+from+multi%2Dhead+checkpoints%2E+In+EMNLP%2C) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
4. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models. CoRR, abs/2402.17463, 2024. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+CoRR%2C+abs%2F2402%2E17463%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E17463%5D)
5. Anthropic. Claude 3.7 Sonnet, 2025. URL [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet). [↗](https://www.anthropic.com/news/claude-3-7-sonnet)
6. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+I%2E+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+J%2E+Cai%2C+Michael+Terry%2C+Quoc+V%2E+Le%2C+and+Charles+Sutton%2E+Program+synthesis+with+large+la) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
7. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Yunfei+Chu%2C+Zeyu+Cui%2C+Kai+Dang%2C+Xiaodong+Deng%2C+Yang+Fan%2C+Wenbin+Ge%2C+Yu+Han%2C+Fei+Huang%2C+Binyuan+Hui%2C+Luo+Ji%2C+Mei+Li%2C+Junyang+Lin%2C+Runji+Lin%2C+Dayiheng+Liu%2C+Gao+Liu%2C+Chengqiang+Lu%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
8. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923, 2025. [find](https://scholar.google.com/scholar?q=Shuai+Bai%2C+Keqin+Chen%2C+Xuejing+Liu%2C+Jialin+Wang%2C+Wenbin+Ge%2C+Sibo+Song%2C+Kai+Dang%2C+Peng+Wang%2C+Shijie+Wang%2C+Jun+Tang%2C+et+al%2E+Qwen2%2E5%2DVL+technical+report%2E+arXiv+preprint+arXiv%3A2502%2E13923%2C+2025%2E) [our card](https://md.research.elamur.ai/free/en/papers/bai-2025-qwen2-5-vl-technical/)
9. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The Belebele benchmark: A parallel reading comprehension dataset in 122 language variants. CoRR, abs/2308.16884, 2023. [find](https://scholar.google.com/scholar?q=Lucas+Bandarkar%2C+Davis+Liang%2C+Benjamin+Muller%2C+Mikel+Artetxe%2C+Satya+Narayan+Shukla%2C+Donald+Husa%2C+Naman+Goyal%2C+Abhinandan+Krishnan%2C+Luke+Zettlemoyer%2C+and+Madian+Khabsa%2E+The+Belebele+benchmark%3A+A+parall) [our card](https://md.research.elamur.ai/free/en/papers/bandarkar-2023-belebele-benchmark-parallel-reading-comprehension/)
10. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
11. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation.… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sydney+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q%2E+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+)
12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mi… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde%CC%81+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harrison+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+Training+ve) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
14. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models.… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
15. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 933–941. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Yann+N%2E+Dauphin%2C+Angela+Fan%2C+Michael+Auli%2C+and+David+Grangier%2E+Language+modeling+with+gated+convolutional+networks%2E+In+ICML%2C+volume+70+of+Proceedings+of+Machine+Learning+Research%2C+pp%2E+933%E2%80%93941%2E+PMLR%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
16. Google DeepMind. Gemini 2.5, 2025. URL [https://blog.google/technology/google-deepmind/gemi](https://blog.google/technology/google-deepmind/gemi) ni-model-thinking-updates-march-2025/. [↗](https://blog.google/technology/google-deepmind/gemi)
17. Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puig… [find](https://scholar.google.com/scholar?q=Mostafa+Dehghani%2C+Josip+Djolonga%2C+Basil+Mustafa%2C+Piotr+Padlewski%2C+Jonathan+Heek%2C+Justin+Gilmer%2C+Andreas+Peter+Steiner%2C+Mathilde+Caron%2C+Robert+Geirhos%2C+Ibrahim+Alabdulmohsin%2C+Rodolphe+Jenatton%2C+Lucas+B)
18. Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502.14739, 2025. [find](https://scholar.google.com/scholar?q=Xinrun+Du%2C+Yifan+Yao%2C+Kaijing+Ma%2C+Bingli+Wang%2C+Tianyu+Zheng%2C+King+Zhu%2C+Minghao+Liu%2C+Yiming+Liang%2C+Xiaolong+Jin%2C+Zhenlin+Wei%2C+et+al%2E+SuperGPQA%3A+Scaling+LLM+evaluation+across+285+graduate+disciplines%2E+a)
19. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez… [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anthony+Hartshorn%2C+Aobo+Yang%2C+Archi+M)
20. Simin Fan, Matteo Pagliardini, and Martin Jaggi. DoGE: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393, 2023. — [arXiv:2310.15393](https://arxiv.org/abs/2310.15393) [find](https://scholar.google.com/scholar?q=Simin+Fan%2C+Matteo+Pagliardini%2C+and+Martin+Jaggi%2E+DoGE%3A+Domain+reweighting+with+generalization+estimation%2E+arXiv+preprint+arXiv%3A2310%2E15393%2C+2023%2E+%E2%80%94+%5BarXiv%3A2310%2E15393%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2310%2E15393%29)
21. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with MMLU? CoRR, abs/2406.04127, 2024. [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema%2C+Joshua+Ong+Jun+Leang%2C+Giwon+Hong%2C+Alessio+Devoto%2C+Alberto+Carlo+Maria+Mancino%2C+Rohit+Saxena%2C+Xuanli+He%2C+Yu+Zhao%2C+Xiaotang+Du%2C+Mohammad+Reza+Ghasemi+Madani%2C+et+al%2E+Are+we+done+with+)
22. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024. [find](https://scholar.google.com/scholar?q=Alex+Gu%2C+Baptiste+Rozie%CC%80re%2C+Hugh+Leather%2C+Armando+Solar%2DLezama%2C+Gabriel+Synnaeve%2C+and+Sida+I%2E+Wang%2E+CRUXEval%3A+A+benchmark+for+code+reasoning%2C+understanding+and+execution%2E+arXiv+preprint+arXiv%3A2401%2E030) [our card](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/)
23. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Dejian+Yang%2C+Haowei+Zhang%2C+Junxiao+Song%2C+Ruoyu+Zhang%2C+Runxin+Xu%2C+Qihao+Zhu%2C+Shirong+Ma%2C+Peiyi+Wang%2C+Xiao+Bi%2C+et+al%2E+DeepSeek%2DR1%3A+Incentivizing+reasoning+capability+in+LLMs+via+reinforcement+) [our card](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-r1-incentivizing-reasoning-capability/)
24. Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, et al. Multi-IF: Benchmarking LLMs on multi-turn and multilingual instructions following. arXiv preprint arXiv:2410.15553, 2024. — [arXiv:2410.15553](https://arxiv.org/abs/2410.15553) [find](https://scholar.google.com/scholar?q=Yun+He%2C+Di+Jin%2C+Chaoqi+Wang%2C+Chloe+Bi%2C+Karishma+Mandyam%2C+Hejia+Zhang%2C+Chen+Zhu%2C+Ning+Li%2C+Tengyu+Xu%2C+Hongjiang+Lv%2C+et+al%2E+Multi%2DIF%3A+Benchmarking+LLMs+on+multi%2Dturn+and+multilingual+instructions+followi)
25. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR. OpenReview.net, 2021a. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Steven+Basart%2C+Andy+Zou%2C+Mantas+Mazeika%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+massive+multitask+language+understanding%2E+In+ICLR%2E+OpenReview%2Enet%2C+2021a%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
26. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, 2021b. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+MATH+dataset%2E+In+NeurIPS+Datasets+a) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
27. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? CoRR, abs/2404.06654, 2024. [find](https://scholar.google.com/scholar?q=Cheng%2DPing+Hsieh%2C+Simeng+Sun%2C+Samuel+Kriman%2C+Shantanu+Acharya%2C+Dima+Rekesh%2C+Fei+Jia%2C+Yang+Zhang%2C+and+Boris+Ginsburg%2E+RULER%3A+What%E2%80%99s+the+real+context+size+of+your+long%2Dcontext+language+models%3F+CoRR%2C+abs) [our card](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/)
28. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-Eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Yuzhen+Huang%2C+Yuzhuo+Bai%2C+Zhihao+Zhu%2C+Junlei+Zhang%2C+Jinghan+Zhang%2C+Tangjun+Su%2C+Junteng+Liu%2C+Chuancheng+Lv%2C+Yikai+Zhang%2C+Jiayi+Lei%2C+Yao+Fu%2C+Maosong+Sun%2C+and+Junxian+He%2E+C%2DEval%3A+A+multi%2Dlevel+multi%2Ddisc) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/)
29. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report. CoRR, abs/2409.12186, 2024. [find](https://scholar.google.com/scholar?q=Binyuan+Hui%2C+Jian+Yang%2C+Zeyu+Cui%2C+Jiaxi+Yang%2C+Dayiheng+Liu%2C+Lei+Zhang%2C+Tianyu+Liu%2C+Jiajun+Zhang%2C+Bowen+Yu%2C+Keming+Lu%2C+et+al%2E+Qwen2%2E5%2DCoder+technical+report%2E+CoRR%2C+abs%2F2409%2E12186%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
30. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024. [find](https://scholar.google.com/scholar?q=Naman+Jain%2C+King+Han%2C+Alex+Gu%2C+Wen%2DDing+Li%2C+Fanjia+Yan%2C+Tianjun+Zhang%2C+Sida+Wang%2C+Armando+Solar%2DLezama%2C+Koushik+Sen%2C+and+Ion+Stoica%2E+LiveCodeBench%3A+Holistic+and+contamination+free+evaluation+of+large+) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
31. Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Z. Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and efficient pre-LN Transformers. CoRR, abs/2305.14858, 2023. — [arXiv:2305.14858](https://arxiv.org/abs/2305.14858) [find](https://scholar.google.com/scholar?q=Zixuan+Jiang%2C+Jiaqi+Gu%2C+Hanqing+Zhu%2C+and+David+Z%2E+Pan%2E+Pre%2DRMSNorm+and+Pre%2DCRMSNorm+Transformers%3A+Equivalent+and+efficient+pre%2DLN+Transformers%2E+CoRR%2C+abs%2F2305%2E14858%2C+2023%2E+%E2%80%94+%5BarXiv%3A2305%2E14858%5D%28https%3A%2F)
32. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smi… [find](https://scholar.google.com/scholar?q=Nathan+Lambert%2C+Jacob+Morrison%2C+Valentina+Pyatkin%2C+Shengyi+Huang%2C+Hamish+Ivison%2C+Faeze+Brahman%2C+Lester+James+V%2E+Miranda%2C+Alisa+Liu%2C+Nouha+Dziri%2C+Shane+Lyu%2C+Yuling+Gu%2C+Saumya+Malik%2C+Victoria+Graf%2C+Jena) [our card](https://md.research.elamur.ai/free/en/papers/lambert-2024-tulu-3-pushing-frontiers-open/)
33. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. CoRR, abs/2406.11939, 2024. [find](https://scholar.google.com/scholar?q=Tianle+Li%2C+Wei%2DLin+Chiang%2C+Evan+Frick%2C+Lisa+Dunlap%2C+Tianhao+Wu%2C+Banghua+Zhu%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+From+crowdsourced+data+to+high%2Dquality+benchmarks%3A+Arena%2DHard+and+BenchBuilder+pipeline) [our card](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/)
34. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. CoRR, abs/2305.20050, 2023. [find](https://scholar.google.com/scholar?q=Hunter+Lightman%2C+Vineet+Kosaraju%2C+Yura+Burda%2C+Harri+Edwards%2C+Bowen+Baker%2C+Teddy+Lee%2C+Jan+Leike%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Karl+Cobbe%2E+Let%E2%80%99s+verify+step+by+step%2E+CoRR%2C+abs%2F2305%2E20050%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
35. Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. ZebraLogic: On the scaling limits of LLMs for logical reasoning. CoRR, abs/2502.01100, 2025. [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Ronan+Le+Bras%2C+Kyle+Richardson%2C+Ashish+Sabharwal%2C+Radha+Poovendran%2C+Peter+Clark%2C+and+Yejin+Choi%2E+ZebraLogic%3A+On+the+scaling+limits+of+LLMs+for+logical+reasoning%2E+CoRR%2C+abs%2F2502%2E01100%2C)
36. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024a. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+et+al%2E+DeepSeek%2DV3+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E19437%2C+2024a)
37. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In NeurIPS, 2023a. [find](https://scholar.google.com/scholar?q=Jiawei+Liu%2C+Chunqiu+Steven+Xia%2C+Yuyao+Wang%2C+and+Lingming+Zhang%2E+Is+your+code+generated+by+ChatGPT+really+correct%3F+Rigorous+evaluation+of+large+language+models+for+code+generation%2E+In+NeurIPS%2C+2023a%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/)
38. Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. RegMix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492, 2024b. [find](https://scholar.google.com/scholar?q=Qian+Liu%2C+Xiaosen+Zheng%2C+Niklas+Muennighoff%2C+Guangtao+Zeng%2C+Longxu+Dou%2C+Tianyu+Pang%2C+Jing+Jiang%2C+and+Min+Lin%2E+RegMix%3A+Data+mixture+as+regression+for+language+model+pre%2Dtraining%2E+arXiv+preprint+arXiv%3A2)
39. Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. AlignBench: Benchmarking Chinese alignment of large language models. CoRR, abs/2311.1874… [find](https://scholar.google.com/scholar?q=Xiao+Liu%2C+Xuanyu+Lei%2C+Shengyuan+Wang%2C+Yue+Huang%2C+Zhuoer+Feng%2C+Bosi+Wen%2C+Jiale+Cheng%2C+Pei+Ke%2C+Yifan+Xu%2C+Weng+Lam+Tam%2C+Xiaohan+Zhang%2C+Lichao+Sun%2C+Hongning+Wang%2C+Jing+Zhang%2C+Minlie+Huang%2C+Yuxiao+Dong%2C+an)
40. Meta-AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025. URL [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/). [↗](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)
41. OpenAI. Hello GPT-4o, 2024. URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). [↗](https://openai.com/index/hello-gpt-4o/)
42. OpenAI. Multilingual massive multitask language understanding, 2024. URL [https://huggingface.co/datasets/openai/MMMLU](https://huggingface.co/datasets/openai/MMMLU). [↗](https://huggingface.co/datasets/openai/MMMLU)
43. OpenAI. Learning to reason with LLMs, 2024. URL [https://openai.com/index/learning-to-reaso](https://openai.com/index/learning-to-reaso) n-with-llms/. [↗](https://openai.com/index/learning-to-reaso)
44. OpenAI. Introducing openai o3 and o4-mini, 2025. URL [https://openai.com/index/introducing-o](https://openai.com/index/introducing-o) 3-and-o4-mini/. [↗](https://openai.com/index/introducing-o)
45. Samuel J. Paech. Creative writing v3, 2024. URL [https://eqbench.com/creative](https://eqbench.com/creative) writing.html. [↗](https://eqbench.com/creative)
46. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. CoRR, abs/2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+CoRR%2C+abs%2F2309%2E00071%2C+2023%2E)
47. Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models. CoRR, abs/2501.11873, 2025. [find](https://scholar.google.com/scholar?q=Zihan+Qiu%2C+Zeyu+Huang%2C+Bo+Zheng%2C+Kaiyue+Wen%2C+Zekun+Wang%2C+Rui+Men%2C+Ivan+Titov%2C+Dayiheng+Liu%2C+Jingren+Zhou%2C+and+Junyang+Lin%2E+Demons+in+the+detail%3A+On+implementing+load+balancing+loss+for+training+specia)
48. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking competition-level code generation of LLMs with human-comparable El… [find](https://scholar.google.com/scholar?q=Shanghaoran+Quan%2C+Jiaxi+Yang%2C+Bowen+Yu%2C+Bo+Zheng%2C+Dayiheng+Liu%2C+An+Yang%2C+Xuancheng+Ren%2C+Bofei+Gao%2C+Yibo+Miao%2C+Yunlong+Feng%2C+Zekun+Wang%2C+Jian+Yang%2C+Zeyu+Cui%2C+Yang+Fan%2C+Yichang+Zhang%2C+Binyuan+Hui%2C+and+J)
49. Qwen Team. QwQ: Reflect deeply on the boundaries of the unknown, November 2024. URL [https://qwenlm.github.io/blog/qwq-32b-preview/](https://qwenlm.github.io/blog/qwq-32b-preview/). [↗](https://qwenlm.github.io/blog/qwq-32b-preview/)
50. Qwen Team. QwQ-32B: Embracing the power of reinforcement learning, March 2025. URL [https://qwenlm.github.io/blog/qwq-32b/](https://qwenlm.github.io/blog/qwq-32b/). [↗](https://qwenlm.github.io/blog/qwq-32b/)
51. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. CoRR, abs/2311.12022, 2023. [find](https://scholar.google.com/scholar?q=David+Rein%2C+Betty+Li+Hou%2C+Asa+Cooper+Stickland%2C+Jackson+Petty%2C+Richard+Yuanzhe+Pang%2C+Julien+Dirani%2C+Julian+Michael%2C+and+Samuel+R%2E+Bowman%2E+GPQA%3A+A+graduate%2Dlevel+Google%2Dproof+Q%26A+benchmark%2E+CoRR%2C+abs%2F2) [our card](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
52. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, A… [find](https://scholar.google.com/scholar?q=Angelika+Romanou%2C+Negar+Foroutan%2C+Anna+Sotnikova%2C+Zeming+Chen%2C+Sree+Harsha+Nelaturu%2C+Shivalika+Singh%2C+Rishabh+Maheshwary%2C+Micol+Altomare%2C+Mohamed+A%2E+Haggag%2C+Snegha+A%2C+Alfonso+Amayuelas%2C+Azril+Hafizi+A)
53. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL (1). The Association for Computer Linguistics, 2016. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+In+ACL+%281%29%2E+The+Association+for+Computer+Linguistics%2C+2016%2E)
54. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. [find](https://scholar.google.com/scholar?q=Zhihong+Shao%2C+Peiyi+Wang%2C+Qihao+Zhu%2C+Runxin+Xu%2C+Junxiao+Song%2C+Mingchuan+Zhang%2C+Y%2E+K%2E+Li%2C+Y%2E+Wu%2C+and+Daya+Guo%2E+DeepSeekMath%3A+Pushing+the+limits+of+mathematical+reasoning+in+open+language+models%2E+CoRR%2C+) [our card](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/)
55. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In ICLR. OpenReview.net, 2023. [find](https://scholar.google.com/scholar?q=Freda+Shi%2C+Mirac+Suzgun%2C+Markus+Freitag%2C+Xuezhi+Wang%2C+Suraj+Srivats%2C+Soroush+Vosoughi%2C+Hyung+Won+Chung%2C+Yi+Tay%2C+Sebastian+Ruder%2C+Denny+Zhou%2C+Dipanjan+Das%2C+and+Jason+Wei%2E+Language+models+are+multilingu) [our card](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/)
56. Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. Linguistic generalizability of test-time scaling in mathematical reasoning. CoRR, abs/2502.17407, 2025. [find](https://scholar.google.com/scholar?q=Guijin+Son%2C+Jiwoo+Hong%2C+Hyunwoo+Ko%2C+and+James+Thorne%2E+Linguistic+generalizability+of+test%2Dtime+scaling+in+mathematical+reasoning%2E+CoRR%2C+abs%2F2502%2E17407%2C+2025%2E)
57. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced Transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Murtadha+H%2E+M%2E+Ahmed%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Wen+Bo%2C+and+Yunfeng+Liu%2E+Roformer%3A+Enhanced+Transformer+with+rotary+position+embedding%2E+Neurocomputing%2C+568%3A127063%2C+2024%2E)
58. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In ACL (Findings), pp. 13003–13051. Association for Computatio… [find](https://scholar.google.com/scholar?q=Mirac+Suzgun%2C+Nathan+Scales%2C+Nathanael+Scha%CC%88rli%2C+Sebastian+Gehrmann%2C+Yi+Tay%2C+Hyung+Won+Chung%2C+Aakanksha+Chowdhery%2C+Quoc+V%2E+Le%2C+Ed+H%2E+Chi%2C+Denny+Zhou%2C+and+Jason+Wei%2E+Challenging+BIG%2DBench+tasks+and+whe) [our card](https://md.research.elamur.ai/free/en/papers/suzgun-2023-challenging-big-bench-tasks-whether/)
59. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Rame%CC%81%2C+Morgane+Rivie%CC%80re%2C+et+al%2E+Gemma+3+technical+report%2E+arXiv+)
60. Changhan Wang, Kyunghyun Cho, and Jiatao Gu. Neural machine translation with byte-level subwords. In AAAI, pp. 9154–9160. AAAI Press, 2020. [find](https://scholar.google.com/scholar?q=Changhan+Wang%2C+Kyunghyun+Cho%2C+and+Jiatao+Gu%2E+Neural+machine+translation+with+byte%2Dlevel+subwords%2E+In+AAAI%2C+pp%2E+9154%E2%80%939160%2E+AAAI+Press%2C+2020%2E)
61. Yiming Wang, Pei Zhang, Jialong Tang, Haoran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. PolyMath: Evaluating mathematical reasoning in multilingual contexts, 2025. [find](https://scholar.google.com/scholar?q=Yiming+Wang%2C+Pei+Zhang%2C+Jialong+Tang%2C+Haoran+Wei%2C+Baosong+Yang%2C+Rui+Wang%2C+Chenshu+Sun%2C+Feitong+Sun%2C+Jiran+Zhang%2C+Junxuan+Wu%2C+Qiqian+Cang%2C+Yichang+Zhang%2C+Fei+Huang%2C+Junyang+Lin%2C+Fei+Huang%2C+and+Jingren+)
62. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. CoRR… [find](https://scholar.google.com/scholar?q=Yubo+Wang%2C+Xueguang+Ma%2C+Ge+Zhang%2C+Yuansheng+Ni%2C+Abhranil+Chandra%2C+Shiguang+Guo%2C+Weiming+Ren%2C+Aaran+Arulraj%2C+Xuan+He%2C+Ziyan+Jiang%2C+Tianle+Li%2C+Max+Ku%2C+Kai+Wang%2C+Alex+Zhuang%2C+Rongqi+Fan%2C+Xiang+Yue%2C+and+W) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
63. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-free LLM benchmark. CoRR, a… [find](https://scholar.google.com/scholar?q=Colin+White%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Arka+Pal%2C+Benjamin+Feuer%2C+Siddhartha+Jain%2C+Ravid+Shwartz%2DZiv%2C+Neel+Jain%2C+Khalid+Saifullah%2C+Siddartha+Naidu%2C+Chinmay+Hegde%2C+Yann+LeCun%2C+Tom+Goldstein%2C+Willie)
64. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A comprehensive benchmark for generative writing. CoRR, abs/2503.05244, 2025. xAI. Grok 3 beta — the age of reasoning agents, 2025. URL [https://x.ai/news/gr…](https://x.ai/news/gr…) — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) [↗](https://x.ai/news/gr…)
65. Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798–69818, 2023. [find](https://scholar.google.com/scholar?q=Sang+Michael+Xie%2C+Hieu+Pham%2C+Xuanyi+Dong%2C+Nan+Du%2C+Hanxiao+Liu%2C+Yifeng+Lu%2C+Percy+S+Liang%2C+Quoc+V+Le%2C+Tengyu+Ma%2C+and+Adams+Wei+Yu%2E+Doremi%3A+Optimizing+data+mixtures+speeds+up+language+model+pretraining%2E+)
66. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Jingyu+Liu%2C+Igor+Molybog%2C+Hejia+Zhang%2C+Prajjwal+Bhargava%2C+Rui+Hou%2C+Louis+Martin%2C+Rashi+Rungta%2C+Karthik+Abinav+Sankararaman%2C+Barlas+Oguz%2C+Madian+Khabsa%2C+Han+Fang%2C+Yashar+Mehdad%2C+Sharan+Na) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
67. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/8](https://gorilla.cs.berkeley.edu/blogs/8) ber keley function calling leaderboard.html, 2024. [↗](https://gorilla.cs.berkeley.edu/blogs/8)
68. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
69. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024b. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+Qwen2%2E5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2C+2024b%2E)
70. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409.12122, 2024c. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bofei+Gao%2C+Bowen+Yu%2C+Chengpeng+Li%2C+Dayiheng+Liu%2C+Jianhong+Tu%2C+Jingren+Zhou%2C+Junyang+Lin%2C+et+al%2E+Qwen2%2E5%2DMath+technical+report%3A+Toward+mathematical+expert+model+via) [our card](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
71. Yidan Zhang, Boyi Deng, Yu Wan, Baosong Yang, Haoran Wei, Fei Huang, Bowen Yu, Junyang Lin, and Jingren Zhou. P-MMEval: A parallel multilingual multitask benchmark for consistent evaluation of LLMs. CoRR, abs/2411.09116, 2024. [find](https://scholar.google.com/scholar?q=Yidan+Zhang%2C+Boyi+Deng%2C+Yu+Wan%2C+Baosong+Yang%2C+Haoran+Wei%2C+Fei+Huang%2C+Bowen+Yu%2C+Junyang+Lin%2C+and+Jingren+Zhou%2E+P%2DMMEval%3A+A+parallel+multilingual+multitask+benchmark+for+consistent+evaluation+of+LLMs%2E+C)
72. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. CoRR, abs/2311.07911, 2023. [find](https://scholar.google.com/scholar?q=Jeffrey+Zhou%2C+Tianjian+Lu%2C+Swaroop+Mishra%2C+Siddhartha+Brahma%2C+Sujoy+Basu%2C+Yi+Luan%2C+Denny+Zhou%2C+and+Le+Hou%2E+Instruction%2Dfollowing+evaluation+for+large+language+models%2E+CoRR%2C+abs%2F2311%2E07911%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)
73. Qin Zhu, Fei Huang, Runyu Peng, Keming Lu, Bowen Yu, Qinyuan Cheng, Xipeng Qiu, Xuanjing Huang, and Junyang Lin. AutoLogi: Automated generation of logic puzzles for evaluating reasoning abilities of large language models. CoRR, abs/2502.16906, 2025. — [arXiv:2502.16906](https://arxiv.org/abs/2502.16906) [find](https://scholar.google.com/scholar?q=Qin+Zhu%2C+Fei+Huang%2C+Runyu+Peng%2C+Keming+Lu%2C+Bowen+Yu%2C+Qinyuan+Cheng%2C+Xipeng+Qiu%2C+Xuanjing+Huang%2C+and+Junyang+Lin%2E+AutoLogi%3A+Automated+generation+of+logic+puzzles+for+evaluating+reasoning+abilities+of+l)