---
title: "Qwen2.5-1M Technical Report — Yang 2025"
description: "Most open LLMs top out around 128K tokens of context."
authors: ["An Yang", "Bowen Yu", "Chengyuan Li", "Dayiheng Liu", "Fei Huang", "Haoyan Huang", "Jiandong Jiang", "Jianhong Tu", "Jianwei Zhang", "Jingren Zhou", "Junyang Lin", "Kai Dang", "Kexin Yang", "Le Yu", "Mei Li", "Minmin Sun", "Qin Zhu", "Rui Men", "Tao He", "Weijia Xu", "Wenbiao Yin", "Wenyuan Yu", "Xiafei Qiu", "Xingzhang Ren", "Xinlong Yang", "Yong Li", "Zhiying Xu", "Zipeng Zhang"]
year: 2025
venue: "arXiv:2501.15383"
source: "https://arxiv.org/abs/2501.15383"
doi: "10.48550/arxiv.2501.15383"
arxiv: "2501.15383"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-qwen2-5-1m-technical/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-qwen2-5-1m-technical/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Qwen2.5-1M Technical Report

# Qwen2.5-1M Technical Report

**Authors:** [An Yang](https://md.research.elamur.ai/free/en/authors/an-yang/), [Bowen Yu](https://md.research.elamur.ai/free/en/authors/bowen-yu/), Chengyuan Li, [Dayiheng Liu](https://md.research.elamur.ai/free/en/authors/dayiheng-liu/), [Fei Huang](https://md.research.elamur.ai/free/en/authors/fei-huang/), Haoyan Huang, Jiandong Jiang, [Jianhong Tu](https://md.research.elamur.ai/free/en/authors/jianhong-tu/), [Jianwei Zhang](https://md.research.elamur.ai/free/en/authors/jianwei-zhang/), [Jingren Zhou](https://md.research.elamur.ai/free/en/authors/jingren-zhou/), [Junyang Lin](https://md.research.elamur.ai/free/en/authors/junyang-lin/), [Kai Dang](https://md.research.elamur.ai/free/en/authors/kai-dang/) et al. (28 authors) · **Year:** 2025 · **Venue:** arXiv:2501.15383 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2501.15383) · [PDF](https://arxiv.org/pdf/2501.15383) · [DOI 10.48550/arxiv.2501.15383](https://doi.org/10.48550/arxiv.2501.15383) · [arXiv 2501.15383](https://arxiv.org/abs/2501.15383) · [OpenAlex](https://openalex.org/W4406880389)

**Citations:** 12 citations (openalex-snapshot · 2026-10-14) · [88](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most open LLMs top out around 128K tokens of context. That rules out tasks like repository-level coding or analysis of large document collections. The Qwen Team extends Qwen2.5 to a 1-million-token window. They release two open-weight models, Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, plus Qwen2.5-Turbo, an MoE model available through an API.

Pre-training mixes natural long texts with synthetic tasks that force the model to use distant context, and the window grows in stages up to 256K tokens. Post-training relies on model-generated long instructions, multi-stage SFT and RL. At inference, a training-free extrapolation method (DCA+YaRN) stretches the context to 1M. Sparse attention with chunked prefill and engine-level optimizations reduce the cost, and the whole framework is open-sourced.

Short-task quality matches the 128K predecessors. The 14B model outperforms GPT-4o-mini on long-context benchmarks, and prefill at 1M tokens runs several times faster. This makes million-token workloads practical with open models.

## Key points
- **Released models:** both open models are licensed under Apache 2.0, with a 1M context and 8K generation limit. The 7B model has 28 layers and 28/4 Q/KV heads; the 14B model has 48 layers and 40/8 heads.
- **Pre-training data:** natural long texts come from Common Crawl, arXiv, books and code. Synthetic tasks include fill-in-the-middle, keyword- and position-based retrieval, and paragraph reordering.
- **Progressive context schedule:** training runs from 4K to 32K tokens with ABF, raising the RoPE base from 10,000 to 1,000,000. It then moves through 64K, 128K and 256K, with RoPE bases of 1M, 5M and 10M. Each of these stages uses 75% full-length and 25% shorter sequences.
- **RULER at 128K (Qwen2.5-14B-1M):** the score climbs from 37.6 after the 32K stage to 87.6 after the 256K stage. Training on longer sequences thus also helps at shorter lengths.
- **Speed at 1M tokens:** time to first token improves 3.2–6.7× across models and GPUs (H20, A100). On H20, the 14B model drops from 12.2 minutes to 109 seconds, and Qwen2.5-Turbo drops from 4.9 minutes to 68 seconds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

88 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 84 not yet.

- [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/)
- [Improving Dialect Robustness in Large Language Models via LoRA and Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/maheshwari-2026-improving-dialect-robustness-large-language/)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/xie-2025-logic-rl-unleashing-llm/)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{yang2025qwen2,
  title = {{Qwen2.5-1M Technical Report}},
  author = {Yang, An and Yu, Bowen and Li, Chengyuan and Liu, Dayiheng and Huang, Fei and Huang, Haoyan and Jiang, Jiandong and Tu, Jianhong and Zhang, Jianwei and Zhou, Jingren and Lin, Junyang and Dang, Kai and Yang, Kexin and Yu, Le and Li, Mei and Sun, Minmin and Zhu, Qin and Men, Rui and He, Tao and Xu, Weijia and Yin, Wenbiao and Yu, Wenyuan and Qiu, Xiafei and Ren, Xingzhang and Yang, Xinlong and Li, Yong and Xu, Zhiying and Zhang, Zipeng},
  year = {2025},
  doi = {10.48550/arxiv.2501.15383},
  eprint = {2501.15383},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.15383}
}
```

- [← previous Prompt Engineering Techniques for Context-dependent Text-to-SQL in Arabic](https://md.research.elamur.ai/free/en/papers/almohaimeed-2025-prompt-engineering-techniques-context-dependent/)
- [next → Qwen2.5-Omni Technical Report](https://md.research.elamur.ai/free/en/papers/xu-2025-qwen2-5-omni-technical/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

53 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query Transformer models from multi-head checkpoints. In EMNLP, pp. 4895–4901. Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Joshua+Ainslie%2C+James+Lee%2DThorp%2C+Michiel+de+Jong%2C+Yury+Zemlyanskiy%2C+Federico+Lebro%CC%81n%2C+and+Sumit+Sanghai%2E+GQA%3A+Training+generalized+multi%2Dquery+Transformer+models+from+multi%2Dhead+checkpoints%2E+In+EMNLP%2C) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/)
2. Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models. CoRR, abs/2402.17463, 2024a. — [arXiv:2402.17463](https://arxiv.org/abs/2402.17463) [find](https://scholar.google.com/scholar?q=Chenxin+An%2C+Fei+Huang%2C+Jun+Zhang%2C+Shansan+Gong%2C+Xipeng+Qiu%2C+Chang+Zhou%2C+and+Lingpeng+Kong%2E+Training%2Dfree+long%2Dcontext+scaling+of+large+language+models%2E+CoRR%2C+abs%2F2402%2E17463%2C+2024a%2E+%E2%80%94+%5BarXiv%3A2402%2E17463)
3. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of llms fall short?, 2024b. URL https://arxiv.org/abs/2410.1 8745. [↗](https://arxiv.org/abs/2410.1)
4. Anthropic. Introducing Claude, 2023a. URL [https://www.anthropic.com/index/introducing-claude](https://www.anthropic.com/index/introducing-claude). [↗](https://www.anthropic.com/index/introducing-claude)
5. Anthropic. Claude 2. Technical report, Anthropic, 2023b. URL [https://www-files.anthropic.com/pro](https://www-files.anthropic.com/pro) duction/images/Model-Card-Claude-2.pdf. [↗](https://www-files.anthropic.com/pro)
6. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Technical report, Anthropic, AI, 2024. URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model) Card Claud e 3.pdf. [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model)
7. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. [find](https://scholar.google.com/scholar?q=Jacob+Austin%2C+Augustus+Odena%2C+Maxwell+I%2E+Nye%2C+Maarten+Bosma%2C+Henryk+Michalewski%2C+David+Dohan%2C+Ellen+Jiang%2C+Carrie+J%2E+Cai%2C+Michael+Terry%2C+Quoc+V%2E+Le%2C+and+Charles+Sutton%2E+Program+synthesis+with+large+la) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
8. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… [find](https://scholar.google.com/scholar?q=Jinze+Bai%2C+Shuai+Bai%2C+Yunfei+Chu%2C+Zeyu+Cui%2C+Kai+Dang%2C+Xiaodong+Deng%2C+Yang+Fan%2C+Wenbin+Ge%2C+Yu+Han%2C+Fei+Huang%2C+Binyuan+Hui%2C+Luo+Ji%2C+Mei+Li%2C+Junyang+Lin%2C+Runji+Lin%2C+Dayiheng+Liu%2C+Gao+Liu%2C+Chengqiang+Lu%2C+) [our card](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/)
9. Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In EMNLP (Findings), pp. 1376–1395. Association for Computational Linguistics, 2024. [find](https://scholar.google.com/scholar?q=Yushi+Bai%2C+Xin+Lv%2C+Jiajie+Zhang%2C+Yuze+He%2C+Ji+Qi%2C+Lei+Hou%2C+Jie+Tang%2C+Yuxiao+Dong%2C+and+Juanzi+Li%2E+LongAlign%3A+A+recipe+for+long+context+alignment+of+large+language+models%2E+In+EMNLP+%28Findings%29%2C+pp%2E+1376%E2%80%931) [our card](https://md.research.elamur.ai/free/en/papers/bai-2024-longalign-recipe-long-context/)
10. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR, abs/2207.14255, 2022. doi: 10.48550/ARXIV.2207.14255. URL https://doi.org/10.48550/arXiv.2207.14255. [↗](https://doi.org/10.48550/arXiv.2207.14255) [our card](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/)
11. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
12. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: A scalable and polyglot approach to benchmarking neural code generation.… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sydney+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q%2E+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+)
13. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mi… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde%CC%81+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harrison+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. [find](https://scholar.google.com/scholar?q=Karl+Cobbe%2C+Vineet+Kosaraju%2C+Mohammad+Bavarian%2C+Mark+Chen%2C+Heewoo+Jun%2C+Lukasz+Kaiser%2C+Matthias+Plappert%2C+Jerry+Tworek%2C+Jacob+Hilton%2C+Reiichiro+Nakano%2C+Christopher+Hesse%2C+and+John+Schulman%2E+Training+ve) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
15. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022. URL [http://papers.nips.cc/paper](http://papers.nips.cc/paper) fil es/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html. [↗](http://papers.nips.cc/paper)
16. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In ICML, volume 70 of Proceedings of Machine Learning Research, pp. 933–941. PMLR, 2017. [find](https://scholar.google.com/scholar?q=Yann+N%2E+Dauphin%2C+Angela+Fan%2C+Michael+Auli%2C+and+David+Grangier%2E+Language+modeling+with+gated+convolutional+networks%2E+In+ICML%2C+volume+70+of+Proceedings+of+Machine+Learning+Research%2C+pp%2E+933%E2%80%93941%2E+PMLR%2C+2) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez… [find](https://scholar.google.com/scholar?q=Abhimanyu+Dubey%2C+Abhinav+Jauhri%2C+Abhinav+Pandey%2C+Abhishek+Kadian%2C+Ahmad+Al%2DDahle%2C+Aiesha+Letman%2C+Akhil+Mathur%2C+Alan+Schelten%2C+Amy+Yang%2C+Angela+Fan%2C+Anirudh+Goyal%2C+Anthony+Hartshorn%2C+Aobo+Yang%2C+Archi+M)
18. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? CoRR, abs/2406.04127, 2024. [find](https://scholar.google.com/scholar?q=Aryo+Pradipta+Gema%2C+Joshua+Ong+Jun+Leang%2C+Giwon+Hong%2C+Alessio+Devoto%2C+Alberto+Carlo+Maria+Mancino%2C+Rohit+Saxena%2C+Xuanli+He%2C+Yu+Zhao%2C+Xiaotang+Du%2C+Mohammad+Reza+Ghasemi+Madani%2C+et+al%2E+Are+we+done+with+)
19. Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Technical report, Google, 2024. URL [https://storage.googleapis.com/deepmind-media/gemini/gemi](https://storage.googleapis.com/deepmind-media/gemini/gemi) ni v1 5 report.pdf. [↗](https://storage.googleapis.com/deepmind-media/gemini/gemi) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
20. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, 2021. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Saurav+Kadavath%2C+Akul+Arora%2C+Steven+Basart%2C+Eric+Tang%2C+Dawn+Song%2C+and+Jacob+Steinhardt%2E+Measuring+mathematical+problem+solving+with+the+MATH+dataset%2E+In+NeurIPS+Datasets+a) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
21. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? CoRR, abs/2404.06654, 2024. [find](https://scholar.google.com/scholar?q=Cheng%2DPing+Hsieh%2C+Simeng+Sun%2C+Samuel+Kriman%2C+Shantanu+Acharya%2C+Dima+Rekesh%2C+Fei+Jia%2C+Yang+Zhang%2C+and+Boris+Ginsburg%2E+RULER%3A+What%E2%80%99s+the+real+context+size+of+your+long%2Dcontext+language+models%3F+CoRR%2C+abs) [our card](https://md.research.elamur.ai/free/en/papers/hsieh-2024-ruler-s-real-context-size/)
22. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2.5-Coder technical report. CoRR, abs/2409.12186, 2024. [find](https://scholar.google.com/scholar?q=Binyuan+Hui%2C+Jian+Yang%2C+Zeyu+Cui%2C+Jiaxi+Yang%2C+Dayiheng+Liu%2C+Lei+Zhang%2C+Tianyu+Liu%2C+Jiajun+Zhang%2C+Bowen+Yu%2C+Keming+Lu%2C+et+al%2E+Qwen2%2E5%2DCoder+technical+report%2E+CoRR%2C+abs%2F2409%2E12186%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
23. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024. [find](https://scholar.google.com/scholar?q=Naman+Jain%2C+King+Han%2C+Alex+Gu%2C+Wen%2DDing+Li%2C+Fanjia+Yan%2C+Tianjun+Zhang%2C+Sida+Wang%2C+Armando+Solar%2DLezama%2C+Koushik+Sen%2C+and+Ion+Stoica%2E+LiveCodeBench%3A+Holistic+and+contamination+free+evaluation+of+large+) [our card](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/)
24. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
25. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+Las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
26. Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. arXiv preprint arXiv:2407.02490, 2024b. [find](https://scholar.google.com/scholar?q=Huiqiang+Jiang%2C+Yucheng+Li%2C+Chengruidong+Zhang%2C+Qianhui+Wu%2C+Xufang+Luo%2C+Surin+Ahn%2C+Zhenhua+Han%2C+Amir+H+Abdi%2C+Dongsheng+Li%2C+Chin%2DYew+Lin%2C+Yuqing+Yang%2C+and+Lili+Qiu%2E+Minference+1%2E0%3A+Accelerating+pre%2Dfil) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2024-minference-1-0-accelerating-pre/)
27. Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, and David Z. Pan. Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and efficient pre-LN Transformers. CoRR, abs/2305.14858, 2023b. — [arXiv:2305.14858](https://arxiv.org/abs/2305.14858) [find](https://scholar.google.com/scholar?q=Zixuan+Jiang%2C+Jiaqi+Gu%2C+Hanqing+Zhu%2C+and+David+Z%2E+Pan%2E+Pre%2DRMSNorm+and+Pre%2DCRMSNorm+Transformers%3A+Equivalent+and+efficient+pre%2DLN+Transformers%2E+CoRR%2C+abs%2F2305%2E14858%2C+2023b%2E+%E2%80%94+%5BarXiv%3A2305%2E14858%5D%28https%3A)
28. Gregory Kamradt. Needle in a haystack - pressure testing LLMs, 2023. URL [https://github.com/gkamr](https://github.com/gkamr) adt/LLMTest NeedleInAHaystack. [↗](https://github.com/gkamr)
29. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. [find](https://scholar.google.com/scholar?q=Woosuk+Kwon%2C+Zhuohan+Li%2C+Siyuan+Zhuang%2C+Ying+Sheng%2C+Lianmin+Zheng%2C+Cody+Hao+Yu%2C+Joseph+E%2E+Gonzalez%2C+Hao+Zhang%2C+and+Ion+Stoica%2E+Efficient+memory+management+for+large+language+model+serving+with+pagedat)
30. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-Hard and BenchBuilder pipeline. CoRR, abs/2406.11939, 2024. [find](https://scholar.google.com/scholar?q=Tianle+Li%2C+Wei%2DLin+Chiang%2C+Evan+Frick%2C+Lisa+Dunlap%2C+Tianhao+Wu%2C+Banghua+Zhu%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+From+crowdsourced+data+to+high%2Dquality+benchmarks%3A+Arena%2DHard+and+BenchBuilder+pipeline) [our card](https://md.research.elamur.ai/free/en/papers/li-2024-crowdsourced-data-high-quality-benchmarks/)
31. OpenAI. GPT4 technical report. CoRR, abs/2303.08774, 2023. [find](https://scholar.google.com/scholar?q=OpenAI%2E+GPT4+technical+report%2E+CoRR%2C+abs%2F2303%2E08774%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
32. OpenAI. Hello GPT-4o, 2024. URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/). [↗](https://openai.com/index/hello-gpt-4o/)
33. Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. Llama 3 gradient: A series of long context models, 2024. URL [https://gradient.ai/blog/scaling-rotational-embeddings-for-l](https://gradient.ai/blog/scaling-rotational-embeddings-for-l) ong-context-language-models. [↗](https://gradient.ai/blog/scaling-rotational-embeddings-for-l)
34. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. CoRR, abs/2309.00071, 2023. [find](https://scholar.google.com/scholar?q=Bowen+Peng%2C+Jeffrey+Quesnelle%2C+Honglu+Fan%2C+and+Enrico+Shippole%2E+YaRN%3A+Efficient+context+window+extension+of+large+language+models%2E+CoRR%2C+abs%2F2309%2E00071%2C+2023%2E)
35. Qwen Team. Code with CodeQwen1.5, 2024a. URL [https://qwenlm.github.io/blog/codeqwen1.5/](https://qwenlm.github.io/blog/codeqwen1.5/). [↗](https://qwenlm.github.io/blog/codeqwen1.5/)
36. Qwen Team. Generalizing an llm from 8k to 1m context using qwen-agent, May 2024b. URL [https://qwenlm.github.io/blog/qwen-agent-2405/](https://qwenlm.github.io/blog/qwen-agent-2405/). [↗](https://qwenlm.github.io/blog/qwen-agent-2405/)
37. Qwen Team. Introducing Qwen2-Math, 2024c. URL [https://qwenlm.github.io/blog/qwen2-math/](https://qwenlm.github.io/blog/qwen2-math/). [↗](https://qwenlm.github.io/blog/qwen2-math/)
38. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Rafael+Rafailov%2C+Archit+Sharma%2C+Eric+Mitchell%2C+Christopher+D%2E+Manning%2C+Stefano+Ermon%2C+and+Chelsea+Finn%2E+Direct+preference+optimization%3A+Your+language+model+is+secretly+a+reward+model%2E+In+NeurIPS%2C+2023)
39. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. CoRR, abs/2311.12022, 2023. [find](https://scholar.google.com/scholar?q=David+Rein%2C+Betty+Li+Hou%2C+Asa+Cooper+Stickland%2C+Jackson+Petty%2C+Richard+Yuanzhe+Pang%2C+Julien+Dirani%2C+Julian+Michael%2C+and+Samuel+R%2E+Bowman%2E+GPQA%3A+A+graduate%2Dlevel+Google%2Dproof+Q%26A+benchmark%2E+CoRR%2C+abs%2F2) [our card](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
40. Jianlin Su. The magical effect of the Bias term: RoPE + Bias = better length extrapolation, 2023. URL [https://spaces.ac.cn/archives/9577](https://spaces.ac.cn/archives/9577). [↗](https://spaces.ac.cn/archives/9577)
41. Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced Transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [find](https://scholar.google.com/scholar?q=Jianlin+Su%2C+Murtadha+H%2E+M%2E+Ahmed%2C+Yu+Lu%2C+Shengfeng+Pan%2C+Wen+Bo%2C+and+Yunfeng+Liu%2E+Roformer%3A+Enhanced+Transformer+with+rotary+position+embedding%2E+Neurocomputing%2C+568%3A127063%2C+2024%2E)
42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. CoRR, abs/2… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Thibaut+Lavril%2C+Gautier+Izacard%2C+Xavier+Martinet%2C+Marie%2DAnne+Lachaux%2C+Timothe%CC%81e+Lacroix%2C+Baptiste+Rozie%CC%80re%2C+Naman+Goyal%2C+Eric+Hambro%2C+Faisal+Azhar%2C+Aure%CC%81lien+Rodriguez%2C+Armand+Joulin%2C+Ed)
43. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton%2DFe)
44. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. CoRR… [find](https://scholar.google.com/scholar?q=Yubo+Wang%2C+Xueguang+Ma%2C+Ge+Zhang%2C+Yuansheng+Ni%2C+Abhranil+Chandra%2C+Shiguang+Guo%2C+Weiming+Ren%2C+Aaran+Arulraj%2C+Xuan+He%2C+Ziyan+Jiang%2C+Tianle+Li%2C+Max+Ku%2C+Kai+Wang%2C+Alex+Zhuang%2C+Rongqi+Fan%2C+Xiang+Yue%2C+and+W) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
45. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-free LLM benchmark. CoRR, a… [find](https://scholar.google.com/scholar?q=Colin+White%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Arka+Pal%2C+Benjamin+Feuer%2C+Siddhartha+Jain%2C+Ravid+Shwartz%2DZiv%2C+Neel+Jain%2C+Khalid+Saifullah%2C+Siddartha+Naidu%2C+Chinmay+Hegde%2C+Yann+LeCun%2C+Tom+Goldstein%2C+Willie)
46. Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.… [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Jingyu+Liu%2C+Igor+Molybog%2C+Hejia+Zhang%2C+Prajjwal+Bhargava%2C+Rui+Hou%2C+Louis+Martin%2C+Rashi+Rungta%2C+Karthik+Abinav+Sankararaman%2C+Barlas+Oguz%2C+Madian+Khabsa%2C+Han+Fang%2C+Yashar+Mehdad%2C+Sharan+Na) [our card](https://md.research.elamur.ai/free/en/papers/xiong-2023-effective-long-context-scaling-foundation/)
47. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chang+Zhou%2C+Chengpeng+Li%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Guanting+Dong%2C+Haoran+Wei%2C+Huan+Lin%2C+Jialong+Tang%2C+Jialin+Wang%2C+Jian+Yang%2C+Jian)
48. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-Math technical report: Toward mathematical expert model via self-improvement. CoRR, abs/2409.12122, 2024b. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bofei+Gao%2C+Bowen+Yu%2C+Chengpeng+Li%2C+Dayiheng+Liu%2C+Jianhong+Tu%2C+Jingren+Zhou%2C+Junyang+Lin%2C+et+al%2E+Qwen2%2E5%2DMath+technical+report%3A+Toward+mathematical+expert+model+via) [our card](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/)
49. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+Yang%2C+Jin)
50. Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. LV-Eval: A balanced long-context benchmark with 5 length levels up to 256K. CoRR, abs/2402.05136, 2024. [find](https://scholar.google.com/scholar?q=Tao+Yuan%2C+Xuefei+Ning%2C+Dong+Zhou%2C+Zhijie+Yang%2C+Shiyao+Li%2C+Minghui+Zhuang%2C+Zheyue+Tan%2C+Zhuyu+Yao%2C+Dahua+Lin%2C+Boxun+Li%2C+Guohao+Dai%2C+Shengen+Yan%2C+and+Yu+Wang%2E+LV%2DEval%3A+A+balanced+long%2Dcontext+benchmark+w)
51. Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, R… [find](https://scholar.google.com/scholar?q=Aohan+Zeng%2C+Bin+Xu%2C+Bowen+Wang%2C+Chenhui+Zhang%2C+Da+Yin%2C+Diego+Rojas%2C+Guanyu+Feng%2C+Hanlin+Zhao%2C+Hanyu+Lai%2C+Hao+Yu%2C+Hongning+Wang%2C+Jiadai+Sun%2C+Jiajie+Zhang%2C+Jiale+Cheng%2C+Jiayi+Gui%2C+Jie+Tang%2C+Jing+Zhang%2C+) [our card](https://md.research.elamur.ai/free/en/papers/glm-2024-chatglm-family-large-language-models/)
52. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+P%2E+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Judging+LLM%2Das%2Da%2Djud)
53. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. CoRR, abs/2311.07911, 2023. [find](https://scholar.google.com/scholar?q=Jeffrey+Zhou%2C+Tianjian+Lu%2C+Swaroop+Mishra%2C+Siddhartha+Brahma%2C+Sujoy+Basu%2C+Yi+Luan%2C+Denny+Zhou%2C+and+Le+Hou%2E+Instruction%2Dfollowing+evaluation+for+large+language+models%2E+CoRR%2C+abs%2F2311%2E07911%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/)