--- title: "gpt-oss-120b & gpt-oss-20b Model Card — Agarwal 2025" description: "This model card from OpenAI describes gpt-oss-120b and gpt-oss-20b." authors: ["OpenAI: Sandhini Agarwal", "Lama Ahmad", "Jason Ai", "Sam Altman", "Andy Applebaum", "Edwin Arbus", "Rahul K. Arora", "Yu Bai", "Bowen Baker", "Haiming Bao", "Boaz Barak", "Ally Bennett", "Tyler Bertao", "Nivedita Brett", "Eugene Brevdo", "Greg Brockman", "Sebastien Bubeck", "Che Chang", "Kai Chen", "Mark Chen", "Enoch Cheung", "Aidan Clark", "Dan Cook", "Marat Dukhan", "Casey Dvorak", "Kevin Fives", "Vlad Fomenko", "Timur Garipov", "Kristian Georgiev", "Mia Glaese", "Tarun Gogineni", "Adam Goucher", "Lukas Gross", "Katia Gil Guzman", "John Hallman", "Jackie Hehir", "Johannes Heidecke", "Alec Helyar", "Haitang Hu", "Romain Huet", "Jacob Huh", "Saachi Jain", "Zach Johnson", "Chris Koch", "Irina Kofman", "Dominik Kundel", "Jason Kwon", "Volodymyr Kyrylov", "Elaine Ya Le", "Guillaume Leclerc", "James Park Lennon", "Scott Lessans", "Mario Lezcano-Casado", "Yuanzhi Li", "Zhuohan Li", "Ji Lin", "Jordan Liss", "Lily Liu", "Jiancheng Liu", "Kevin Lu", "Chris Lu", "Zoran Martinovic", "Lindsay McCallum", "Josh McGrath", "Scott McKinney", "Aidan McLaughlin", "Song Mei", "Steve Mostovoy", "Tong Mu", "Gideon Myles", "Alexander Neitz", "Alex Nichol", "Jakub Pachocki", "Alex Paino", "Dana Palmie", "Ashley Pantuliano", "Giambattista Parascandolo", "Jongsoo Park", "Leher Pathak", "Carolina Paz", "Ludovic Peran", "Dmitry Pimenov", "Michelle Pokrass", "Elizabeth Proehl", "Huida Qiu", "Gaby Raila", "Filippo Raso", "Hongyu Ren", "Kimmy Richardson", "David Robinson", "Bob Rotsted", "Hadi Salman", "Suvansh Sanjeev", "Max Schwarzer", "D. Sculley", "Harshit Sikchi", "Kendal Simon", "Karan Singhal", "Yang Song", "Dane Stuckey", "Zhiqing Sun", "Philippe Tillet", "Sam Toizer", "Foivos Tsimpourlas", "Nikhil Vyas", "Eric Wallace", "Xin Wang", "Miles Wang", "Olivia Watkins", "Kevin Weil", "Amy Wendling", "Kevin Whinnery", "Cedric Whitney", "Hannah Wong", "Lin Yang", "Yu Yang", "Michihiro Yasunaga", "Kristen Ying", "Wojciech Zaremba", "Wenting Zhan", "Cyril Zhang", "Brian Zhang", "Eddie Zhang", "Shengjia Zhao"] year: 2025 venue: "arXiv:2508.10925" source: "https://arxiv.org/abs/2508.10925" doi: "10.48550/arxiv.2508.10925" arxiv: "2508.10925" licence: "CC BY 4.0 (arXiv)" licence_url: "http://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) / gpt-oss-120b & gpt-oss-20b Model Card # gpt-oss-120b & gpt-oss-20b Model Card **Authors:** OpenAI: Sandhini Agarwal, [Lama Ahmad](https://md.research.elamur.ai/free/en/authors/lama-ahmad/), Jason Ai, [Sam Altman](https://md.research.elamur.ai/free/en/authors/sam-altman/), Andy Applebaum, Edwin Arbus, [Rahul K. Arora](https://md.research.elamur.ai/free/en/authors/rahul-k-arora/), [Yu Bai](https://md.research.elamur.ai/free/en/authors/yu-bai/), [Bowen Baker](https://md.research.elamur.ai/free/en/authors/bowen-baker/), Haiming Bao, Boaz Barak, Ally Bennett et al. (124 authors) · **Year:** 2025 · **Venue:** arXiv:2508.10925 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2508.10925) · [PDF](https://arxiv.org/pdf/2508.10925) · [DOI 10.48550/arxiv.2508.10925](https://doi.org/10.48550/arxiv.2508.10925) · [arXiv 2508.10925](https://arxiv.org/abs/2508.10925) · [OpenAlex](https://openalex.org/W7075994024) **Citations:** 0 citations (openalex-snapshot · 2026-10-14) · [552](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary This model card from OpenAI describes gpt-oss-120b and gpt-oss-20b. They are text-only reasoning models with open weights under the Apache 2.0 license. They are built for agentic workflows: web search, Python execution, function calling and Structured Outputs. Users can set how much effort the model spends on reasoning, and the full chain-of-thought is visible. Both models are Mixture-of-Experts transformers derived from GPT-2/GPT-3. They were pretrained on trillions of text tokens focused on STEM, code and general knowledge, with hazardous biosecurity material filtered out. Post-training used chain-of-thought reinforcement learning similar to OpenAI o3, together with a new "harmony" chat format that defines message roles and channels. On math, science, coding and tool-use benchmarks, the larger model beats o3-mini and gets close to o4-mini. The smaller model stays competitive at roughly one sixth the size. Much of the card is about safety. OpenAI fine-tuned the 120b model adversarially on biology and cyber tasks to simulate an attacker, and it still stayed below the "High" capability threshold of the Preparedness Framework. This matters because it brings strong reasoning to modest hardware and gives a worst-case misuse assessment for releasing open weights. ## Key points - **Sizes:** gpt-oss-120b has 36 layers, 116.8B total parameters, 5.1B active per token and 128 experts. gpt-oss-20b has 24 layers, 20.9B total parameters, 3.6B active and 32 experts. Both models route each token to its top 4 experts. - **Memory:** the MoE weights are quantized to MXFP4 (about 4.25 bits per parameter). This lets the 120b model fit on one 80GB GPU and the 20b model run in 16GB of memory. - **Architecture:** attention alternates between dense layers and 128-token banded-window layers, with GQA (8 KV heads). Context reaches 131,072 tokens via YaRN. The tokenizer is o200k_harmony, with 201,088 tokens. - **Training:** the 120b model took about 2.1M H100-hours, and the 20b model needed nearly 10× less. Knowledge cutoff is June 2024. - **Instruction priority:** when instructions conflict, the models follow System > Developer > User > Assistant > Tool. Developers deploying the models may need to add their own safeguards. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 552 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 524 not yet. - [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/) - [Olmo 3](https://md.research.elamur.ai/free/en/papers/ettinger-2025-olmo-3/) - [AraDynFact: Dynamic Evaluation of Factual Knowledge in Arabic](https://md.research.elamur.ai/free/en/papers/iacobacci-2026-aradynfact-dynamic-evaluation-factual-knowledge/) - [When Similar Means Different: Evaluating LLMs on Arabic–Hebrew Cognates](https://md.research.elamur.ai/free/en/papers/langstream-2026-liang-similar-means-different-evaluating-llms/) - [Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs](https://md.research.elamur.ai/free/en/papers/langstream-2026-shmidman-dicta-lm-3-0-advancing/) - [Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus](https://md.research.elamur.ai/free/en/papers/mannion-2026-is-biomedical-specialization-still-worth/) - [Multilingual Target-Stance Extraction](https://md.research.elamur.ai/free/en/papers/mines-2026-multilingual-target-stance-extraction/) - [A Dialogue System for Second Language Learning with Dynamic Adaptation to In-Dialogue Difficulty Changes](https://md.research.elamur.ai/free/en/papers/morioka-2026-dialogue-system-second-language-learning/) - [Tree-Based Interview Topic Guidance for Collecting Target Information under Adaptive Topic continuation/switching](https://md.research.elamur.ai/free/en/papers/nagasawa-2026-tree-based-interview-topic-guidance/) - [Meaning Over Morphology: A Multi-Metric Benchmark of LLMs for Bangla Dialect Translation](https://md.research.elamur.ai/free/en/papers/niloy-2026-meaning-over-morphology-multi-metric/) - [InstructSum: A Benchmark to Evaluate Instruction-Following Capability of Large Language Models in Summarization](https://md.research.elamur.ai/free/en/papers/nishida-2026-instructsum-benchmark-evaluate-instruction-following/) - [Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches](https://md.research.elamur.ai/free/en/papers/park-2026-identifying-imaging-follow-up-radiology/) - [HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection](https://md.research.elamur.ai/free/en/papers/patterson-2026-hybridcodeauthorship-benchmark-dataset-line-level/) - [Information Asymmetry across Language Varieties: A Case Study on Cantonese-Mandarin and Bavarian-German QA](https://md.research.elamur.ai/free/en/papers/pei-2026-information-asymmetry-across-language-varieties/) - [Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset](https://md.research.elamur.ai/free/en/papers/plum-2026-do-llms-judge-distantly-supervised/) - [On the Robustness of Morphosyntactic Transformation with Large Language Models: The Case of Quechua Collao](https://md.research.elamur.ai/free/en/papers/pocco-2026-robustness-morphosyntactic-transformation-large-language/) - [Common Sense vs. Morality: The Curious Case of Narrative Focus Bias in LLMs](https://md.research.elamur.ai/free/en/papers/purkayastha-2026-common-sense-vs-morality-curious/) - [Is This Idea Novel? An Automated Benchmark for Judgment of Research Ideas](https://md.research.elamur.ai/free/en/papers/schopf-2026-is-this-idea-novel-automated/) - [Children's English Reading Story Generation via Supervised Fine-Tuning of Compact LLMs with Controllable Difficulty and Safety](https://md.research.elamur.ai/free/en/papers/shen-2026-children-s-english-reading-story/) - [Prompting Instruction-tuned LLMs for Semantic Similarity Values](https://md.research.elamur.ai/free/en/papers/snelder-2026-prompting-instruction-tuned-llms-semantic/) - [STRUDEL: Unrolling a Benchmark for Evaluating Vision-Language Models on Structured Diagram Understanding across Domains](https://md.research.elamur.ai/free/en/papers/steinigen-2026-strudel-unrolling-benchmark-evaluating-vision/) - [Evaluating the Effect of Question Wording Variations on Answer Consistency in Large Language Models](https://md.research.elamur.ai/free/en/papers/takayama-2026-evaluating-effect-question-wording-variations/) - [HOTATE: A Japanese Dialogue Corpus Annotated with Responses of Private Thoughts and Public Statements](https://md.research.elamur.ai/free/en/papers/toda-2026-hotate-japanese-dialogue-corpus-annotated/) - [The Moral Foundations Reddit Corpus](https://md.research.elamur.ai/free/en/papers/trager-2026-moral-foundations-reddit-corpus/) - [CAIT: A Syntactic Parsing Toolkit for Child–Adult InTeractions](https://md.research.elamur.ai/free/en/papers/ud-2026-padovani-cait-syntactic-parsing-toolkit-child/) - [Quality and Appropriateness of Large Text Datasets for Irish NLP](https://md.research.elamur.ai/free/en/papers/walsh-2026-quality-appropriateness-large-text-datasets/) - [The Surge of Anti-Semitism in German Social Media following the October 7 Attacks](https://md.research.elamur.ai/free/en/papers/wiedemann-2026-surge-anti-semitism-german-social/) - [Rhetorical Questions in LLM Representations: A Linear Probing Study](https://md.research.elamur.ai/free/en/papers/yao-2026-rhetorical-questions-llm-representations-linear/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{agarwal2025gpt, title = {{gpt-oss-120b \& gpt-oss-20b Model Card}}, author = {Agarwal, OpenAI: Sandhini and Ahmad, Lama and Ai, Jason and Altman, Sam and Applebaum, Andy and Arbus, Edwin and Arora, Rahul K. and Bai, Yu and Baker, Bowen and Bao, Haiming and Barak, Boaz and Bennett, Ally and Bertao, Tyler and Brett, Nivedita and Brevdo, Eugene and Brockman, Greg and Bubeck, Sebastien and Chang, Che and Chen, Kai and Chen, Mark and Cheung, Enoch and Clark, Aidan and Cook, Dan and Dukhan, Marat and Dvorak, Casey and Fives, Kevin and Fomenko, Vlad and Garipov, Timur and Georgiev, Kristian and Glaese, Mia and Gogineni, Tarun and Goucher, Adam and Gross, Lukas and Guzman, Katia Gil and Hallman, John and Hehir, Jackie and Heidecke, Johannes and Helyar, Alec and Hu, Haitang and Huet, Romain and Huh, Jacob and Jain, Saachi and Johnson, Zach and Koch, Chris and Kofman, Irina and Kundel, Dominik and Kwon, Jason and Kyrylov, Volodymyr and Le, Elaine Ya and Leclerc, Guillaume and Lennon, James Park and Lessans, Scott and Lezcano-Casado, Mario and Li, Yuanzhi and Li, Zhuohan and Lin, Ji and Liss, Jordan and Liu, Lily and Liu, Jiancheng and Lu, Kevin and Lu, Chris and Martinovic, Zoran and McCallum, Lindsay and McGrath, Josh and McKinney, Scott and McLaughlin, Aidan and Mei, Song and Mostovoy, Steve and Mu, Tong and Myles, Gideon and Neitz, Alexander and Nichol, Alex and Pachocki, Jakub and Paino, Alex and Palmie, Dana and Pantuliano, Ashley and Parascandolo, Giambattista and Park, Jongsoo and Pathak, Leher and Paz, Carolina and Peran, Ludovic and Pimenov, Dmitry and Pokrass, Michelle and Proehl, Elizabeth and Qiu, Huida and Raila, Gaby and Raso, Filippo and Ren, Hongyu and Richardson, Kimmy and Robinson, David and Rotsted, Bob and Salman, Hadi and Sanjeev, Suvansh and Schwarzer, Max and Sculley, D. and Sikchi, Harshit and Simon, Kendal and Singhal, Karan and Song, Yang and Stuckey, Dane and Sun, Zhiqing and Tillet, Philippe and Toizer, Sam and Tsimpourlas, Foivos and Vyas, Nikhil and Wallace, Eric and Wang, Xin and Wang, Miles and Watkins, Olivia and Weil, Kevin and Wendling, Amy and Whinnery, Kevin and Whitney, Cedric and Wong, Hannah and Yang, Lin and Yang, Yu and Yasunaga, Michihiro and Ying, Kristen and Zaremba, Wojciech and Zhan, Wenting and Zhang, Cyril and Zhang, Brian and Zhang, Eddie and Zhao, Shengjia}, year = {2025}, doi = {10.48550/arxiv.2508.10925}, eprint = {2508.10925}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2508.10925} } ``` - [← previous GPT-NER: Named Entity Recognition via Large Language Models](https://md.research.elamur.ai/free/en/papers/wang-2025-gpt-ner-named-entity-recognition/) - [next → GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning](https://md.research.elamur.ai/free/en/papers/ramos-2025-grammamt-improving-machine-translation-grammar/) - [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/2/) ## References 35 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of Advances in Neural Information Processing Systems, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Vaswani%2C+N%2E+Shazeer%2C+N%2E+Parmar%2C+J%2E+Uszkoreit%2C+L%2E+Jones%2C+A%2E+N%2E+Gomez%2C+L%2E+Kaiser%2C+and+I%2E+Polosukhin%2C+%E2%80%9CAttention+is+all+you+need%2C%E2%80%9D+in+Proceedings+of+Advances+in+Neural+Information+Processing+Systems%2C+) 2. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 2017. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+A%2E+Mirhoseini%2C+K%2E+Maziarz%2C+A%2E+Davis%2C+Q%2E+Le%2C+G%2E+Hinton%2C+and+J%2E+Dean%2C+%E2%80%9COutrageously+large+neural+networks%3A+The+sparsely%2Dgated+mixture%2Dof%2Dexperts+layer%2C%E2%80%9D+2017%2E) 3. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Lepikhin%2C+H%2E+Lee%2C+Y%2E+Xu%2C+D%2E+Chen%2C+O%2E+Firat%2C+Y%2E+Huang%2C+M%2E+Krikun%2C+N%2E+Shazeer%2C+and+Z%2E+Chen%2C+%E2%80%9CGshard%3A+Scaling+giant+models+with+conditional+computation+and+automatic+sharding%2C%E2%80%9D+arXiv+preprint+arXiv%3A20) 4. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in International conference on machine learning, pp. 5547–5569, PMLR, 2022. [find](https://scholar.google.com/scholar?q=N%2E+Du%2C+Y%2E+Huang%2C+A%2E+M%2E+Dai%2C+S%2E+Tong%2C+D%2E+Lepikhin%2C+Y%2E+Xu%2C+M%2E+Krikun%2C+Y%2E+Zhou%2C+A%2E+W%2E+Yu%2C+O%2E+Firat%2C+et+al%2E%2C+%E2%80%9CGlam%3A+Efficient+scaling+of+language+models+with+mixture%2Dof%2Dexperts%2C%E2%80%9D+in+International+conferen) [our card](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/) 5. O. C. Project, “OCP Microscaling Formats (MX) Specification Version 1.0,” technical report, Open Compute Project, Sept. 2023. [find](https://scholar.google.com/scholar?q=O%2E+C%2E+Project%2C+%E2%80%9COCP+Microscaling+Formats+%28MX%29+Specification+Version+1%2E0%2C%E2%80%9D+technical+report%2C+Open+Compute+Project%2C+Sept%2E+2023%2E) 6. B. Zhang and R. Sennrich, “Root mean square layer normalization,” 2019. [find](https://scholar.google.com/scholar?q=B%2E+Zhang+and+R%2E+Sennrich%2C+%E2%80%9CRoot+mean+square+layer+normalization%2C%E2%80%9D+2019%2E) 7. R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu, “On layer normalization in the transformer architecture,” 2020. [find](https://scholar.google.com/scholar?q=R%2E+Xiong%2C+Y%2E+Yang%2C+D%2E+He%2C+K%2E+Zheng%2C+S%2E+Zheng%2C+C%2E+Xing%2C+H%2E+Zhang%2C+Y%2E+Lan%2C+L%2E+Wang%2C+and+T%2E%2DY%2E+Liu%2C+%E2%80%9COn+layer+normalization+in+the+transformer+architecture%2C%E2%80%9D+2020%2E) 8. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Radford%2C+J%2E+Wu%2C+R%2E+Child%2C+D%2E+Luan%2C+D%2E+Amodei%2C+I%2E+Sutskever%2C+et+al%2E%2C+%E2%80%9CLanguage+models+are+unsupervised+multitask+learners%2C%E2%80%9D+OpenAI+blog%2C+2019%2E) 9. N. Shazeer, “GLU variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+%E2%80%9CGLU+variants+improve+transformer%2C%E2%80%9D+arXiv+preprint+arXiv%3A2002%2E05202%2C+2020%2E) 10. R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Child%2C+S%2E+Gray%2C+A%2E+Radford%2C+and+I%2E+Sutskever%2C+%E2%80%9CGenerating+long+sequences+with+sparse+transformers%2C%E2%80%9D+arXiv+preprint+arXiv%3A1904%2E10509%2C+2019%2E) 11. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” NeurIPS, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+et+al%2E%2C+%E2%80%9CLanguage+models+are+few%2Dshot+learners%2C%E2%80%9D+NeurIPS%2C+2020%2E) 12. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” 2023. [find](https://scholar.google.com/scholar?q=J%2E+Ainslie%2C+J%2E+Lee%2DThorp%2C+M%2E+de+Jong%2C+Y%2E+Zemlyanskiy%2C+F%2E+Lebr%C3%B3n%2C+and+S%2E+Sanghai%2C+%E2%80%9CGQA%3A+Training+generalized+multi%2Dquery+transformer+models+from+multi%2Dhead+checkpoints%2C%E2%80%9D+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/) 13. N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019. [find](https://scholar.google.com/scholar?q=N%2E+Shazeer%2C+%E2%80%9CFast+transformer+decoding%3A+One+write%2Dhead+is+all+you+need%2C%E2%80%9D+arXiv+preprint+arXiv%3A1911%2E02150%2C+2019%2E) 14. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing, 2024. [find](https://scholar.google.com/scholar?q=J%2E+Su%2C+M%2E+Ahmed%2C+Y%2E+Lu%2C+S%2E+Pan%2C+W%2E+Bo%2C+and+Y%2E+Liu%2C+%E2%80%9CRoformer%3A+Enhanced+transformer+with+rotary+position+embedding%2C%E2%80%9D+Neurocomputing%2C+2024%2E) 15. B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “YaRN: Efficient context window extension of large language models,” arXiv preprint arXiv:2309.00071, 2023. [find](https://scholar.google.com/scholar?q=B%2E+Peng%2C+J%2E+Quesnelle%2C+H%2E+Fan%2C+and+E%2E+Shippole%2C+%E2%80%9CYaRN%3A+Efficient+context+window+extension+of+large+language+models%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E00071%2C+2023%2E) 16. E. Miller, “Attention is off by one (2023),” URL [https://www.evanmiller.org/attention-is-off-by-one.html](https://www.evanmiller.org/attention-is-off-by-one.html). [↗](https://www.evanmiller.org/attention-is-off-by-one.html) 17. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” arXiv preprint arXiv:2309.17453, 2023. [find](https://scholar.google.com/scholar?q=G%2E+Xiao%2C+Y%2E+Tian%2C+B%2E+Chen%2C+S%2E+Han%2C+and+M%2E+Lewis%2C+%E2%80%9CEfficient+streaming+language+models+with+attention+sinks%2C%E2%80%9D+arXiv+preprint+arXiv%3A2309%2E17453%2C+2023%2E) 18. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “GPT-4o system card,” arXiv preprint arXiv:2410.21276, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Hurst%2C+A%2E+Lerer%2C+A%2E+P%2E+Goucher%2C+A%2E+Perelman%2C+A%2E+Ramesh%2C+A%2E+Clark%2C+A%2E+Ostrow%2C+A%2E+Welihinda%2C+A%2E+Hayes%2C+A%2E+Radford%2C+et+al%2E%2C+%E2%80%9CGPT%2D4o+system+card%2C%E2%80%9D+arXiv+preprint+arXiv%3A2410%2E21276%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/) 19. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Paszke%2C+S%2E+Gross%2C+F%2E+Massa%2C+A%2E+Lerer%2C+J%2E+Bradbury%2C+G%2E+Chanan%2C+T%2E+Killeen%2C+Z%2E+Lin%2C+N%2E+Gimelshein%2C+L%2E+Antiga%2C+et+al%2E%2C+%E2%80%9CPytorch%3A+An+imperative+style%2C+high%2Dperformance+deep+learning+library%2C%E2%80%9D+Advances+) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/) 20. P. Tillet, H.-T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019. [find](https://scholar.google.com/scholar?q=P%2E+Tillet%2C+H%2E%2DT%2E+Kung%2C+and+D%2E+Cox%2C+%E2%80%9CTriton%3A+an+intermediate+language+and+compiler+for+tiled+neural+network+computations%2C%E2%80%9D+in+Proceedings+of+the+3rd+ACM+SIGPLAN+International+Workshop+on+Machine+Learni) [our card](https://md.research.elamur.ai/free/en/papers/tillet-2019-triton-intermediate-language-compiler-tiled/) 21. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” 2022. [find](https://scholar.google.com/scholar?q=T%2E+Dao%2C+D%2E+Y%2E+Fu%2C+S%2E+Ermon%2C+A%2E+Rudra%2C+and+C%2E+R%C3%A9%2C+%E2%80%9CFlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2C%E2%80%9D+2022%2E) 22. OpenAI, “Chowdhury, neil and aung, james and shern, chan jun and jaffe, oliver and sherburn, dane and starace, giulio and mays, evan and dias, rachel and aljubeh, marwan and glaese, mia and jimenez, carlos e and yang, john and ho, leyton and patwardhan, tejal and liu, kevin and madry, aleksander.” h… [find](https://scholar.google.com/scholar?q=OpenAI%2C+%E2%80%9CChowdhury%2C+neil+and+aung%2C+james+and+shern%2C+chan+jun+and+jaffe%2C+oliver+and+sherburn%2C+dane+and+starace%2C+giulio+and+mays%2C+evan+and+dias%2C+rachel+and+aljubeh%2C+marwan+and+glaese%2C+mia+and+jimenez%2C+c) 23. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. [find](https://scholar.google.com/scholar?q=S%2E+Yao%2C+N%2E+Shinn%2C+P%2E+Razavi%2C+and+K%2E+Narasimhan%2C+%E2%80%9C%CF%84+%2Dbench%3A+A+benchmark+for+tool%2Dagent%2Duser+interaction+in+real%2Dworld+domains%2C%E2%80%9D+arXiv+preprint+arXiv%3A2406%2E12045%2C+2024%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/) 24. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof QA benchmark,” in COLM, 2024. [find](https://scholar.google.com/scholar?q=D%2E+Rein%2C+B%2E+L%2E+Hou%2C+A%2E+C%2E+Stickland%2C+J%2E+Petty%2C+R%2E+Y%2E+Pang%2C+J%2E+Dirani%2C+J%2E+Michael%2C+and+S%2E+R%2E+Bowman%2C+%E2%80%9CGPQA%3A+A+graduate%2Dlevel+google%2Dproof+QA+benchmark%2C%E2%80%9D+in+COLM%2C+2024%2E) 25. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020. [find](https://scholar.google.com/scholar?q=D%2E+Hendrycks%2C+C%2E+Burns%2C+S%2E+Basart%2C+A%2E+Zou%2C+M%2E+Mazeika%2C+D%2E+Song%2C+and+J%2E+Steinhardt%2C+%E2%80%9CMeasuring+massive+multitask+language+understanding%2C%E2%80%9D+arXiv+preprint+arXiv%3A2009%2E03300%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) 26. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al., “Humanity’s last exam,” arXiv preprint arXiv:2501.14249, 2025. [find](https://scholar.google.com/scholar?q=L%2E+Phan%2C+A%2E+Gatti%2C+Z%2E+Han%2C+N%2E+Li%2C+J%2E+Hu%2C+H%2E+Zhang%2C+C%2E+B%2E+C%2E+Zhang%2C+M%2E+Shaaban%2C+J%2E+Ling%2C+S%2E+Shi%2C+et+al%2E%2C+%E2%80%9CHumanity%E2%80%99s+last+exam%2C%E2%80%9D+arXiv+preprint+arXiv%3A2501%2E14249%2C+2025%2E) [our card](https://md.research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/) 27. N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, E. Mays, R. Dias, M. Aljubeh, M. Glaese, C. E. Jimenez, J. Yang, L. Ho, T. Patwardhan, K. Liu, and A. Madry, “Introducing SWE-bench Verified,” OpenAI, 2024. [find](https://scholar.google.com/scholar?q=N%2E+Chowdhury%2C+J%2E+Aung%2C+C%2E+J%2E+Shern%2C+O%2E+Jaffe%2C+D%2E+Sherburn%2C+G%2E+Starace%2C+E%2E+Mays%2C+R%2E+Dias%2C+M%2E+Aljubeh%2C+M%2E+Glaese%2C+C%2E+E%2E+Jimenez%2C+J%2E+Yang%2C+L%2E+Ho%2C+T%2E+Patwardhan%2C+K%2E+Liu%2C+and+A%2E+Madry%2C+%E2%80%9CIntroducing+SWE%2Dben) 28. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al., “HealthBench: Evaluating large language models towards improved human health,” arXiv preprint arXiv:2505.08775, 2025. [find](https://scholar.google.com/scholar?q=R%2E+K%2E+Arora%2C+J%2E+Wei%2C+R%2E+S%2E+Hicks%2C+P%2E+Bowman%2C+J%2E+Qui%C3%B1onero%2DCandela%2C+F%2E+Tsimpourlas%2C+M%2E+Sharman%2C+M%2E+Shah%2C+A%2E+Vallone%2C+A%2E+Beutel%2C+et+al%2E%2C+%E2%80%9CHealthBench%3A+Evaluating+large+language+models+towards+improved+h) [our card](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/) 29. M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese, “Deliberative alignment: Reasoning enables safer language models,” arXiv preprint arXiv:2412.16339, 2024. [find](https://scholar.google.com/scholar?q=M%2E+Y%2E+Guan%2C+M%2E+Joglekar%2C+E%2E+Wallace%2C+S%2E+Jain%2C+B%2E+Barak%2C+A%2E+Helyar%2C+R%2E+Dias%2C+A%2E+Vallone%2C+H%2E+Ren%2C+J%2E+Wei%2C+H%2E+W%2E+Chung%2C+S%2E+Toyer%2C+J%2E+Heidecke%2C+A%2E+Beutel%2C+and+A%2E+Glaese%2C+%E2%80%9CDeliberative+alignment%3A+Reasoning) 30. E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The instruction hierarchy: Training LLMs to prioritize privileged instructions,” arXiv preprint arXiv:2404.13208, 2024. [find](https://scholar.google.com/scholar?q=E%2E+Wallace%2C+K%2E+Xiao%2C+R%2E+Leike%2C+L%2E+Weng%2C+J%2E+Heidecke%2C+and+A%2E+Beutel%2C+%E2%80%9CThe+instruction+hierarchy%3A+Training+LLMs+to+prioritize+privileged+instructions%2C%E2%80%9D+arXiv+preprint+arXiv%3A2404%2E13208%2C+2024%2E) 31. A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, et al., “A strongreject for empty jailbreaks,” arXiv preprint arXiv:2402.10260, 2024. [find](https://scholar.google.com/scholar?q=A%2E+Souly%2C+Q%2E+Lu%2C+D%2E+Bowen%2C+T%2E+Trinh%2C+E%2E+Hsieh%2C+S%2E+Pandey%2C+P%2E+Abbeel%2C+J%2E+Svegliato%2C+S%2E+Emmons%2C+O%2E+Watkins%2C+et+al%2E%2C+%E2%80%9CA+strongreject+for+empty+jailbreaks%2C%E2%80%9D+arXiv+preprint+arXiv%3A2402%2E10260%2C+2024%2E) 32. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman, “BBQ: A hand-built bias benchmark for question answering,” arXiv preprint arXiv:2110.08193, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Parrish%2C+A%2E+Chen%2C+N%2E+Nangia%2C+V%2E+Padmakumar%2C+J%2E+Phang%2C+J%2E+Thompson%2C+P%2E+M%2E+Htut%2C+and+S%2E+R%2E+Bowman%2C+%E2%80%9CBBQ%3A+A+hand%2Dbuilt+bias+benchmark+for+question+answering%2C%E2%80%9D+arXiv+preprint+arXiv%3A2110%2E08193%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/parrish-2022-bbq-hand-built-bias-benchmark/) 33. T. Patwardhan, K. Liu, T. Markov, N. Chowdhury, D. Leet, N. Cone, C. Maltbie, J. Huizinga, C. Wainwright, S. Jackson, S. Adler, R. Casagrande, and A. Madry, “Building an early warning system for LLM-aided biological threat creation,” OpenAI, 2023. [find](https://scholar.google.com/scholar?q=T%2E+Patwardhan%2C+K%2E+Liu%2C+T%2E+Markov%2C+N%2E+Chowdhury%2C+D%2E+Leet%2C+N%2E+Cone%2C+C%2E+Maltbie%2C+J%2E+Huizinga%2C+C%2E+Wainwright%2C+S%2E+Jackson%2C+S%2E+Adler%2C+R%2E+Casagrande%2C+and+A%2E+Madry%2C+%E2%80%9CBuilding+an+early+warning+system+for+LLM%2Da) 34. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques, “LAB-Bench: Measuring capabilities of language models for biology research,” 2024. [find](https://scholar.google.com/scholar?q=J%2E+M%2E+Laurent%2C+J%2E+D%2E+Janizek%2C+M%2E+Ruzo%2C+M%2E+M%2E+Hinks%2C+M%2E+J%2E+Hammerling%2C+S%2E+Narayanan%2C+M%2E+Ponnapati%2C+A%2E+D%2E+White%2C+and+S%2E+G%2E+Rodriques%2C+%E2%80%9CLAB%2DBench%3A+Measuring+capabilities+of+language+models+for+biology+re) 35. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan, “PaperBench: Evaluating ai’s ability to replicate ai research.” [https://openai.com/index/paperbench/](https://openai.com/index/paperbench/), 2025. [↗](https://openai.com/index/paperbench/)