---
title: "Scaling Laws for Code: Every Programming Language Matters"
description: "This study runs more than 1000 pre-training experiments from scratch, costing roughly 336,000 H800… 26 references, cited by 2 papers here."
authors: ["Jian Yang", "Shuyue Guo", "Linzheng Chai", "Wei Zhang", "Aishan Liu", "Chuan Hao", "Zhoujun Li", "Xin Zhao", "Xianglong Liu", "Weifeng Lv", "Bryan Dai"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.487/"
doi: "10.18653/v1/2026.findings-acl.487"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2026-scaling-laws-code-every-programming/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2026-scaling-laws-code-every-programming/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/) / Scaling Laws for Code: Every Programming Language Matters

# Scaling Laws for Code: Every Programming Language Matters

**Authors:** [Jian Yang](https://md.research.elamur.ai/free/en/authors/jian-yang/), [Shuyue Guo](https://md.research.elamur.ai/free/en/authors/shuyue-guo/), [Linzheng Chai](https://md.research.elamur.ai/free/en/authors/linzheng-chai/), [Wei Zhang](https://md.research.elamur.ai/free/en/authors/wei-zhang/), [Aishan Liu](https://md.research.elamur.ai/free/en/authors/aishan-liu/), Chuan Hao, [Zhoujun Li](https://md.research.elamur.ai/free/en/authors/zhoujun-li/), [Xin Zhao](https://md.research.elamur.ai/free/en/authors/xin-zhao/), [Xianglong Liu](https://md.research.elamur.ai/free/en/authors/xianglong-liu/), Weifeng Lv and [Bryan Dai](https://md.research.elamur.ai/free/en/authors/bryan-dai/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.487/) · [PDF](https://aclanthology.org/2026.findings-acl.487.pdf) · [DOI 10.18653/v1/2026.findings-acl.487](https://doi.org/10.18653/v1/2026.findings-acl.487) · [OpenAlex](https://openalex.org/W4417459239)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Classic scaling laws treat code as a single, language-agnostic domain. In practice, programming languages (PLs) contribute unequally during pre-training, so loss predictions for multilingual code LLMs can be inaccurate. This study runs more than 1000 pre-training experiments from scratch, costing roughly 336,000 H800 GPU hours, with models from 0.2B to 14B parameters and data budgets up to 1T tokens. The authors first fit Chinchilla-style laws separately for seven PLs. They then test how pairing two languages affects the target language and compare data-organization schemes for cross-lingual ability. Finally, they combine these results into a proportion-dependent multilingual scaling law that includes language-specific parameters and transfer terms. Interpreted languages gain more from scale than compiled ones. Similar languages help each other, and parallel code pairs improve translation. A token mix chosen with the fitted law achieves lower loss and better downstream scores than uniform allocation, without hurting any single language. This gives practitioners a principled way to choose data mixtures without costly ablations.

## Key points
- Seven languages were examined individually: Rust, Go, C#, TypeScript, JavaScript, Java and Python. Each was trained at 10 model sizes (0.1B–3.1B) and 6 token budgets (2B–64B), giving 420 runs. All models use a LLaMA-2-like architecture (SwiGLU, RoPE, MHA, RMSNorm).
- Fitted irreducible loss L∞ ranks the languages as C# \< Java ≈ Rust \< Go \< TypeScript \< JavaScript \< Python. Strictly typed languages are easier to predict. Python has the largest α_N and α_D, while Rust's exponents are notably small.
- Bilingual tests used a 128B-token budget. Pairing 64B tokens of the target PL with 64B of another PL usually beat repeating the target data twice. Syntactically close pairs such as Java–C# and JavaScript–TypeScript transferred best.
- Parallel pairing, meaning concatenating translated snippets either directly or with prompts, clearly outperformed the baseline on code translation. Translation into Python scored noticeably lower than other directions.
- Final comparison: two models were trained on 400B tokens (350B code plus 50B FineWeb-Edu text), one with uniform and one with law-guided allocation. The guided mix shifts tokens toward Python and away from fast-saturating languages such as Rust. It improved average BLEU on translation and Pass@1 on MultiPL-E.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Beihang University](https://md.research.elamur.ai/free/en/universities/beihang-university/) — University, Beijing, China · 65 works · 8,452 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Jian Yang, Shuyue Guo, Linzheng Chai, Wei Zhang, Aishan Liu, Chuan Hao, Zhoujun Li, Xin Zhao, Xianglong Liu, Weifeng Lv and Bryan Dai. 2026. Scaling Laws for Code: Every Programming Language Matters. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2026scaling,
  title = {{Scaling Laws for Code: Every Programming Language Matters}},
  author = {Jian Yang, Shuyue Guo, Linzheng Chai, Wei Zhang, Aishan Liu, Chuan Hao, Zhoujun Li, Xin Zhao, Xianglong Liu, Weifeng Lv and Bryan Dai},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.487},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.487}
}
```

- [← previous SCALER: Synthetic Scalable Adaptive Learning Environment for Reasoning](https://md.research.elamur.ai/free/en/papers/xu-2026-scaler-synthetic-scalable-adaptive-learning/)
- [next → Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework](https://md.research.elamur.ai/free/en/papers/yan-2026-sculpting-vector-space-efficient-multi/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/9/)

## References

26 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. [SantaCoder: Don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/) arXiv preprint arXiv:2301.03988.
2. Anysphere Inc. 2025. Cursor features. [https://cursor.com/features](https://cursor.com/features). [↗](https://cursor.com/features)
3. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. [find](https://scholar.google.com/scholar?q=Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+et+al%2E+2024%2E+Deepseek+llm%3A+Scaling+open%2Dsource+language+models+with+longtermism%2E+)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. [Codebert: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/). In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-…
7. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. [Graphcodebert: Pre-training code representations with da…](https://md.research.elamur.ai/free/en/papers/guo-2021-graphcodebert-pre-training-code-representations/)
8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
9. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code)
10. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2019. [Deep learning scaling is predictable, empirically](https://md.research.elamur.ai/free/en/papers/hestness-2017-deep-learning-scaling-predictable-empirically/). arXiv preprint arXiv:1712.00409.
11. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
12. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas… [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
13. Xianzhen Luo, Wenzhen Zheng, Qingfu Zhu, Rongyi Zhang, Houyi Li, Siming Huang, YuanTao Fan, and Wanxiang Che. 2025. [Scaling laws for code: A more data-hungry regime](https://md.research.elamur.ai/free/en/papers/luo-2026-scaling-laws-code-more-data/). arXiv preprint arXiv:2510.08702.
14. Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. 2025. Os agents: A survey on mllm-based agents for general computing devices use. arXiv preprint arXiv:2508.04482. — [arXiv:2508.04482](https://arxiv.org/abs/2508.04482) [find](https://scholar.google.com/scholar?q=Xueyu+Hu%2C+Tao+Xiong%2C+Biao+Yi%2C+Zishu+Wei%2C+Ruixuan+Xiao%2C+Yurun+Chen%2C+Jiasheng+Ye%2C+Meiling+Tao%2C+Xiangxin+Zhou%2C+Ziyu+Zhao%2C+et+al%2E+2025%2E+Os+agents%3A+A+survey+on+mllm%2Dbased+agents+for+general+computing+devic)
15. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-… [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2023%2E+CodeGen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+In+)
16. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
17. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/).
18. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
19. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. [Are emergent abilities of large language models a mirage?](https://md.research.elamur.ai/free/en/papers/schaeffer-2023-emergent-abilities-large-language-models/) Advances in neural information processing systems, 36:55565–55581.
20. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
21. Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. 2019. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(112):1–49. [find](https://scholar.google.com/scholar?q=Christopher+J+Shallue%2C+Jaehoon+Lee%2C+Joseph+Antognini%2C+Jascha+Sohl%2DDickstein%2C+Roy+Frostig%2C+and+George+E+Dahl%2E+2019%2E+Measuring+the+effects+of+data+parallelism+on+neural+network+training%2E+Journal+of+Mach)
22. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. [Coderl: Mastering code generation through pretrained models and deep reinforcement learning](https://md.research.elamur.ai/free/en/papers/le-2022-coderl-mastering-code-generation-through/). In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems…
23. Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. 2024. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890. — [arXiv:2411.04890](https://arxiv.org/abs/2411.04890) [find](https://scholar.google.com/scholar?q=Shuai+Wang%2C+Weiwen+Liu%2C+Jingxuan+Chen%2C+Yuqi+Zhou%2C+Weinan+Gan%2C+Xingshan+Zeng%2C+Yuhan+Che%2C+Shuai+Yu%2C+Xinlong+Hao%2C+Kun+Shao%2C+et+al%2E+2024%2E+Gui+agents+with+foundation+models%3A+A+comprehensive+survey%2E+arXiv+p)
24. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+2021%2E+CodeT5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2109%2E00859%2E)
25. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). arXiv preprint arXiv:2206.07682.
26. Jian Yang, Xianglong Liu, Weifeng Lv, Ken Deng, Shawn Guo, Lin Jing, Yizhi Li, Shark Liu, Xianzhen Luo, Yuyu Luo, et al. 2025. From code foundation models to agents and applications: A comprehensive survey and practical guide to code intelligence. arXiv preprint arXiv:2511.18538. — [arXiv:2511.18538](https://arxiv.org/abs/2511.18538) [find](https://scholar.google.com/scholar?q=Jian+Yang%2C+Xianglong+Liu%2C+Weifeng+Lv%2C+Ken+Deng%2C+Shawn+Guo%2C+Lin+Jing%2C+Yizhi+Li%2C+Shark+Liu%2C+Xianzhen+Luo%2C+Yuyu+Luo%2C+et+al%2E+2025%2E+From+code+foundation+models+to+agents+and+applications%3A+A+comprehensive+s)

*Updated 31 October 2026*