---
title: "Language Models Encode the Value of Numbers Linearly (2025)"
description: "The paper asks whether large language models internally represent the numeric magnitude of… 36 references, cited by 14 papers here."
authors: ["Fangwei Zhu", "Damai Dai", "Zhifang Sui"]
year: 2025
venue: "Proceedings of the 31st International Conference on Computational Linguistics"
source: "https://aclanthology.org/2025.coling-main.47/"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhu-2025-language-models-encode-value-numbers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhu-2025-language-models-encode-value-numbers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / Language Models Encode the Value of Numbers Linearly

# Language Models Encode the Value of Numbers Linearly

**Authors:** Fangwei Zhu, [Damai Dai](https://md.research.elamur.ai/free/en/authors/damai-dai/) and [Zhifang Sui](https://md.research.elamur.ai/free/en/authors/zhifang-sui/) · **Year:** 2025 · **Venue:** [Proceedings of the 31st International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2025](https://md.research.elamur.ai/free/en/venues/coling/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.coling-main.47/) · [PDF](https://aclanthology.org/2025.coling-main.47.pdf) · [OpenAlex](https://openalex.org/W4390722398)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/zhu-2025-language-models-encode-value-numbers/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether large language models internally represent the numeric magnitude of numbers in their input, and how. The authors generated a synthetic set of addition questions with operands of 2 to 10 digits. They fed these to LLaMA-2-7B, LLaMA-2-13B and Mistral-7B without fine-tuning, recorded the residual-stream hidden states at every layer, and fitted per-layer Ridge-regression probes to predict the base-2 logarithm of each operand and of the model's answer. The probes achieved strong correlation and R² from the very first layer onward, so numeric values are largely linearly decodable. Exact recovery, however, was poor: approximate accuracy never reached 50%. Decoding quality peaks in intermediate layers and degrades somewhat in later ones. The authors also showed that activation patching and simple vector additions can shift the model's computed result, which indicates the encoded values are actually used. The work clarifies how LLMs handle arithmetic and suggests directions such as dedicated number encoders and error-correction modules.

## Key points
- **Dataset:** 1,000 addition prompts per operand length (2–10 digits), split 80/10/10. Subtraction gave similar results. Multiplication was excluded because models scored near 0% even on 5-digit cases.
- **Probing setup:** Ridge regression with λ = 0.1 on log₂ values. Metrics were Pearson ρ, out-of-sample R², approximate accuracy (AAcc, within 1% error) and MSE.
- **Readout positions:** operand values were read at their final digit token, and the model's answer at the last input token.
- **Layer-wise behaviour in LLaMA-2-7B:**
  - Layer 0 mostly reflects how many digits a number has.
  - Layer 10 yields near-linear reconstructions.
  - Late layers (e.g., 30) add uniformly spread noise.
- **Resources:** code and data are released as NumProbe (github.com/solitaryzero/NumProbe). Closed models like GPT-4 were not tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Peking University](https://md.research.elamur.ai/free/en/universities/peking-university/) — University, Beijing, China · 244 works · 39,359 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [➗ Mathematics](https://md.research.elamur.ai/free/en/shelves/mathematics/) · 177 works · [➗ Statistics and Probability](https://md.research.elamur.ai/free/en/shelves/mathematics/statistics-and-probability/) · 16 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Fangwei Zhu, Damai Dai and Zhifang Sui. 2025. Language Models Encode the Value of Numbers Linearly. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhu2025language,
  title = {{Language Models Encode the Value of Numbers Linearly}},
  author = {Fangwei Zhu, Damai Dai and Zhifang Sui},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.47/}
}
```

- [← previous L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models](https://md.research.elamur.ai/free/en/papers/jeon-2025-l4q-parameter-efficient-quantization-aware/)
- [next → LLäMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratch](https://md.research.elamur.ai/free/en/papers/pfister-2025-llammlein-transparent-compact-competitive-german/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

36 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. [find](https://scholar.google.com/scholar?q=Guillaume+Alain+and+Yoshua+Bengio%2E+2016%2E+Understanding+intermediate+layers+using+linear+classifier+probes%2E+arXiv+preprint+arXiv%3A1610%2E01644%2E)
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403.
3. Yonatan Belinkov. 2022. [Probing classifiers: Promises, shortcomings, and advances](https://md.research.elamur.ai/free/en/papers/belinkov-2022-probing-classifiers-promises-shortcomings-advances/). Computational Linguistics, 48(1):207–219.
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. 2024. [Faith and fate: Limits of transformers on compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/). Advances in Neural Information Processing Systems, 36.
8. Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. 2024. Not all language model features are linear. arXiv preprint arXiv:2405.14860. — [arXiv:2405.14860](https://arxiv.org/abs/2405.14860) [find](https://scholar.google.com/scholar?q=Joshua+Engels%2C+Isaac+Liao%2C+Eric+J+Michaud%2C+Wes+Gurnee%2C+and+Max+Tegmark%2E+2024%2E+Not+all+language+model+features+are+linear%2E+arXiv+preprint+arXiv%3A2405%2E14860%2E+%E2%80%94+%5BarXiv%3A2405%2E14860%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F24)
9. Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. 2023. xval: A continuous number encoding for large language models. In NeurIPS 2023 AI for Science Workshop. [find](https://scholar.google.com/scholar?q=Siavash+Golkar%2C+Mariel+Pettee%2C+Michael+Eickenberg%2C+Alberto+Bietti%2C+Miles+Cranmer%2C+Geraud+Krawezik%2C+Francois+Lanusse%2C+Michael+McCabe%2C+Ruben+Ohana%2C+Liam+Parker%2C+et+al%2E+2023%2E+xval%3A+A+continuous+number+en)
10. Wes Gurnee and Max Tegmark. 2023. Language models represent space and time. arXiv preprint arXiv:2310.02207. [find](https://scholar.google.com/scholar?q=Wes+Gurnee+and+Max+Tegmark%2E+2023%2E+Language+models+represent+space+and+time%2E+arXiv+preprint+arXiv%3A2310%2E02207%2E)
11. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2024. [How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). Advances in Neural Information Processing Systems, 36.
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
13. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
14. Belinda Z Li, Maxwell Nye, and Jacob Andreas. 2021. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pap… [find](https://scholar.google.com/scholar?q=Belinda+Z+Li%2C+Maxwell+Nye%2C+and+Jacob+Andreas%2E+2021%2E+Implicit+representations+of+meaning+in+neural+language+models%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
15. Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2022. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Kenneth+Li%2C+Aspen+K+Hopkins%2C+David+Bau%2C+Fernanda+Vi%C3%A9gas%2C+Hanspeter+Pfister%2C+and+Martin+Wattenberg%2E+2022%2E+Emergent+world+representations%3A+Exploring+a+sequence+model+trained+on+a+synthetic+task%2E+In+The+)
16. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023a. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+et+al%2E+2023a%2E+Starcoder%3A+may+the+source+be+with+you%21+ar)
17. Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Poria. 2023b. Chain of knowledge: A framework for grounding large language models with structured knowledge bases. arXiv preprint arXiv:2305.13269. [find](https://scholar.google.com/scholar?q=Xingxuan+Li%2C+Ruochen+Zhao%2C+Yew+Ken+Chia%2C+Bosheng+Ding%2C+Lidong+Bing%2C+Shafiq+Joty%2C+and+Soujanya+Poria%2E+2023b%2E+Chain+of+knowledge%3A+A+framework+for+grounding+large+language+models+with+structured+knowledg)
18. Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771. [find](https://scholar.google.com/scholar?q=Thomas+McGrath%2C+Matthew+Rahtz%2C+Janos+Kramar%2C+Vladimir+Mikulik%2C+and+Shane+Legg%2E+2023%2E+The+hydra+effect%3A+Emergent+self%2Drepair+in+language+model+computations%2E+arXiv+preprint+arXiv%3A2307%2E15771%2E)
19. Jean-Michel Muller, Nicolas Brisebarre, Florent De Dinechin, Claude-Pierre Jeannerod, Vincent Lefevre, Guillaume Melquiond, Nathalie Revol, Damien Stehlé, Serge Torres, et al. 2018. Handbook of floating-point arithmetic. Springer. [find](https://scholar.google.com/scholar?q=Jean%2DMichel+Muller%2C+Nicolas+Brisebarre%2C+Florent+De+Dinechin%2C+Claude%2DPierre+Jeannerod%2C+Vincent+Lefevre%2C+Guillaume+Melquiond%2C+Nathalie+Revol%2C+Damien+Stehl%C3%A9%2C+Serge+Torres%2C+et+al%2E+2018%2E+Handbook+of+floati)
20. Neel Nanda, Andrew Lee, and Martin Wattenberg. 2023. Emergent linear representations in world models of self-supervised sequence models. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16–30. [find](https://scholar.google.com/scholar?q=Neel+Nanda%2C+Andrew+Lee%2C+and+Martin+Wattenberg%2E+2023%2E+Emergent+linear+representations+in+world+models+of+self%2Dsupervised+sequence+models%2E+In+Proceedings+of+the+6th+BlackboxNLP+Workshop%3A+Analyzing+and+I)
21. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+In+)
22. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do nlp models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
23. Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. 2023. [Math-shepherd: A label-free step-by-step verifier for llms in mathematical reasoning](https://md.research.elamur.ai/free/en/papers/wang-2024-math-shepherd-verify-reinforce-llms/). arXiv preprint arXiv:2312.08935.
24. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
25. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744.
27. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. [Does string-based neural mt learn source syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1526– 1534.
28. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7035–7052. [find](https://scholar.google.com/scholar?q=Alessandro+Stolfo%2C+Yonatan+Belinkov%2C+and+Mrinmaya+Sachan%2E+2023%2E+A+mechanistic+interpretation+of+arithmetic+reasoning+in+language+models+using+causal+mediation+analysis%2E+In+Proceedings+of+the+2023+Conf)
29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Advances in Neural Information Processing Systems, 35:24824–24837.
30. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
31. Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023. [Verify-and-edit: A knowledge-enhanced chain-of-thought framework](https://md.research.elamur.ai/free/en/papers/zhao-2023-verify-edit-knowledge-enhanced-chain/). arXiv preprint arXiv:2305.03268.
32. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. 2024. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Ziqian+Zhong%2C+Ziming+Liu%2C+Max+Tegmark%2C+and+Jacob+Andreas%2E+2024%2E+The+clock+and+the+pizza%3A+Two+stories+in+mechanistic+explanation+of+neural+networks%2E+Advances+in+Neural+Information+Processing+Systems%2C+3)
33. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [find](https://scholar.google.com/scholar?q=Rohan+Taori%2C+Ishaan+Gulrajani%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Xuechen+Li%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Stanford+alpaca%3A+An+instruction%2Dfollowing+llama+model%2E)
34. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. 2018. [What do you learn from context? probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/). In International Conference on Learning Re…
35. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
36. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.

*Updated 30 October 2026*