---
title: "LLMs Know More About Numbers than They Can Say (2026)"
description: "The paper investigates whether LLMs actually understand numerical magnitude. 99% accuracy. 18 references."
authors: ["Fengting Yuchi", "Li Du", "Jason Eisner"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2026.eacl-short.47/"
doi: "10.18653/v1/2026.eacl-short.47"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yuchi-2026-llms-know-more-about-numbers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yuchi-2026-llms-know-more-about-numbers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/) / LLMs Know More About Numbers than They Can Say

# LLMs Know More About Numbers than They Can Say

**Authors:** Fengting Yuchi, [Li Du](https://md.research.elamur.ai/free/en/authors/li-du/) and [Jason Eisner](https://md.research.elamur.ai/free/en/authors/jason-eisner/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-short.47/) · [PDF](https://aclanthology.org/2026.eacl-short.47.pdf) · [DOI 10.18653/v1/2026.eacl-short.47](https://doi.org/10.18653/v1/2026.eacl-short.47) · [OpenAlex](https://openalex.org/W7140088556)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper investigates whether LLMs actually understand numerical magnitude. Frontier non-reasoning models such as GPT-4.1 still make errors when comparing numbers written in different notations, for example scientific notation versus a plain decimal. The authors studied open-weight 7B–8B models (base, instruction-tuned and distilled) using two kinds of data: a synthetic cross-notation comparison dataset and arXiv text drawn from peS2o. They trained linear probes on hidden states. Regression probes predicted log₂ magnitudes, and logistic-regression classifiers predicted which numeral in a pair is larger.

Magnitude turns out to be linearly decodable, and the hidden state at the end of the prompt already contains the comparison result with high accuracy. Yet when the same models are asked to state the answer, they perform barely above chance. Probe quality in the earliest layers correlates with how well a model answers verbally. Adding the classifier's loss as an auxiliary objective during finetuning raises verbal accuracy further. The authors interpret this as evidence that strengthening internal numeric representations can improve the numeracy a model expresses in its answers.

## Key points
- **Magnitude probing results for Mistral-7B:**
  - A single linear probe on one late layer recovers numeral magnitudes with about 2.3% median relative error on synthetic prompts (R² > 0.998 with mixed notations).
  - On arXiv text the error rises to 19.06% (R² = 0.51), probably because numbers in real papers play more varied roles.
- **Gap between probing and verbal answers:**
  - Comparison classifiers reach about 93–99% accuracy across models.
  - One-shot verbal answers reach only 50–70%. For example, Qwen3-8B scores 70.00% verbally but 98.88% with the probe, and Mistral-7B-v0.1 scores 50.00% versus 96.44%.
  - For reference, GPT-4.1 and GPT-4.1-mini score 94.19% and 92.12% verbally.
- **Limits of regression-based comparison:** Comparing two regression-predicted magnitudes breaks down when one number is less than twice the other, and falls to chance when |log₂(a/b)| \< 0.2. The dedicated classifier handles close pairs much better.
- **Early layers predict verbal skill:** Average probe performance over the first three layers correlates strongly with verbalized accuracy across the seven open models.
- **Finetuning with an auxiliary probe loss:**
  - The training objective adds the classifier's loss, weighted by β = 0.02, to the language-modeling loss, with the probe placed at 90% of model depth.
  - This gives an extra 3.22% verbal accuracy compared with ordinary finetuning.
  - Two of the three Mistral- and Qwen-based models reach at least 99% accuracy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 247 works · 106,235 citations
- [Shanghai Jiao Tong University](https://md.research.elamur.ai/free/en/universities/shanghai-jiao-tong-university/) — University, Shanghai, China · 212 works · 19,544 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🧮 Computational Theory and Mathematics](https://md.research.elamur.ai/free/en/shelves/computer-science/computational-theory-and-mathematics/) · 321 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Fengting Yuchi, Li Du and Jason Eisner. 2026. LLMs Know More About Numbers than They Can Say. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yuchi2026llms,
  title = {{LLMs Know More About Numbers than They Can Say}},
  author = {Fengting Yuchi, Li Du and Jason Eisner},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-short.47},
  url = {https://doi.org/10.18653/v1/2026.eacl-short.47}
}
```

- [← previous LLMs for Now, Fine-Tuning for Later: An Ensemble Approach to Data Drift in Domain-Specific Tasks](https://md.research.elamur.ai/free/en/papers/lu-2026-llms-now-fine-tuning-later/)
- [next → LLMs Meet Isolation Kernel: Lightweight, Learning-free Binary Embeddings for Fast Retrieval](https://md.research.elamur.ai/free/en/papers/zhang-2026-llms-meet-isolation-kernel-lightweight/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/40/)

## References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. [How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model](https://md.research.elamur.ai/free/en/papers/hanna-2023-gpt-2-compute-greater-than/). In Thirty-seventh Conference on Neural Information Processing Systems.
2. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. [LoRA: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). In International Conference on Learning Representations.
3. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
4. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. [SWE-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations.
5. Haoyang Li, Xuejia Chen, Zhanchao XU, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, and Lei Chen. 2025. Exposing numeracy gaps: A benchmark to evaluate fundamental numerical abilities in large language models. Preprint, arXiv:2502.11075. [find](https://scholar.google.com/scholar?q=Haoyang+Li%2C+Xuejia+Chen%2C+Zhanchao+XU%2C+Darian+Li%2C+Nicole+Hu%2C+Fei+Teng%2C+Yiming+Li%2C+Luyu+Qiu%2C+Chen+Jason+Zhang%2C+Qing+Li%2C+and+Lei+Chen%2E+2025%2E+Exposing+numeracy+gaps%3A+A+benchmark+to+evaluate+fundamental+nu)
6. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others… [find](https://scholar.google.com/scholar?q=Jack+Lindsey%2C+Wes+Gurnee%2C+Emmanuel+Ameisen%2C+Brian+Chen%2C+Adam+Pearce%2C+Nicholas+L%2E+Turner%2C+Craig+Citro%2C+David+Abrahams%2C+Shan+Carter%2C+Basil+Hosmer%2C+Jonathan+Marcus%2C+Michael+Sklar%2C+Adly+Templeton%2C+Trenton)
7. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. [S2ORC: The semantic scholar open research corpus](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
8. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307–5315, Hong Kong, China. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+Preprint%2C+arXiv%3A1711%2E05101%2E+in+Natural+Language+Processing+and+the+9th+International+Joint+Conference+on+Natural+Language)
9. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. [BloombergGPT: A large language model for finance](https://md.research.elamur.ai/free/en/papers/wu-2023-bloomberggpt-large-language-model-finance/). Preprint, arXiv:2303.17564.
10. Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, and Muhan Zhang. 2025. Number cookbook: Number understanding of language models and how to improve it. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Haotong+Yang%2C+Yi+Hu%2C+Shijia+Kang%2C+Zhouchen+Lin%2C+and+Muhan+Zhang%2E+2025%2E+Number+cookbook%3A+Number+understanding+of+language+models+and+how+to+improve+it%2E+In+The+Thirteenth+International+Conference+on+Lea)
11. Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Mengyao Zhang, Jinlu Zhang, Jiyu Cui, Renjun Xu, Hongyang Chen, Xiaohui Fan, and 2 others. 2025. Scientific large language models: A survey o… [find](https://scholar.google.com/scholar?q=Qiang+Zhang%2C+Keyan+Ding%2C+Tianwen+Lv%2C+Xinda+Wang%2C+Qingyu+Yin%2C+Yiwen+Zhang%2C+Jing+Yu%2C+Yuhao+Wang%2C+Xiaotong+Li%2C+Zhuoyi+Xiang%2C+Xiang+Zhuang%2C+Zeyuan+Wang%2C+Ming+Qin%2C+Mengyao+Zhang%2C+Jinlu+Zhang%2C+Jiyu+Cui%2C+Ren)
12. Fangwei Zhu, Damai Dai, and Zhifang Sui. 2025. [Language models encode the value of numbers linearly](https://md.research.elamur.ai/free/en/papers/zhu-2025-language-models-encode-value-numbers/). In Proceedings of the 31st International Conference on Computational Linguistics, pages 693–709, Abu Dhabi, UAE.
13. Luca Soldaini and Kyle Lo. 2023. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI. ODC-By, [https://github.com/allenai/pes2o](https://github.com/allenai/pes2o). [↗](https://github.com/allenai/pes2o)
14. Xingyou Song and Dara Bahri. 2025. Decoding-based regression. Computing Research Repository, arXiv:2501.19383. — [arXiv:2501.19383](https://arxiv.org/abs/2501.19383) [find](https://scholar.google.com/scholar?q=Xingyou+Song+and+Dara+Bahri%2E+2025%2E+Decoding%2Dbased+regression%2E+Computing+Research+Repository%2C+arXiv%3A2501%2E19383%2E+%E2%80%94+%5BarXiv%3A2501%2E19383%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E19383%29)
15. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Alessandro+Stolfo%2C+Yonatan+Belinkov%2C+and+Mrinmaya+Sachan%2E+2023%2E+A+mechanistic+interpretation+of+arithmetic+reasoning+in+language+models+using+causal+mediation+analysis%2E+In+Proceedings+of+the+2023+Conf)
16. Eric Tang, Bangding Yang, and Xingyou Song. 2025. Understanding LLM embeddings for regression. Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Eric+Tang%2C+Bangding+Yang%2C+and+Xingyou+Song%2E+2025%2E+Understanding+LLM+embeddings+for+regression%2E+Transactions+on+Machine+Learning+Research%2E)
17. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). Preprint, arXiv:2211.09085.
18. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? Probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods

*Updated 2 November 2026*