---
title: "Do Language Embeddings capture Scales? — Zhang 2020"
description: "The paper investigates whether pretrained text representations encode the typical… 0.18 accuracy. 20 references, cited by 40 papers here."
authors: ["Xikun Zhang", "Deepak Ramachandran", "Ian Tenney", "Yanai Elazar", "Dan Roth"]
year: 2020
venue: "Findings of the Association for Computational Linguistics: EMNLP 2020"
source: "https://aclanthology.org/2020.findings-emnlp.439/"
doi: "10.18653/v1/2020.findings-emnlp.439"
arxiv: "2010.05345"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhang-2020-do-language-embeddings-capture-scales/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhang-2020-do-language-embeddings-capture-scales/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/) / Do Language Embeddings capture Scales?

# Do Language Embeddings capture Scales?

**Authors:** [Xikun Zhang](https://md.research.elamur.ai/free/en/authors/xikun-zhang/), [Deepak Ramachandran](https://md.research.elamur.ai/free/en/authors/deepak-ramachandran/), [Ian Tenney](https://md.research.elamur.ai/free/en/authors/ian-tenney/), [Yanai Elazar](https://md.research.elamur.ai/free/en/authors/yanai-elazar/) and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2020 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2020](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2020](https://md.research.elamur.ai/free/en/venues/findings/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.findings-emnlp.439/) · [PDF](https://aclanthology.org/2020.findings-emnlp.439.pdf) · [DOI 10.18653/v1/2020.findings-emnlp.439](https://doi.org/10.18653/v1/2020.findings-emnlp.439) · [arXiv 2010.05345](https://arxiv.org/abs/2010.05345) · [OpenAlex](https://openalex.org/W3100879603)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/zhang-2020-do-language-embeddings-capture-scales/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper investigates whether pretrained text representations encode the typical numeric magnitude of everyday objects, such as how heavy, long or expensive something is. The authors call this task scalar probing. They use DoQ (Distributions over Quantities), a web-derived resource of value counts, and focus on nouns for MASS, LENGTH and PRICE. Values are grouped into 12 logarithmic buckets, so the target is an order-of-magnitude distribution. They freeze Word2vec, ELMo and BERT and fit linear probes on top. One probe is a regression on the median value. The other is a multi-class classifier trained against the full distribution. They also pretrain NumBERT, a BERT variant that sees all numbers in scientific notation.

Embeddings do contain scale knowledge, but the best probes close only part of the gap between a simple baseline and an approximate upper bound. Contextual encoders beat Word2vec even though the task has no context. Number formatting also changes the results. This suggests that numeracy and context are concrete ways to improve commonsense reasoning about quantities.

## Key points
- **Probe design:** The distribution-based classifier (mcc) consistently beats the median regression (rgr). The regression often fails to outperform an aggregate baseline that predicts the global bucket distribution.
- **Accuracy by encoder:**
  - ELMo reaches the highest probe accuracy, up to about 0.43, likely helped by its character-level tokenization.
  - The approximate upper bounds on accuracy come to 0.570 (lengths), 0.537 (masses) and 0.476 (prices).
- **NumBERT:** It replaces numbers with a mantissa plus an exponent token, so 314.1 becomes 3141[EXP]2. It gets the best Earth Mover's Distance, but slightly worse MSE and accuracy, so the effect depends on the metric.
- **Narrow domains:** Probes trained only on animal masses perform better, because ambiguity like "crane" (bird or machine) disappears.
- **Zero-shot transfer:**
  - On VerbPhysics (Forbes and Choi), rgr with NumBERT scores 0.76 on test, close to the DoQ oracle (0.77) but below a specialized model (0.87).
  - On Amazon product prices, mcc with NumBERT is best on all three metrics, with 0.18 accuracy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations
- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations
- [University of Pennsylvania](https://md.research.elamur.ai/free/en/universities/university-of-pennsylvania/) — University, Philadelphia, United States · 233 works · 65,583 citations
- [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) — University, Ramat Gan, Israel · 146 works · 24,845 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar and Dan Roth. 2020. Do Language Embeddings capture Scales?. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zhang2020do,
  title = {{Do Language Embeddings capture Scales?}},
  author = {Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar and Dan Roth},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.439},
  eprint = {2010.05345},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.439}
}
```

- [← previous Bridging Anaphora Resolution as Question Answering](https://md.research.elamur.ai/free/en/papers/hou-2020-bridging-anaphora-resolution-as-question/)
- [next → The Tiberian Pronunciation Tradition of Biblical Hebrew](https://md.research.elamur.ai/free/en/papers/khan-2020-tiberian-pronunciation-tradition-biblical/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/6/)

## References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. L Baringhaus and N Henze. 2017. Cramér–von mises distance: probabilistic interpretation, confidence intervals, and neighbourhood-of-model validation. Journal of Nonparametric Statistics, 29(2):167–188. [find](https://scholar.google.com/scholar?q=L+Baringhaus+and+N+Henze%2E+2017%2E+Crame%CC%81r%E2%80%93von+mises+distance%3A+probabilistic+interpretation%2C+confidence+intervals%2C+and+neighbourhood%2Dof%2Dmodel+validation%2E+Journal+of+Nonparametric+Statistics%2C+29%282%29%3A167%E2%80%9318)
2. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. [Piqa: Reasoning about physical commonsense in natural language](https://md.research.elamur.ai/free/en/papers/bisk-2019-piqa-reasoning-about-physical-commonsense/). In Thirty-Fourth AAAI Conference on Artificial Intelligence.
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
4. Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Abhijit+Mahabal%2C+Deepak+Ramachandran%2C+Tania+Bedrax%2DWeiss%2C+and+Dan+Roth%2E+2019%2E+How+large+are+lions%3F+inducing+distributions+over+quantitative+attributes%2E+In+Association+for+Computational+L)
5. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. 2016. [Probing for semantic evidence of composition by means of simple classification tasks](https://md.research.elamur.ai/free/en/papers/ettinger-2016-probing-semantic-evidence-composition-means/). In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP, pages 134–139.
6. Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 266–276, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Maxwell+Forbes+and+Yejin+Choi%2E+2017%2E+Verb+physics%3A+Relative+physical+knowledge+of+actions+and+objects%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
7. Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. [Injecting numerical reasoning skills into language models](https://md.research.elamur.ai/free/en/papers/geva-2020-injecting-numerical-reasoning-skills-into/). arXiv preprint arXiv:2004.04487.
8. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. arXiv preprint arXiv:1909.03368. [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+arXiv+preprint+arXiv%3A1909%2E03368%2E)
9. Julian McAuley Jianmo Ni, Jiacheng Li. 2019. Justifying recommendations using distantly-labeled reviews and fined-grained aspects. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=Julian+McAuley+Jianmo+Ni%2C+Jiacheng+Li%2E+2019%2E+Justifying+recommendations+using+distantly%2Dlabeled+reviews+and+fined%2Dgrained+aspects%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
10. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. [Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models](https://md.research.elamur.ai/free/en/papers/lin-2020-birds-have-four-legs-numersense/). arXiv preprint arXiv:2005.00683.
11. Hong Liu, Mingsheng Long, Jianmin Wang, and Michael I. Jordan. 2019. Transferable adversarial training: A general approach to adapting deep classifiers. In Proceedings of the 36th International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Hong+Liu%2C+Mingsheng+Long%2C+Jianmin+Wang%2C+and+Michael+I%2E+Jordan%2E+2019%2E+Transferable+adversarial+training%3A+A+general+approach+to+adapting+deep+classifiers%2E+In+Proceedings+of+the+36th+International+Confer)
12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in neural information processing systems, pages 3111–3119.
13. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arXiv preprint arXiv:1802.05365.
14. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. [Information-theoretic probing for linguistic structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/). arXiv preprint arXiv:2004.03061.
15. Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 1998. A metric for distributions with applications to image databases. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), pages 59–66. IEEE. [find](https://scholar.google.com/scholar?q=Yossi+Rubner%2C+Carlo+Tomasi%2C+and+Leonidas+J+Guibas%2E+1998%2E+A+metric+for+distributions+with+applications+to+image+databases%2E+In+Sixth+International+Conference+on+Computer+Vision+%28IEEE+Cat%2E+No%2E+98CH36271%29)
16. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. [Does string-based neural mt learn source syntax?](https://md.research.elamur.ai/free/en/papers/shi-2016-does-string-based-neural-mt/) In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534.
17. Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. 2016. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747–1756. [find](https://scholar.google.com/scholar?q=Aaron+Van+Oord%2C+Nal+Kalchbrenner%2C+and+Koray+Kavukcuoglu%2E+2016%2E+Pixel+recurrent+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+pages+1747%E2%80%931756%2E)
18. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. [Do NLP models know numbers? probing numeracy in embeddings](https://md.research.elamur.ai/free/en/papers/wallace-2019-do-nlp-models-know-numbers/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
19. Yiben Yang, Larry Birnbaum, Ji-Ping Wang, and Doug Downey. 2018. Extracting commonsense properties from embeddings with limited human guidance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 644–649, Melbourne, Australia. As… [find](https://scholar.google.com/scholar?q=Yiben+Yang%2C+Larry+Birnbaum%2C+Ji%2DPing+Wang%2C+and+Doug+Downey%2E+2018%2E+Extracting+commonsense+properties+from+embeddings+with+limited+human+guidance%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Associa)
20. Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ben+Zhou%2C+Qiang+Ning%2C+Daniel+Khashabi%2C+and+Dan+Roth%2E+2020%2E+Temporal+common+sense+acquisition+with+minimal+supervision%2E+In+Association+for+Computational+Linguistics%2E)

*Updated 2 November 2026*