---
title: "Graduating the Benchmark Scale: Lessons from Thermometry"
description: "The paper introduces no new data or models. 23 references. Among them: 0."
authors: ["Sean Trott", "Oisín Parkinson-Coombs"]
year: 2026
venue: "Proceedings of the Workshop on Evaluating Evaluations (EvalEval)"
source: "https://aclanthology.org/2026.evaleval-1.21/"
doi: "10.18653/v1/2026.evaleval-1.21"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/trott-2026-graduating-benchmark-scale-lessons-thermometry/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/trott-2026-graduating-benchmark-scale-lessons-thermometry/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/) / Graduating the Benchmark Scale: Lessons from Thermometry

# Graduating the Benchmark Scale: Lessons from Thermometry

**Authors:** [Sean Trott](https://md.research.elamur.ai/free/en/authors/sean-trott/) and Oisín Parkinson-Coombs · **Year:** 2026 · **Venue:** [Proceedings of the Workshop on Evaluating Evaluations (EvalEval)](https://md.research.elamur.ai/free/en/venues/evaleval/) ([EvalEval 2026](https://md.research.elamur.ai/free/en/venues/evaleval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evaleval-1.21/) · [PDF](https://aclanthology.org/2026.evaleval-1.21.pdf) · [DOI 10.18653/v1/2026.evaleval-1.21](https://doi.org/10.18653/v1/2026.evaleval-1.21) · [OpenAlex](https://openalex.org/W7166879501)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
This position paper from the EvalEval 2026 workshop looks at a neglected part of LLM benchmark validity. It asks whether a benchmark score can be treated as a graded measure of an underlying capability, and what justifies reading equal score gaps as equal differences in ability. The authors frame this with Hasok Chang's "problem of nomic measurement" from the history of thermometry. The problem is that the function linking an observable reading to a hidden quantity cannot be tested without already knowing that quantity.

The paper introduces no new data or models. Instead, it compares hypothetical score-to-capability mappings. It then shows how thermometry escaped this circularity through epistemic iteration: step-by-step, mutually correcting refinement of instruments, fixed points and theory. LLM capabilities are more abstract and lack stable anchors, so the analogy only partly holds.

The central claim is that the problem's severity depends on what a benchmark is for. Because the assumed mapping can change deployment and regulatory choices, researchers should state their goals and theoretical commitments explicitly.

## Key points
- Figure 1 sketches six possible links between measured and "true" capability: null, linear, categorical, logarithmic, sigmoidal and exponential.
- The authors illustrate the issue with an imagined 100-item Python benchmark. A jump from 30 to 50 and a jump from 50 to 70 mean the same thing only under a linear mapping. A logarithmic mapping makes the first gap larger, and an exponential one makes the second gap larger.
- Chang (2004) describes three stages in thermometry:
  - bodily sensation;
  - ordinal thermoscopes;
  - numerical thermometers built on fixed points and instrument comparison, with Henri Regnault's work on air thermometers as the example.
  
  Along the way, the boiling point of water was later replaced by steam temperature as a fixed point.
- Three benchmark goals are distinguished:
  - ranking models, which works under every mapping except null;
  - detecting a behavior, which needs a meaningful threshold;
  - measuring degrees of a capability, where the problem is hardest.
- Human baselines (METR, 2026) are suggested as possible anchors. However, they are judged less stable than physical fixed points and may not be comparable with LLM behavior.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Rutgers, The State University of New Jersey](https://md.research.elamur.ai/free/en/universities/rutgers-the-state-university-of-new-jersey/) — University, New Brunswick, United States · 40 works · 12,486 citations
- [University of Bergen](https://md.research.elamur.ai/free/en/universities/university-of-bergen/) — University, Bergen, Norway · 9 works · 583 citations

📚 Shelves (8): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 Safety Research](https://md.research.elamur.ai/free/en/shelves/social-sciences/safety-research/) · 626 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Management Science and Operations Research](https://md.research.elamur.ai/free/en/shelves/decision-sciences/management-science-and-operations-research/) · 134 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Sean Trott and Oisín Parkinson-Coombs. 2026. Graduating the Benchmark Scale: Lessons from Thermometry. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{trott2026graduating,
  title = {{Graduating the Benchmark Scale: Lessons from Thermometry}},
  author = {Sean Trott and Oisín Parkinson-Coombs},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.21},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.21}
}
```

- [← previous Gradient-Guided Multi-Judge Prompt Optimization](https://md.research.elamur.ai/free/en/papers/zhao-2026-gradient-guided-multi-judge-prompt/)
- [next → Grammar Search for Multi-Agent Systems](https://md.research.elamur.ai/free/en/papers/singh-2026-grammar-search-multi-agent-systems/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/35/)

## References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366. [find](https://scholar.google.com/scholar?q=Inioluwa+Deborah+Raji%2C+Emily+M+Bender%2C+Amandalynne+Paullada%2C+Emily+Denton%2C+and+Alex+Hanna%2E+2021%2E+Ai+and+the+everything+in+the+whole+wide+world+benchmark%2E+arXiv+preprint+arXiv%3A2111%2E15366%2E)
2. Anna Alexandrova and Daniel M. Haybron. 2016. Is Construct Validation Valid? Philosophy of Science, 83(5):1098–1109. [find](https://scholar.google.com/scholar?q=Anna+Alexandrova+and+Daniel+M%2E+Haybron%2E+2016%2E+Is+Construct+Validation+Valid%3F+Philosophy+of+Science%2C+83%285%29%3A1098%E2%80%931109%2E)
3. Andrew M Bean, Ryan Othniel Kearns, Angelika Romanou, Franziska Sofia Hafner, Harry Mayne, Jan Batzner, Negar Foroutan, Chris Schmitz, Karolina Korgul, Hunar Batra, and 1 others. 2025. Measuring what matters: Construct validity in large language model benchmarks. arXiv preprint arXiv:2511.04703. — [arXiv:2511.04703](https://arxiv.org/abs/2511.04703) [find](https://scholar.google.com/scholar?q=Andrew+M+Bean%2C+Ryan+Othniel+Kearns%2C+Angelika+Romanou%2C+Franziska+Sofia+Hafner%2C+Harry+Mayne%2C+Jan+Batzner%2C+Negar+Foroutan%2C+Chris+Schmitz%2C+Karolina+Korgul%2C+Hunar+Batra%2C+and+1+others%2E+2025%2E+Measuring+what+)
4. Denny Borsboom, Gideon J. Mellenbergh, and Jaap Van Heerden. 2004. The Concept of Validity. Psychological Review, 111(4):1061–1071. [find](https://scholar.google.com/scholar?q=Denny+Borsboom%2C+Gideon+J%2E+Mellenbergh%2C+and+Jaap+Van+Heerden%2E+2004%2E+The+Concept+of+Validity%2E+Psychological+Review%2C+111%284%29%3A1061%E2%80%931071%2E)
5. Hasok Chang. 2004. Inventing Temperature: Measurement and Scientific Progress. Oxford University Press. Google-Books-ID: yVOuV8qJkxMC. [find](https://scholar.google.com/scholar?q=Hasok+Chang%2E+2004%2E+Inventing+Temperature%3A+Measurement+and+Scientific+Progress%2E+Oxford+University+Press%2E+Google%2DBooks%2DID%3A+yVOuV8qJkxMC%2E)
6. Hasok Chang. 2015. The rising of chemical natural kinds through epistemic iteration. In Natural kinds and classification in scientific practice, pages 33–46. Routledge. [find](https://scholar.google.com/scholar?q=Hasok+Chang%2E+2015%2E+The+rising+of+chemical+natural+kinds+through+epistemic+iteration%2E+In+Natural+kinds+and+classification+in+scientific+practice%2C+pages+33%E2%80%9346%2E+Routledge%2E)
7. Hasok Chang. 2017. Epistemic iteration and natural kinds: Realism and pluralism in taxonomy. Philosophical issues in psychiatry IV: Psychiatric nosology, pages 229–245. [find](https://scholar.google.com/scholar?q=Hasok+Chang%2E+2017%2E+Epistemic+iteration+and+natural+kinds%3A+Realism+and+pluralism+in+taxonomy%2E+Philosophical+issues+in+psychiatry+IV%3A+Psychiatric+nosology%2C+pages+229%E2%80%93245%2E)
8. Jennifer Hu, Felix Sosa, and Tomer Ullman. 2025. Re-evaluating Theory of Mind evaluation in large language models. Philosophical Transactions of the Royal Society B: Biological Sciences, 380(1932):20230499. [find](https://scholar.google.com/scholar?q=Jennifer+Hu%2C+Felix+Sosa%2C+and+Tomer+Ullman%2E+2025%2E+Re%2Devaluating+Theory+of+Mind+evaluation+in+large+language+models%2E+Philosophical+Transactions+of+the+Royal+Society+B%3A+Biological+Sciences%2C+380%281932%29%3A202)
9. Anna A. Ivanova. 2025. How to evaluate the cognitive abilities of LLMs. Nature Human Behaviour, 9(2):230–233. [find](https://scholar.google.com/scholar?q=Anna+A%2E+Ivanova%2E+2025%2E+How+to+evaluate+the+cognitive+abilities+of+LLMs%2E+Nature+Human+Behaviour%2C+9%282%29%3A230%E2%80%93233%2E)
10. Cameron Jones and Benjamin Bergen. 2026. Lies, damned lies, and language statistics: a comprehensive review of risks from manipulation, persuasion, and deception with large language models. Artificial Intelligence Review. [find](https://scholar.google.com/scholar?q=Cameron+Jones+and+Benjamin+Bergen%2E+2026%2E+Lies%2C+damned+lies%2C+and+language+statistics%3A+a+comprehensive+review+of+risks+from+manipulation%2C+persuasion%2C+and+deception+with+large+language+models%2E+Artificial)
11. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
12. Tom Kouwenhoven, Michiel van der Meer, and Max van Duijn. 2026. Traces of social competence in large language models. arXiv preprint arXiv:2603.04161. [find](https://scholar.google.com/scholar?q=Tom+Kouwenhoven%2C+Michiel+van+der+Meer%2C+and+Max+van+Duijn%2E+2026%2E+Traces+of+social+competence+in+large+language+models%2E+arXiv+preprint+arXiv%3A2603%2E04161%2E)
13. Cristian Larroulet Philippi. 2021. Valid for What? On the Very Idea of Unconditional Validity. Philosophy of the Social Sciences, 51(2):151–175. Publisher: SAGE Publications Inc. [find](https://scholar.google.com/scholar?q=Cristian+Larroulet+Philippi%2E+2021%2E+Valid+for+What%3F+On+the+Very+Idea+of+Unconditional+Validity%2E+Philosophy+of+the+Social+Sciences%2C+51%282%29%3A151%E2%80%93175%2E+Publisher%3A+SAGE+Publications+Inc%2E)
14. METR. 2026. Task-completion time horizons of frontier ai models. [https://metr.org/](https://metr.org/) time-horizons/. [↗](https://metr.org/)
15. Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. 2025. Measurement to meaning: A validity-centered framework for ai evaluation. arXiv preprint arXiv:2505.10573. — [arXiv:2505.10573](https://arxiv.org/abs/2505.10573) [find](https://scholar.google.com/scholar?q=Olawale+Salaudeen%2C+Anka+Reuel%2C+Ahmed+Ahmed%2C+Suhana+Bedi%2C+Zachary+Robertson%2C+Sudharsan+Sundar%2C+Ben+Domingue%2C+Angelina+Wang%2C+and+Sanmi+Koyejo%2E+2025%2E+Measurement+to+meaning%3A+A+validity%2Dcentered+framework)
16. Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. 2024. Benchmarks as Microscopes: A Call for Model Metrology. arXiv preprint. ArXiv:2407.16711 [cs]. — [arXiv:2407.16711](https://arxiv.org/abs/2407.16711) [find](https://scholar.google.com/scholar?q=Michael+Saxon%2C+Ari+Holtzman%2C+Peter+West%2C+William+Yang+Wang%2C+and+Naomi+Saphra%2E+2024%2E+Benchmarks+as+Microscopes%3A+A+Call+for+Model+Metrology%2E+arXiv+preprint%2E+ArXiv%3A2407%2E16711+%5Bcs%5D%2E+%E2%80%94+%5BarXiv%3A2407%2E16711%5D%28h)
17. Rylan Schaeffer, Punit Singh Koura, Binh Tang, Ranjan Subramanian, Aaditya K Singh, Todor Mihaylov, Prajjwal Bhargava, Lovish Madaan, Niladri S Chatterji, Vedanuj Goswami, and 1 others. 2025. Correlating and predicting human evaluations of language models from natural language processing benchmarks.… — [arXiv:2502.18339](https://arxiv.org/abs/2502.18339) [find](https://scholar.google.com/scholar?q=Rylan+Schaeffer%2C+Punit+Singh+Koura%2C+Binh+Tang%2C+Ranjan+Subramanian%2C+Aaditya+K+Singh%2C+Todor+Mihaylov%2C+Prajjwal+Bhargava%2C+Lovish+Madaan%2C+Niladri+S+Chatterji%2C+Vedanuj+Goswami%2C+and+1+others%2E+2025%2E+Correlat)
18. Sean Trott, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. 2023. Do Large Language Models Know What Humans Know? Cognitive Science, 47(7):e13309. [find](https://scholar.google.com/scholar?q=Sean+Trott%2C+Cameron+Jones%2C+Tyler+Chang%2C+James+Michaelov%2C+and+Benjamin+Bergen%2E+2023%2E+Do+Large+Language+Models+Know+What+Humans+Know%3F+Cognitive+Science%2C+47%287%29%3Ae13309%2E)
19. Sean Trott, Samuel Taylor, Cameron Jones, James A. Michaelov, and Pamela D. Rivière. 2026. [Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs](https://md.research.elamur.ai/free/en/papers/trott-2026-language-statistics-false-belief-reasoning/). arXiv preprint. ArXiv:2602.16085 [cs].
20. Tomer Ullman. 2023. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks. arXiv preprint. ArXiv:2302.08399 [cs]. [find](https://scholar.google.com/scholar?q=Tomer+Ullman%2E+2023%2E+Large+Language+Models+Fail+on+Trivial+Alterations+to+Theory%2Dof%2DMind+Tasks%2E+arXiv+preprint%2E+ArXiv%3A2302%2E08399+%5Bcs%5D%2E)
21. Hanna Wallach, Meera Desai, A. Feder Cooper, Angelina Wang, Chad Atalla, Solon Barocas, Su Lin Blodgett, Alexandra Chouldechova, Emily Corvi, P. Alex Dow, Jean Garcia-Gathright, Alexandra Olteanu, Nicholas J Pangakis, Stefanie Reed, Emily Sheng, Dan Vann, Jennifer Wortman Vaughan, Matthew Vogel, Han… [find](https://scholar.google.com/scholar?q=Hanna+Wallach%2C+Meera+Desai%2C+A%2E+Feder+Cooper%2C+Angelina+Wang%2C+Chad+Atalla%2C+Solon+Barocas%2C+Su+Lin+Blodgett%2C+Alexandra+Chouldechova%2C+Emily+Corvi%2C+P%2E+Alex+Dow%2C+Jean+Garcia%2DGathright%2C+Alexandra+Olteanu%2C+Nic)
22. Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Deep Ganguli, Sanmi Koyejo, and William Isaac. 2025. Toward an evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336. — [arXiv:2503.05336](https://arxiv.org/abs/2503.05336) [find](https://scholar.google.com/scholar?q=Laura+Weidinger%2C+Inioluwa+Deborah+Raji%2C+Hanna+Wallach%2C+Margaret+Mitchell%2C+Angelina+Wang%2C+Olawale+Salaudeen%2C+Rishi+Bommasani%2C+Deep+Ganguli%2C+Sanmi+Koyejo%2C+and+William+Isaac%2E+2025%2E+Toward+an+evaluation+s)
23. Tal Yarkoni. 2022. The generalizability crisis. Behavioral and Brain Sciences, 45:e1. [find](https://scholar.google.com/scholar?q=Tal+Yarkoni%2E+2022%2E+The+generalizability+crisis%2E+Behavioral+and+Brain+Sciences%2C+45%3Ae1%2E)

*Updated 1 November 2026*