---
title: "Mapping global dynamics of benchmark creation and saturation"
description: "The authors proposed a way to concisely map the emergence, use, and saturation of benchmarks… 12 references, cited by 21 papers here."
authors: ["Simon Ott", "Adriano Barbosa-Silva", "Kathrin Blagec", "Jan Brauner", "Matthias Samwald"]
year: 2022
venue: "arXiv:2203.04592 · Nature Communications volume 13, Article number: 6793 (2022)"
source: "https://arxiv.org/abs/2203.04592"
doi: "10.1038/s41467-022-34591-0"
arxiv: "2203.04592"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ott-2022-mapping-global-dynamics-benchmark-creation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ott-2022-mapping-global-dynamics-benchmark-creation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Mapping global dynamics of benchmark creation and saturation in artificial intelligence

# Mapping global dynamics of benchmark creation and saturation in artificial intelligence

**Authors:** [Simon Ott](https://md.research.elamur.ai/free/en/authors/simon-ott/), Adriano Barbosa-Silva, Kathrin Blagec, [Jan Brauner](https://md.research.elamur.ai/free/en/authors/jan-brauner/) and [Matthias Samwald](https://md.research.elamur.ai/free/en/authors/matthias-samwald/) · **Year:** 2022 · **Venue:** [arXiv:2203.04592 · Nature Communications volume 13, Article number: 6793 (2022)](https://md.research.elamur.ai/free/en/venues/nature-communications/) ([Nature Communications 2022](https://md.research.elamur.ai/free/en/venues/nature-communications/2022/)) · **Publisher:** [Nature Portfolio](https://md.research.elamur.ai/free/en/publishers/nature-portfolio/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1038/s41467-022-34591-0) · [PDF](https://arxiv.org/pdf/2203.04592) · [DOI 10.1038/s41467-022-34591-0](https://doi.org/10.1038/s41467-022-34591-0) · [arXiv 2203.04592](https://arxiv.org/abs/2203.04592) · [OpenAlex](https://openalex.org/W4308885736)

**Citations:** 72 citations (OpenAlex, 2026-10-14) · [21](https://md.research.elamur.ai/free/en/papers/ott-2022-mapping-global-dynamics-benchmark-creation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Benchmarks not only measure progress in AI but also set its direction. At the same time, many of them quickly reach saturation or are barely used. The authors proposed a way to concisely map the emergence, use, and saturation of benchmarks across entire fields. To do this, they took a Papers With Code dump and built from it the ontology and knowledge graph ITO (Intelligence Task Ontology) in OWL format. They manually curated the hierarchy of tasks and metrics. The analysis covered 3765 benchmarks for 947 computer vision and NLP tasks. To make different metrics comparable, the authors normalized SOTA trajectories via relative improvement, calculated from the first result to the best one. The shapes of the curves were clustered using self-organizing maps (SOM). It turned out that a substantial share of benchmarks approach saturation very early. Many of them are hardly used by anyone, and sharp jumps in performance on individual tasks occur unpredictably. The practical conclusion: a new benchmark, including one for the Ukrainian language, has a better chance of being adopted if it is versatile, broad in coverage, and useful for real-world applications.

## Key points
- **Data volume.** 3765 benchmarks and 947 tasks were analyzed. Of the 1318 NLP benchmarks that have at least one result, only 661 (50%) have results at three or more distinct points in time.
- **Three curve types.** SOM grouped 1079 metric trajectories from 654 benchmarks into three clusters: steady growth (460), early saturation or stagnation (378), and stagnation followed by a late breakthrough (241).
- **Dynamics over time.** The number of benchmarks began to grow in 2013. It increased fastest in 2017–2018, after which growth slowed, and in 2020 it stagnated, most noticeably in NLP. The number of active benchmarks peaked in 2020: 432 in NLP versus 1100 in computer vision.
- **Popularity.** The most popular and least used datasets were compared using one-sided t-tests. According to the authors, popularity is associated with versatility, breadth of coverage, and practical usefulness.
- **Open resources.** The ITO knowledge graph is published on Zenodo (doi:10.5281/zenodo.7097305). The code and the interactive "Global Map of AI Benchmark Dynamics" are available in the GitHub repository OpenBioLink/ITO.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations
- [Medical University of Vienna](https://md.research.elamur.ai/free/en/universities/medical-university-of-vienna/) — University, Vienna, Austria · 3 works · 161 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 61 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

💡 Concepts (1): [💡 Artificial intelligence](https://md.research.elamur.ai/free/en/concepts/artificial-intelligence/) · 49 works

## Cite

Cite as: Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner and Matthias Samwald. 2022. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. arXiv:2203.04592 · Nature Communications volume 13, Article number: 6793 (2022).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{ott2022mapping,
  title = {{Mapping global dynamics of benchmark creation and saturation in artificial intelligence}},
  author = {Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner and Matthias Samwald},
  year = {2022},
  doi = {10.1038/s41467-022-34591-0},
  eprint = {2203.04592},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41467-022-34591-0}
}
```

- [← previous SciFact-Open: Towards open-domain scientific claim verification](https://md.research.elamur.ai/free/en/papers/wadden-2022-scifact-open-open-domain-scientific/)
- [next → Tailor: Generating and Perturbing Text with Semantic Controls](https://md.research.elamur.ai/free/en/papers/ross-2022-tailor-generating-perturbing-text-semantic/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Deng, J. et al. ImageNet: A large-scale hierarchical image database. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009). doi:10.1109/CVPR.2009.5206848. [find](https://scholar.google.com/scholar?q=Deng%2C+J%2E+et+al%2E+ImageNet%3A+A+large%2Dscale+hierarchical+image+database%2E+in+2009+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+248%E2%80%93255+%28IEEE%2C+2009%29%2E+doi%3A10%2E1109%2FCVPR%2E2009%2E5206848%2E)
2. Soomro, K., Zamir, A. R. & Shah, M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXiv (2012). [find](https://scholar.google.com/scholar?q=Soomro%2C+K%2E%2C+Zamir%2C+A%2E+R%2E+%26+Shah%2C+M%2E+UCF101%3A+A+Dataset+of+101+Human+Actions+Classes+From+Videos+in+The+Wild%2E+arXiv+%282012%29%2E)
3. Borgwardt, K. M. et al. Protein function prediction via graph kernels. Bioinformatics 21 Suppl 1, i47-56 (2005). [find](https://scholar.google.com/scholar?q=Borgwardt%2C+K%2E+M%2E+et+al%2E+Protein+function+prediction+via+graph+kernels%2E+Bioinformatics+21+Suppl+1%2C+i47%2D56+%282005%29%2E)
4. Kiela, D. et al. Dynabench: rethinking benchmarking in NLP. in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 4110–4124 (Association for Computational Linguistics, 2021). doi:10.18653/v1/2021.naacl-main.3… [find](https://scholar.google.com/scholar?q=Kiela%2C+D%2E+et+al%2E+Dynabench%3A+rethinking+benchmarking+in+NLP%2E+in+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Technol)
5. Bowman, S. R. & Dahl, G. What will it take to fix benchmarking in natural language understanding? in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 4843–4855 (Association for Computational Linguistics, 20… [find](https://scholar.google.com/scholar?q=Bowman%2C+S%2E+R%2E+%26+Dahl%2C+G%2E+What+will+it+take+to+fix+benchmarking+in+natural+language+understanding%3F+in+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation)
6. Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020). [find](https://scholar.google.com/scholar?q=Geirhos%2C+R%2E+et+al%2E+Shortcut+learning+in+deep+neural+networks%2E+Nat%2E+Mach%2E+Intell%2E+2%2C+665%E2%80%93673+%282020%29%2E)
7. Krizhevsky, A. Learning multiple layers of features from tiny images. (2009). [find](https://scholar.google.com/scholar?q=Krizhevsky%2C+A%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+%282009%29%2E)
8. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D. & Parikh, D. [Making the V in VQA matter: elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/). in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 6325–6334 (IEEE, 2017). doi:10.1109/CVPR.2017.670.
9. Zhang, D. et al. The AI Index 2021 Annual Report. Preprint at arXiv: https://arxiv.org/abs/2103.06312 (2021). — [arXiv:2103.06312](https://arxiv.org/abs/2103.06312) [↗](https://arxiv.org/abs/2103.06312)
10. Martínez-Plumed, F., Barredo, P., hÉigeartaigh, S. Ó. & Hernández-Orallo, J. Research community dynamics behind popular AI benchmarks. Nat. Mach. Intell. (2021) doi:10.1038/s42256-021-00339-6. — [doi:10.1038/s42256-021-00339-6](https://doi.org/10.1038/s42256-021-00339-6) [find](https://scholar.google.com/scholar?q=Mart%C3%ADnez%2DPlumed%2C+F%2E%2C+Barredo%2C+P%2E%2C+h%C3%89igeartaigh%2C+S%2E+%C3%93%2E+%26+Hern%C3%A1ndez%2DOrallo%2C+J%2E+Research+community+dynamics+behind+popular+AI+benchmarks%2E+Nat%2E+Mach%2E+Intell%2E+%282021%29+doi%3A10%2E1038%2Fs42256%2D021%2D00339%2D6%2E+%E2%80%94+%5Bdoi%3A)
11. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). arXiv:1606.05250 [cs] (2016).
12. Koch, B., Denton, E., Hanna, A. & Foster, J. G. Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research. (2021). \#\# 13. Dehghani, M. et al. The Benchmark Lottery. arXiv (2021). \#\# 14. Nie, Y. et al. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). in Proc…

*Updated 31 October 2026*