---
title: "BenchNavigator — paper by Sokol et al. (2026)"
description: "The authors combine a practitioner survey, which builds on ten earlier interviews, with an analysis of metadata from several… 20 references."
authors: ["Anna Sokol", "Inge Vejsbjerg", "Elizabeth M. Daly", "David Piorkowski", "Michael Hind", "Nuno Moniz", "Nitesh V. Chawla"]
year: 2026
venue: "Proceedings of the Workshop on Evaluating Evaluations (EvalEval)"
source: "https://aclanthology.org/2026.evaleval-1.29/"
doi: "10.18653/v1/2026.evaleval-1.29"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sokol-2026-benchnavigator-discovery-interface-comparing-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sokol-2026-benchnavigator-discovery-interface-comparing-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/) / BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks

# BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks

**Authors:** Anna Sokol, [Inge Vejsbjerg](https://md.research.elamur.ai/free/en/authors/inge-vejsbjerg/), [Elizabeth M. Daly](https://md.research.elamur.ai/free/en/authors/elizabeth-m-daly/), [David Piorkowski](https://md.research.elamur.ai/free/en/authors/david-piorkowski/), Michael Hind, [Nuno Moniz](https://md.research.elamur.ai/free/en/authors/nuno-moniz/) and [Nitesh V. Chawla](https://md.research.elamur.ai/free/en/authors/nitesh-v-chawla/) · **Year:** 2026 · **Venue:** [Proceedings of the Workshop on Evaluating Evaluations (EvalEval)](https://md.research.elamur.ai/free/en/venues/evaleval/) ([EvalEval 2026](https://md.research.elamur.ai/free/en/venues/evaleval/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.evaleval-1.29/) · [PDF](https://aclanthology.org/2026.evaleval-1.29.pdf) · [DOI 10.18653/v1/2026.evaleval-1.29](https://doi.org/10.18653/v1/2026.evaleval-1.29) · [OpenAlex](https://openalex.org/W7166839270)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The number of LLM benchmarks keeps growing, and information about each one is spread across papers, repositories and dataset cards that use different fields, terms and levels of detail. This makes it hard to pick a benchmark that fits a real use case. Earlier efforts such as BenchmarkCards, BenchHub and BetterBench document or grade individual benchmarks, but none gives a consistent way to compare candidates while searching.

The authors combine a practitioner survey, which builds on ten earlier interviews, with an analysis of metadata from several sources to find which attributes matter for discovery. They then build BenchNavigator, a prototype that:
- maps existing metadata onto a shared schema with controlled vocabularies,
- reconciles conflicting or missing values,
- records the provenance of each value,
- presents everything through one filterable interface.

The prototype shows that comparable presentation is achievable without asking benchmark creators to adopt new reporting standards. The authors frame the system as discovery infrastructure, not a quality score. They treat gaps in reporting, such as on contamination and validity, as an explicit finding.

## Key points
- The paper poses three research questions: current selection workflows and pain points, the attributes practitioners prioritize, and whether those attributes can be unified across heterogeneous sources in one interface.
- The method has three layers: schema alignment with controlled vocabularies, normalization of conflicting or absent fields, and a uniform presentation layer that keeps provenance.
- Benchmark growth is illustrated using ArXiv, GitHub and Hugging Face. The authors cite Liang et al.'s study of over 32,000 model cards, which found that voluntary documentation is often incomplete.
- BenchNavigator does not rank a single "best" benchmark. It shows metadata and filters so users can choose based on their own context. Survey respondents reported little trust in leaderboards and vendor-reported scores.
- Planned extensions include adding EvalCards and similar formats, and showing adoption trends such as paper mentions over time and saturation.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [IBM](https://md.research.elamur.ai/free/en/companies/ibm/) — Company, Armonk, United States · 154 works · 72,606 citations
- [University of Notre Dame](https://md.research.elamur.ai/free/en/universities/university-of-notre-dame/) — University, Notre Dame, United States · 59 works · 37,127 citations
- [IBM Research - Ireland](https://md.research.elamur.ai/free/en/labs/ibm-research-ireland/) — Research institute, Dublin, Ireland · 11 works · 904 citations
- [IBM (Ireland)](https://md.research.elamur.ai/free/en/companies/ibm-ireland/) — Company, Dublin, Ireland · 6 works · 274 citations

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 52 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works

## Cite

Cite as: Anna Sokol, Inge Vejsbjerg, Elizabeth M. Daly, David Piorkowski, Michael Hind, Nuno Moniz and Nitesh V. Chawla. 2026. BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sokol2026benchnavigator,
  title = {{BenchNavigator: A Discovery Interface for Comparing LLM Benchmarks}},
  author = {Anna Sokol, Inge Vejsbjerg, Elizabeth M. Daly, David Piorkowski, Michael Hind, Nuno Moniz and Nitesh V. Chawla},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.29},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.29}
}
```

- [← previous Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces](https://md.research.elamur.ai/free/en/papers/shi-2026-benchmarking-web-agent-safety-under/)
- [next → Bengali-English and Hindi-English Code Mixed Speech Data with Disfluencies](https://md.research.elamur.ai/free/en/papers/mitra-2026-bengali-english-hindi-english-code/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/24/)

## References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. NIST AI. 2023. Artificial intelligence risk management framework (ai rmf 1.0). URL: [https://nvlpubs.nist](https://nvlpubs.nist). gov/nistpubs/ai/nist. ai, pages 100–1. [↗](https://nvlpubs.nist)
2. Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, and 1 others. 2024. [When benchmarks are targets: Revealing the sensitivity of large language model leaderboards](https://md.research.elamur.ai/free/en/papers/alzahrani-2024-benchmarks-targets-revealing-sensitivity-large/). In Proceedings of…
3. Frank Bagehorn, Kristina Brimijoin, Elizabeth M Daly, Jessica He, Michael Hind, Luis Garces-Erice, Christopher Giblin, Ioana Giurgiu, Jacquelyn Martino, Rahul Nair, and 1 others. 2025. Ai risk atlas: Taxonomy and tooling for navigating ai risks and resources. arXiv preprint arXiv:2503.05780. — [arXiv:2503.05780](https://arxiv.org/abs/2503.05780) [find](https://scholar.google.com/scholar?q=Frank+Bagehorn%2C+Kristina+Brimijoin%2C+Elizabeth+M+Daly%2C+Jessica+He%2C+Michael+Hind%2C+Luis+Garces%2DErice%2C+Christopher+Giblin%2C+Ioana+Giurgiu%2C+Jacquelyn+Martino%2C+Rahul+Nair%2C+and+1+others%2E+2025%2E+Ai+risk+atlas%3A+)
4. Vanessa Bracamonte, Sebastian Pape, Sascha Löbner, and Frederic Tronnier. 2023. Effectiveness and information quality perception of an ai model card: a study among non-experts. In 2023 20th Annual International Conference on Privacy, Security and Trust (PST), pages 1–7. IEEE. [find](https://scholar.google.com/scholar?q=Vanessa+Bracamonte%2C+Sebastian+Pape%2C+Sascha+L%C3%B6bner%2C+and+Frederic+Tronnier%2E+2023%2E+Effectiveness+and+information+quality+perception+of+an+ai+model+card%3A+a+study+among+non%2Dexperts%2E+In+2023+20th+Annual+Int)
5. Ryan Burnell, Wout Schellaert, John Burden, Tomer D. Ullman, Fernando Martinez-Plumed, Joshua B. Tenenbaum, Danaja Rutar, Lucy G. Cheke, Jascha Sohl-Dickstein, Melanie Mitchell, Douwe Kiela, Murray Shanahan, Ellen M. Voorhees, Anthony G. Cohn, Joel Z. Leibo, and Jose Hernandez-Orallo. 2023. Rethink… [find](https://scholar.google.com/scholar?q=Ryan+Burnell%2C+Wout+Schellaert%2C+John+Burden%2C+Tomer+D%2E+Ullman%2C+Fernando+Martinez%2DPlumed%2C+Joshua+B%2E+Tenenbaum%2C+Danaja+Rutar%2C+Lucy+G%2E+Cheke%2C+Jascha+Sohl%2DDickstein%2C+Melanie+Mitchell%2C+Douwe+Kiela%2C+Murray+Sh)
6. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. [A survey on evaluation of large language models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/). ACM transactions on intelligent systems and technology, 15(3):1–45.
7. Ruchira Dhar, Danae Sanchez Villegas, Antonia Karamolegkou, Alice Schiavone, Yifei Yuan, Xinyi Chen, Jiaang Li, Stella Frank, Laura De Grazia, Monorama Swain, and 1 others. 2025. Evalcards: A framework for standardized evaluation reporting. arXiv preprint arXiv:2511.21695. — [arXiv:2511.21695](https://arxiv.org/abs/2511.21695) [find](https://scholar.google.com/scholar?q=Ruchira+Dhar%2C+Danae+Sanchez+Villegas%2C+Antonia+Karamolegkou%2C+Alice+Schiavone%2C+Yifei+Yuan%2C+Xinyi+Chen%2C+Jiaang+Li%2C+Stella+Frank%2C+Laura+De+Grazia%2C+Monorama+Swain%2C+and+1+others%2E+2025%2E+Evalcards%3A+A+framewor)
8. Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation. Preprint, arXiv:2502.06559. — [arXiv:2502.06559](https://arxiv.org/abs/2502.06559) [find](https://scholar.google.com/scholar?q=Maria+Eriksson%2C+Erasmo+Purificato%2C+Arman+Noroozian%2C+Joao+Vinagre%2C+Guillaume+Chaslot%2C+Emilia+Gomez%2C+and+David+Fernandez%2DLlorca%2E+2025%2E+Can+we+trust+ai+benchmarks%3F+an+interdisciplinary+review+of+current+)
9. Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness gym: Unifying the nlp evaluation landscape. arXiv preprint arXiv:2101.04840. [find](https://scholar.google.com/scholar?q=Karan+Goel%2C+Nazneen+Rajani%2C+Jesse+Vig%2C+Samson+Tan%2C+Jason+Wu%2C+Stephan+Zheng%2C+Caiming+Xiong%2C+Mohit+Bansal%2C+and+Christopher+R%C3%A9%2E+2021%2E+Robustness+gym%3A+Unifying+the+nlp+evaluation+landscape%2E+arXiv+preprint)
10. Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder, Allie Griffith, Dylan M Asmar, Sanmi Koyejo, Michael S Bernstein, and Mykel John Kochenderfer. 2025. More than marketing? on the information value of ai benchmarks for practitioners. In Proceedings of the 30th International Conference on I… [find](https://scholar.google.com/scholar?q=Amelia+Hardy%2C+Anka+Reuel%2C+Kiana+Jafari+Meimandi%2C+Lisa+Soder%2C+Allie+Griffith%2C+Dylan+M+Asmar%2C+Sanmi+Koyejo%2C+Michael+S+Bernstein%2C+and+Mykel+John+Kochenderfer%2E+2025%2E+More+than+marketing%3F+on+the+informatio)
11. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, and 1 others. 2021. Dynabench: Rethinking benchmarking in nlp. arXiv preprint arXiv:2104.14337. [find](https://scholar.google.com/scholar?q=Douwe+Kiela%2C+Max+Bartolo%2C+Yixin+Nie%2C+Divyansh+Kaushik%2C+Atticus+Geiger%2C+Zhengxuan+Wu%2C+Bertie+Vidgen%2C+Grusha+Prasad%2C+Amanpreet+Singh%2C+Pratik+Ringshia%2C+and+1+others%2E+2021%2E+Dynabench%3A+Rethinking+benchmark)
12. Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel, Amit Agarwal, and Alice Oh. 2025. Benchhub: A unified benchmark suite for holistic and customizable llm evaluation. arXiv preprint arXiv:2506.00482. — [arXiv:2506.00482](https://arxiv.org/abs/2506.00482) [find](https://scholar.google.com/scholar?q=Eunsu+Kim%2C+Haneul+Yoo%2C+Guijin+Son%2C+Hitesh+Patel%2C+Amit+Agarwal%2C+and+Alice+Oh%2E+2025%2E+Benchhub%3A+A+unified+benchmark+suite+for+holistic+and+customizable+llm+evaluation%2E+arXiv+preprint+arXiv%3A2506%2E00482%2E+%E2%80%94+)
13. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+and+1+others%2E+2022%2E+Holistic+evaluation+of+language+m)
14. Weixin Liang, Nazneen Rajani, Xinyu Yang, Ezinwanne Ozoani, Eric Wu, Yiqun Chen, Daniel Scott Smith, and James Zou. 2024. Systematic analysis of 32,111 ai model cards characterizes documentation practice in ai. Nature Machine Intelligence, 6(7):744–753. [find](https://scholar.google.com/scholar?q=Weixin+Liang%2C+Nazneen+Rajani%2C+Xinyu+Yang%2C+Ezinwanne+Ozoani%2C+Eric+Wu%2C+Yiqun+Chen%2C+Daniel+Scott+Smith%2C+and+James+Zou%2E+2024%2E+Systematic+analysis+of+32%2C111+ai+model+cards+characterizes+documentation+pract)
15. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. [find](https://scholar.google.com/scholar?q=Moran+Mizrahi%2C+Guy+Kaplan%2C+Dan+Malkin%2C+Rotem+Dror%2C+Dafna+Shahaf%2C+and+Gabriel+Stanovsky%2E+2024%2E+State+of+what+art%3F+a+call+for+multi%2Dprompt+llm+evaluation%2E+Transactions+of+the+Association+for+Computation)
16. Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366. [find](https://scholar.google.com/scholar?q=Inioluwa+Deborah+Raji%2C+Emily+M+Bender%2C+Amandalynne+Paullada%2C+Emily+Denton%2C+and+Alex+Hanna%2E+2021%2E+Ai+and+the+everything+in+the+whole+wide+world+benchmark%2E+arXiv+preprint+arXiv%3A2111%2E15366%2E)
17. Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, and Mykel J Kochenderfer. 2024. Betterbench: Assessing ai benchmarks, uncovering issues, and establishing best practices. Advances in Neural Information Processing Systems, 37:21763–21813. [find](https://scholar.google.com/scholar?q=Anka+Reuel%2C+Amelia+Hardy%2C+Chandler+Smith%2C+Max+Lamparth%2C+Malcolm+Hardy%2C+and+Mykel+J+Kochenderfer%2E+2024%2E+Betterbench%3A+Assessing+ai+benchmarks%2C+uncovering+issues%2C+and+establishing+best+practices%2E+Advance)
18. Anna Sokol, Elizabeth Daly, Michael Hind, David Piorkowski, Xiangliang Zhang, Nuno Moniz, and Nitesh V. Chawla. 2024. Benchmarkcards: Standardized documentation for large language model benchmarks. ArXiv. [find](https://scholar.google.com/scholar?q=Anna+Sokol%2C+Elizabeth+Daly%2C+Michael+Hind%2C+David+Piorkowski%2C+Xiangliang+Zhang%2C+Nuno+Moniz%2C+and+Nitesh+V%2E+Chawla%2E+2024%2E+Benchmarkcards%3A+Standardized+documentation+for+large+language+model+benchmarks%2E+Ar)
19. Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, and 1 others. 2024. [Benchmark data contamination of large language models: A survey](https://md.research.elamur.ai/free/en/papers/xu-2024-benchmark-data-contamination-large/). arXiv preprint arXiv:2406.04244.
20. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. [Don’t make your llm an evaluation benchmark cheater](https://md.research.elamur.ai/free/en/papers/zhou-2023-don-t-make-llm-evaluation/). arXiv preprint arXiv:2311.01964. *Figure 1: Cumulative growth of language model benchmarks across arXiv, GitHub, and Hugging F…

*Updated 2 November 2026*