---
title: "MetaLead — paper by Timmer et al. (2026) — summary and ideas"
description: "The authors build MetaLead, a fully manually annotated resource that extends the SciLead document set of Şahinüç et al.… 47 references."
authors: ["Roelien C. Timmer", "Necva Bölücü", "Stephen Wan"]
year: 2026
venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.eacl-long.196/"
doi: "10.18653/v1/2026.eacl-long.196"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/timmer-2026-metalead-comprehensive-human-curated-leaderboard/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/timmer-2026-metalead-comprehensive-human-curated-leaderboard/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments

# MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments

**Authors:** Roelien C. Timmer, Necva Bölücü and [Stephen Wan](https://md.research.elamur.ai/free/en/authors/stephen-wan/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.196/) · [PDF](https://aclanthology.org/2026.eacl-long.196.pdf) · [DOI 10.18653/v1/2026.eacl-long.196](https://doi.org/10.18653/v1/2026.eacl-long.196) · [OpenAlex](https://openalex.org/W7140144222)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
Leaderboards are the standard way to compare machine learning models, but curating them by hand is costly. The datasets used to train automatic leaderboard extractors keep only each paper's top score and little context. This hides baselines, ablations and negative findings, which feeds "performance myopia" and publication bias. It also leaves unclear whether a score reflects in-domain or out-of-domain evaluation. The authors build MetaLead, a fully manually annotated resource that extends the SciLead document set of Şahinüç et al. (2024). For every paper they record all reported results, not just the headline one. They also add two schema fields: an experiment-type label (baseline, proposed method, or variation of the proposed method) and separate train and test datasets. They benchmark several LLMs on extracting this richer structure and propose a metric that takes experiment type into account. The experiments show that complete, structured extraction remains difficult. MetaLead therefore serves as a reference standard for transparent ML reporting, cost-aware model selection and meta-analysis of failures.

## Key points
- Tuple schema: each result becomes a six-element tuple covering the task, the datasets used for training and for testing, the evaluation metric, the reported score, and the kind of experiment, instead of the usual four elements of task, dataset, metric and score.
- The authors added 3,285 new records to SciLead's 295 best-result records, giving 3,568 annotated results from 43 English-language ML papers.
- The annotators were the authors themselves, working under CSIRO ethics guidance. Results shown only in figures (e.g., line graphs) were excluded.
- The paper benchmarks several LLMs on the extraction task and introduces a new evaluation metric for experiment-type-guided comparison.
- The base data is licensed CC-BY-NC 4.0, so MetaLead is for non-commercial research only. Data and code are at github.com/RoelTim/metalead.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Data61](https://md.research.elamur.ai/free/en/labs/data61/) — Research institute, Sydney, Australia · 16 works · 2,532 citations
- [Commonwealth Scientific and Industrial Research Organisation](https://md.research.elamur.ai/free/en/government/commonwealth-scientific-and-industrial-research-organisation/) — Government body, Canberra, Australia · 15 works · 1,898 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 41 works · [🧱 Materials Science](https://md.research.elamur.ai/free/en/shelves/materials-science/) · 26 works · [🧱 Materials Chemistry](https://md.research.elamur.ai/free/en/shelves/materials-science/materials-chemistry/) · 26 works

🧭 Topics (1): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Roelien C. Timmer, Necva Bölücü and Stephen Wan. 2026. MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{timmer2026metalead,
  title = {{MetaLead: A Comprehensive Human-Curated Leaderboard Dataset for Transparent Reporting of Machine Learning Experiments}},
  author = {Roelien C. Timmer, Necva Bölücü and Stephen Wan},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.196},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.196}
}
```

- [← previous MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking](https://md.research.elamur.ai/free/en/papers/ramamoorthy-2026-merlin-testbed-multilingual-multimodal-entity/)
- [next → Mind the Gap in Cultural Alignment: Task-Aware Culture Management for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2026-mind-gap-cultural-alignment-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. [Construction of the Literature Graph in Semantic Scholar](https://md.research.elamur.ai/free/en/papers/ammar-2018-construction-literature-graph-semantic-scholar/). In Proceedings of NAACL-HLT, pages 84–91. arXiv. 2025. arXiv Monthly Submissi…
2. Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. 2022. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274–19289. [find](https://scholar.google.com/scholar?q=Christina+Baek%2C+Yiding+Jiang%2C+Aditi+Raghunathan%2C+and+J+Zico+Kolter%2E+2022%2E+Agreement%2Don%2Dthe%2Dline%3A+Predicting+the+performance+of+neural+networks+under+distribution+shift%2E+Advances+in+Neural+Information+)
3. Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):1–15. [find](https://scholar.google.com/scholar?q=Lutz+Bornmann%2C+Robin+Haunschild%2C+and+Ru%CC%88diger+Mutz%2E+2021%2E+Growth+rates+of+modern+science%3A+a+latent+piecewise+growth+curve+approach+to+model+publication+numbers+from+established+and+new+literature+data)
4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. [find](https://scholar.google.com/scholar?q=Jacob+Cohen%2E+1960%2E+A+coefficient+of+agreement+for+nominal+scales%2E+Educational+and+psychological+measurement%2C+20%281%29%3A37%E2%80%9346%2E)
5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. [Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/). arXiv prep…
6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
7. Jennifer D’Souza, Salomon Kabongo, Hamed Babaei Giglou, and Yue Zhang. 2024. Overview of the CLEF 2024 simpletext task 4: SOTA? tracking the state-ofthe-art in scholarly publications. Working Notes of CLEF. [find](https://scholar.google.com/scholar?q=Jennifer+D%E2%80%99Souza%2C+Salomon+Kabongo%2C+Hamed+Babaei+Giglou%2C+and+Yue+Zhang%2E+2024%2E+Overview+of+the+CLEF+2024+simpletext+task+4%3A+SOTA%3F+tracking+the+state%2Dofthe%2Dart+in+scholarly+publications%2E+Working+Notes+of)
8. Elicit. 2023. Elicit: The AI Research Assistant. [find](https://scholar.google.com/scholar?q=Elicit%2E+2023%2E+Elicit%3A+The+AI+Research+Assistant%2E)
9. Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the Eye of the User: A Critique of NLP Leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846–4853, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh+and+Dan+Jurafsky%2E+2020%2E+Utility+is+in+the+Eye+of+the+User%3A+A+Critique+of+NLP+Leaderboards%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28E)
10. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86– 92. [find](https://scholar.google.com/scholar?q=Timnit+Gebru%2C+Jamie+Morgenstern%2C+Briana+Vecchione%2C+Jennifer+Wortman+Vaughan%2C+Hanna+Wallach%2C+Hal+Daume%CC%81+Iii%2C+and+Kate+Crawford%2E+2021%2E+Datasheets+for+datasets%2E+Communications+of+the+ACM%2C+64%2812%29%3A86%E2%80%93+92%2E)
11. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, et al. 2025. [Towards an AI co-scientist](https://md.research.elamur.ai/free/en/papers/gottweis-2025-accelerating-scientific-discovery-co/). arXiv preprint arXiv:2502.18864.
12. Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages… [find](https://scholar.google.com/scholar?q=Yufang+Hou%2C+Charles+Jochim%2C+Martin+Gleize%2C+Francesca+Bonin%2C+and+Debasis+Ganguly%2E+2019%2E+Identification+of+Tasks%2C+Datasets%2C+Evaluation+Metrics%2C+and+Numeric+Scores+for+Scientific+Leaderboards+Constructio)
13. Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2021. TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:… [find](https://scholar.google.com/scholar?q=Yufang+Hou%2C+Charles+Jochim%2C+Martin+Gleize%2C+Francesca+Bonin%2C+and+Debasis+Ganguly%2E+2021%2E+TDMSci%3A+A+Specialized+Corpus+for+Scientific+Literature+Entity+Tagging+of+Tasks+Datasets+and+Metrics%2E+In+Proceedin)
14. Shizhou Huang, Bo Xu, Yang Yu, Changqun Li, and Xin Alex Lin. 2025. GuideNER: Annotation Guidelines Are Better than Examples for In-Context Named Entity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24159–24166. [find](https://scholar.google.com/scholar?q=Shizhou+Huang%2C+Bo+Xu%2C+Yang+Yu%2C+Changqun+Li%2C+and+Xin+Alex+Lin%2E+2025%2E+GuideNER%3A+Annotation+Guidelines+Are+Better+than+Examples+for+In%2DContext+Named+Entity+Recognition%2E+In+Proceedings+of+the+AAAI+Confere)
15. Hugging Face. 2024. Mistral Large Model Card. Accessed: 2025-07-24. [find](https://scholar.google.com/scholar?q=Hugging+Face%2E+2024%2E+Mistral+Large+Model+Card%2E+Accessed%3A+2025%2D07%2D24%2E)
16. Fech Scen Khoo, Megan Mark, Roelien C. Timmer, Marcella Scoczynski Ribeiro Martins, Emily Fos-hee, Kaylin Bugbee, Gregory Renard, and Anamaria Berea. 2023. Building knowledge graphs in helio-physics and astrophysics. In Natural Language Processing and Information Systems, pages 215–228, Cham. Spring… [find](https://scholar.google.com/scholar?q=Fech+Scen+Khoo%2C+Megan+Mark%2C+Roelien+C%2E+Timmer%2C+Marcella+Scoczynski+Ribeiro+Martins%2C+Emily+Fos%2Dhee%2C+Kaylin+Bugbee%2C+Gregory+Renard%2C+and+Anamaria+Berea%2E+2023%2E+Building+knowledge+graphs+in+helio%2Dphysics+a)
17. Esther Landhuis. 2016. Scientific literature: Information overload. Nature, 535(7612):457–458. [find](https://scholar.google.com/scholar?q=Esther+Landhuis%2E+2016%2E+Scientific+literature%3A+Information+overload%2E+Nature%2C+535%287612%29%3A457%E2%80%93458%2E)
18. Ruizhe Li, Chenghua Lin, Matthew Collinson, Xiao Li, and Guanyi Chen. 2019. A Dual-Attention Hierarchical Recurrent Neural Network for Dialogue Act Classification. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 383–392. [find](https://scholar.google.com/scholar?q=Ruizhe+Li%2C+Chenghua+Lin%2C+Matthew+Collinson%2C+Xiao+Li%2C+and+Guanyi+Chen%2E+2019%2E+A+Dual%2DAttention+Hierarchical+Recurrent+Neural+Network+for+Dialogue+Act+Classification%2E+In+Proceedings+of+the+23rd+Conferenc)
19. Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. [SciREX: A challenge dataset for document-level information extraction](https://md.research.elamur.ai/free/en/papers/jain-2020-scirex-challenge-dataset-document-level/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7506– 7516, Online. Association for Computational Li…
20. Pengfei Liu, Jinlan Fu, Yang Xiao, Weizhe Yuan, Shuaichen Chang, Junqi Dai, Yixin Liu, Zihuiwen Ye, and Graham Neubig. 2021. ExplainaBoard: An explainable leaderboard for NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… [find](https://scholar.google.com/scholar?q=Pengfei+Liu%2C+Jinlan+Fu%2C+Yang+Xiao%2C+Weizhe+Yuan%2C+Shuaichen+Chang%2C+Junqi+Dai%2C+Yixin+Liu%2C+Zihuiwen+Ye%2C+and+Graham+Neubig%2E+2021%2E+ExplainaBoard%3A+An+explainable+leaderboard+for+NLP%2E+In+Proceedings+of+the+59)
21. David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-Farland, and Dan Jurafsky. 2018. [Measuring the evolution of a scientific field through citation frames](https://md.research.elamur.ai/free/en/papers/jurgens-2018-measuring-evolution-scientific-field-through/). Transactions of the Association for Computational Linguistics, 6:391–406.
22. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. [Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction](https://md.research.elamur.ai/free/en/papers/luan-2018-multi-task-identification-entities-relations/). pages 3219–3232.
23. Salomon Kabongo, Jennifer D’Souza, and Sören Auer. 2024a. Effective context selection in llm-based leaderboard generation: An empirical study. In International Conference on Applications of Natural Language to Information Systems, pages 150–160. Springer. [find](https://scholar.google.com/scholar?q=Salomon+Kabongo%2C+Jennifer+D%E2%80%99Souza%2C+and+So%CC%88ren+Auer%2E+2024a%2E+Effective+context+selection+in+llm%2Dbased+leaderboard+generation%3A+An+empirical+study%2E+In+International+Conference+on+Applications+of+Natural+L)
24. Salomon Kabongo, Jennifer D’Souza, and Sören Auer. 2024b. ORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph. International Journal on Digital Libraries, 25(1):41–54. [find](https://scholar.google.com/scholar?q=Salomon+Kabongo%2C+Jennifer+D%E2%80%99Souza%2C+and+So%CC%88ren+Auer%2E+2024b%2E+ORKG%2DLeaderboards%3A+a+systematic+workflow+for+mining+leaderboards+as+a+knowledge+graph%2E+International+Journal+on+Digital+Libraries%2C+25%281%29%3A41%E2%80%935)
25. Souvik Mandal, Ashish Talewar, Paras Ahuja, and Prathamesh Juvatkar. 2025. Nanonets-OCR-S: A model for transforming documents into structured markdown with intelligent content recognition and semantic tagging. [find](https://scholar.google.com/scholar?q=Souvik+Mandal%2C+Ashish+Talewar%2C+Paras+Ahuja%2C+and+Prathamesh+Juvatkar%2E+2025%2E+Nanonets%2DOCR%2DS%3A+A+model+for+transforming+documents+into+structured+markdown+with+intelligent+content+recognition+and+semantic)
26. Nick McGreivy and Ammar Hakim. 2024. Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature machine intelligence, 6(10):1256– 1269. [find](https://scholar.google.com/scholar?q=Nick+McGreivy+and+Ammar+Hakim%2E+2024%2E+Weak+baselines+and+reporting+biases+lead+to+overoptimism+in+machine+learning+for+fluid%2Drelated+partial+differential+equations%2E+Nature+machine+intelligence%2C+6%2810%29%3A1)
27. Salomon Kabongo, Jennifer D’Souza, and Sören Auer. 2021. Automated Mining of Leaderboards for Empirical AI Research. volume 13133 LNCS. [find](https://scholar.google.com/scholar?q=Salomon+Kabongo%2C+Jennifer+D%E2%80%99Souza%2C+and+So%CC%88ren+Auer%2E+2021%2E+Automated+Mining+of+Leaderboards+for+Empirical+AI+Research%2E+volume+13133+LNCS%2E)
28. Ishani Mondal, Yufang Hou, and Charles Jochim. 2021. End-to-end construction of NLP knowledge graph. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1885–1895. [find](https://scholar.google.com/scholar?q=Ishani+Mondal%2C+Yufang+Hou%2C+and+Charles+Jochim%2E+2021%2E+End%2Dto%2Dend+construction+of+NLP+knowledge+graph%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+ACL%2DIJCNLP+2021%2C+pages+1885%E2%80%931895%2E)
29. Marcin Kardas, Piotr Czapla, Pontus Stenetorp, Sebastian Ruder, Sebastian Riedel, Ross Taylor, and Robert Stojnic. 2020. AxCell: Automatic extraction of results from machine learning papers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8580… [find](https://scholar.google.com/scholar?q=Marcin+Kardas%2C+Piotr+Czapla%2C+Pontus+Stenetorp%2C+Sebastian+Ruder%2C+Sebastian+Riedel%2C+Ross+Taylor%2C+and+Robert+Stojnic%2E+2020%2E+AxCell%3A+Automatic+extraction+of+results+from+machine+learning+papers%2E+In+Procee)
30. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. [Don’t Give Me the Details, Just the Summary!](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/) Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisti…
31. Florian Karl, Lukas Malte Kemeter, Gabriel Dax, and Paulina Sierak. 2024. Position: embracing negative results in machine learning. In Proceedings of the 41st International Conference on Machine Learning, pages 23256–23265. [find](https://scholar.google.com/scholar?q=Florian+Karl%2C+Lukas+Malte+Kemeter%2C+Gabriel+Dax%2C+and+Paulina+Sierak%2E+2024%2E+Position%3A+embracing+negative+results+in+machine+learning%2E+In+Proceedings+of+the+41st+International+Conference+on+Machine+Learn)
32. OpenAI. 2024a. GPT-4.1. [https://openai.com/ind](https://openai.com/ind) ex/gpt-4-1/. Accessed: 2025-07-24. [↗](https://openai.com/ind)
33. OpenAI. 2024b. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). Technical Report arXiv:2410.21276, OpenAI, San Francisco, CA. Accessed: 2025-07-24.
34. OpenAI. 2025. Deep Research. [https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/). AI research assistant. [↗](https://openai.com/index/introducing-deep-research/)
35. OpenAI. 2025. Introducing OpenAI o3 and o4-mini. Technical report, OpenAI. Accessed: 2025-07-24. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+OpenAI+o3+and+o4%2Dmini%2E+Technical+report%2C+OpenAI%2E+Accessed%3A+2025%2D07%2D24%2E)
36. Wolfgang Otto, Matthäus Zloch, Lu Gan, Saurav Karmakar, and Stefan Dietze. 2023. GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8166– 8176. [find](https://scholar.google.com/scholar?q=Wolfgang+Otto%2C+Mattha%CC%88us+Zloch%2C+Lu+Gan%2C+Saurav+Karmakar%2C+and+Stefan+Dietze%2E+2023%2E+GSAP%2DNER%3A+A+Novel+Task%2C+Corpus%2C+and+Baseline+for+Scholarly+Entity+Extraction+Focused+on+Machine+Learning+Models+and+Da)
37. Roelien C. Timmer, Megan Mark, Fech Scen Khoo, Marcella Scoczynski Ribeiro Martins, Anamaria Berea, Gregory Renard, and Kaylin Bugbee. 2023. Nasa science mission directorate knowledge graph discovery. In Companion Proceedings of the ACM Web Conference 2023, WWW ’23 Companion, page 795–799, New York,… [find](https://scholar.google.com/scholar?q=Roelien+C%2E+Timmer%2C+Megan+Mark%2C+Fech+Scen+Khoo%2C+Marcella+Scoczynski+Ribeiro+Martins%2C+Anamaria+Berea%2C+Gregory+Renard%2C+and+Kaylin+Bugbee%2E+2023%2E+Nasa+science+mission+directorate+knowledge+graph+discovery%2E)
38. William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4):1– 38. [find](https://scholar.google.com/scholar?q=William+Webber%2C+Alistair+Moffat%2C+and+Justin+Zobel%2E+2010%2E+A+similarity+measure+for+indefinite+rankings%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C+28%284%29%3A1%E2%80%93+38%2E)
39. Evan F Risko and Sam J Gilbert. 2016. Cognitive offloading. Trends in cognitive sciences, 20(9):676– 688. [find](https://scholar.google.com/scholar?q=Evan+F+Risko+and+Sam+J+Gilbert%2E+2016%2E+Cognitive+offloading%2E+Trends+in+cognitive+sciences%2C+20%289%29%3A676%E2%80%93+688%2E)
40. Sean Yang, Chris Tensmeyer, and Curtis Wigington. 2022. TELIN: Table Entity LINker for Extracting Leaderboards from Machine Learning Publications. [find](https://scholar.google.com/scholar?q=Sean+Yang%2C+Chris+Tensmeyer%2C+and+Curtis+Wigington%2E+2022%2E+TELIN%3A+Table+Entity+LINker+for+Extracting+Leaderboards+from+Machine+Learning+Publications%2E)
41. Oscar Sainz, Iker Garcı́a-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://md.research.elamur.ai/free/en/papers/sainz-2023-gollie-annotation-guidelines-improve-zero/). In The Twelfth International Conference on Learning Representations. Fatih Şahinüç, Thi Thao Tran, Yuliya Grishina, Yuf…
42. Mayank Singh, Rajdeep Sarkar, Atharva Vyas, Pawan Goyal, Animesh Mukherjee, and Soumen Chakrabarti. 2019. Automated early leaderboard generation from comparative tables. In Advances in Information Retrieval: 41st European Conference on IR Research, ECIR 2019, Cologne, Germany, April 14–18, 2019, Pro… [find](https://scholar.google.com/scholar?q=Mayank+Singh%2C+Rajdeep+Sarkar%2C+Atharva+Vyas%2C+Pawan+Goyal%2C+Animesh+Mukherjee%2C+and+Soumen+Chakrabarti%2E+2019%2E+Automated+early+leaderboard+generation+from+comparative+tables%2E+In+Advances+in+Information+Ret)
43. Shruti Singh, Shoaib Alam, Husain Malwat, and Mayank Singh. 2024. LEGOBench: Scientific Leaderboard Generation Benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14598–14613. [find](https://scholar.google.com/scholar?q=Shruti+Singh%2C+Shoaib+Alam%2C+Husain+Malwat%2C+and+Mayank+Singh%2E+2024%2E+LEGOBench%3A+Scientific+Leaderboard+Generation+Benchmark%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2024%2C+page)
44. Zihan Song, Gyo-Yeob Hwang, Xin Zhang, Shan Huang, and Byung-Kwon Park. 2025. A scientific-article key-insight extraction system based on multi-actor of fine-tuned open-source large language models. Scientific Reports, 15(1):1608. [find](https://scholar.google.com/scholar?q=Zihan+Song%2C+Gyo%2DYeob+Hwang%2C+Xin+Zhang%2C+Shan+Huang%2C+and+Byung%2DKwon+Park%2E+2025%2E+A+scientific%2Darticle+key%2Dinsight+extraction+system+based+on+multi%2Dactor+of+fine%2Dtuned+open%2Dsource+large+language+models%2E+S)
45. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530.
46. Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. 2020. On the value of out-of-distribution testing: An example of Goodhart’s Law. Advances in neural information processing systems, 33:407–417. [find](https://scholar.google.com/scholar?q=Damien+Teney%2C+Ehsan+Abbasnejad%2C+Kushal+Kafle%2C+Robik+Shrestha%2C+Christopher+Kanan%2C+and+Anton+Van+Den+Hengel%2E+2020%2E+On+the+value+of+out%2Dof%2Ddistribution+testing%3A+An+example+of+Goodhart%E2%80%99s+Law%2E+Advances+in+)
47. Roelien C. Timmer, Yufang Hou, and Stephen Wan. 2025. A Position Paper on the Automatic Generation of Machine Learning Leaderboards. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 30761–30784, Suzhou, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roelien+C%2E+Timmer%2C+Yufang+Hou%2C+and+Stephen+Wan%2E+2025%2E+A+Position+Paper+on+the+Automatic+Generation+of+Machine+Learning+Leaderboards%2E+In+Proceedings+of+the+2025+Conference+on+Empirical+Methods+in+Natur)

*Updated 31 October 2026*