--- title: "SCALAR — paper by Wang et al. (2026) — summary and ideas" description: "SCALAR thus offers a realistic, renewable test of long-context reasoning in the scientific domain. 3 references." authors: ["Renxi Wang", "Honglin Mu", "Liqun Ma", "Lizhi Lin", "Yunlong Feng", "Timothy Baldwin", "Xudong Han", "Haonan Li"] year: 2026 venue: "Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2026.eacl-long.366/" doi: "10.18653/v1/2026.eacl-long.366" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2026-scalar-scientific-citation-based-live/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-scalar-scientific-citation-based-live/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/) / SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning # SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning **Authors:** Renxi Wang, Honglin Mu, [Liqun Ma](https://md.research.elamur.ai/free/en/authors/liqun-ma/), Lizhi Lin, [Yunlong Feng](https://md.research.elamur.ai/free/en/authors/yunlong-feng/), [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/), [Xudong Han](https://md.research.elamur.ai/free/en/authors/xudong-han/) and [Haonan Li](https://md.research.elamur.ai/free/en/authors/haonan-li/) · **Year:** 2026 · **Venue:** [Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/eacl/) ([EACL 2026](https://md.research.elamur.ai/free/en/venues/eacl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.eacl-long.366/) · [PDF](https://aclanthology.org/2026.eacl-long.366.pdf) · [DOI 10.18653/v1/2026.eacl-long.366](https://doi.org/10.18653/v1/2026.eacl-long.366) · [OpenAlex](https://openalex.org/W7140091707) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Evaluating how well LLMs actually understand long documents is difficult. Synthetic retrieval tests such as needle-in-a-haystack are easy for current models, and benchmarks built from existing corpora risk contamination and allow shortcut answers. SCALAR addresses this with academic papers whose citations act as expert-made labels, so no manual annotation is needed. The authors gather arXiv versions of accepted ICLR 2024 and ICLR 2025 papers, parse them into structured form, and mask in-text citations. Models must recover the masked citations in two tasks: - **Multiple-choice QA (MCQA):** pick one correct reference among four candidates. - **Cloze-style prediction:** fill four masks at once from a shared candidate list. Difficulty (Easy, Medium, Hard) is controlled by citation type, distractor sampling and context length. A live update mechanism adds new papers over time to limit contamination. Human experts score above 90% on MCQA, while most LLMs struggle. The cloze task is harder still, with no model reaching 50%. SCALAR thus offers a realistic, renewable test of long-context reasoning in the scientific domain. ## Key points - A contamination analysis over four years of ICLR papers found that pre-2023 papers are often memorized by models, while 2024–2025 papers appear largely unseen. This finding motivated the benchmark's continuous updating. - Difficulty is governed by several factors: - whether the citation is attributional or descriptive; - how distractor candidates are drawn (at random or by nearest neighbor); - question scope (section vs. full paper); - candidate representation (abstract, body, or full text). - Two PhD candidates in computer science answered 30 MCQA questions spread across the three difficulty levels. They were perfect on Easy and highly accurate on Medium and Hard. - In cloze prediction, open-source models lose about 10 points relative to MCQA, and proprietary models can lose up to half their score. The best average accuracy is below 40%. - Results for Qwen2.5-7B-Instruct-1M: - On Easy, accuracy falls from 0.88 to 0.40–0.42 when candidates are given as full papers. - In the hardest setting (descriptive, nearest-neighbor, full text), it scores 22%, below the 25% chance level. - Error analysis of 72 incorrect predictions: in over 70% of them, the model chose a topically similar but wrong paper. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 415 works · 94,028 citations - [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 201 works · 17,173 citations - [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 102 works · 22,618 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 249 works · [🎯 Statistics, Probability and Uncertainty](https://md.research.elamur.ai/free/en/shelves/decision-sciences/statistics-probability-and-uncertainty/) · 65 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works ## Cite Cite as: Renxi Wang, Honglin Mu, Liqun Ma, Lizhi Lin, Yunlong Feng, Timothy Baldwin, Xudong Han and Haonan Li. 2026. SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2026scalar, title = {{SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning}}, author = {Renxi Wang, Honglin Mu, Liqun Ma, Lizhi Lin, Yunlong Feng, Timothy Baldwin, Xudong Han and Haonan Li}, booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2026}, doi = {10.18653/v1/2026.eacl-long.366}, url = {https://doi.org/10.18653/v1/2026.eacl-long.366} } ``` - [← previous Scalable Video-Based Search in the VGT Dictionary](https://md.research.elamur.ai/free/en/papers/vandendriessche-2026-scalable-video-based-search-vgt/) - [next → ScaleBox: Enabling High-Fidelity and Scalable Code Verification for Large Language Models](https://md.research.elamur.ai/free/en/papers/zheng-2026-scalebox-enabling-high-fidelity-scalable/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/48/) ## References 3 references, 3 of them with a card of ours. ↗ leads to the work. 1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Sali… — [Phi-4 Technical Report](https://md.research.elamur.ai/free/en/papers/abdin-2024-phi-4-technical-report/) 2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774. 3. Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. 2024. [Litsearch: A retrieval benchmark for scientific literature search](https://md.research.elamur.ai/free/en/papers/ajith-2024-litsearch-retrieval-benchmark-scientific-literature/). *Updated 1 November 2026*