--- title: "Experiments or Outcomes? Probing Scientific Feasibility in Large" description: "They build a controlled-knowledge protocol in which each hypothesis is shown in one of four ways: alone, with extracted… 10 references." authors: ["Seyedali Mohammadi", "Manas Gaur", "Francis Ferraro"] year: 2026 venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/2026.acl-short.50/" doi: "10.18653/v1/2026.acl-short.50" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/mohammadi-2026-experiments-or-outcomes-probing-scientific/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mohammadi-2026-experiments-or-outcomes-probing-scientific/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) / Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models # Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models **Authors:** [Seyedali Mohammadi](https://md.research.elamur.ai/free/en/authors/seyedali-mohammadi/), [Manas Gaur](https://md.research.elamur.ai/free/en/authors/manas-gaur/) and [Francis Ferraro](https://md.research.elamur.ai/free/en/authors/francis-ferraro/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.acl-short.50/) · [PDF](https://aclanthology.org/2026.acl-short.50.pdf) · [DOI 10.18653/v1/2026.acl-short.50](https://doi.org/10.18653/v1/2026.acl-short.50) · [OpenAlex](https://openalex.org/W7166868990) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary LLMs are increasingly used in research pipelines, but it is unclear how well they judge whether a scientific claim is feasible, meaning whether it fits existing knowledge and can be tested experimentally. The authors treat this as a binary decision (FEASIBLE/INFEASIBLE) accompanied by a short justification. They build a controlled-knowledge protocol in which each hypothesis is shown in one of four ways: alone, with extracted experiment descriptions, with extracted outcomes, or with both. A stability analysis then randomly removes parts of the experiment and outcome lists to see how predictions degrade. Five proprietary models are tested on MATTER-OF-FACT and REASONS, with post-cutoff filtering to reduce pretraining leakage. Outcome evidence generally improves accuracy beyond what the models know internally. Experiment descriptions are fragile: incomplete context can push performance below the hypothesis-only baseline, often in a non-monotonic way. The authors trace this to three behaviours: matching on surface topic, failing to check whether the evidence is relevant, and anchoring on partial fragments. The framework matters because it reveals weaknesses that standard claim-verification and RAG evaluations miss. ## Key points - **Protocol:** There are four conditions (H, H+E, H+O, H+E+O). Reveal fractions k1 (experiments) and k2 (outcomes) take values in {0, 0.5, 1.0}, with R = 5 random subsamples per configuration, reported as mean and standard deviation. - **Metrics:** Accuracy, macro-F1 and MCC are reported, plus a below-baseline rate. A lexical-overlap score (ROUGE) against gold explanations is used on MATTER-OF-FACT only as a diagnostic. - **Models:** gpt-5.1, gpt-4o, Gemini-2.5-Pro, Gemini-2.5-Flash and Grok-4.1-fast, all prompted with identical instructions. - **Data:** - MATTER-OF-FACT (Jansen et al., 2025) contains claims that a source paper may support, contradict or only partly address. - REASONS (Saxena et al., 2025) is a positive-feasibility set. - Leakage control relies on model knowledge cutoffs: GPT-4o (Oct 2023) uses the full test split, while GPT-5.1 (Sep 2024) and Gemini-2.5 (Jan 2025) use narrower 2024–2025 subsets. - **Evidence and resources:** Experiments and outcomes are extracted once per instance and the full paper text is never shown; a manual audit checked extraction coverage. Code and prompts are available at github.com/mohammadi-ali/scify (DARPA SciFy program). *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Maryland, Baltimore County](https://md.research.elamur.ai/free/en/universities/university-of-maryland-baltimore-county/) — University, Baltimore, United States · 14 works · 915 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works 🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🔬 Interpretability](https://md.research.elamur.ai/free/en/topics/interpretability/) · 130 works ## Cite Cite as: Seyedali Mohammadi, Manas Gaur and Francis Ferraro. 2026. Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{mohammadi2026experiments, title = {{Experiments or Outcomes? Probing Scientific Feasibility in Large Language Models}}, author = {Seyedali Mohammadi, Manas Gaur and Francis Ferraro}, booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2026}, doi = {10.18653/v1/2026.acl-short.50}, url = {https://doi.org/10.18653/v1/2026.acl-short.50} } ``` - [← previous Experience-driven Multi-turn Reinforcement Learning for GUI Agents](https://md.research.elamur.ai/free/en/papers/lu-2026-experience-driven-multi-turn-reinforcement/) - [next → Expert Calibration Lens for Pruning Mixture of Experts](https://md.research.elamur.ai/free/en/papers/salim-2026-expert-calibration-lens-pruning-mixture/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/29/) ## References 10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, and Tristan Miller. 2025. [Transforming science with large language models: A survey on ai-assisted scien…](https://md.research.elamur.ai/free/en/papers/eger-2025-transforming-science-large-language-models/) 2. Peter Jansen, Samiah Hassan, and Ruoyao Wang. 2025. Matter-of-fact: A benchmark for verifying the feasibility of literature-supported claims in materials science. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 4090–4102, Suzhou, China. Association fo… [find](https://scholar.google.com/scholar?q=Peter+Jansen%2C+Samiah+Hassan%2C+and+Ruoyao+Wang%2E+2025%2E+Matter%2Dof%2Dfact%3A+A+benchmark+for+verifying+the+feasibility+of+literature%2Dsupported+claims+in+materials+science%2E+In+Proceedings+of+the+2025+Conference) 3. Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, and Hyunseok Oh. 2025. Beyond designer’s knowledge: Generating materials design hypotheses via a large language model. Acta Materialia, 297:121307. [find](https://scholar.google.com/scholar?q=Quanliang+Liu%2C+Maciej+P%2E+Polak%2C+So+Yeon+Kim%2C+MD+Al+Amin+Shuvo%2C+Hrishikesh+Shridhar+Deodhar%2C+Jeongsoo+Han%2C+Dane+Morgan%2C+and+Hyunseok+Oh%2E+2025%2E+Beyond+designer%E2%80%99s+knowledge%3A+Generating+materials+design+h) 4. Seyedali Mohammadi, Bhaskara Hanuma Vedula, He-mank Lamba, Edward Raff, Ponnurangam Kumaraguru, Francis Ferraro, and Manas Gaur. 2025. Do LLMs adhere to label definitions? examining their receptivity to external label definitions. In Proceedings of the 2025 Conference on Empirical Methods in Natural… [find](https://scholar.google.com/scholar?q=Seyedali+Mohammadi%2C+Bhaskara+Hanuma+Vedula%2C+He%2Dmank+Lamba%2C+Edward+Raff%2C+Ponnurangam+Kumaraguru%2C+Francis+Ferraro%2C+and+Manas+Gaur%2E+2025%2E+Do+LLMs+adhere+to+label+definitions%3F+examining+their+receptivity+) 5. Yumi Nakamura, Hiroshi Watanabe, Aiko Tanaka, Masato Yasui, Jun Nishihira, and Norihito Murayama. 2020. Effect of increased daily water intake and hydration on health in japanese adults. Nutrients, 12(4). [find](https://scholar.google.com/scholar?q=Yumi+Nakamura%2C+Hiroshi+Watanabe%2C+Aiko+Tanaka%2C+Masato+Yasui%2C+Jun+Nishihira%2C+and+Norihito+Murayama%2E+2020%2E+Effect+of+increased+daily+water+intake+and+hydration+on+health+in+japanese+adults%2E+Nutrients%2C+12) 6. Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. 2023. Large language models are zero shot hypothesis proposers. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. [find](https://scholar.google.com/scholar?q=Biqing+Qi%2C+Kaiyan+Zhang%2C+Haoxiang+Li%2C+Kai+Tian%2C+Sihang+Zeng%2C+Zhang%2DRen+Chen%2C+and+Bowen+Zhou%2E+2023%2E+Large+language+models+are+zero+shot+hypothesis+proposers%2E+In+NeurIPS+2023+Workshop+on+Instruction+Tun) 7. Delip Rao, Weiqiu You, Eric Wong, and Chris Callison-Burch. 2025. [NSF-SciFy: Mining the NSF awards database for scientific claims](https://md.research.elamur.ai/free/en/papers/rao-2026-nsf-scify-mining-nsf-awards/). In Proceedings of The 5th New Frontiers in Summarization Workshop, pages 183–198, Hybrid. Association for Computational Linguistics. 8. Yash Saxena, Deepa Tilwani, Ali Mohammadi, Edward Raff, Amit Sheth, Srinivasan Parthasarathy, and Manas Gaur. 2025. Attribution in scientific literature: New benchmark and methods. Preprint, arXiv:2405.02228. [find](https://scholar.google.com/scholar?q=Yash+Saxena%2C+Deepa+Tilwani%2C+Ali+Mohammadi%2C+Edward+Raff%2C+Amit+Sheth%2C+Srinivasan+Parthasarathy%2C+and+Manas+Gaur%2E+2025%2E+Attribution+in+scientific+literature%3A+New+benchmark+and+methods%2E+Preprint%2C+arXiv%3A240) 9. Zonglin Yang, Xinya Du, Junxian Li, Jie Zheng, Soujanya Poria, and Erik Cambria. 2024. Large language models for automated open-domain scientific hypotheses discovery. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13545–13565, Bangkok, Thailand. Association for Comput… [find](https://scholar.google.com/scholar?q=Zonglin+Yang%2C+Xinya+Du%2C+Junxian+Li%2C+Jie+Zheng%2C+Soujanya+Poria%2C+and+Erik+Cambria%2E+2024%2E+Large+language+models+for+automated+open%2Ddomain+scientific+hypotheses+discovery%2E+In+Findings+of+the+Association+f) 10. Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. 2025. From automation to autonomy: A survey on large language models in scientific discovery. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17733–17750… [find](https://scholar.google.com/scholar?q=Tianshi+Zheng%2C+Zheye+Deng%2C+Hong+Ting+Tsang%2C+Weiqi+Wang%2C+Jiaxin+Bai%2C+Zihao+Wang%2C+and+Yangqiu+Song%2E+2025%2E+From+automation+to+autonomy%3A+A+survey+on+large+language+models+in+scientific+discovery%2E+In+Proce) *Updated 1 November 2026*