--- title: "Questionable Answers in Question Answering Research:" description: "The paper examines how reproducible neural results in NLP really are. Cited by 25 papers here." authors: ["Matt Crane"] year: 2018 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/Q18-1018/" doi: "10.1162/tacl_a_00018" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/crane-2018-questionable-answers-question-answering-research/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/crane-2018-questionable-answers-question-answering-research/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) / Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results # Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results **Authors:** Matt Crane · **Year:** 2018 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2018](https://md.research.elamur.ai/free/en/venues/tacl/2018/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/Q18-1018/) · [PDF](https://aclanthology.org/Q18-1018.pdf) · [DOI 10.1162/tacl\_a\_00018](https://doi.org/10.1162/tacl_a_00018) · [OpenAlex](https://openalex.org/W2799424953) **Citations:** 70 citations (OpenAlex, 2026-10-14) · [25](https://md.research.elamur.ai/free/en/papers/crane-2018-questionable-answers-question-answering-research/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary The paper examines how reproducible neural results in NLP really are. It argues that releasing source code does not guarantee that others can reproduce a reported number. Crane uses answer selection for open-domain question answering as a test case. The exemplar model is the simple Siamese CNN by Severyn and Moschitti (2015), in the PyTorch reimplementation by Sequiera et al. (2017). It is evaluated on TrecQA (RAW variant) and WikiQA with AP and RR. Hyper-parameters are held fixed throughout. Only one environmental factor is changed at a time, for example the code revision used. Differences within each condition are checked with a paired Wilcoxon signed rank test. The main finding is that these normally unreported factors can move scores by more than most published state-of-the-art gains. Code without a reproducible environment is therefore of limited value. The paper also shows that a learning-free baseline beats several older published systems, exposing the use of weak baselines. For each factor, it recommends ways to make reported results reproducible. ## Key points - A non-learning baseline sums IDF weights of the terms shared by question and candidate, after removing 127 nltk stopwords. On TrecQA RAW it reaches 0.701 AP and 0.769 RR. This beats a number of earlier state-of-the-art systems, while Yih et al. (2013) had reported only 0.6531 AP and 0.7071 RR for it. - Successive state-of-the-art gains on TrecQA and WikiQA, compiled from the ACL Wiki and from papers, are mostly small. Many are around 0.01–0.02 AP or less. - Running different commits of the same model repository changed TrecQA AP from 0.7495 (cf0e269) to about 0.67 in later versions. These drops were significant at p \< 0.01. - Some revisions would not run inside the Docker environment. Others gave different results from one run to the next. - The TrecQA RAW and CLEAN variants are not comparable with each other. CLEAN removes questions that have no positive answers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of Waterloo](https://md.research.elamur.ai/free/en/universities/university-of-waterloo/) — University, Waterloo, Canada · 67 works · 13,592 citations 📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🎯 Decision Sciences](https://md.research.elamur.ai/free/en/shelves/decision-sciences/) · 209 works · [🎯 Information Systems and Management](https://md.research.elamur.ai/free/en/shelves/decision-sciences/information-systems-and-management/) · 41 works 🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 441 works ## Cite Cite as: Matt Crane. 2018. Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{crane2018questionable, title = {{Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results}}, author = {Matt Crane}, journal = {Transactions of the Association for Computational Linguistics}, year = {2018}, doi = {10.1162/tacl\_a\_00018}, url = {https://doi.org/10.1162/tacl_a_00018} } ``` - [← previous Overview of the EVALITA 2018 Task on Irony Detection in Italian Tweets (IronITA)](https://md.research.elamur.ai/free/en/papers/cignarella-2018-overview-evalita-2018-task-irony/) - [next → Shami: A Corpus of Levantine Arabic Dialects](https://md.research.elamur.ai/free/en/papers/kwaik-2018-shami-corpus-levantine-arabic-dialects/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/4/) *Updated 30 October 2026*