---
title: "Replicability Analysis for Natural Language Processing: Testing"
description: "The authors show that this practice does not control the chance of making at least one false claim… 71 references, cited by 35 papers here."
authors: ["Rotem Dror", "Gili Baumer", "Marina Bogomolov", "Roi Reichart"]
year: 2017
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q17-1033/"
doi: "10.1162/tacl_a_00074"
arxiv: "1709.09500"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/dror-2017-replicability-analysis-natural-language-processing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/dror-2017-replicability-analysis-natural-language-processing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/) / Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

# Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets

**Authors:** [Rotem Dror](https://md.research.elamur.ai/free/en/authors/rotem-dror/), Gili Baumer, Marina Bogomolov and [Roi Reichart](https://md.research.elamur.ai/free/en/authors/roi-reichart/) · **Year:** 2017 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2017](https://md.research.elamur.ai/free/en/venues/tacl/2017/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q17-1033/) · [PDF](https://aclanthology.org/Q17-1033.pdf) · [DOI 10.1162/tacl\_a\_00074](https://doi.org/10.1162/tacl_a_00074) · [arXiv 1709.09500](https://arxiv.org/abs/1709.09500) · [OpenAlex](https://openalex.org/W2963583956)

**Citations:** 63 citations (OpenAlex, 2026-10-14) · [35](https://md.research.elamur.ai/free/en/papers/dror-2017-replicability-analysis-natural-language-processing/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
NLP systems are now routinely evaluated on many datasets spanning languages, domains and genres. A common practice is to run a significance test on each dataset and count how many fall below a threshold such as 0.05. The authors show that this practice does not control the chance of making at least one false claim, and the problem grows as more datasets are added. They adapt replicability analysis from statistics to answer two questions: on how many datasets one algorithm truly beats another (counting), and which datasets those are (identification). For counting, they use the partial conjunction approach of Benjamini and Heller, with Fisher-based p-values for independent datasets and Bonferroni-based p-values for dependent ones. For identification, they use Holm's procedure, which bounds the family-wise error. They test the framework on synthetic examples and four tasks: dependency parsing across several domains, part-of-speech tagging in many languages, sentiment classification transferred between domains, and predicting word similarity. The sound estimates often differ substantially from naive counts. They can be higher when datasets are independent and lower when datasets overlap. This gives NLP researchers a principled and compact way to report results across many evaluation sets.

## Key points
- The paper appeared in TACL vol. 5 (2017), and the code is released at github.com/rtmdrr/replicability-analysis-NLP.
- In the Mate vs. Redshift parsing comparison, the analysis finds evidence of an effect on 5 of 7 evaluation sets, whereas naive counting gives only 2.
- In multilingual POS tagging (MIMICK vs. Char→Tag), the analysis supports 16 of 23 sets with an effect, compared with 11 by naive counting.
- For dependent setups, naive counting overstates the gains of new algorithms. In sentiment classification at α = 0.05, it reports 10 of 12 sets, while the Bonferroni-based estimate is 6.
- Suggested extensions include Simes-based tests, Hochberg's or Hommel's procedures under positive dependence, and FDR control (Benjamini–Hochberg) when some errors are acceptable.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Technion – Israel Institute of Technology](https://md.research.elamur.ai/free/en/universities/technion-israel-institute-of-technology/) — University, Haifa, Israel · 59 works · 7,590 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 19,503 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 18,805 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,959 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,789 works

## Cite

Cite as: Rotem Dror, Gili Baumer, Marina Bogomolov and Roi Reichart. 2017. Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{dror2017replicability,
  title = {{Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets}},
  author = {Rotem Dror, Gili Baumer, Marina Bogomolov and Roi Reichart},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2017},
  doi = {10.1162/tacl\_a\_00074},
  eprint = {1709.09500},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00074}
}
```

- [← previous Event Coreference Resolution by Iteratively Unfolding Inter-dependencies among Events](https://md.research.elamur.ai/free/en/papers/choubey-2017-event-coreference-resolution-iteratively-unfolding/)
- [next → Don't Throw Those Morphological Analyzers Away Just Yet: Neural Morphological Disambiguation for Arabic](https://md.research.elamur.ai/free/en/papers/zalmout-2017-don-t-throw-those-morphological/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/4/)

## References

71 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. [A study on similarity and relatedness using distributional and WordNet-based approaches](https://md.research.elamur.ai/free/en/papers/agirre-2009-study-similarity-relatedness-distributional-wordnet/). In Proceedings of HLT-NAACL.
2. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. [Polyglot: Distributed word representations for multilingual NLP](https://md.research.elamur.ai/free/en/papers/rfou-2013-polyglot-distributed-word-representations-multilingual/). In Proceedings of CoNLL.
3. Simon Baker, Roi Reichart, and Anna Korhonen. 2014. An unsupervised model for instance level subcategorization acquisition. In Proceedings of EMNLP. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014. Don’t count, predict! a systematic comparison of context-counting vs. context-predicting se… [find](https://scholar.google.com/scholar?q=Simon+Baker%2C+Roi+Reichart%2C+and+Anna+Korhonen%2E+2014%2E+An+unsupervised+model+for+instance+level+subcategorization+acquisition%2E+In+Proceedings+of+EMNLP%2E+Marco+Baroni%2C+Georgiana+Dinu%2C+and+Germa%CC%81n+Kruszewsk)
4. C. Glenn Begley and Lee M. Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature, 483(7391):531–533. [find](https://scholar.google.com/scholar?q=C%2E+Glenn+Begley+and+Lee+M%2E+Ellis%2E+2012%2E+Drug+development%3A+Raise+standards+for+preclinical+cancer+research%2E+Nature%2C+483%287391%29%3A531%E2%80%93533%2E)
5. Yoav Benjamini and Ruth Heller. 2008. Screening for partial conjunction hypotheses. Biometrics, 64(4):1215–1222. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini+and+Ruth+Heller%2E+2008%2E+Screening+for+partial+conjunction+hypotheses%2E+Biometrics%2C+64%284%29%3A1215%E2%80%931222%2E)
6. Yoav Benjamini and Yosef Hochberg. 1995. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), pages 289– 300. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini+and+Yosef+Hochberg%2E+1995%2E+Controlling+the+false+discovery+rate%3A+A+practical+and+powerful+approach+to+multiple+testing%2E+Journal+of+the+Royal+Statistical+Society%2E+Series+B+%28Methodological)
7. Yoav Benjamini, Abba M. Krieger, and Daniel Yekutieli. 2006. Adaptive linear step-up procedures that control the false discovery rate. Biometrika, pages 491–507. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini%2C+Abba+M%2E+Krieger%2C+and+Daniel+Yekutieli%2E+2006%2E+Adaptive+linear+step%2Dup+procedures+that+control+the+false+discovery+rate%2E+Biometrika%2C+pages+491%E2%80%93507%2E)
8. Yoav Benjamini, Ruth Heller, and Daniel Yekutieli. 2009. Selective inference in complex research. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 367(1906):4255–4271. [find](https://scholar.google.com/scholar?q=Yoav+Benjamini%2C+Ruth+Heller%2C+and+Daniel+Yekutieli%2E+2009%2E+Selective+inference+in+complex+research%2E+Philosophical+Transactions+of+the+Royal+Society+of+London+A%3A+Mathematical%2C+Physical+and+Engineering+Sc)
9. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. 2012. An empirical investigation of statistical significance in NLP. In Proceedings of EMNLP-CoNLL. [find](https://scholar.google.com/scholar?q=Taylor+Berg%2DKirkpatrick%2C+David+Burkett%2C+and+Dan+Klein%2E+2012%2E+An+empirical+investigation+of+statistical+significance+in+NLP%2E+In+Proceedings+of+EMNLP%2DCoNLL%2E)
10. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. [Domain adaptation with structural correspondence learning](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/). In Proceedings of EMNLP.
11. John Blitzer, Mark Dredze, and Fernando Pereira. 2007. [Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification](https://md.research.elamur.ai/free/en/papers/blitzer-2007-biographies-bollywood-boom-boxes-blenders/). In Proceedings of ACL.
12. Bernd Bohnet. 2010. [Very high accuracy and fast dependency parsing is not a contradiction](https://md.research.elamur.ai/free/en/papers/bohnet-2010-top-accuracy-fast-dependency-parsing/). In Proceedings of COLING.
13. Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012. [Distributional semantics in technicolor](https://md.research.elamur.ai/free/en/papers/bruni-2012-distributional-semantics-technicolor/). In Proceedings of ACL.
14. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of Artificial Intelligence Research (JAIR), 49:1–47. [find](https://scholar.google.com/scholar?q=Elia+Bruni%2C+Nam%2DKhanh+Tran%2C+and+Marco+Baroni%2E+2014%2E+Multimodal+distributional+semantics%2E+Journal+of+Artificial+Intelligence+Research+%28JAIR%29%2C+49%3A1%E2%80%9347%2E)
15. Sabine Buchholz and Erwin Marsi. 2006. [CoNLL-x shared task on multilingual dependency parsing](https://md.research.elamur.ai/free/en/papers/buchholz-2006-conll-x-shared-task-multilingual/). In Proceedings of CoNLL.
16. Yee Seng Chan and Hwee Tou Ng. 2007. Domain adaptation with active learning for word sense disambiguation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Yee+Seng+Chan+and+Hwee+Tou+Ng%2E+2007%2E+Domain+adaptation+with+active+learning+for+word+sense+disambiguation%2E+In+Proceedings+of+ACL%2E)
17. Eugene Charniak. 2000. [A maximum-entropy-inspired parser](https://md.research.elamur.ai/free/en/papers/charniak-2000-maximum-entropy-inspired-parser/). In Proceedings of HLT-NAACL.
18. Minmin Chen, Yixin Chen, and Kilian Q. Weinberger. 2011. Automatic feature decomposition for single view co-training. In Proceedings of ICML. [find](https://scholar.google.com/scholar?q=Minmin+Chen%2C+Yixin+Chen%2C+and+Kilian+Q%2E+Weinberger%2E+2011%2E+Automatic+feature+decomposition+for+single+view+co%2Dtraining%2E+In+Proceedings+of+ICML%2E)
19. Jinho D. Choi, Joel Tetreault, and Amanda Stent. 2015. [It depends: Dependency parser comparison using a web-based evaluation tool](https://md.research.elamur.ai/free/en/papers/choi-2015-it-depends-dependency-parser-comparison/). In Proceedings of ACL.
20. Open Science Collaboration. 2012. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6):657–660. [find](https://scholar.google.com/scholar?q=Open+Science+Collaboration%2E+2012%2E+An+open%2C+large%2Dscale%2C+collaborative+effort+to+estimate+the+reproducibility+of+psychological+science%2E+Perspectives+on+Psychological+Science%2C+7%286%29%3A657%E2%80%93660%2E)
21. Michael Collins. 2003. [Head-driven statistical models for natural language parsing](https://md.research.elamur.ai/free/en/papers/collins-2003-head-driven-statistical-models-natural/). Computational linguistics, 29(4):589–637. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of ACL.
22. Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Stanford dependencies: A cross-linguistic typology. In Proceedings of LREC. Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets… [find](https://scholar.google.com/scholar?q=Marie%2DCatherine+De+Marneffe%2C+Timothy+Dozat%2C+Natalia+Silveira%2C+Katri+Haverinen%2C+Filip+Ginter%2C+Joakim+Nivre%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Stanford+dependencies%3A+A+cross%2Dlinguistic+typology%2E+In+Proc)
23. Thomas G. Dietterich. 1998. Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation, 10(7):1895–1923. [find](https://scholar.google.com/scholar?q=Thomas+G%2E+Dietterich%2E+1998%2E+Approximate+statistical+tests+for+comparing+supervised+classification+learning+algorithms%2E+Neural+computation%2C+10%287%29%3A1895%E2%80%931923%2E)
24. Bradley Efron and Robert J. Tibshirani. 1994. An introduction to the bootstrap. CRC press. [find](https://scholar.google.com/scholar?q=Bradley+Efron+and+Robert+J%2E+Tibshirani%2E+1994%2E+An+introduction+to+the+bootstrap%2E+CRC+press%2E)
25. Alessio Farcomeni. 2007. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Statistical Methods in Medical Research. [find](https://scholar.google.com/scholar?q=Alessio+Farcomeni%2E+2007%2E+A+review+of+modern+multiple+hypothesis+testing%2C+with+particular+attention+to+the+false+discovery+proportion%2E+Statistical+Methods+in+Medical+Research%2E)
26. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at wordvec-tors.org. In Proceedings of the ACL: System Demonstrations. [find](https://scholar.google.com/scholar?q=Manaal+Faruqui+and+Chris+Dyer%2E+2014%2E+Community+evaluation+and+exchange+of+word+vectors+at+wordvec%2Dtors%2Eorg%2E+In+Proceedings+of+the+ACL%3A+System+Demonstrations%2E)
27. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001a. Placing search in context: The concept revisited. In Proceedings of WWW. [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Evgeniy+Gabrilovich%2C+Yossi+Matias%2C+Ehud+Rivlin%2C+Zach+Solan%2C+Gadi+Wolfman%2C+and+Eytan+Ruppin%2E+2001a%2E+Placing+search+in+context%3A+The+concept+revisited%2E+In+Proceedings+of+WWW%2E)
28. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001b. Placing search in context: The concept revisited. In Proceedings of WWW. [find](https://scholar.google.com/scholar?q=Lev+Finkelstein%2C+Evgeniy+Gabrilovich%2C+Yossi+Matias%2C+Ehud+Rivlin%2C+Zach+Solan%2C+Gadi+Wolfman%2C+and+Eytan+Ruppin%2E+2001b%2E+Placing+search+in+context%3A+The+concept+revisited%2E+In+Proceedings+of+WWW%2E)
29. Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang, Xian Wu, and Zhong Su. 2009. Domain adaptation with latent semantic association for named entity recognition. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Honglei+Guo%2C+Huijia+Zhu%2C+Zhili+Guo%2C+Xiaoxun+Zhang%2C+Xian+Wu%2C+and+Zhong+Su%2E+2009%2E+Domain+adaptation+with+latent+semantic+association+for+named+entity+recognition%2E+In+Proceedings+of+HLT%2DNAACL%2E)
30. Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012. Large-scale learning of word relatedness with constraints. In Proceedings of ACM SIGKDD. [find](https://scholar.google.com/scholar?q=Guy+Halawi%2C+Gideon+Dror%2C+Evgeniy+Gabrilovich%2C+and+Yehuda+Koren%2E+2012%2E+Large%2Dscale+learning+of+word+relatedness+with+constraints%2E+In+Proceedings+of+ACM+SIGKDD%2E)
31. Ruth Heller, Marina Bogomolov, and Yoav Benjamini. 2014. Deciding whether follow-up studies have replicated findings in a preliminary large-scale omics study. Proceedings of the National Academy of Sciences, 111(46):16262–16267. [find](https://scholar.google.com/scholar?q=Ruth+Heller%2C+Marina+Bogomolov%2C+and+Yoav+Benjamini%2E+2014%2E+Deciding+whether+follow%2Dup+studies+have+replicated+findings+in+a+preliminary+large%2Dscale+omics+study%2E+Proceedings+of+the+National+Academy+of+Sc)
32. Thomas Herndon, Michael Ash, and Robert Pollin. 2014. Does high public debt consistently stifle economic growth? a critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38(2):257–279. [find](https://scholar.google.com/scholar?q=Thomas+Herndon%2C+Michael+Ash%2C+and+Robert+Pollin%2E+2014%2E+Does+high+public+debt+consistently+stifle+economic+growth%3F+a+critique+of+Reinhart+and+Rogoff%2E+Cambridge+Journal+of+Economics%2C+38%282%29%3A257%E2%80%93279%2E)
33. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. [Simlex-999: Evaluating semantic models with (genuine) similarity estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/). Computational Linguistics, 41(4):665–695.
34. Felix Hill, Roi Reichart, and Anna Korhonen. 2016. [Simlex-999: Evaluating semantic models with (genuine) similarity estimation](https://md.research.elamur.ai/free/en/papers/hill-2015-simlex-999-evaluating-semantic-models/).
35. Computational Linguistics. Yosef Hochberg. 1988. A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4):800–802. [find](https://scholar.google.com/scholar?q=Computational+Linguistics%2E+Yosef+Hochberg%2E+1988%2E+A+sharper+Bonferroni+procedure+for+multiple+tests+of+significance%2E+Biometrika%2C+75%284%29%3A800%E2%80%93802%2E)
36. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70. [find](https://scholar.google.com/scholar?q=Sture+Holm%2E+1979%2E+A+simple+sequentially+rejective+multiple+test+procedure%2E+Scandinavian+Journal+of+Statistics%2C+6%282%29%3A65%E2%80%9370%2E)
37. Gerhard Hommel. 1988. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2):383–386. [find](https://scholar.google.com/scholar?q=Gerhard+Hommel%2E+1988%2E+A+stagewise+rejective+multiple+test+procedure+based+on+a+modified+Bonferroni+test%2E+Biometrika%2C+75%282%29%3A383%E2%80%93386%2E)
38. Matthew Honnibal and Mark Johnson. 2015. [An improved non-monotonic transition system for dependency parsing](https://md.research.elamur.ai/free/en/papers/honnibal-2015-improved-non-monotonic-transition-system/). In Proceedings of EMNLP.
39. Matthew Honnibal, Yoav Goldberg, and Mark Johnson. 2013. [A non-monotonic arc-eager transition system for dependency parsing](https://md.research.elamur.ai/free/en/papers/honnibal-2013-non-monotonic-arc-eager-transition/). In Proceedings of CoNLL.
40. Philipp Koehn and Josh Schroeder. 2007. [Experiments in domain adaptation for statistical machine translation](https://md.research.elamur.ai/free/en/papers/koehn-2007-experiments-domain-adaptation-statistical-machine/). In Proceedings of the Second Workshop on Statistical Machine Translation.
41. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the tenth Machine Translation Summit. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+Proceedings+of+the+tenth+Machine+Translation+Summit%2E)
42. Jeffrey T. Leek and Roger D Peng. 2015. Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences, 112(6):1645–1646. [find](https://scholar.google.com/scholar?q=Jeffrey+T%2E+Leek+and+Roger+D+Peng%2E+2015%2E+Opinion%3A+Reproducible+research+can+still+be+wrong%3A+Adopting+a+prevention+approach%2E+Proceedings+of+the+National+Academy+of+Sciences%2C+112%286%29%3A1645%E2%80%931646%2E)
43. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Dependency%2Dbased+word+embeddings%2E+In+Proceedings+of+ACL%2E)
44. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. [Finding function in form: Compositional character models for open vocabulary word representation](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/). In Proceedings of EMNLP.
45. Thomas M. Loughin. 2004. A systematic comparison of methods for combining p-values from independent tests. Computational Statistics & Data Analysis, 47(3):467–485. [find](https://scholar.google.com/scholar?q=Thomas+M%2E+Loughin%2E+2004%2E+A+systematic+comparison+of+methods+for+combining+p%2Dvalues+from+independent+tests%2E+Computational+Statistics+%26+Data+Analysis%2C+47%283%29%3A467%E2%80%93485%2E)
46. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. [Better word representations with recursive neural networks for morphology](https://md.research.elamur.ai/free/en/papers/luong-2013-better-word-representations-recursive-neural/). In Proceedings of CoNLL.
47. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. [Building a large annotated corpus of English: The Penn Treebank](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/). Computational linguistics, 19(2):313–330.
48. Edison Marrese-Taylor and Yutaka Matsuo. 2017. Replication issues in syntax-based aspect extraction for opinion mining. In Proceedings of the Student Research Workshop at EACL. [find](https://scholar.google.com/scholar?q=Edison+Marrese%2DTaylor+and+Yutaka+Matsuo%2E+2017%2E+Replication+issues+in+syntax%2Dbased+aspect+extraction+for+opinion+mining%2E+In+Proceedings+of+the+Student+Research+Workshop+at+EACL%2E)
49. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. [find](https://scholar.google.com/scholar?q=Quinn+McNemar%2E+1947%2E+Note+on+the+sampling+error+of+the+difference+between+correlated+proportions+or+percentages%2E+Psychometrika%2C+12%282%29%3A153%E2%80%93157%2E)
50. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Proceedings of NIPS.
51. George A. Miller and Walter G. Charles. 1991. Contextual correlates of semantic similarity. Language and cognitive processes, 6(1):1–28. [find](https://scholar.google.com/scholar?q=George+A%2E+Miller+and+Walter+G%2E+Charles%2E+1991%2E+Contextual+correlates+of+semantic+similarity%2E+Language+and+cognitive+processes%2C+6%281%29%3A1%E2%80%9328%2E)
52. Ramal Moonesinghe, Muin J. Khoury, and A. Cecile J. W. Janssens. 2007. Most published research findings are false but a little replication goes a long way. PLoS Med, 4(2):e28. Aurélie Névéol, Cyril Grouin, Kevin Bretonnel Cohen, and Aude Robert. 2016. Replicability of research in biomedical natur… [find](https://scholar.google.com/scholar?q=Ramal+Moonesinghe%2C+Muin+J%2E+Khoury%2C+and+A%2E+Cecile+J%2E+W%2E+Janssens%2E+2007%2E+Most+published+research+findings+are+false+but+a+little+replication+goes+a+long+way%2E+PLoS+Med%2C+4%282%29%3Ae28%2E+Aure%CC%81lie+Ne%CC%81ve%CC%81ol%2C+Cyril)
53. Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. [find](https://scholar.google.com/scholar?q=Jens+Nilsson%2C+Sebastian+Riedel%2C+and+Deniz+Yuret%2E+2007%2E)
54. [The CoNLL 2007 shared task on dependency parsing](https://md.research.elamur.ai/free/en/papers/nivre-2007-conll-2007-shared-task-dependency/). In Proceedings of CoNLL. Diarmuid Ó Séaghdha and Anna Korhonen. 2014. Probabilistic distributional semantics. Computational Linguistics, 40(3):587–631.
55. Prasad Patil, Roger D. Peng, and Jeffrey Leek. 2016. A statistical definition for reproducibility and replicability. bioRxiv. [find](https://scholar.google.com/scholar?q=Prasad+Patil%2C+Roger+D%2E+Peng%2C+and+Jeffrey+Leek%2E+2016%2E+A+statistical+definition+for+reproducibility+and+replicability%2E+bioRxiv%2E)
56. Roger D. Peng. 2011. Reproducible research in computational science. Science, 334(6060):1226–1227. [find](https://scholar.google.com/scholar?q=Roger+D%2E+Peng%2E+2011%2E+Reproducible+research+in+computational+science%2E+Science%2C+334%286060%29%3A1226%E2%80%931227%2E)
57. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. [GloVe: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Proceedings of EMNLP.
58. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). [find](https://scholar.google.com/scholar?q=Slav+Petrov+and+Ryan+McDonald%2E+2012%2E+Overview+of+the+2012+shared+task+on+parsing+the+web%2E+In+Notes+of+the+First+Workshop+on+Syntactic+Analysis+of+Non%2DCanonical+Language+%28SANCL%29%2E)
59. Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of EMNLP. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis us… [find](https://scholar.google.com/scholar?q=Yuval+Pinter%2C+Robert+Guthrie%2C+and+Jacob+Eisenstein%2E+2017%2E+Mimicking+word+embeddings+using+subword+RNNs%2E+In+Proceedings+of+EMNLP%2E+Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Hwee+Tou+Ng%2C+Anders+)
60. Kira Radinsky, Eugene Agichtein, Evgeniy Gabrilovich, and Shaul Markovitch. 2011. A word at a time: Computing word relatedness using temporal semantic analysis. In Proceedings of WWW. [find](https://scholar.google.com/scholar?q=Kira+Radinsky%2C+Eugene+Agichtein%2C+Evgeniy+Gabrilovich%2C+and+Shaul+Markovitch%2E+2011%2E+A+word+at+a+time%3A+Computing+word+relatedness+using+temporal+semantic+analysis%2E+In+Proceedings+of+WWW%2E)
61. Herbert Rubenstein and John B. Goodenough. 1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633. [find](https://scholar.google.com/scholar?q=Herbert+Rubenstein+and+John+B%2E+Goodenough%2E+1965%2E+Contextual+correlates+of+synonymy%2E+Communications+of+the+ACM%2C+8%2810%29%3A627%E2%80%93633%2E)
62. Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Roi+Reichart%2C+and+Ari+Rappoport%2E+2015%2E+Symmetric+pattern+based+word+embeddings+for+improved+word+similarity+prediction%2E+In+Proceedings+of+CoNLL%2E)
63. Carina Silberer and Mirella Lapata. 2014. [Learning grounded meaning representations with autoencoders](https://md.research.elamur.ai/free/en/papers/silberer-2014-learning-grounded-meaning-representations-autoencoders/). In Proceedings of ACL.
64. R. John Simes. 1986. An improved Bonferroni procedure for multiple tests of significance. Biometrika, pages 751–754. [find](https://scholar.google.com/scholar?q=R%2E+John+Simes%2E+1986%2E+An+improved+Bonferroni+procedure+for+multiple+tests+of+significance%2E+Biometrika%2C+pages+751%E2%80%93754%2E)
65. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. [Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/). In Proceedings of EMNLP. Anders Søgaard, Anders Johannsen, Barbara Plank, Dirk Hovy, and Héctor Martı́nez Alonso. 2014. What’s in a p-v…
66. Anders Søgaard. 2013. Estimating effect size across datasets. In Proceedings of HLT-NAACL. [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard%2E+2013%2E+Estimating+effect+size+across+datasets%2E+In+Proceedings+of+HLT%2DNAACL%2E)
67. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245–251. [find](https://scholar.google.com/scholar?q=James+H%2E+Steiger%2E+1980%2E+Tests+for+comparing+elements+of+a+correlation+matrix%2E+Psychological+Bulletin%2C+87%282%29%3A245%E2%80%93251%2E)
68. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. [find](https://scholar.google.com/scholar?q=Ralph+Weischedel%2C+Eduard+Hovy%2C+Mitchell+Marcus%2C+Martha+Palmer%2C+Robert+Belvin%2C+Sameer+Pradhan%2C+Lance+Ramshaw%2C+and+Nianwen+Xue%2E+2011%2E+OntoNotes%3A+A+large+training+corpus+for+enhanced+processing%2E+Handbook)
69. Springer. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80–83. [find](https://scholar.google.com/scholar?q=Springer%2E+Frank+Wilcoxon%2E+1945%2E+Individual+comparisons+by+ranking+methods%2E+Biometrics+bulletin%2C+1%286%29%3A80%E2%80%9383%2E)
70. Dongqiang Yang and David M.W. Powers. Verb similarity on the taxonomy of WordNet. In Proceedings of the 3rd International WordNet Conference. Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=Dongqiang+Yang+and+David+M%2EW%2E+Powers%2E+Verb+similarity+on+the+taxonomy+of+WordNet%2E+In+Proceedings+of+the+3rd+International+WordNet+Conference%2E+Alexander+Yeh%2E+2000%2E+More+accurate+tests+for+the+statistic)
71. Yftah Ziser and Roi Reichart. 2017. Neural structural correspondence learning for domain adaptation. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=Yftah+Ziser+and+Roi+Reichart%2E+2017%2E+Neural+structural+correspondence+learning+for+domain+adaptation%2E+In+Proceedings+of+CoNLL%2E)

*Updated 3 November 2026*