---
title: "Exploring Lexical Irregularities in Hypothesis-Only Models of"
description: "The papers that build automatically recast datasets have not examined these giveaways closely. 23 references."
authors: ["Qingyuan Hu", "Yi Zhang", "Kanishka Misra", "Julia Rayz"]
year: 2021
venue: "arXiv:2101.07397"
source: "https://arxiv.org/abs/2101.07397"
doi: "10.48550/arxiv.2101.07397"
arxiv: "2101.07397"
licence: "CC BY-NC-ND 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by-nc-nd/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/hu-2021-exploring-lexical-irregularities-hypothesis-only/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/hu-2021-exploring-lexical-irregularities-hypothesis-only/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/8/) / Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference

# Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference

**Authors:** Qingyuan Hu, [Yi Zhang](https://md.research.elamur.ai/free/en/authors/yi-zhang/), [Kanishka Misra](https://md.research.elamur.ai/free/en/authors/kanishka-misra/) and Julia Rayz · **Year:** 2021 · **Venue:** arXiv:2101.07397 · **Licence:** [CC BY-NC-ND 4.0 (arXiv)](http://creativecommons.org/licenses/by-nc-nd/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2101.07397) · [PDF](https://arxiv.org/pdf/2101.07397) · [DOI 10.48550/arxiv.2101.07397](https://doi.org/10.48550/arxiv.2101.07397) · [arXiv 2101.07397](https://arxiv.org/abs/2101.07397) · [OpenAlex](https://openalex.org/W3171676470)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
NLI benchmarks are meant to test whether a system can relate a premise to a hypothesis. Earlier work by Poliak et al. showed that models which see only the hypothesis can still perform surprisingly well. That suggests datasets carry statistical giveaways unrelated to inference. The papers that build automatically recast datasets have not examined these giveaways closely.

This paper looks at one such recast resource, the Semantic Proto-Roles (SPR) dataset from White et al. It trains hypothesis-only models on SPR and inspects which individual words push predictions towards a label. On this data, hypothesis-only baselines had already been reported to clearly beat the majority baseline and to approach the then state of the art, a modified InferSent. The authors find signs of lexical biases in the hypotheses that could inflate reported accuracy, so models may be "right for the wrong reasons".

The aim is diagnosis, not a stronger model. The work warns that template-generated inference data needs word-level auditing before its scores are trusted as evidence of understanding.

## Key points
- Poliak et al. compared hypothesis-only baselines across 10 NLI datasets. They grouped the datasets into three kinds:
  - human-elicited, such as SNLI;
  - human-judged, such as SciTail;
  - automatically recast, such as SPR, DPR and FN+.
- Earlier findings cited by the paper:
  - in SNLI, negation words such as "nobody" strongly signal contradiction;
  - in recast FN+, more grammatical hypotheses tend to be labelled entailed.
- SPR breaks semantic roles into 16 finer properties, such as instigation. Annotators rated how likely each property was on a 1–5 scale. Scores of 1–3 became "not entailed" and 4–5 became "entailed", with no neutral class.
- The reference model is InferSent, a BiLSTM sentence encoder with pooling. Hypothesis-only models on SPR came close to its performance.
- Future work proposed by the authors: test whether existing models actually exploit the found biases by comparing their outputs on biased versus debiased instances.

## Interesting ideas
- **Partial-input baselines:** feed a model only part of the input (here, the hypothesis without the premise). If accuracy stays high, the benchmark leaks the answer through surface cues.
- **Word-level label scan:** check, word by word, how strongly each word's presence predicts a label. This cheaply surfaces giveaway tokens in automatically produced data.
- **Recasting templates leave fingerprints:** converting a semantic resource into inference pairs by template produces repeated phrasing, which models can latch onto instead of meaning.
- **Roles as bundles of graded properties:** SPR describes an argument through likelihoods of properties like instigation rather than a single role label. That gives finer, more explainable role judgements.
- **Surface quality as a confound:** in FN+, how grammatical a hypothesis was correlated with its label. Text quality alone can therefore act as a hidden signal.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Add "question-only" and "text-shuffled" runs as standing negative controls in evaluation. A reader that still scores well without the story reveals a leaky test set.
- **I2** (learn): Run this scan over silver data and LLM-teacher-generated items before training. Flag words whose label distribution is extreme so they can be balanced or removed.
- **I3** (answer): When generating school-course tests or fable QA from templates, vary the wording and check for template-specific cues. This keeps reader scores reflecting real comprehension.
- **I4** (frames): Alongside VerbNet roles, frames could store proto-role properties (caused it, changed state, etc.). This would make who-did-what-to-whom decisions more transparent.
- **I5** (back): When generated sentences are used as test items or training pairs, keep fluency comparable across true and false variants. That prevents quality from betraying the answer.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Purdue University West Lafayette](https://md.research.elamur.ai/free/en/universities/purdue-university-west-lafayette/) — University, West Lafayette, United States · 9 works · 1,254 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Qingyuan Hu, Yi Zhang, Kanishka Misra and Julia Rayz. 2021. Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference. arXiv:2101.07397.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{hu2021exploring,
  title = {{Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference}},
  author = {Qingyuan Hu, Yi Zhang, Kanishka Misra and Julia Rayz},
  year = {2021},
  doi = {10.48550/arxiv.2101.07397},
  eprint = {2101.07397},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2101.07397}
}
```

- [← previous Exploratory Arabic Offensive Language Dataset Analysis](https://md.research.elamur.ai/free/en/papers/husain-2021-exploratory-arabic-offensive-language-dataset/)
- [next → Fisseha G. Demoze and William H. Armstrong. Ethiopian Amharic Proverbs](https://md.research.elamur.ai/free/en/papers/roamndo-2021-fisseha-g-demoze-william-h/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/8/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning, “A large annotated corpus for learning natural language inference,” arXiv:1508.05326 [cs], Aug. 2015, arXiv: 1508.05326. [Online]. Available: http://arxiv.org/abs/1508.05326 [↗](https://arxiv.org/abs/1508.05326)
2. A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme, “Hypothesis Only Baselines in Natural Language Inference,” in Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics. New Orleans, Louisiana: Association for Computational Linguistics, 2018, pp. 180–191. [Online]. Available: http://aclweb.org/anthology/S18-2023 [↗](https://aclanthology.org/S18-2023/)
3. I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, ser. Lecture Notes in Computer Science, J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. d’Alché Buc, Eds. Berlin, Heidelberg: Springer, 2006, pp. 177–190. [find](https://scholar.google.com/scholar?q=I%2E+Dagan%2C+O%2E+Glickman%2C+and+B%2E+Magnini%2C+%E2%80%9CThe+pascal+recognising+textual+entailment+challenge%2C%E2%80%9D+in+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classification%2C+and+Recog)
4. T. Khot, A. Sabharwal, and P. Clark, “Scitail: A textual entailment dataset from science question answering,” 2018. [Online]. Available: [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17368](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17368) [↗](https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17368)
5. A. S. White, P. Rastogi, K. Duh, and B. Van Durme, “[Inference is everything: Recasting semantic resources into a unified evaluation framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/),” in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 996–1005.
6. T. McCoy, E. Pavlick, and T. Linzen, “[Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/),” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 3428–3448.
7. J. J. Katz, Semantic Theory. New York: Harper & Row, 1972. [find](https://scholar.google.com/scholar?q=J%2E+J%2E+Katz%2C+Semantic+Theory%2E+New+York%3A+Harper+%26+Row%2C+1972%2E)
8. S. Bowman and X. Zhu, “Deep learning for natural language inference,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 6–8. [Online]. Available: https://www.aclweb.org/anthology/N19-5002 [↗](https://aclanthology.org/N19-5002/)
9. S. Wang and J. Jiang, “[Learning natural language inference with lstm](https://md.research.elamur.ai/free/en/papers/wang-2016-learning-natural-language-inference-lstm/),” arXiv preprint arXiv:1512.08849, 2015. [↗](https://arxiv.org/abs/1512.08849)
10. O. Glickman and I. Dagan, “Web based probabilistic textual entailment,” in In Proceedings of the 1st Pascal Challenge Workshop, 2005. [find](https://scholar.google.com/scholar?q=O%2E+Glickman+and+I%2E+Dagan%2C+%E2%80%9CWeb+based+probabilistic+textual+entailment%2C%E2%80%9D+in+In+Proceedings+of+the+1st+Pascal+Challenge+Workshop%2C+2005%2E)
11. Y. Mehdad, A. Moschitti, and F. M. Zanzotto, “Semker: Syntactic/semantic kernels for recognizing textual entailment,” Theory and Applications of Categories, 2009. [find](https://scholar.google.com/scholar?q=Y%2E+Mehdad%2C+A%2E+Moschitti%2C+and+F%2E+M%2E+Zanzotto%2C+%E2%80%9CSemker%3A+Syntactic%2Fsemantic+kernels+for+recognizing+textual+entailment%2C%E2%80%9D+Theory+and+Applications+of+Categories%2C+2009%2E)
12. B. MacCartney, M. Galley, and C. D. Manning, “A phrase-based alignment model for natural language inference,” in Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 2008, pp. 802–811. [find](https://scholar.google.com/scholar?q=B%2E+MacCartney%2C+M%2E+Galley%2C+and+C%2E+D%2E+Manning%2C+%E2%80%9CA+phrase%2Dbased+alignment+model+for+natural+language+inference%2C%E2%80%9D+in+Proceedings+of+the+2008+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
13. P. Clark and P. Harrison, “An inference-based approach to recognizing entailment,” Theory and Applications of Categories, 2009. [find](https://scholar.google.com/scholar?q=P%2E+Clark+and+P%2E+Harrison%2C+%E2%80%9CAn+inference%2Dbased+approach+to+recognizing+entailment%2C%E2%80%9D+Theory+and+Applications+of+Categories%2C+2009%2E)
14. H. Wang, D. Sun, and E. P. Xing, “What if we simply swap the two text fragments? a straightforward yet effective way to test the robustness of methods to confounding signals in nature language inference tasks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 7136–7143. [find](https://scholar.google.com/scholar?q=H%2E+Wang%2C+D%2E+Sun%2C+and+E%2E+P%2E+Xing%2C+%E2%80%9CWhat+if+we+simply+swap+the+two+text+fragments%3F+a+straightforward+yet+effective+way+to+test+the+robustness+of+methods+to+confounding+signals+in+nature+language+inferen)
15. H. Wang, A. Meghawat, L. Morency, and E. P. Xing, “Select-additive learning: Improving cross-individual generalization in multimodal sentiment analysis,” CoRR, vol. abs/1609.05244, 2016. [Online]. Available: http://arxiv.org/abs/1609.05244 [↗](https://arxiv.org/abs/1609.05244)
16. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “[Making the V in VQA matter: Elevating the role of image understanding in visual question answering](https://md.research.elamur.ai/free/en/papers/goyal-2016-making-v-vqa-matter-elevating/),” CoRR, vol. abs/1612.00837, 2016. [Online]. Available: http://arxiv.org/abs/1612.00837 [↗](https://arxiv.org/abs/1612.00837)
17. T. Yue and H. Wang, “Deep learning for genomics: A concise overview,” 2018. [↗](https://doi.org/10.48550/arxiv.1802.00810)
18. R. Cooper, D. Crouch, J. V. Eijck, C. Fox, J. V. Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, and K. Konrad, “Using the framework,” 1996. [find](https://scholar.google.com/scholar?q=R%2E+Cooper%2C+D%2E+Crouch%2C+J%2E+V%2E+Eijck%2C+C%2E+Fox%2C+J%2E+V%2E+Genabith%2C+J%2E+Jaspars%2C+H%2E+Kamp%2C+D%2E+Milward%2C+M%2E+Pinkal%2C+M%2E+Poesio%2C+S%2E+Pulman%2C+T%2E+Briscoe%2C+H%2E+Maier%2C+and+K%2E+Konrad%2C+%E2%80%9CUsing+the+framework%2C%E2%80%9D+1996%2E)
19. D. Reisinger, R. Rudinger, F. Ferraro, C. Harman, K. Rawlins, and B. Van Durme, “[Semantic proto-roles](https://md.research.elamur.ai/free/en/papers/reisinger-2015-semantic-proto-roles/),” Transactions of the Association for Computational Linguistics, vol. 3, pp. 475–488, 2015. [↗](https://doi.org/10.1162/tacl_a_00152)
20. E. Pavlick, T. Wolfe, P. Rastogi, C. Callison-Burch, M. Dredze, and B. Van Durme, “[FrameNet+: Fast paraphrastic tripling of FrameNet](https://md.research.elamur.ai/free/en/papers/pavlick-2015-framenet-fast-paraphrastic-tripling-framenet/),” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Beijing, China: Association for Computational Linguistics, Jul. 2015, pp. 408– 413. [Online]. Available: https://www.aclweb.org/anthology/P15-2067 [↗](https://aclanthology.org/P15-2067/)
21. A. Rahman and V. Ng, “[Resolving complex cases of definite pronouns: The Winograd schema challenge](https://md.research.elamur.ai/free/en/papers/rahman-2012-resolving-complex-cases-definite-pronouns/),” in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Jeju Island, Korea: Association for Computational Linguistics, Jul. 2012, pp. 777–789. [Online]. Available: https://www.aclweb.org/anthology/D12-1071 [↗](https://aclanthology.org/D12-1071/)
22. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, “[Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/),” CoRR, vol. abs/1705.02364, 2017. [Online]. Available: http://arxiv.org/abs/1705.02364 [↗](https://arxiv.org/abs/1705.02364)
23. A. Williams, N. Nangia, and S. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2018, pp. 1112–1122. [Online]. Available: http://aclweb.org/anthology/N18-1101 [↗](https://aclanthology.org/N18-1101/)

*Updated 24 October 2026*