---
title: "Not another Negation Benchmark — paper by Truong et al. (2022)"
description: "The authors build NaN-NLI, a hand-made NLI test suite aimed at sub-clausal negation, where the… 26 references, cited by 20 papers here."
authors: ["Hung Thinh Truong", "Yulia Otmakhova", "Timothy Baldwin", "Trevor Cohn", "Jey Han Lau", "Karin Verspoor"]
year: 2022
venue: "Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.aacl-main.65/"
doi: "10.18653/v1/2022.aacl-main.65"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/truong-2022-not-another-negation-benchmark-nan/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/truong-2022-not-another-negation-benchmark-nan/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

# Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation

**Authors:** Hung Thinh Truong, Yulia Otmakhova, [Timothy Baldwin](https://md.research.elamur.ai/free/en/authors/timothy-baldwin/), [Trevor Cohn](https://md.research.elamur.ai/free/en/authors/trevor-cohn/), [Jey Han Lau](https://md.research.elamur.ai/free/en/authors/jey-han-lau/) and [Karin Verspoor](https://md.research.elamur.ai/free/en/authors/karin-verspoor/) · **Year:** 2022 · **Venue:** [Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/aacl/) ([AACL 2022](https://md.research.elamur.ai/free/en/venues/aacl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.aacl-main.65/) · [PDF](https://aclanthology.org/2022.aacl-main.65.pdf) · [DOI 10.18653/v1/2022.aacl-main.65](https://doi.org/10.18653/v1/2022.aacl-main.65) · [OpenAlex](https://openalex.org/W4404783524)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [20](https://md.research.elamur.ai/free/en/papers/truong-2022-not-another-negation-benchmark-nan/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pre-trained language models often mishandle negation, and existing negation benchmarks mostly add a plain *not* to a main verb. That yields verbal, analytic, clausal negation, which is the easy case. The authors build NaN-NLI, a hand-made NLI test suite aimed at sub-clausal negation, where the negator covers only part of a clause (e.g. *not one but three*). Premises are example sentences taken from the grammar of Pullum and Huddleston (2002). Hypotheses are minimal edits of each premise built with ten operation types. Every pair carries labels for negation type, construction, and the edits used, so errors can be traced to specific phenomena. In experiments with pre-trained models, NaN-NLI proves harder than earlier negation-focused NLI sets. Scores drop further on NaN-Quant, the subset where negation interacts with quantifier bounds. The resource matters because it targets scope ambiguity, which surface cues cannot resolve, and because its annotations support diagnostic rather than aggregate evaluation.

## Key points
- The typology has four axes: verbal/non-verbal, analytic/synthetic, clausal/sub-clausal, and ordinary/meta-linguistic. Meta-linguistic cases are left out because they are rare.
- Covered constructions include:
  - *not* + quantifiers, *not a little*, *not even*/*not only*;
  - degree expressions and *not un-* adjectives;
  - *not* in coordination, with PPs, and in verbless clauses;
  - *not that*;
  - absolute and approximate negators (*seldom*, *rarely*).
- There are 10 hypothesis-building operations, and over half of the hypotheses combine two or more of them. Comparative quantifier changes are the hardest (error rate 0.650), while syntactic changes produce no errors.
- Over half of the samples (133) involve quantification. Overall F1 falls from 0.629 on NaN-NLI to 0.486 on NaN-Quant, and Contradiction drops from 0.692 to 0.477.
- Swapping gendered names and pronouns did not change results. The Neutral class is underrepresented. Data and code are released at github.com/joey234/nan-nli.

## Interesting ideas
- Pull test premises from a reference grammar's examples instead of a corpus, so every construction is covered deliberately, including rare ones.
- Annotate each test item with the edit operation that turned premise into hypothesis, then report error rates per operation to see which kind of change breaks the system.
- Keep hypotheses as minimal edits of the premise, so the system cannot succeed by noticing large lexical differences.
- Treat negation plus quantifiers as a question of upper and lower bounds. For example, *not one, not two, but three* makes "more than two" true and "more than three" false.
- Separate clausal from sub-clausal negation: *not for the first time* or *in no time* leave the clause positive.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (answer): Build a negation and scope quiz from grammar-book sentences to check whether the world model gives the correct polarity for each construction.
- **I2** (learn): Tag our negative-control items by edit type so evaluation can report which transformations (quantifier change, negator movement) Mova's pipeline fails on.
- **I3** (learn): When generating contrast pairs for self-play or silver checks, restrict changes to function words so that only structural reasoning can separate the labels.
- **I4** (world): Represent quantities in the text world as intervals with explicit bounds, so entailment about counts is computed rather than guessed from shared numbers.
- **I5** (frames): Attach negation to the constituent it scopes over in the UD tree, not automatically to the main predicate's frame, so the event is still asserted as happening.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Melbourne](https://md.research.elamur.ai/free/en/universities/the-university-of-melbourne/) — University, Melbourne, Australia · 60 works · 21,199 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 47 works · 4,456 citations
- [RMIT University](https://md.research.elamur.ai/free/en/universities/rmit-university/) — University, Melbourne, Australia · 6 works · 477 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 685 works

## Cite

Cite as: Hung Thinh Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau and Karin Verspoor. 2022. Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation. Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{truong2022not,
  title = {{Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation}},
  author = {Hung Thinh Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Jey Han Lau and Karin Verspoor},
  booktitle = {Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-main.65},
  url = {https://doi.org/10.18653/v1/2022.aacl-main.65}
}
```

- [← previous Low-Resource Authorship Style Transfer: Can Non-Famous Authors Be Imitated?](https://md.research.elamur.ai/free/en/papers/patel-2022-low-resource-authorship-style-transfer/)
- [next → A New Amharic Speech Emotion Dataset and Classification Benchmark](https://md.research.elamur.ai/free/en/papers/retta-2022-new-amharic-speech-emotion-dataset/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

26 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Catherine Anderson. 2018. Essentials of Linguistics. McMaster University. [find](https://scholar.google.com/scholar?q=Catherine+Anderson%2E+2018%2E+Essentials+of+Linguistics%2E+McMaster+University%2E)
2. Ron Artstein and Massimo Poesio. 2008. [Inter-coder agreement for computational linguistics](https://md.research.elamur.ai/free/en/papers/artstein-2008-survey-article-inter-coder-agreement/). Computational linguistics, 34(4):555–596. [↗](https://doi.org/10.1162/coli.07-034-r2)
3. Jeremy Barnes, Erik Velldal, and Lilja Øvrelid. 2021. Improving sentiment analysis with multi-task learning of negation. Natural Language Engineering, 27(2):249–269. [↗](https://doi.org/10.1017/s1351324920000510)
4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. [↗](https://doi.org/10.1177/001316446002000104)
5. K. Bretonnel Cohen, Christophe Roeder, William A. Baumgartner Jr., Lawrence E. Hunter, and Karin Verspoor. 2010. Test suite design for biomedical ontology concept recognition systems. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=K%2E+Bretonnel+Cohen%2C+Christophe+Roeder%2C+William+A%2E+Baumgartner+Jr%2E%2C+Lawrence+E%2E+Hunter%2C+and+Karin+Verspoor%2E+2010%2E+Test+suite+design+for+biomedical+ontology+concept+recognition+systems%2E+In+Proceedings+o)
6. Allyson Ettinger. 2020. [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48. [↗](https://doi.org/10.1162/tacl_a_00298)
7. Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020. [An analysis of natural language inference benchmarks through the lens of negation](https://md.research.elamur.ai/free/en/papers/hossain-2020-analysis-natural-language-inference-benchmarks/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9106–9118, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-main.732)
8. Salud María Jiménez-Zafra, Noa P Cruz-Díaz, Maite Taboada, and María Teresa Martín-Valdivia. 2021. Negation detection for sentiment analysis: A case study in spanish. Natural Language Engineering, 27(2):225–248. [↗](https://doi.org/10.1017/s1351324920000376)
9. Nora Kassner and Hinrich Schütze. 2020. [Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.
10. Aditya Khandelwal and Suraj Sawant. 2020. NegBERT: A transfer learning approach for negation detection and scope resolution. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5739–5748, Marseille, France. European Language Resources Association. [↗](https://doi.org/10.48550/arxiv.1911.04211)
11. Paul Kroeger. 2018. Analyzing Meaning: An Introduction to Semantics and Pragmatics. Language Science Press. [↗](https://doi.org/10.5281/zenodo.1164112)
12. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1654536.1654538)
13. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
14. Mareike Hartmann, Miryam de Lhoneux, Daniel Hershcovich, Yova Kementchedjhieva, Lukas Nielsen, Chen Qiu, and Anders Søgaard. 2021. A multilingual benchmark for probing negation-awareness with minimal pairs. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 244–257, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mareike+Hartmann%2C+Miryam+de+Lhoneux%2C+Daniel+Hershcovich%2C+Yova+Kementchedjhieva%2C+Lukas+Nielsen%2C+Chen+Qiu%2C+and+Anders+S%C3%B8gaard%2E+2021%2E+A+multilingual+benchmark+for+probing+negation%2Dawareness+with+minimal+)
15. Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022. An analysis of negation in natural language understanding corpora. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 716–723, Dublin, Ireland. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Md+Mosharaf+Hossain%2C+Dhivya+Chinnappa%2C+and+Eduardo+Blanco%2E+2022%2E+An+analysis+of+negation+in+natural+language+understanding+corpora%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Com)
16. Sabine Lehmann, Stephan Oepen, Sylvie Regnier-Prost, Klaus Netter, Veronika Lux, Judith Klein, Kirsten Falkedal, Frederik Fouvry, Dominique Estival, Eva Dauphin, Herve Compagnion, Judith Baur, Lorna Balkan, and Doug Arnold. 1996. TSNLP - test suites for natural language processing. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sabine+Lehmann%2C+Stephan+Oepen%2C+Sylvie+Regnier%2DProst%2C+Klaus+Netter%2C+Veronika+Lux%2C+Judith+Klein%2C+Kirsten+Falkedal%2C+Frederik+Fouvry%2C+Dominique+Estival%2C+Eva+Dauphin%2C+Herve+Compagnion%2C+Judith+Baur%2C+Lorna+B)
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
18. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/p19-1334)
19. Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. 2018. [NegBio: a high-performance tool for negation and uncertainty detection in radiology reports](https://md.research.elamur.ai/free/en/papers/peng-2017-negbio-high-performance-tool-negation/). AMIA Summits on Translational Science Proceedings, 2018:188. [↗](https://doi.org/10.48550/arxiv.1712.05898)
20. Geoffrey K. Pullum and Rodney Huddleston. 2002. Negation, chapter 9. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Geoffrey+K%2E+Pullum+and+Rodney+Huddleston%2E+2002%2E+Negation%2C+chapter+9%2E+Cambridge+University+Press%2E)
21. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.442)
22. Gongbo Tang, Philipp Rönchen, Rico Sennrich, and Joakim Nivre. 2021. Revisiting negation in neural machine translation. Transactions of the Association for Computational Linguistics, 9:740–755. A All models are implemented using the transformers package from Hugging-Face (Wolf et al., 2020). We use the base variant of RoBERTa. For fine-tuning on NegNLI, we split the dataset into training/validation sets with a 85:15 ratio. [↗](https://doi.org/10.1162/tacl_a_00395)
23. Thinh Truong, Timothy Baldwin, Trevor Cohn, and Karin Verspoor. 2022. Improving negation detection with negation-focused pre-training. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4188–4193, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2205.04012)
24. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [SuperGLUE: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems.
25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-5446)
26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. Implementation Details Hyper-parameter Value batch size lr epochs optimizer 16 3e-5 3 Adam *Table 11: Hyper-parameters for fine-tuning on MNLI* Hyper-parameter Value batch size lr epochs optimizer 16 2e-5 5 Adam *Table 12: Hyper-parameters for fine-tuning on NegNLI* [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)

*Updated 24 October 2026*