---
title: "Mitigating Dataset Artifacts in Natural Language Inference"
description: "They then propose acda (automatic contextual data augmentation), which generates related variants of… 31 references, cited by 3 papers here."
authors: ["Michail Mersinias", "Panagiotis Valvis"]
year: 2022
venue: "Proceedings of the Thirteenth Language Resources and Evaluation Conference"
source: "https://aclanthology.org/2022.lrec-1.45/"
licence: "CC BY-NC 4.0 (ELRA, LREC proceedings)"
licence_url: "https://creativecommons.org/licenses/by-nc/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mersinias-2022-mitigating-dataset-artifacts-natural-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mersinias-2022-mitigating-dataset-artifacts-natural-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/) / Mitigating Dataset Artifacts in Natural Language Inference Through Automatic Contextual Data Augmentation and Learning Optimization

# Mitigating Dataset Artifacts in Natural Language Inference Through Automatic Contextual Data Augmentation and Learning Optimization

**Authors:** Michail Mersinias and Panagiotis Valvis · **Year:** 2022 · **Venue:** [Proceedings of the Thirteenth Language Resources and Evaluation Conference](https://md.research.elamur.ai/free/en/venues/lrec/) ([LREC 2022](https://md.research.elamur.ai/free/en/venues/lrec/2022/)) · **Publisher:** [European Language Resources Association (ELRA)](https://md.research.elamur.ai/free/en/publishers/european-language-resources-association-elra/) · **Licence:** [CC BY-NC 4.0 (ELRA, LREC proceedings)](https://creativecommons.org/licenses/by-nc/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.lrec-1.45/) · [PDF](https://aclanthology.org/2022.lrec-1.45.pdf) · [OpenAlex](https://openalex.org/W7160234384)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 3 in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper addresses a known weakness of pre-trained models on natural language inference (NLI). They score very well on SNLI and MNLI but often rely on dataset artifacts, which are shallow cues that break on slightly altered inputs. The authors first diagnose these artifacts. They write 12 adversarial premise–hypothesis pairs, and the strongest fine-tuned baseline gets every one of them wrong. They then propose acda (automatic contextual data augmentation), which generates related variants of training examples with no human annotation. The authors present it as much cheaper than TextAttack or Checklist. Two training changes support it:
- **Contrastive learning over instance bundles**, borrowed from reading comprehension work (Dua et al., 2021). acda produces groups of closely related examples, and the model is trained over these groups.
- **A hybrid loss** that adds a likelihood term to cross-entropy, meant to sharpen the decision boundary.

Models trained with acda and these techniques beat their plain fine-tuned Huggingface counterparts on SNLI, MNLI and the adversarial set. The broader point is that robustness gains can come from cheap, automatic data generation rather than expensive manual adversarial annotation.

## Key points
- **Benchmarks and baselines:** SNLI and MNLI, with three labels (ENT, NEU, CON). The baselines are fine-tuned pre-trained models from the Huggingface Transformers repository.
- **Adversarial probe result:** on 12 hand-annotated adversarial examples, the best baseline is 0/12 correct. Its errors cluster around the neutral and entailment classes.
- **Diagnosed artifacts:**
  - The model seems to learn a premise–hypothesis embedding distance, so it treats distant words, such as hyponyms, as contradictions.
  - It links particular hypothesis words to labels, citing Wallace et al. (2019): "not"/"least" for ENT, "joyously" for NEU, "nobody"/"never" for CON.
- **Typical failures:**
  - Synonym swaps break predictions, e.g. "to go" for "takeaway"/"food".
  - A hyponym such as "lasagna" for "food" should give neutral, but the model predicts contradiction.
- **Method and results:** acda is fully automatic and cheaper than TextAttack and Checklist. The number of bundle members grows exponentially with hypothesis length. It is combined with contrastive learning and a cross-entropy plus NLL hybrid loss. The exact scores fall in the part of the text missing from our extract.

## Interesting ideas
- A dozen targeted, hand-written minimal edits (synonym, hyponym, specificity shifts) can expose a model that looks excellent on its benchmark, which makes a tiny probe set a cheap sanity check.
- Generating bundles of near-identical variants whose labels differ, and training over each bundle as a group, makes the model attend to the small difference that matters.
- The paper's analysis shows that replacing a word with a more specific one ("food" → "lasagna") turns entailment into neutral, while a true synonym should keep the label. Direction in the lexical hierarchy therefore decides the inference.
- Words like "nobody" or "never" act as label triggers regardless of context, so auditing which surface tokens predict an output is a quick way to catch shortcut learning.
- Adding a likelihood term to cross-entropy is described as encouraging the model to learn the fine structure of the decision boundary rather than only the majority signal.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Keep a small hand-built set of lexical-shift pairs as a standing negative control, so that benchmark gains are not trusted unless the probe set also holds.
- **I2** (answer): Fable and school-test QA could be checked on bundles of minimally edited stories, confirming each answer changes only when the world of the text actually changes.
- **I3** (world): Entailment checks between frames can use the WordNet ontology explicitly: a more general filler preserves truth, a more specific one yields "unknown", and a disjoint sibling suggests conflict.
- **I4** (frames): Verify that negation and quantifier effects come from their scope in the UD tree and frames, by testing the same trigger words in contexts where they should not flip the speech act or polarity.
- **I5** (learn): When training gparse on silver data, a boundary-focused auxiliary loss on contested arcs (where the witness parsers disagree) could be tried and evaluated against plain cross-entropy.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The University of Texas at Austin](https://md.research.elamur.ai/free/en/universities/the-university-of-texas-at-austin/) — University, Austin, United States · 72 works · 55,722 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Michail Mersinias and Panagiotis Valvis. 2022. Mitigating Dataset Artifacts in Natural Language Inference Through Automatic Contextual Data Augmentation and Learning Optimization. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{mersinias2022mitigating,
  title = {{Mitigating Dataset Artifacts in Natural Language Inference Through Automatic Contextual Data Augmentation and Learning Optimization}},
  author = {Michail Mersinias and Panagiotis Valvis},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/45snhhbhahiy},
  url = {https://doi.org/10.63317/45snhhbhahiy}
}
```

- [← previous Increasing Robustness for Cross-domain Dialogue Act Classification on Social Media Data](https://md.research.elamur.ai/free/en/papers/vielsted-2022-increasing-robustness-cross-domain-dialogue/)
- [next → On the feature (in)deficiency of dedicated impersonal pronouns: the view from Jordanian Arabic](https://md.research.elamur.ai/free/en/papers/alhailawani-2022-feature-deficiency-dedicated-impersonal-pronouns/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/6/)

## References

31 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bartolo, M., Roberts, A., Welbl, J., Riedel, S., and Stenetorp, P. (2020). Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662–678. [↗](https://doi.org/10.1162/tacl_a_00338)
2. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal, September. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1508.05326)
3. Chen, Q., Zhu, X., Ling, Z., Wei, S., Jiang, H., and Inkpen, D. (2016). Enhanced lstm for natural language inference. arXiv preprint arXiv:1609.06038. [↗](https://arxiv.org/abs/1609.06038)
4. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805. [↗](https://arxiv.org/abs/1810.04805)
5. Dua, D., Dasigi, P., Singh, S., and Gardner, M. (2021). Learning with instance bundles for reading comprehension. arXiv preprint arXiv:2104.08735. [↗](https://arxiv.org/abs/2104.08735)
6. Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020). [Evaluating models’ local decision boundaries via contrast sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/). arXiv preprint arXiv:2004.02709. [↗](https://arxiv.org/abs/2004.02709)
7. Ghaeini, R., Hasan, S. A., Datla, V., Liu, J., Lee, K., Qadir, A., Ling, Y., Prakash, A., Fern, X. Z., and Farri, O. (2018). Dr-bilstm: Dependent reading bidirectional lstm for natural language inference. arXiv preprint arXiv:1802.05577. [↗](https://arxiv.org/abs/1802.05577)
8. Glockner, M., Shwartz, V., and Goldberg, Y. (2018). [Breaking nli systems with sentences that require simple lexical inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/). arXiv preprint arXiv:1805.02266. [↗](https://arxiv.org/abs/1805.02266)
9. Jia, R. and Liang, P. (2017). [Adversarial examples for evaluating reading comprehension systems](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/). arXiv preprint arXiv:1707.07328. [↗](https://arxiv.org/abs/1707.07328)
10. Liu, N. F., Schwartz, R., and Smith, N. A. (2019). [Inoculation by fine-tuning: A method for analyzing challenge datasets](https://md.research.elamur.ai/free/en/papers/liu-2019-inoculation-fine-tuning-method-analyzing/). arXiv preprint arXiv:1904.02668. [↗](https://arxiv.org/abs/1904.02668)
11. MacCartney, B. and Manning, C. D. (2008). Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 521–528. [find](https://scholar.google.com/scholar?q=MacCartney%2C+B%2E+and+Manning%2C+C%2E+D%2E+%282008%29%2E+Modeling+semantic+containment+and+exclusion+in+natural+language+inference%2E+In+Proceedings+of+the+22nd+International+Conference+on+Computational+Linguistics+%28C)
12. McCoy, R. T., Pavlick, E., and Linzen, T. (2019). [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). arXiv preprint arXiv:1902.01007. [↗](https://arxiv.org/abs/1902.01007)
13. Meissner, J. M., Thumwanit, N., Sugawara, S., and Aizawa, A. (2021). [Embracing ambiguity: Shifting the training target of nli models](https://md.research.elamur.ai/free/en/papers/meissner-2021-embracing-ambiguity-shifting-training-target/). arXiv preprint arXiv:2106.03020. [↗](https://arxiv.org/abs/2106.03020)
14. Miller, G. A. (1995). [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM, 38(11):39– 41.
15. Morris, J. X., Lifland, E., Yoo, J. Y., Grigsby, J., Jin, D., and Qi, Y. (2020). [Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp](https://md.research.elamur.ai/free/en/papers/morris-2020-textattack-framework-adversarial-attacks-data/). arXiv preprint arXiv:2005.05909. [↗](https://arxiv.org/abs/2005.05909)
16. Munkhdalai, T. and Yu, H. (2017). Neural tree indexers for text understanding. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 1, page 11. NIH Public Access. [find](https://scholar.google.com/scholar?q=Munkhdalai%2C+T%2E+and+Yu%2C+H%2E+%282017%29%2E+Neural+tree+indexers+for+text+understanding%2E+In+Proceedings+of+the+conference%2E+Association+for+Computational+Linguistics%2E+Meeting%2C+volume+1%2C+page+11%2E+NIH+Public+Acces)
17. Nie, Y., Zhou, X., and Bansal, M. (2020). [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) arXiv preprint arXiv:2010.03532. [↗](https://arxiv.org/abs/2010.03532)
18. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (1802). [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). arxiv 2018. arXiv preprint arXiv:1802.05365, 12. [↗](https://arxiv.org/abs/1802.05365)
19. Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., and Van Durme, B. (2018). Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042. [↗](https://arxiv.org/abs/1805.01042)
20. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Narasimhan%2C+K%2E%2C+Salimans%2C+T%2E%2C+and+Sutskever%2C+I%2E+%282018%29%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
21. Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.04118. [↗](https://arxiv.org/abs/2005.04118)
22. Rocktäschel, T., Grefenstette, E., Hermann, K. M., Kočiskỳ, T., and Blunsom, P. (2015). [Reasoning about entailment with neural attention](https://md.research.elamur.ai/free/en/papers/rocktaschel-2015-reasoning-about-entailment-neural-attention/). arXiv preprint arXiv:1509.06664. [↗](https://arxiv.org/abs/1509.06664)
23. Sha, L., Chang, B., Sui, Z., and Li, S. (2016). Reading and thinking: Re-read lstm unit for textual entailment recognition. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2870–2879. [find](https://scholar.google.com/scholar?q=Sha%2C+L%2E%2C+Chang%2C+B%2E%2C+Sui%2C+Z%2E%2C+and+Li%2C+S%2E+%282016%29%2E+Reading+and+thinking%3A+Re%2Dread+lstm+unit+for+textual+entailment+recognition%2E+In+Proceedings+of+COLING+2016%2C+the+26th+International+Conference+on+Computat)
24. Smith, N. A. and Eisner, J. (2005). [Contrastive estimation: Training log-linear models on unlabeled data](https://md.research.elamur.ai/free/en/papers/smith-2005-contrastive-estimation-training-log-linear/). In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 354–362.
25. Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., and Choi, Y. (2020). Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795. [↗](https://arxiv.org/abs/2009.10795)
26. Wallace, E., Feng, S., Kandpal, N., Gardner, M., and Singh, S. (2019). Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125. [↗](https://arxiv.org/abs/1908.07125)
27. Williams, A., Nangia, N., and Bowman, S. R. (2017). A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. [↗](https://arxiv.org/abs/1704.05426)
28. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019). Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv preprint arXiv:1910.03771. [↗](https://arxiv.org/abs/1910.03771)
29. Yaghoobzadeh, Y., Mehri, S., Tachet, R., Hazen, T. J., and Sordoni, A. (2019). Increasing robustness to spurious correlations using forgettable examples. arXiv preprint arXiv:1911.03861. [↗](https://arxiv.org/abs/1911.03861)
30. Zhang, Z., Wu, Y., Zhao, H., Li, Z., Zhang, S., Zhou, X., and Zhou, X. (2020). Semantics-aware bert for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9628–9635. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Wu%2C+Y%2E%2C+Zhao%2C+H%2E%2C+Li%2C+Z%2E%2C+Zhang%2C+S%2E%2C+Zhou%2C+X%2E%2C+and+Zhou%2C+X%2E+%282020%29%2E+Semantics%2Daware+bert+for+language+understanding%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+volume)
31. Zhou, X. and Bansal, M. (2020). [Towards robustifying nli models against lexical dataset biases](https://md.research.elamur.ai/free/en/papers/zhou-2020-robustifying-nli-models-against-lexical/). arXiv preprint arXiv:2005.04732. [↗](https://arxiv.org/abs/2005.04732)

*Updated 24 October 2026*