--- title: "Embracing Ambiguity: Shifting the Training Target of NLI Models" description: "They build AmbiNLI, a trial resource of 91,717 premise–hypothesis pairs. 1 reference, cited by 21 papers here." authors: ["Johannes Mario Meissner", "Napat Thumwanit", "Saku Sugawara", "Akiko Aizawa"] year: 2021 venue: "Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)" source: "https://aclanthology.org/2021.acl-short.109/" doi: "10.18653/v1/2021.acl-short.109" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/meissner-2021-embracing-ambiguity-shifting-training-target/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/meissner-2021-embracing-ambiguity-shifting-training-target/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) / Embracing Ambiguity: Shifting the Training Target of NLI Models # Embracing Ambiguity: Shifting the Training Target of NLI Models **Authors:** Johannes Mario Meissner, Napat Thumwanit, [Saku Sugawara](https://md.research.elamur.ai/free/en/authors/saku-sugawara/) and [Akiko Aizawa](https://md.research.elamur.ai/free/en/authors/akiko-aizawa/) · **Year:** 2021 · **Venue:** [Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2021](https://md.research.elamur.ai/free/en/venues/acl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.acl-short.109/) · [PDF](https://aclanthology.org/2021.acl-short.109.pdf) · [DOI 10.18653/v1/2021.acl-short.109](https://doi.org/10.18653/v1/2021.acl-short.109) · [OpenAlex](https://openalex.org/W3174102190) **Citations:** 15 citations (OpenAlex, 2026-10-14) · [21](https://md.research.elamur.ai/free/en/papers/meissner-2021-embracing-ambiguity-shifting-training-target/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary NLI benchmarks contain many pairs where annotators disagree, yet models are usually trained on a single gold label. The authors ask whether training directly on the annotator label distribution helps. They build AmbiNLI, a trial resource of 91,717 premise–hypothesis pairs. Each pair carries a probability distribution derived from the 5-annotator labels in SNLI and MNLI, or from UNLI's scalar scores via a piecewise-linear mapping. Items overlapping with ChaosNLI are removed. A BERT-base model, first trained on SNLI+MNLI gold labels, is then finetuned with cross-entropy against these soft targets. It is evaluated on ChaosNLI, which has 100 annotations per item, using Jensen-Shannon Divergence (JSD) and accuracy. Soft-label finetuning on the SNLI/MNLI part (AmbiSM) cuts JSD by about a fifth. It also beats an otherwise identical model trained on gold labels in accuracy, and gives better frozen-encoder features for transfer tasks. UNLI-derived data hurts unless near-certain items are filtered out. The authors present this as evidence that collecting and releasing multi-annotator labels is worthwhile, not as a final dataset. ## Key points - AmbiNLI contains 18,152 SNLI, 18,048 MNLI and 55,517 UNLI examples. The authors explicitly discourage reusing it and call for new multi-annotation collection instead. - UNLI conversion maps a value p to a 3-way distribution: - p \< 0.5 becomes (0, 2p, 1−2p). - Otherwise it becomes (2p−1, 2−2p, 0). - Setup: BERT-base with a softmax head, batch size 128 and learning rate 1e-5, after 3 epochs of gold-label pretraining on SNLI+MNLI. - With AmbiSM, ChaosSNLI JSD falls from 0.2379 to 0.1893 and ChaosMNLI JSD from 0.3349 to 0.2619 (about 20.5% and 21.7% decreases). Accuracy rises slightly over the same-data gold-label model. - UNLI items with p \< 0.05 or p > 0.97 produce extreme distributions. Removing them lowers AmbiU JSD by roughly 20%. - Three-fold cross-validation on ChaosMNLI shows average accuracy of 0.5210 with soft labels versus 0.5009 with gold labels. - JSD stays stable across three entropy bins, while accuracy drops as entropy grows. ## Interesting ideas - Instead of collapsing annotator votes into one label, train on the vote proportions as a soft target; the model then learns how contested an item is, not just its majority answer - Compare two models trained on exactly the same examples, differing only in hard versus soft targets, so any gain is attributable to the target itself - A graded scalar judgment can be turned into a categorical distribution with a simple piecewise-linear map, but saturated values near the extremes skew training and are worth filtering - Bin the evaluation set by annotator-disagreement entropy and report metrics per bin, to check that the model's own uncertainty tracks how ambiguous items really are - Judge representation quality by freezing the encoder and training only a small head on downstream tasks ## For Mova How these ideas could serve Mova, our small open language model — its files, stages and tests: - **I1** (learn): When several witness parsers disagree on silver data, the fraction agreeing on each head or label could serve as a soft training target, rather than discarding or majority-voting the disputed arcs. - **I2** (learn): Any change to how silver labels are aggregated should be tested with this same-data control, alongside existing negative controls, to isolate the effect of the labeling scheme. - **I3** (frames): Graded confidence scores for implicature or irony could be mapped into label distributions the same way, with near-certain extremes capped or dropped to avoid overconfident training. - **I4** (answer): Test questions for the readers could be grouped by how much humans or teachers disagree, verifying that answer confidence falls on genuinely ambiguous fable or word-problem items. - **I5** (tree): An encoder in gparse trained with soft attachment targets could be probed with frozen layers on frame or tagging tasks, to see whether it carries more transferable structure than one trained on hard labels. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [The University of Tokyo](https://md.research.elamur.ai/free/en/universities/the-university-of-tokyo/) — University, Tokyo, Japan · 45 works · 8,692 citations - [National Institute of Informatics](https://md.research.elamur.ai/free/en/labs/national-institute-of-informatics/) — Research institute, Tokyo, Japan · 21 works · 3,045 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works ## Cite Cite as: Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara and Akiko Aizawa. 2021. Embracing Ambiguity: Shifting the Training Target of NLI Models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{meissner2021embracing, title = {{Embracing Ambiguity: Shifting the Training Target of NLI Models}}, author = {Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara and Akiko Aizawa}, booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)}, year = {2021}, doi = {10.18653/v1/2021.acl-short.109}, url = {https://doi.org/10.18653/v1/2021.acl-short.109} } ``` - [← previous Top-down Discourse Parsing via Sequence Labelling](https://md.research.elamur.ai/free/en/papers/koto-2021-top-down-discourse-parsing-sequence/) - [next → Exploiting Document Structures and Cluster Consistencies for Event Coreference Resolution](https://md.research.elamur.ai/free/en/papers/tran-2021-exploiting-document-structures-cluster-consistencies/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/5/) ## References 1 references, 1 of them with a card of ours. ↗ leads to the work. 1. UNLI Pearson↑ MSE↓ IMDB CE Loss↓ Acc.↑ 1 Layer AmbiSM G AmbiSM .6331(0.9) .6354(1.0) .0758(0.5) .0754(0.4) .4727(1.7) .4701(1.5) .7758(6.4) .7783(6.1) 2 Layers AmbiSM G AmbiSM .6266(5.9) .6290(4.1) .0765(1.0) .0762(0.7) .4431(0.8) .4392(1.2) .7906(4.3) .7939(3.3) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. [Abductive commonsense reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/). In International Conference on Learning Representations. *Updated 24 October 2026*