---
title: "The RepEval 2017 Shared Task — paper by Nangia et al. (2017)"
description: "The paper reports on a shared competition at RepEval 2017. 8 references, cited by 40 papers here."
authors: ["Nikita Nangia", "Adina Williams", "Angeliki Lazaridou", "Samuel R. Bowman"]
year: 2017
venue: "Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP"
source: "https://aclanthology.org/W17-5301/"
doi: "10.18653/v1/w17-5301"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nangia-2017-repeval-2017-shared-task-multi/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nangia-2017-repeval-2017-shared-task-multi/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

# The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations

**Authors:** [Nikita Nangia](https://md.research.elamur.ai/free/en/authors/nikita-nangia/), [Adina Williams](https://md.research.elamur.ai/free/en/authors/adina-williams/), [Angeliki Lazaridou](https://md.research.elamur.ai/free/en/authors/angeliki-lazaridou/) and [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2017 · **Venue:** [Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP](https://md.research.elamur.ai/free/en/venues/workshop-on-evaluating-vector-space-representations-for-nlp/) ([Workshop on Evaluating Vector Space Representations for NLP 2017](https://md.research.elamur.ai/free/en/venues/workshop-on-evaluating-vector-space-representations-for-nlp/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/W17-5301/) · [PDF](https://aclanthology.org/W17-5301.pdf) · [DOI 10.18653/v1/w17-5301](https://doi.org/10.18653/v1/w17-5301) · [OpenAlex](https://openalex.org/W2737504179)

**Citations:** 87 citations (OpenAlex, 2026-10-14) · [40](https://md.research.elamur.ai/free/en/papers/nangia-2017-repeval-2017-shared-task-multi/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper reports on a shared competition at RepEval 2017. Its goal was to measure how well neural sentence encoders capture meaning. Each sentence had to be compressed into a single fixed-length vector, and the vectors were then used for three-way natural language inference (entailment, neutral, contradiction) on the MultiNLI corpus. Mechanisms that let the two sentences in a pair look at each other, such as cross-sentence attention or memory over token sequences, were banned. This kept the comparison focused on the encoders. Two test conditions were run. In the matched condition, test genres also appear in training. In the mismatched condition, the five test genres were never seen in training. Five teams took part, and every one of them outperformed the BiLSTM and CBOW baselines. All the submitted systems were built on BiLSTMs. The strongest single model reached 74.5% matched accuracy using stacked BiLSTMs with residual (shortcut) connections. The gap between matched and mismatched accuracy turned out to be small. Results were also fairly even across the linguistic-phenomenon subsets. The organizers read this as evidence that the encoders learned fairly genre-independent representations, while NLI remains well below human level.

## Key points
- MultiNLI has about 393k sentence pairs (392,702 for training) drawn from ten genres of written and spoken English. Only five genres (FICTION, GOVERNMENT, SLATE, TELEPHONE, TRAVEL) come with training data. Each genre has 2,000 dev and 2,000 test pairs.
- Scoring ran on Kaggle in Class. The public leaderboard showed only a random 25% of test labels. Final scores came from the hidden 75%, using each team's two chosen systems.
- Final matched / mismatched accuracy:

  | System | Matched | Mismatched |
  |---|---|---|
  | Chen et al. ensemble ("alpha") | 74.9% | 74.9% |
  | Nie and Bansal, single model | 74.5% | 73.5% |
  | BiLSTM baseline | 67.0% | 67.6% |

- Competition rules:
  - Training on the dev sets was forbidden.
  - Any publicly available outside data was allowed, and SNLI was explicitly encouraged.
  - Teams could submit at most two runs per day during a two-week evaluation window.
- Teams also supplied vectors for automatically generated probe sentences. In these, vector similarity separated negation, subject identity and tense, but continuous tenses were not reliably distinguished. Behaviour on the probes did not clearly track overall accuracy.

## Interesting ideas
- Holding out whole genres from training (the mismatched split) shows whether a model learned general meaning or just the style of its training text.
- Banning cross-sentence interaction keeps the evaluation on the component of interest, so alignment tricks cannot mask weak representations.
- Minimal-pair probe sentences that vary only negation, role assignment or tense show which distinctions a representation actually encodes.
- Showing only a small public slice of the test labels and capping daily submissions discourages tuning to the evaluation set.
- Reporting annotator agreement per genre next to model accuracy gives a realistic ceiling and flags genres where the gold labels themselves are shaky.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Evaluate each trained component on a genre deliberately absent from its silver data, and report the matched-vs-unseen gap next to the main score.
- **I2** (answer): Build the world of each text independently, then decide entailment-style questions by comparing the two worlds, so errors point at world construction rather than the matcher.
- **I3** (frames): Generate such minimal pairs automatically and check that the resulting frames differ exactly on polarity, agent/patient or tense, catching silent role swaps.
- **I4** (learn): Keep a hidden portion of the gold test data that repair rules and self-play loops never see, consulted only for occasional final checks.
- **I5** (learn): Store agreement among witness parsers per genre and use it to set expectations and to down-weight low-consensus silver data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [University of Trento](https://md.research.elamur.ai/free/en/universities/university-of-trento/) — University, Trento, Italy · 62 works · 14,868 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Nikita Nangia, Adina Williams, Angeliki Lazaridou and Samuel R. Bowman. 2017. The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations. Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nangia2017repeval,
  title = {{The RepEval 2017 Shared Task: Multi-Genre Natural Language Inference with Sentence Representations}},
  author = {Nikita Nangia, Adina Williams, Angeliki Lazaridou and Samuel R. Bowman},
  booktitle = {Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP},
  year = {2017},
  doi = {10.18653/v1/w17-5301},
  url = {https://doi.org/10.18653/v1/w17-5301}
}
```

- [← previous Annotation, Modelling and Analysis of Fine-Grained Emotions on a Stance and Sentiment Detection Corpus](https://md.research.elamur.ai/free/en/papers/schuff-2017-annotation-modelling-analysis-fine-grained/)
- [next → How much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT](https://md.research.elamur.ai/free/en/papers/morey-2017-how-much-progress-have-we/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, and Yutaka Matsuo. 2017. Refining raw sentence representations for textual entailment recognition via attention. In The Second Workshop on Evaluating Vector Space Representations for NLP (RepEval 2017). [find](https://scholar.google.com/scholar?q=Jorge+Balazs%2C+Edison+Marrese%2DTaylor%2C+Pablo+Loyola%2C+and+Yutaka+Matsuo%2E+2017%2E+Refining+raw+sentence+representations+for+textual+entailment+recognition+via+attention%2E+In+The+Second+Workshop+on+Evaluating)
2. Shuohang Wang and Jing Jiang. 2016. [Learning natural language inference with LSTM](https://md.research.elamur.ai/free/en/papers/wang-2016-learning-natural-language-inference-lstm/). In Proc. NAACL.
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proc. EMNLP. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. CoRR abs/1704.05426. [↗](https://arxiv.org/abs/1704.05426)
5. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017a. Enhanced LSTM for natural language inference. In Proc. ACL. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017a%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+Proc%2E+ACL%2E)
6. Han Yang, Marta R. Costa-jussà, and José A. R. Fonollosa. 2017. Character-level intra attention network for natural language inference. In Proceedings of The Second Workshop on Evaluating Vector Space Representations for NLP (RepEval 2017). [find](https://scholar.google.com/scholar?q=Han+Yang%2C+Marta+R%2E+Costa%2Djussa%CC%80%2C+and+Jose%CC%81+A%2E+R%2E+Fonollosa%2E+2017%2E+Character%2Dlevel+intra+attention+network+for+natural+language+inference%2E+In+Proceedings+of+The+Second+Workshop+on+Evaluating+Vector+Spa)
7. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Recurrent neural network-based sentence encoder with gated attention for natural language inference. In Proceedings of The Second Workshop on Evaluating Vector Space Representations for NLP (RepEval 2017). [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017b%2E+Recurrent+neural+network%2Dbased+sentence+encoder+with+gated+attention+for+natural+language+inference%2E+In+Proceedings+o)
8. Gennaro Chierchia and Sally McConnell-Ginet. 1991. Meaning and Grammar. MIT Press, Cambridge, MA. [find](https://scholar.google.com/scholar?q=Gennaro+Chierchia+and+Sally+McConnell%2DGinet%2E+1991%2E+Meaning+and+Grammar%2E+MIT+Press%2C+Cambridge%2C+MA%2E)

*Updated 24 October 2026*