--- title: "A Decomposable Attention Model for Natural — Parikh 2016" description: "Natural language inference (NLI) means deciding whether a hypothesis follows from a premise, contradicts it, or is neutral." authors: ["Ankur Parikh", "Oscar Täckström", "Dipanjan Das", "Jakob Uszkoreit"] year: 2016 venue: "Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/D16-1244/" doi: "10.18653/v1/d16-1244" arxiv: "1606.01933" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / A Decomposable Attention Model for Natural Language Inference # A Decomposable Attention Model for Natural Language Inference **Authors:** [Ankur Parikh](https://md.research.elamur.ai/free/en/authors/ankur-parikh/), [Oscar Täckström](https://md.research.elamur.ai/free/en/authors/oscar-tackstrom/), [Dipanjan Das](https://md.research.elamur.ai/free/en/authors/dipanjan-das/), [Jakob Uszkoreit](https://md.research.elamur.ai/free/en/authors/jakob-uszkoreit/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D16-1244/) · [PDF](https://aclanthology.org/D16-1244.pdf) · [DOI 10.18653/v1/d16-1244](https://doi.org/10.18653/v1/d16-1244) · [arXiv 1606.01933](https://arxiv.org/abs/1606.01933) · [OpenAlex](https://openalex.org/W2413794162) **Citations:** 1296 citations (openalex-snapshot · 2026-10-14) · [293](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Natural language inference (NLI) means deciding whether a hypothesis follows from a premise, contradicts it, or is neutral. Earlier neural approaches built large sentence encoders from LSTMs or CNNs. Those models have millions of parameters and read text one word after another. The authors argue that most of the task can be solved by matching small fragments of the two sentences and then combining the evidence. Their model has three feed-forward stages that are trained together. Attend softly aligns word embeddings across the sentence pair. Compare turns each word and the material aligned to it into a vector. Aggregate sums these vectors and classifies the result. The basic version ignores word order entirely. An optional intra-sentence attention step adds a distance-based bias, which gives the model a small amount of ordering information. On SNLI, this design matched or beat the best published accuracy with about ten times fewer parameters than the strongest LSTM model. It can also run in parallel across sentence positions. The results suggest that for this task, pairwise alignment matters more than a representation of the whole sentence. ## Key points - **Data:** the paper uses the SNLI corpus with three classes. After dropping pairs that have no gold label, it has 549,367 training, 9,842 development and 9,824 test pairs. - **Results:** the vanilla model reaches 86.3% test accuracy with 382K parameters, not counting embeddings. Adding intra-sentence attention raises this to 86.8% with 582K parameters. For comparison, the LSTMN of Cheng et al. scores 86.3% with 3.4M parameters, and the mLSTM scores 86.1% with 1.9M. - **Efficiency:** each attention score is computed as F(a_i)ᵀF(b_j), so the network F runs only ℓ_a + ℓ_b times instead of once per word pair. The total cost is O(ℓd² + ℓ²d), the same as an LSTM with attention. When sentence length is below the embedding size, the cost drops to that of a plain LSTM, and unlike an LSTM it can be parallelized across sentence length. - **Setup:** the model uses fixed 300D GloVe vectors projected down to 200D, and out-of-vocabulary words are hashed into 100 random vectors. The networks have 2 layers of 200 ReLU units. Training uses Adagrad with dropout 0.2, implemented in TensorFlow. - **Error analysis:** most gains over earlier models come from neutral pairs, and most losses come from contradictions. The model still fails when the answer depends on word order, on counting, or on a single opposing word such as *closes* vs *open*. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 293 papers of our library cite it (of the 140884 whose bibliographies we read); 41 of them have a page here, 252 not yet. - [Integrating Stance Detection and Fact Checking in a Unified Corpus](https://md.research.elamur.ai/free/en/papers/baly-2018-integrating-stance-detection-fact-checking/) - [Interpretable Neural Predictions with Differentiable Binary Variables](https://md.research.elamur.ai/free/en/papers/bastings-2019-interpretable-neural-predictions-differentiable-binary/) - [Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/) - [e-SNLI: Natural Language Inference with Natural Language Explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/) - [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/) - [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/) - [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/) - [Transformers as Soft Reasoners over Language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/) - [Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/) - [Sarcasm Analysis Using Conversation Context](https://md.research.elamur.ai/free/en/papers/ghosh-2018-sarcasm-analysis-conversation-context/) - [Breaking NLI Systems with Sentences that Require Simple Lexical Inferences](https://md.research.elamur.ai/free/en/papers/glockner-2018-breaking-nli-systems-sentences-that/) - [Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual](https://md.research.elamur.ai/free/en/papers/he-2019-unlearn-dataset-bias-natural-language/) - [Learning to Write with Cooperative Discriminators](https://md.research.elamur.ai/free/en/papers/holtzman-2018-learning-write-cooperative-discriminators/) - [Attention is not Explanation](https://md.research.elamur.ai/free/en/papers/jain-2019-attention-is-not-explanation/) - [Certified Robustness to Adversarial Word Substitutions](https://md.research.elamur.ai/free/en/papers/jia-2019-certified-robustness-adversarial-word-substitutions/) - [Explainable Automated Fact-Checking for Public Health Claims](https://md.research.elamur.ai/free/en/papers/kotonya-2020-explainable-automated-fact-checking-public/) - [Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation](https://md.research.elamur.ai/free/en/papers/kumar-2019-submodular-optimization-based-diverse-paraphrasing/) - [Paraphrase Generation with Deep Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/li-2018-paraphrase-generation-deep-reinforcement-learning/) - [A Structured Self-attentive Sentence Embedding](https://md.research.elamur.ai/free/en/papers/lin-2017-structured-self-attentive-sentence-embedding/) - [Fine-grained Fact Verification with Kernel Graph Attention Network](https://md.research.elamur.ai/free/en/papers/liu-2020-fine-grained-fact-verification-kernel/) - [Learned in Translation: Contextualized Word Vectors](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/) - [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/) - [Are Sixteen Heads Really Better than One?](https://md.research.elamur.ai/free/en/papers/michel-2019-sixteen-heads-really-better-than/) - [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://md.research.elamur.ai/free/en/papers/pasunuru-2018-multi-reward-reinforced-summarization-saliency/) - [Probing Natural Language Inference Models through Semantic Fragments](https://md.research.elamur.ai/free/en/papers/richardson-2019-probing-natural-language-inference-models/) - [Lessons from Natural Language Inference in the Clinical Domain](https://md.research.elamur.ai/free/en/papers/romanov-2018-lessons-natural-language-inference-clinical/) - [Interpretation of Natural Language Rules in Conversational Machine Reading](https://md.research.elamur.ai/free/en/papers/saeidi-2018-interpretation-natural-language-rules-conversational/) - [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://md.research.elamur.ai/free/en/papers/shen-2018-baseline-needs-more-love-simple/) - [Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold](https://md.research.elamur.ai/free/en/papers/swayamdipta-2017-frame-semantic-parsing-softmax-margin/) - [Deep Semantic Role Labeling with Self-Attention](https://md.research.elamur.ai/free/en/papers/tan-2017-deep-semantic-role-labeling-self/) - [Reasoning with Sarcasm by Reading In-Between](https://md.research.elamur.ai/free/en/papers/tay-2018-reasoning-sarcasm-reading-between/) - [The Fact Extraction and VERification (FEVER) Shared Task](https://md.research.elamur.ai/free/en/papers/thorne-2018-fact-extraction-verification-fever-shared/) - [FEVER: a Large-scale Dataset for Fact Extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/) - [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://md.research.elamur.ai/free/en/papers/tsai-2019-multimodal-transformer-unaligned-multimodal-language/) - [Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/) - [Bilateral Multi-Perspective Matching for Natural Language Sentences](https://md.research.elamur.ai/free/en/papers/wang-2017-bilateral-multi-perspective-matching-natural/) - [Gated Self-Matching Networks for Reading Comprehension and Question Answering](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) - [Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework](https://md.research.elamur.ai/free/en/papers/white-2017-inference-is-everything-recasting-semantic/) - [TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification](https://md.research.elamur.ai/free/en/papers/yin-2018-twowingos-two-wing-optimization-strategy/) - [PAWS: Paraphrase Adversaries from Word Scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/) - [GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification](https://md.research.elamur.ai/free/en/papers/zhou-2019-gear-graph-based-evidence-aggregating/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{parikh2016decomposable, title = {{A Decomposable Attention Model for Natural Language Inference}}, author = {Parikh, Ankur and Täckström, Oscar and Das, Dipanjan and Uszkoreit, Jakob}, booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing}, year = {2016}, doi = {10.18653/v1/d16-1244}, eprint = {1606.01933}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/d16-1244} } ``` - [← previous A Corpus of Preposition Supersenses](https://md.research.elamur.ai/free/en/papers/aclbulk-2016-schneider-corpus-preposition-supersenses/) - [next → A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks](https://md.research.elamur.ai/free/en/papers/poria-2016-deeper-look-into-sarcastic-tweets/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. [Abadi et al.2015] Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2015. TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow. org. [Andreas et al.2… [find](https://scholar.google.com/scholar?q=%5BAbadi+et+al%2E2015%5D+Mart%C4%B1%CC%81n+Abadi%2C+Ashish+Agarwal%2C+Paul+Barham%2C+Eugene+Brevdo%2C+Zhifeng+Chen%2C+Craig+Citro%2C+Greg+S+Corrado%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+et+al%2E+2015%2E+TensorFlow%3A+Large%2Dscale+) 2. Denker, D. Henderson, R.E. Howard, W. Hubbard, and L.D. Jackel. 1990. Handwritten digit recognition with a back-propagation network. In Advances in NIPS. [MacCartney and Manning2009] Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the IWCS. [Ma… [find](https://scholar.google.com/scholar?q=Denker%2C+D%2E+Henderson%2C+R%2EE%2E+Howard%2C+W%2E+Hubbard%2C+and+L%2ED%2E+Jackel%2E+1990%2E+Handwritten+digit+recognition+with+a+back%2Dpropagation+network%2E+In+Advances+in+NIPS%2E+%5BMacCartney+and+Manning2009%5D+Bill+MacCartney+a) 3. [van Benthem2008] Johan van Benthem. 2008. A brief history of natural logic. College Publications. [Vendrov et al.2015] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2015. Order-embeddings of images and language. In Proceedings of ICLR. [Wang and Jiang2016] Shuohang Wang and Jing Jiang… [find](https://scholar.google.com/scholar?q=%5Bvan+Benthem2008%5D+Johan+van+Benthem%2E+2008%2E+A+brief+history+of+natural+logic%2E+College+Publications%2E+%5BVendrov+et+al%2E2015%5D+Ivan+Vendrov%2C+Ryan+Kiros%2C+Sanja+Fidler%2C+and+Raquel+Urtasun%2E+2015%2E+Order%2Dembeddin)