--- title: "GLUE: A Multi-Task Benchmark and Analysis — Wang 2018" description: "Most NLU systems that work above the word level are built for one task and tend to fail when the data comes from a different domain." authors: ["Alex Wang", "Amanpreet Singh", "Julian Michael", "Felix Hill", "Omer Levy", "Samuel R. Bowman"] year: 2018 venue: "Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP" source: "https://aclanthology.org/W18-5446/" doi: "10.18653/v1/w18-5446" arxiv: "1804.07461" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding # GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding **Authors:** [Alex Wang](https://md.research.elamur.ai/free/en/authors/alex-wang/), [Amanpreet Singh](https://md.research.elamur.ai/free/en/authors/amanpreet-singh/), [Julian Michael](https://md.research.elamur.ai/free/en/authors/julian-michael/), [Felix Hill](https://md.research.elamur.ai/free/en/authors/felix-hill/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), [Samuel R. Bowman](https://md.research.elamur.ai/free/en/authors/samuel-r-bowman/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/W18-5446/) · [PDF](https://aclanthology.org/W18-5446.pdf) · [DOI 10.18653/v1/w18-5446](https://doi.org/10.18653/v1/w18-5446) · [arXiv 1804.07461](https://arxiv.org/abs/1804.07461) · [OpenAlex](https://openalex.org/W2923014074) **Citations:** 4074 citations (openalex-snapshot · 2026-10-14) · [3062](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Most NLU systems that work above the word level are built for one task and tend to fail when the data comes from a different domain. To push the field toward general-purpose language understanding, the authors introduce GLUE. It brings together nine existing English sentence-level tasks in three groups: single-sentence classification, similarity/paraphrase, and inference. The tasks differ widely in training-set size and genre. Results are reported for each task and as an average. Evaluation runs through a public online platform with a leaderboard, and test labels for four tasks are kept private to keep comparisons fair. The authors also release a small diagnostic set of NLI pairs, built by experts and tagged by linguistic phenomenon. The baselines are a multi-task BiLSTM with a shared encoder and a separate head for each task, the same model trained on one task at a time, ELMo-based variants, and pretrained sentence encoders. Models trained directly on GLUE data did best, but every system got fairly low absolute scores, both on the benchmark and on the diagnostic set. GLUE rewards models that learn from few examples and transfer knowledge between tasks. ## Key points - Tasks: CoLA and SST-2 (single sentence); MRPC, QQP and STS-B (similarity/paraphrase); MNLI, QNLI, RTE and WNLI (inference). QNLI is SQuAD recast as NLI, and WNLI is the Winograd Schema Challenge recast as NLI. - Training data ranges from 393k examples (MNLI) and 364k (QQP) down to 2.5k (RTE) and 634 (WNLI). All tasks are binary classification except STS-B, which is regression, and MNLI, which has three classes. - The diagnostic set has 550 hand-built NLI sentence pairs covering phenomena such as logical operators, lexical entailment and world knowledge. It was checked with a hypothesis-only baseline and by six NLP researchers who reviewed a sample. - Average scores: the multi-task BiLSTM got 69.0, its single-task version 64.8, GenSen 66.6 (best of the pretrained encoders) and CBoW 58.9. - Results were weakest on the small tasks. The best CoLA result was 35.0 Matthews correlation, from the single-task model, and no system beat 65.1 on WNLI. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 3062 papers of our library cite it (of the 140673 whose bibliographies we read); 271 of them have a page here, the first 100 below, 2791 not yet. - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://md.research.elamur.ai/free/en/papers/abdul-2021-mageed-arbert-marbert-deep-bidirectional-transformers/) - [The SADID Evaluation Datasets for Low-Resource Spoken Language Machine Translation of Arabic Dialects](https://md.research.elamur.ai/free/en/papers/abid-2020-sadid-evaluation-datasets-low-resource/) - [Quantifying Attention Flow in Transformers](https://md.research.elamur.ai/free/en/papers/abnar-2020-quantifying-attention-flow-transformers/) - [AlephBERT: Language Model Pre-training and Evaluation from Sub-Word to Sentence Level](https://md.research.elamur.ai/free/en/papers/acl-2022-seker-alephbert-language-model-pre-training/) - [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/) - [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/) - [Rationale-Guided Distillation for E-Commerce Relevance Classification: Bridging Large Language Models and Lightweight Cross-Encoders](https://md.research.elamur.ai/free/en/papers/agrawal-2025-rationale-guided-distillation-e-commerce/) - [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://md.research.elamur.ai/free/en/papers/aguilar-2020-lince-centralized-benchmark-linguistic-code/) - [Unsupervised Domain Clusters in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/) - [MEGA: Multilingual Evaluation of Generative AI](https://md.research.elamur.ai/free/en/papers/ahuja-2023-mega-multilingual-evaluation-generative-ai/) - [ETC: Encoding Long and Structured Inputs in Transformers](https://md.research.elamur.ai/free/en/papers/ainslie-2020-etc-encoding-long-structured-inputs/) - [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://md.research.elamur.ai/free/en/papers/ainslie-2023-gqa-training-generalized-multi-query/) - [The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction](https://md.research.elamur.ai/free/en/papers/alikaniotis-2019-unreasonable-effectiveness-transformer-language-models/) - [Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data](https://md.research.elamur.ai/free/en/papers/alkhamissi-2024-dreaming-out-loud-self-synthesis/) - [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/) - [Acquiring Linguistic Knowledge from Multimodal Input](https://md.research.elamur.ai/free/en/papers/amariucai-2023-acquiring-linguistic-knowledge-multimodal-input/) - [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/) - [Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models](https://md.research.elamur.ai/free/en/papers/askari-2025-are-babylms-deaf-gricean-maxims/) - [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://md.research.elamur.ai/free/en/papers/augenstein-2019-multifc-real-world-multi-domain/) - [Factuality Challenges in the Era of Large Language Models](https://md.research.elamur.ai/free/en/papers/augenstein-2023-factuality-challenges-era-large-language/) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/) - [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://md.research.elamur.ai/free/en/papers/bao-2020-unilmv2-pseudo-masked-language-models/) - [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://md.research.elamur.ai/free/en/papers/barbieri-2020-tweeteval-unified-benchmark-comparative-evaluation/) - [Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/) - [Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/belinkov-2019-don-t-take-premise-granted/) - [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://md.research.elamur.ai/free/en/papers/ben-2021-zaken-bitfit-simple-parameter-efficient-fine/) - [Abductive Commonsense Reasoning](https://md.research.elamur.ai/free/en/papers/bhagavatula-2019-abductive-commonsense-reasoning/) - [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://md.research.elamur.ai/free/en/papers/bhattacharjee-2022-banglabert-language-model-pretraining-benchmarks/) - [Experience Grounds Language](https://md.research.elamur.ai/free/en/papers/bisk-2020-experience-grounds-language/) - [Systematic Inequalities in Language Technology Performance across the World's Languages](https://md.research.elamur.ai/free/en/papers/blasi-2022-systematic-inequalities-language-technology-performance/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://md.research.elamur.ai/free/en/papers/burns-2023-weak-strong-generalization-eliciting-strong/) - [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/) - [MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-multieurlex-multi-lingual-multi-label/) - [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) - [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/) - [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://md.research.elamur.ai/free/en/papers/chen-2023-benchmarking-large-language-models-retrieval/) - [An Empirical Survey of Data Augmentation for Limited Data Learning in NLP](https://md.research.elamur.ai/free/en/papers/chen-2023-empirical-survey-data-augmentation-limited/) - [BAM! Born-Again Multi-Task Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/) - [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/) - [8-bit Optimizers via Block-wise Quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/) - [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/) - [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/) - [OpenPrompt: An Open-source Framework for Prompt-learning](https://md.research.elamur.ai/free/en/papers/ding-2022-openprompt-open-source-framework-prompt/) - [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://md.research.elamur.ai/free/en/papers/doddapaneni-2023-leaving-no-indic-language-behind/) - [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/) - [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/) - [AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2022-americasnli-evaluating-zero-shot-natural/) - [Reducing Transformer Depth on Demand with Structured Dropout](https://md.research.elamur.ai/free/en/papers/fan-2019-reducing-transformer-depth-demand-structured/) - [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://md.research.elamur.ai/free/en/papers/fei-2023-lawbench-benchmarking-legal-knowledge-large/) - [IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages](https://md.research.elamur.ai/free/en/papers/gala-2023-indictrans2-towards-high-quality-accessible/) - [Condenser: a Pre-training Architecture for Dense Retrieval](https://md.research.elamur.ai/free/en/papers/gao-2021-condenser-pre-training-architecture-dense/) - [Making Pre-trained Language Models Better Few-shot Learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/) - [JABER and SABER: Junior and Senior Arabic BERt](https://md.research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/) - [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://md.research.elamur.ai/free/en/papers/giorgi-2021-declutr-deep-contrastive-learning-unsupervised/) - [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/) - [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://md.research.elamur.ai/free/en/papers/goot-2021-massive-choice-ample-tasks-machamp/) - [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://md.research.elamur.ai/free/en/papers/gordon-2020-compressing-bert-studying-effects-weight/) - [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://md.research.elamur.ai/free/en/papers/goyal-2021-larger-scale-transformers-multilingual-masked/) - [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://md.research.elamur.ai/free/en/papers/gu-2020-domain-specific-language-model-pretraining/) - [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://md.research.elamur.ai/free/en/papers/gu-2022-ppt-pre-trained-prompt-tuning/) - [Context Matters: A Pragmatic Study of PLMs' Negation Understanding](https://md.research.elamur.ai/free/en/papers/gubelmann-2022-context-matters-pragmatic-study-plms/) - [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/) - [Parameter-Efficient Transfer Learning with Diff Pruning](https://md.research.elamur.ai/free/en/papers/guo-2021-parameter-efficient-transfer-learning-diff/) - [Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts](https://md.research.elamur.ai/free/en/papers/guo-2022-auto-debias-debiasing-masked-language/) - [Evaluating Large Language Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/guo-2023-evaluating-large-language-models-comprehensive/) - [X-PuDu at SemEval-2022 Task 6: Multilingual Learning for English and Arabic Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/han-2022-x-pudu-semeval-2022-task/) - [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://md.research.elamur.ai/free/en/papers/han-2024-parameter-efficient-fine-tuning-large/) - [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/) - [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://md.research.elamur.ai/free/en/papers/he-2021-debertav3-improving-deberta-using-electra/) - [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://md.research.elamur.ai/free/en/papers/he-2021-effectiveness-adapter-based-tuning-pretrained/) - [Measuring Massive Multitask Language Understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/) - [TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/) - [An Analysis of Natural Language Inference Benchmarks through the Lens of Negation](https://md.research.elamur.ai/free/en/papers/hossain-2020-analysis-natural-language-inference-benchmarks/) - [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/) - [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://md.research.elamur.ai/free/en/papers/huang-2019-unicoder-universal-language-encoder-pre/) - [Unsupervised Syntactically Controlled Paraphrase Generation with Abstract Meaning Representations](https://md.research.elamur.ai/free/en/papers/huang-2022-unsupervised-syntactically-controlled-paraphrase-generation/) - [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://md.research.elamur.ai/free/en/papers/huang-2023-c-eval-multi-level-multi/) - [Large Language Models Can Self-Improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) - [ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://md.research.elamur.ai/free/en/papers/huang-2023-paraamr-large-scale-syntactically-diverse/) - [TrustLLM: Trustworthiness in Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2024-trustllm-trustworthiness-large-language-models/) - [Editing Models with Task Arithmetic](https://md.research.elamur.ai/free/en/papers/ilharco-2022-editing-models-task-arithmetic/) - [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://md.research.elamur.ai/free/en/papers/iv-2022-cutting-down-prompts-parameters-simple/) - [ArNLI: Arabic Natural Language Inference for Entailment and Contradiction Detection](https://md.research.elamur.ai/free/en/papers/jallad-2022-arnli-arabic-natural-language-inference/) - [E-CONAN (Entailment, CONtradition And Neutral) Benchmarks: Arabic Textual Entailment and Natural Inference Datasets](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral-benchmarks/) - [E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/jallad-2026-e-conan-entailment-contradition-neutral/) - [What Does BERT Learn about the Structure of Language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/) - [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/) - [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://md.research.elamur.ai/free/en/papers/jiang-2020-smart-robust-efficient-fine-tuning/) - [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/) - [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/) - [Debiasing Pre-trained Contextualised Embeddings](https://md.research.elamur.ai/free/en/papers/kaneko-2021-debiasing-pre-trained-contextualised-embeddings/) - [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/) - [GLUECoS: An Evaluation Benchmark for Code-Switched NLP](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/) - [Probing What Different NLP Tasks Teach Machines about Function Word Comprehension](https://md.research.elamur.ai/free/en/papers/kim-2019-probing-what-different-nlp-tasks/) - [I-BERT: Integer-only BERT Quantization](https://md.research.elamur.ai/free/en/papers/kim-2021-i-bert-integer-only-bert/) - [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2018glue, title = {{GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding}}, author = {Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.}, booktitle = {Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP}, year = {2018}, doi = {10.18653/v1/w18-5446}, eprint = {1804.07461}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/w18-5446} } ``` - [← previous Getting Gender Right in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/vanmassenhove-2018-getting-gender-right-neural-machine/) - [next → Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://md.research.elamur.ai/free/en/papers/zhang-2018-graph-convolution-over-pruned-dependency/) - [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) ## References 19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442–451. [find](https://scholar.google.com/scholar?q=lysozyme%2E+Biochimica+et+Biophysica+Acta+%28BBA%29%2DProtein+Structure%2C+405%282%29%3A442%E2%80%93451%2E) 2. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. [find](https://scholar.google.com/scholar?q=Roy+Bar+Haim%2C+Ido+Dagan%2C+Bill+Dolan%2C+Lisa+Ferro%2C+Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+and+Idan+Szpektor%2E+2006%2E+The+second+PASCAL+recognising+textual+entailment+challenge%2E) 3. Allen Nie, Erin D Bennett, and Noah D Goodman. 2017. Dissent: Sentence representation learning from explicit discourse relations. arXiv preprint 1710.04334. — [arXiv:1710.04334](https://arxiv.org/abs/1710.04334) [find](https://scholar.google.com/scholar?q=Allen+Nie%2C+Erin+D+Bennett%2C+and+Noah+D+Goodman%2E+2017%2E+Dissent%3A+Sentence+representation+learning+from+explicit+discourse+relations%2E+arXiv+preprint+1710%2E04334%2E+%E2%80%94+%5BarXiv%3A1710%2E04334%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F) 4. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+Danilo+Giampiccolo%2C+and+Bernardo+Magnini%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E) 5. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL 2018. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL+2018%2E) 6. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. In 11th International Workshop on Semantic Evaluations. [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity%2Dmultilingual+and+cross%2Dlingual+focused+evaluation%2E+In+11th+Internation) 7. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 8. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, De… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/) 9. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+) 10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classificati) 11. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J. Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Christopher+J%2E+Pal%2E+2018%2E+Learning+general+purpose+distributed+sentence+representations+via+large+scale+multi%2Dtask+learning%2E+In+Proceedings+of+I) 12. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of IWP. [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+IWP%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/) 13. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint 1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+1805%2E12471%2E) 14. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Danilo+Giampiccolo%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+and+Bill+Dolan%2E+2007%2E+The+third+PASCAL+recognizing+textual+entailment+challenge%2E+In+Proceedings+of+the+ACL%2DPASCAL+workshop+on+textual+entailment+and+pa) 15. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL 2018. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+NAACL+2018%2E) 16. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Suchin+Gururangan%2C+Swabha+Swayamdipta%2C+Omer+Levy%2C+Roy+Schwartz%2C+Samuel+R%2E+Bowman%2C+and+Noah+A%2E+Smith%2E+2018%2E+Annotation+artifacts+in+natural+language+inference+data%2E+In+Proceedings+of+the+North+American) 17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-Thought vectors. In Advances in neural information processing systems, pages 3294–3302. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2DThought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+page) 18. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+Winograd+schema+challenge%2E+In+Aaai+spring+symposium%3A+Logical+formalizations+of+commonsense+reasoning%2C+volume+46%2C+page+47%2E) 19. Brian W Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage [find](https://scholar.google.com/scholar?q=Brian+W+Matthews%2E+1975%2E+Comparison+of+the+predicted+and+observed+secondary+structure+of+t4+phage)