--- title: "BART: Denoising Sequence-to-Sequence — Lewis 2020" description: "Earlier self-supervised pre-training methods were usually built either for understanding tasks, like BERT-style masked models, or for generation, like…" authors: ["Mike Lewis", "Yinhan Liu", "Naman Goyal", "Marjan Ghazvininejad", "Abdelrahman Mohamed", "Omer Levy", "Veselin Stoyanov", "Luke Zettlemoyer"] year: 2020 venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.acl-main.703/" doi: "10.18653/v1/2020.acl-main.703" arxiv: "1910.13461" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/) / BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension # BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension **Authors:** [Mike Lewis](https://md.research.elamur.ai/free/en/authors/mike-lewis/), [Yinhan Liu](https://md.research.elamur.ai/free/en/authors/yinhan-liu/), [Naman Goyal](https://md.research.elamur.ai/free/en/authors/naman-goyal/), [Marjan Ghazvininejad](https://md.research.elamur.ai/free/en/authors/marjan-ghazvininejad/), [Abdelrahman Mohamed](https://md.research.elamur.ai/free/en/authors/abdelrahman-mohamed/), [Omer Levy](https://md.research.elamur.ai/free/en/authors/omer-levy/), [Veselin Stoyanov](https://md.research.elamur.ai/free/en/authors/veselin-stoyanov/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.703/) · [PDF](https://aclanthology.org/2020.acl-main.703.pdf) · [DOI 10.18653/v1/2020.acl-main.703](https://doi.org/10.18653/v1/2020.acl-main.703) · [arXiv 1910.13461](https://arxiv.org/abs/1910.13461) · [OpenAlex](https://openalex.org/W2982399380) **Citations:** 8244 citations (openalex-snapshot · 2026-10-14) · [4507](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Earlier self-supervised pre-training methods were usually built either for understanding tasks, like BERT-style masked models, or for generation, like GPT-style left-to-right models. Few did both well. The authors propose BART, a sequence-to-sequence Transformer pre-trained as a denoising autoencoder. Input documents are corrupted by a noising function, and a bidirectional encoder works with an autoregressive decoder to restore the original text. The encoder input does not need to line up with the decoder output, so any kind of corruption can be used, including changes in length. The authors ran a controlled comparison of noising strategies and re-implemented earlier objectives. The best overall mix was shuffling sentences together with text infilling, where spans of varying length are each replaced by one mask token. After fine-tuning, the large model performs about as well as RoBERTa on GLUE and SQuAD. It also sets new state-of-the-art results in abstractive summarization, dialogue and generative question answering. This matters because one pre-trained model can handle both comprehension and generation, and the authors released code and weights publicly. ## Key points - **Architecture:** a standard Transformer encoder-decoder with GeLU activations. The base model has 6 encoder and 6 decoder layers (hidden size 768), and the large model has 12 of each. It has about 10% more parameters than a BERT model of the same size. - **Noising schemes tested:** token masking, token deletion, text infilling, sentence permutation and document rotation. In text infilling, span lengths come from a Poisson distribution (λ = 3), and zero-length spans are allowed. Unlike SpanBERT, each span becomes a single [MASK]. - **Generation results:** gains of up to 3.5 ROUGE on XSum, plus new top scores on several dialogue, QA and summarization benchmarks. The main weakness the authors note is a tendency to hallucinate unsupported content. - **Machine translation:** a new, randomly initialized source encoder replaces BART's embedding layer and is trained in two stages. This gives +1.1 BLEU over a back-translation baseline on WMT Romanian-English, using only English (target-side) pre-training. - **Release:** code and pre-trained models are available in fairseq and Hugging Face Transformers. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 4507 papers of our library cite it (of the 140673 whose bibliographies we read); 195 of them have a page here, the first 100 below, 4312 not yet. - [Semitic Root Encoding: Tokenization Based on the Templatic Morphology of Semitic Languages in NMT](https://md.research.elamur.ai/free/en/papers/aclbulk-2025-hatch-semitic-root-encoding-tokenization-based/) - [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-intrinsic-dimensionality-explains-effectiveness-language/) - [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://md.research.elamur.ai/free/en/papers/aghajanyan-2021-muppet-massive-multi-task-representations/) - [Unified Pre-training for Program Understanding and Generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/) - [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://md.research.elamur.ai/free/en/papers/akhtar-2023-exploring-numerical-reasoning-capabilities-language/) - [Improving Cross-lingual Transfer through Subtree-aware Word Reordering](https://md.research.elamur.ai/free/en/papers/arviv-2023-improving-cross-lingual-transfer-through/) - [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/asai-2022-evidentiality-guided-generation-knowledge-intensive/) - [Bridging Laughter Across Languages: Generation of Hindi-English Code-mixed Puns](https://md.research.elamur.ai/free/en/papers/asapu-2025-bridging-laughter-across-languages-generation/) - [Bridging Cognition and Affect: Emotion-Aware Opinion Summarization using LLMs](https://md.research.elamur.ai/free/en/papers/attri-2026-bridging-cognition-affect-emotion-aware/) - [Graph Pre-training for AMR Parsing and Generation](https://md.research.elamur.ai/free/en/papers/bai-2022-graph-pre-training-amr-parsing/) - [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://md.research.elamur.ai/free/en/papers/bang-2023-multitask-multilingual-multimodal-evaluation-chatgpt/) - [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://md.research.elamur.ai/free/en/papers/bao-2020-unilmv2-pseudo-masked-language-models/) - [Exemplification Modeling: Can You Give Me an Example, Please?](https://md.research.elamur.ai/free/en/papers/barba-2021-exemplification-modeling/) - [Efficient Training of Language Models to Fill in the Middle](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/) - [Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys](https://md.research.elamur.ai/free/en/papers/cao-2024-bridging-cultural-nuances-dialogue-agents/) - [Extracting Training Data from Large Language Models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-lexglue-benchmark-dataset-legal-language/) - [A Survey on Evaluation of Large Language Models](https://md.research.elamur.ai/free/en/papers/chang-2023-survey-evaluation-large-language-models/) - [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://md.research.elamur.ai/free/en/papers/chen-2021-dialogsum-real-life-scenario-dialogue/) - [Automatic Evaluate Dialogue Appropriateness by Using Dialogue Act](https://md.research.elamur.ai/free/en/papers/chen-2023-automatic-evaluate-dialogue-appropriateness-dialogue/) - [Binding Language Models in Symbolic Languages](https://md.research.elamur.ai/free/en/papers/cheng-2022-binding-language-models-symbolic-languages/) - [No Context Needed: Contextual Quandary In Idiomatic Reasoning With Pre-Trained Language Models](https://md.research.elamur.ai/free/en/papers/cheng-2024-no-context-needed-contextual-quandary/) - [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://md.research.elamur.ai/free/en/papers/dai-2023-auggpt-leveraging-chatgpt-text-data/) - [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://md.research.elamur.ai/free/en/papers/dasigi-2021-dataset-information-seeking-questions-answers/) - [RobBERT: a Dutch RoBERTa-based Language Model](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/) - [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/) - [8-bit Optimizers via Block-wise Quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/) - [Paragraph-level Simplification of Medical Texts](https://md.research.elamur.ai/free/en/papers/devaraj-2021-paragraph-level-simplification-medical-texts/) - [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://md.research.elamur.ai/free/en/papers/dhole-2023-nl-augmenter-framework-task-sensitive/) - [GSum: A General Framework for Guided Neural Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/dou-2021-gsum-general-framework-guided-neural/) - [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://md.research.elamur.ai/free/en/papers/du-2022-glm-general-language-model-pretraining/) - [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/durmus-2020-feqa-question-answering-evaluation-framework/) - [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/) - [常识问答研究综述](https://md.research.elamur.ai/free/en/papers/fan-2023-commonsense-question-answering/) - [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/) - [A Survey of Data Augmentation Approaches for NLP](https://md.research.elamur.ai/free/en/papers/feng-2021-survey-data-augmentation-approaches-nlp/) - [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://md.research.elamur.ai/free/en/papers/feng-2023-pretraining-data-language-models-downstream/) - [The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)](https://md.research.elamur.ai/free/en/papers/ferreira-2020-2020-bilingual-bi-directional-webnlg/) - [EPIC: Multi-Perspective Annotation of a Corpus of Irony](https://md.research.elamur.ai/free/en/papers/frenda-2023-epic-multi-perspective-annotation-corpus/) - [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://md.research.elamur.ai/free/en/papers/fried-2023-pragmatics-language-grounding-phenomena-tasks/) - [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/) - [ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning](https://md.research.elamur.ai/free/en/papers/golovneva-2022-roscoe-suite-metrics-scoring-step/) - [DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models](https://md.research.elamur.ai/free/en/papers/gong-2023-diffuseq-v2-bridging-discrete-continuous/) - [Evaluating Factuality in Generation with Dependency-level Entailment](https://md.research.elamur.ai/free/en/papers/goyal-2020-evaluating-factuality-generation-dependency-level/) - [Annotating and Modeling Fine-grained Factuality in Summarization](https://md.research.elamur.ai/free/en/papers/goyal-2021-annotating-modeling-fine-grained-factuality/) - [When Truth Matters - Addressing Pragmatic Categories in Natural Language Inference (NLI) by Large Language Models (LLMs)](https://md.research.elamur.ai/free/en/papers/gubelmann-2023-when-truth-matters-addressing-pragmatic/) - [GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution](https://md.research.elamur.ai/free/en/papers/guo-2022-gravl-bert-graphical-visual-linguistic/) - [LongT5: Efficient Text-To-Text Transformer for Long Sequences](https://md.research.elamur.ai/free/en/papers/guo-2022-longt5-efficient-text-text-transformer/) - [Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language](https://md.research.elamur.ai/free/en/papers/hbo-2022-shmidman-introducing-berel-bert-embeddings-rabbinic/) - [Towards a Unified View of Parameter-Efficient Transfer Learning](https://md.research.elamur.ai/free/en/papers/he-2021-towards-unified-view-parameter-efficient/) - [Measuring Mathematical Problem Solving With the MATH Dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/) - [TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://md.research.elamur.ai/free/en/papers/hsu-2021-hubert-self-supervised-speech-representation/) - [Efficient Attentions for Long Document Summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/) - [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://md.research.elamur.ai/free/en/papers/huang-2022-layoutlmv3-pre-training-document-ai/) - [ParaAMR: A Large-Scale Syntactically Diverse Paraphrase Dataset by AMR Back-Translation](https://md.research.elamur.ai/free/en/papers/huang-2023-paraamr-large-scale-syntactically-diverse/) - [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/) - [Transformer-Patcher: One Mistake worth One Neuron](https://md.research.elamur.ai/free/en/papers/huang-2023-transformer-patcher-one-mistake-worth/) - [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://md.research.elamur.ai/free/en/papers/hwang-2020-comet-atomic-2020-symbolic-neural/) - [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/) - [T-STAR: Truthful Style Transfer using AMR Graph as Intermediate Representation](https://md.research.elamur.ai/free/en/papers/jangra-2022-t-star-truthful-style-transfer/) - [Survey of Hallucination in Natural Language Generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/) - [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/) - [BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence](https://md.research.elamur.ai/free/en/papers/jin-2024-bider-bridging-knowledge-inconsistency-efficient/) - [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://md.research.elamur.ai/free/en/papers/jing-2023-multi-source-semantic-graph-based/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [Dense Passage Retrieval for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/) - [RealTime QA: What's the Answer Right Now?](https://md.research.elamur.ai/free/en/papers/kasai-2022-realtime-qa-s-answer-right/) - [Evaluating Large Language Models for Health-related Queries with Presuppositions](https://md.research.elamur.ai/free/en/papers/kaur-2024-evaluating-large-language-models-health/) - [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) - [OCR-free Document Understanding Transformer](https://md.research.elamur.ai/free/en/papers/kim-2021-ocr-free-document-understanding-transformer/) - [Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes](https://md.research.elamur.ai/free/en/papers/kim-2021-perspective-taking-pragmatics-generating-empathetic/) - [Internet-Augmented Dialogue Generation](https://md.research.elamur.ai/free/en/papers/komeili-2022-internet-augmented-dialogue-generation/) - [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/krishna-2020-reformulating-unsupervised-style-transfer-as/) - [Hurdles to Progress in Long-form Question Answering](https://md.research.elamur.ai/free/en/papers/krishna-2021-hurdles-progress-long-form-question/) - [Data Augmentation using Pre-trained Transformer Models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/) - [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://md.research.elamur.ai/free/en/papers/kumar-2022-when-did-you-become-so/) - [Unsupervised Translation of Programming Languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/) - [WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/ladhak-2020-wikilingua-new-benchmark-dataset-cross/) - [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://md.research.elamur.ai/free/en/papers/lai-2023-chatgpt-beyond-english-comprehensive-evaluation/) - [Pre-training via Paraphrasing](https://md.research.elamur.ai/free/en/papers/lewis-2020-pre-training-paraphrasing/) - [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://md.research.elamur.ai/free/en/papers/lewis-2021-paq-65-million-probably-asked/) - [Generationary or «How We Went beyond Word Sense Inventories and Learned to Gloss»](https://md.research.elamur.ai/free/en/papers/lex-2020-bevilacqua-generationary/) - [Document-Level Event Argument Extraction by Conditional Generation](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/) - [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://md.research.elamur.ai/free/en/papers/li-2021-mtop-comprehensive-multilingual-task-oriented/) - [Generative Judge for Evaluating Alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/) - [A New Direction in Stance Detection: Target-Stance Extraction in the Wild](https://md.research.elamur.ai/free/en/papers/li-2023-new-direction-stance-detection-target/) - [Discourse Relation Prediction and Discourse Parsing in Dialogues with Minimal Supervision](https://md.research.elamur.ai/free/en/papers/li-2024-discourse-relation-prediction-discourse-parsing/) - [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://md.research.elamur.ai/free/en/papers/liang-2020-xglue-new-benchmark-dataset-cross/) - [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/) - [GUMsley: Evaluating Entity Salience in Summarization for 12 English Genres](https://md.research.elamur.ai/free/en/papers/lin-2024-gumsley-evaluating-entity-salience-summarization/) - [Coreference-Aware Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-coreference-aware-dialogue-summarization/) - [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://md.research.elamur.ai/free/en/papers/liu-2021-dexperts-decoding-time-controlled-text/) - [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/) - [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/liu-2021-simcls-simple-framework-contrastive-learning/) - [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://md.research.elamur.ai/free/en/papers/liu-2021-tapex-table-pre-training-learning/) - [BRIO: Bringing Order to Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/liu-2022-brio-bringing-order-abstractive-summarization/) - [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://md.research.elamur.ai/free/en/papers/liu-2022-revisiting-gold-standard-grounding-summarization/) - [What Makes Good In-Context Examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lewis2020bart, title = {{BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension}}, author = {Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke}, booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, year = {2020}, doi = {10.18653/v1/2020.acl-main.703}, eprint = {1910.13461}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2020.acl-main.703} } ``` - [← previous BAE: BERT-based Adversarial Examples for Text Classification](https://md.research.elamur.ai/free/en/papers/garg-2020-bae-bert-based-adversarial-examples/) - [next → Becoming Linguistically Mature: Modeling English and German Children's Writing Development Across School Grades](https://md.research.elamur.ai/free/en/papers/kerz-2020-becoming-linguistically-mature-modeling-english/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/) ## References 26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Eneko Agirre, Llu’is M‘arquez, and Richard Wicentowski (eds.). Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). Association for Computational Linguistics, Prague, Czech Republic, June 2007. [find](https://scholar.google.com/scholar?q=Eneko+Agirre%2C+Llu%E2%80%99is+M%E2%80%98arquez%2C+and+Richard+Wicentowski+%28eds%2E%29%2E+Proceedings+of+the+Fourth+International+Workshop+on+Semantic+Evaluations+%28SemEval%2D2007%29%2E+Association+for+Computational+Linguistics%2C+Pragu) 2. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pp. 177– 190. Springer, 2006. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+learning+challenges%2E+evaluating+predictive+uncertainty%2C+visual+object+classification%2C+an) 3. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Yacine+Jernite%2C+Ethan+Perez%2C+David+Grangier%2C+Jason+Weston%2C+and+Michael+Auli%2E+Eli5%3A+Long+form+question+answering%2E+arXiv+preprint+arXiv%3A1907%2E09190%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/) 4. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+Gaussian+error+linear+units+%28gelus%29%2E+arXiv+preprint+arXiv%3A1606%2E08415%2C+2016%2E) 5. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693–1701, 2015. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+neural+information+proce) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/) 6. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Danqi+Chen%2C+Yinhan+Liu%2C+Daniel+S+Weld%2C+Luke+Zettlemoyer%2C+and+Omer+Levy%2E+Spanbert%3A+Improving+pre%2Dtraining+by+representing+and+predicting+spans%2E+arXiv+preprint+arXiv%3A1907%2E10529%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2019-spanbert-improving-pre-training-representing/) 7. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2C+2019%2E) 8. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+arXiv+preprint+arXiv%3A1909%2E11) 9. Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+Winograd+schema+challenge%2E+In+AAAI+Spring+Symposium%3A+Logical+Formalizations+of+Commonsense+Reasoning%2C+volume+46%2C+pp%2E+47%2C+2011%2E) 10. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345, 2019. [find](https://scholar.google.com/scholar?q=Yang+Liu+and+Mirella+Lapata%2E+Text+summarization+with+pretrained+encoders%2E+arXiv+preprint+arXiv%3A1908%2E08345%2C+2019%2E) 11. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. Leveraging pre-trained checkpoints for sequence generation tasks. arXiv preprint arXiv:1907.12461, 2019. [find](https://scholar.google.com/scholar?q=Sascha+Rothe%2C+Shashi+Narayan%2C+and+Aliaksei+Severyn%2E+Leveraging+pre%2Dtrained+checkpoints+for+sequence+generation+tasks%2E+arXiv+preprint+arXiv%3A1907%2E12461%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/rothe-2019-leveraging-pre-trained-checkpoints-sequence/) 12. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+arXiv+preprint+arXiv%3A1704%2E04368%2C+2017%2E) 13. Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Edinburgh+neural+machine+translation+systems+for+WMT+16%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+Papers%2C+) [our card](https://md.research.elamur.ai/free/en/papers/sennrich-2016-edinburgh-neural-machine-translation-systems/) 14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+arXiv+p) 15. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pp. 1631–1642, 2013. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+Procee) 16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2C+2013%2E) 17. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, 2019. [find](https://scholar.google.com/scholar?q=Kaitao+Song%2C+Xu+Tan%2C+Tao+Qin%2C+Jianfeng+Lu%2C+and+Tie%2DYan+Liu%2E+Mass%3A+Masked+sequence+to+sequence+pre%2Dtraining+for+language+generation%2E+In+International+Conference+on+Machine+Learning%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/) 18. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018. [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B+Cohen%2C+and+Mirella+Lapata%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+arXiv+preprint+arXiv%3A1808%2E08745%2C+201) 19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+Advances+in+neural+information+processing+sys) 20. Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06548, 2017. [find](https://scholar.google.com/scholar?q=Gabriel+Pereyra%2C+George+Tucker%2C+Jan+Chorowski%2C+%C5%81ukasz+Kaiser%2C+and+Geoffrey+Hinton%2E+Regularizing+neural+networks+by+penalizing+confident+output+distributions%2E+arXiv+preprint+arXiv%3A1701%2E06548%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/) 21. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1804) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/) 22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2C+2018%2E) 23. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805.12471, 2018. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R%2E+Bowman%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+1805%2E12471%2C+2018%2E) 24. Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for Alec Radford, Karthik Narasimhan, Tim Salimans, sentence understanding through inference. arXiv and Ilya Sutskever. Improving language un-preprint arXiv:1704.05426, 2017. derstanding by generative pre-training.… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+A+broad%2Dcoverage+challenge+corpus+for+Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+sentence+understanding+through+inference%2E+arXiv+and+Ilya+Sutsk) 25. OpenAI Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. [find](https://scholar.google.com/scholar?q=OpenAI+Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex) 26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)