---
title: "CTRL: A Conditional Transformer Language Model — Keskar 2019"
description: "Large pretrained language models produce fluent text, but users have little influence over its style, topic or purpose beyond writing a prompt."
authors: ["Nitish Shirish Keskar", "Bryan McCann", "Lav R. Varshney", "Caiming Xiong", "Richard Socher"]
year: 2019
venue: "arXiv:1909.05858"
source: "https://arxiv.org/abs/1909.05858"
doi: "10.48550/arxiv.1909.05858"
arxiv: "1909.05858"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / CTRL: A Conditional Transformer Language Model for Controllable Generation

# CTRL: A Conditional Transformer Language Model for Controllable Generation

**Authors:** [Nitish Shirish Keskar](https://md.research.elamur.ai/free/en/authors/nitish-shirish-keskar/), [Bryan McCann](https://md.research.elamur.ai/free/en/authors/bryan-mccann/), Lav R. Varshney, [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/), [Richard Socher](https://md.research.elamur.ai/free/en/authors/richard-socher/) · **Year:** 2019 · **Venue:** arXiv:1909.05858 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1909.05858) · [PDF](https://arxiv.org/pdf/1909.05858) · [DOI 10.48550/arxiv.1909.05858](https://doi.org/10.48550/arxiv.1909.05858) · [arXiv 1909.05858](https://arxiv.org/abs/1909.05858) · [OpenAlex](https://openalex.org/W2973049837)

**Citations:** 730 citations (openalex-snapshot · 2026-10-14) · [543](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Large pretrained language models produce fluent text, but users have little influence over its style, topic or purpose beyond writing a prompt. Keskar et al. address this with CTRL, a Transformer language model with 1.63 billion parameters. It always conditions on a control code placed before the text. The authors do not label data by hand. Instead, they derive the codes from metadata that already comes with web-scale corpora: source domain (Wikipedia, Project Gutenberg, Amazon Reviews, individual subreddits), URLs, review ratings, and markers for tasks such as question answering and machine translation. Training uses 140 GB of filtered text and the usual next-token objective. The model therefore stays general while gaining explicit steering controls, and these can be combined, for example a task code with a domain code. Every code points to a specific slice of the training data. This lets the model run in reverse and estimate which source most plausibly produced a given text, a form of model-based source attribution useful for corpus analysis. The authors publicly release full-sized pretrained checkpoints, so others can reuse the model for research on controllable generation.

## Key points
- **Architecture:** a 48-layer Transformer with model dimension 1280, feedforward size 8192 and 16 attention heads. It uses a BPE vocabulary of about 250K tokens built with fastBPE, roughly four times larger than comparable models. Training sequences were 256 or 512 tokens long.
- **Training data:** 180 GB was collected and filtered down to 140 GB. Sources include:
  - Wikipedia (En, De, Es, Fr) and Project Gutenberg
  - 45 subreddits and OpenWebText
  - news corpora (CNN/DailyMail, New York Times, Newsroom) and Amazon Reviews
  - WMT Europarl/UN translation pairs (En-De, En-Es, En-Fr)
  - question-answer pairs from ELI5 and MRQA (SQuAD, NewsQA, TriviaQA, SearchQA, HotpotQA, Natural Questions)
- **Control code design:** a domain code is always the first token of each sequence. Secondary codes can follow it: `Rating:` (1.0–5.0) for Reviews, a full or partial URL for Links, and `Title:`/`Text:` for Reddit data.
- **Training setup:** TensorFlow, running on a Cloud TPU v3 Pod slice with 256 cores, with batch size 1024, for 800k iterations (about 2 weeks). The optimizer was Adagrad with linear warmup to 0.05 and gradient clipping at 0.25. Adagrad matched Adam's convergence while using considerably less memory.
- **Release:** pretrained models are available at github.com/salesforce/ctrl.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

543 papers of our library cite it (of the 140673 whose bibliographies we read); 52 of them have a page here, 491 not yet.

- [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/)
- [Efficient Training of Language Models to Fill in the Middle](https://md.research.elamur.ai/free/en/papers/bavarian-2022-efficient-training-language-models-fill/)
- [Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data](https://md.research.elamur.ai/free/en/papers/bender-2020-climbing-nlu-meaning-form-understanding/)
- [Experience Grounds Language](https://md.research.elamur.ai/free/en/papers/bisk-2020-experience-grounds-language/)
- [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/)
- [TLDR: Extreme Summarization of Scientific Documents](https://md.research.elamur.ai/free/en/papers/cachola-2020-tldr-extreme-summarization-scientific-documents/)
- [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://md.research.elamur.ai/free/en/papers/chuang-2023-dola-decoding-contrasting-layers-improves/)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/)
- [RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/deng-2022-rlprompt-optimizing-discrete-text-prompts/)
- [8-bit Optimizers via Block-wise Quantization](https://md.research.elamur.ai/free/en/papers/dettmers-2021-8-bit-optimizers-block-wise/)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://md.research.elamur.ai/free/en/papers/dziri-2022-origin-hallucinations-conversational-models-is/)
- [Facilitating the Communication of Politeness through Fine-Grained Paraphrasing](https://md.research.elamur.ai/free/en/papers/fu-2020-facilitating-communication-politeness-through-fine/)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://md.research.elamur.ai/free/en/papers/jawahar-2020-automatic-detection-machine-generated-text/)
- [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/)
- [Pretraining Language Models with Human Preferences](https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://md.research.elamur.ai/free/en/papers/krause-2021-gedi-generative-discriminator-guided-sequence/)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://md.research.elamur.ai/free/en/papers/krishna-2020-reformulating-unsupervised-style-transfer-as/)
- [Data Augmentation using Pre-trained Transformer Models](https://md.research.elamur.ai/free/en/papers/kumar-2020-data-augmentation-using-pre-trained/)
- [Diffusion-LM Improves Controllable Text Generation](https://md.research.elamur.ai/free/en/papers/li-2022-diffusion-lm-improves-controllable-text/)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://md.research.elamur.ai/free/en/papers/liu-2021-dexperts-decoding-time-controlled-text/)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer](https://md.research.elamur.ai/free/en/papers/liu-2023-chatgpt-vs-human-authored-text/)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://md.research.elamur.ai/free/en/papers/lu-2022-quark-controllable-text-generation-reinforced/)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://md.research.elamur.ai/free/en/papers/meng-2022-generating-training-data-language-models/)
- [Teaching language models to support answers with verified quotes](https://md.research.elamur.ai/free/en/papers/menick-2022-teaching-language-models-support-answers/)
- [Training Question Answering Models From Synthetic Data](https://md.research.elamur.ai/free/en/papers/puri-2020-training-question-answering-models-synthetic/)
- [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://md.research.elamur.ai/free/en/papers/reynolds-2021-prompt-programming-large-language-models/)
- [Recipes for Building an Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://md.research.elamur.ai/free/en/papers/sap-2020-social-bias-frames-reasoning-about/)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/)
- [Large Language Model Alignment: A Survey](https://md.research.elamur.ai/free/en/papers/shen-2023-large-language-model-alignment-survey/)
- [Towards Controllable Biases in Language Generation](https://md.research.elamur.ai/free/en/papers/sheng-2020-controllable-biases-language-generation/)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://md.research.elamur.ai/free/en/papers/shumailov-2023-curse-recursion-training-generated-data/)
- [Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering](https://md.research.elamur.ai/free/en/papers/siriwardhana-2023-improving-domain-adaptation-retrieval-augmented/)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://md.research.elamur.ai/free/en/papers/smith-2020-can-you-put-it-all/)
- [Release Strategies and the Social Impacts of Language Models](https://md.research.elamur.ai/free/en/papers/solaiman-2019-release-strategies-social-impacts-language/)
- [Text Classification via Large Language Models](https://md.research.elamur.ai/free/en/papers/sun-2023-text-classification-large-language-models/)
- [Authorship Attribution for Neural Text Generation](https://md.research.elamur.ai/free/en/papers/uchendu-2020-authorship-attribution-neural-text-generation/)
- [RSA-Control: A Pragmatics-Grounded Lightweight Controllable Text Generation Framework](https://md.research.elamur.ai/free/en/papers/wang-2024-rsa-control-pragmatics-grounded-lightweight/)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://md.research.elamur.ai/free/en/papers/wu-2020-tod-bert-pre-trained-natural/)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://md.research.elamur.ai/free/en/papers/wu-2021-polyjuice-generating-counterfactuals-explaining-evaluating/)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://md.research.elamur.ai/free/en/papers/yang-2021-fudge-controlled-text-generation-future/)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/)
- [Persona-Aware Evaluation of Cognitive Bias in LLMs: From Benchmark to Applied Decision-Making](https://md.research.elamur.ai/free/en/papers/yoshikawa-2026-persona-aware-evaluation-cognitive-bias/)
- [Secrets of RLHF in Large Language Models Part I: PPO](https://md.research.elamur.ai/free/en/papers/zheng-2023-secrets-rlhf-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{keskar2019ctrl,
  title = {{CTRL: A Conditional Transformer Language Model for Controllable Generation}},
  author = {Keskar, Nitish Shirish and McCann, Bryan and Varshney, Lav R. and Xiong, Caiming and Socher, Richard},
  year = {2019},
  doi = {10.48550/arxiv.1909.05858},
  eprint = {1909.05858},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.05858}
}
```

- [← previous Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task](https://md.research.elamur.ai/free/en/papers/yung-2019-crowdsourcing-discourse-relation-annotations-two/)
- [next → Currents in the Historical Linguistics and Linguistic Dating of the Hebrew Bible: Report on the State of Research as Reflected in Recent Major Publications](https://md.research.elamur.ai/free/en/papers/hbo-2019-rezetko-currents-historical-linguistics-linguistic-dating/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

86 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Annotation and benchmarking on understanding and transparency of machine learning lifecycles (ABOUT ML), 2019. URL [https://www.partnershiponai.org/about-ml/](https://www.partnershiponai.org/about-ml/). Partnership on AI (PAI), v0. [↗](https://www.partnershiponai.org/about-ml/)
2. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pp.… [find](https://scholar.google.com/scholar?q=Mart%C4%B1%CC%81n+Abadi%2C+Paul+Barham%2C+Jianmin+Chen%2C+Zhifeng+Chen%2C+Andy+Davis%2C+Jeffrey+Dean%2C+Matthieu+Devin%2C+Sanjay+Ghemawat%2C+Geoffrey+Irving%2C+Michael+Isard%2C+et+al%2E+Tensorflow%3A+A+system+for+large%2Dscale+machine+l)
3. Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory-efficient adaptive optimization for large-scale learning. arXiv preprint arXiv:1901.11150, 2019. — [arXiv:1901.11150](https://arxiv.org/abs/1901.11150) [find](https://scholar.google.com/scholar?q=Rohan+Anil%2C+Vineet+Gupta%2C+Tomer+Koren%2C+and+Yoram+Singer%2E+Memory%2Defficient+adaptive+optimization+for+large%2Dscale+learning%2E+arXiv+preprint+arXiv%3A1901%2E11150%2C+2019%2E+%E2%80%94+%5BarXiv%3A1901%2E11150%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
4. Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214–223, 2017. [find](https://scholar.google.com/scholar?q=Martin+Arjovsky%2C+Soumith+Chintala%2C+and+Le%CC%81on+Bottou%2E+Wasserstein+generative+adversarial+networks%2E+In+International+conference+on+machine+learning%2C+pp%2E+214%E2%80%93223%2C+2017%2E)
5. Matthew Arnold, Rachel K. E. Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilovic, Ravi Nair, Karthikeyan Natesan Ramamurthy, Darrell Reimer, Alexandra Olteanu, David Piorkowski, Jason Tsay, and Kush R. Varshney. Factsheets: Increasing trust in AI services through supplier’s d… [find](https://scholar.google.com/scholar?q=Matthew+Arnold%2C+Rachel+K%2E+E%2E+Bellamy%2C+Michael+Hind%2C+Stephanie+Houde%2C+Sameep+Mehta%2C+Aleksandra+Mojsilovic%2C+Ravi+Nair%2C+Karthikeyan+Natesan+Ramamurthy%2C+Darrell+Reimer%2C+Alexandra+Olteanu%2C+David+Piorkowski)
6. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041, 2017. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe%2C+Gorka+Labaka%2C+Eneko+Agirre%2C+and+Kyunghyun+Cho%2E+Unsupervised+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1710%2E11041%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/)
7. Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. [find](https://scholar.google.com/scholar?q=Jimmy+Ba%2C+Ryan+Kiros%2C+and+Geoffrey+E%2E+Hinton%2E+Layer+normalization%2E+CoRR%2C+abs%2F1607%2E06450%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
8. Loı̈c Barrault, Ondřej Bojar, Marta R Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al. Findings of the 2019 conference on machine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Vo… [find](https://scholar.google.com/scholar?q=Lo%C4%B1%CC%88c+Barrault%2C+Ondr%CC%8Cej+Bojar%2C+Marta+R+Costa%2Djussa%CC%80%2C+Christian+Federmann%2C+Mark+Fishel%2C+Yvette+Graham%2C+Barry+Haddow%2C+Matthias+Huck%2C+Philipp+Koehn%2C+Shervin+Malmasi%2C+et+al%2E+Findings+of+the+2019+conferenc) [our card](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/)
9. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003. [find](https://scholar.google.com/scholar?q=Yoshua+Bengio%2C+Re%CC%81jean+Ducharme%2C+Pascal+Vincent%2C+and+Christian+Jauvin%2E+A+neural+probabilistic+language+model%2E+Journal+of+machine+learning+research%2C+3%28Feb%29%3A1137%E2%80%931155%2C+2003%2E)
10. Thorsten Brants, Ashok C Popat, Peng Xu, Franz J Och, and Jeffrey Dean. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 858–867, 2007. [find](https://scholar.google.com/scholar?q=Thorsten+Brants%2C+Ashok+C+Popat%2C+Peng+Xu%2C+Franz+J+Och%2C+and+Jeffrey+Dean%2E+Large+language+models+in+machine+translation%2E+In+Proceedings+of+the+2007+Joint+Conference+on+Empirical+Methods+in+Natural+Langua) [our card](https://md.research.elamur.ai/free/en/papers/brants-2007-large-language-models-machine-translation/)
11. Miles Brundage. Artificial intelligence and responsible innovation. In Vincent C. Müller (ed.), Fundamental Issues of Artificial Intelligence, pp. 543–554. Springer, 2016. [find](https://scholar.google.com/scholar?q=Miles+Brundage%2E+Artificial+intelligence+and+responsible+innovation%2E+In+Vincent+C%2E+Mu%CC%88ller+%28ed%2E%29%2C+Fundamental+Issues+of+Artificial+Intelligence%2C+pp%2E+543%E2%80%93554%2E+Springer%2C+2016%2E)
12. Miles Brundage, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, Hyrum Anderson, Heather Roff, Gregory C. Allen, Jacob Steinhardt, Carrick Flynn, Seán Ó hÉigeartaigh, Simon Beard, Haydn Belfield, Sebastian Farquhar, Cla… — [arXiv:1802.07228](https://arxiv.org/abs/1802.07228) [find](https://scholar.google.com/scholar?q=Miles+Brundage%2C+Shahar+Avin%2C+Jack+Clark%2C+Helen+Toner%2C+Peter+Eckersley%2C+Ben+Garfinkel%2C+Allan+Dafoe%2C+Paul+Scharre%2C+Thomas+Zeitzoff%2C+Bobby+Filar%2C+Hyrum+Anderson%2C+Heather+Roff%2C+Gregory+C%2E+Allen%2C+Jacob+Ste)
13. Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged back-translation. arXiv preprint arXiv:1906.06442, 2019. [find](https://scholar.google.com/scholar?q=Isaac+Caswell%2C+Ciprian+Chelba%2C+and+David+Grangier%2E+Tagged+back%2Dtranslation%2E+arXiv+preprint+arXiv%3A1906%2E06442%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/)
14. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172–2180, 2016. [find](https://scholar.google.com/scholar?q=Xi+Chen%2C+Yan+Duan%2C+Rein+Houthooft%2C+John+Schulman%2C+Ilya+Sutskever%2C+and+Pieter+Abbeel%2E+Infogan%3A+Interpretable+representation+learning+by+information+maximizing+generative+adversarial+nets%2E+In+Advances+i)
15. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. [find](https://scholar.google.com/scholar?q=Rewon+Child%2C+Scott+Gray%2C+Alec+Radford%2C+and+Ilya+Sutskever%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2C+2019%2E)
16. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160–167. ACM, 2008. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+international+conference+on+Machine+l)
17. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493–2537, 2011. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+machine+learning+research%2C+12%28Aug%29%3A2493%E2%80%932) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
18. Ruth Schwartz Cowan. The consumption junction: A proposal for research strategies in the sociology of technology. In Wiebe E. Bijker, Thomas P. Hughes, and Trevor J. Pinch (eds.), The Social Construction of Technological Systems, pp. 261–280. MIT Press, Cambridge, MA, USA, 1987. [find](https://scholar.google.com/scholar?q=Ruth+Schwartz+Cowan%2E+The+consumption+junction%3A+A+proposal+for+research+strategies+in+the+sociology+of+technology%2E+In+Wiebe+E%2E+Bijker%2C+Thomas+P%2E+Hughes%2C+and+Trevor+J%2E+Pinch+%28eds%2E%29%2C+The+Social+Construct)
19. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079–3087, 2015. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3079%E2%80%933087%2C+2015%2E)
20. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+William+W+Cohen%2C+Jaime+Carbonell%2C+Quoc+V+Le%2C+and+Ruslan+Salakhutdinov%2E+Transformer%2Dxl%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+arXiv+preprint+arX)
21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+arXiv+preprint+arXiv%3A1810%2E04805%2C+2018%2E)
22. John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159, 2011. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+Journal+of+Machine+Learning+Research%2C+12%28Jul%29%3A2121%E2%80%932159%2C+2011%2E)
23. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017. [find](https://scholar.google.com/scholar?q=Matthew+Dunn%2C+Levent+Sagun%2C+Mike+Higgins%2C+V+Ugur+Guney%2C+Volkan+Cirik%2C+and+Kyunghyun+Cho%2E+Searchqa%3A+A+new+q%26a+dataset+augmented+with+context+from+a+search+engine%2E+arXiv+preprint+arXiv%3A1704%2E05179%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/)
24. Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. arXiv preprint arXiv:1805.04833, 2018. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Mike+Lewis%2C+and+Yann+Dauphin%2E+Hierarchical+neural+story+generation%2E+arXiv+preprint+arXiv%3A1805%2E04833%2C+2018%2E)
25. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190, 2019. [find](https://scholar.google.com/scholar?q=Angela+Fan%2C+Yacine+Jernite%2C+Ethan+Perez%2C+David+Grangier%2C+Jason+Weston%2C+and+Michael+Auli%2E+Eli5%3A+Long+form+question+answering%2E+arXiv+preprint+arXiv%3A1907%2E09190%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/fan-2019-eli5-long-form-question-answering/)
26. Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M Cohen. Stochastic gradient methods with layer-wise adaptive moments for training of deep networks. arXiv preprint arXiv:1905.11286, 2019. — [arXiv:1905.11286](https://arxiv.org/abs/1905.11286) [find](https://scholar.google.com/scholar?q=Boris+Ginsburg%2C+Patrice+Castonguay%2C+Oleksii+Hrinchuk%2C+Oleksii+Kuchaiev%2C+Vitaly+Lavrukhin%2C+Ryan+Leary%2C+Jason+Li%2C+Huyen+Nguyen%2C+and+Jonathan+M+Cohen%2E+Stochastic+gradient+methods+with+layer%2Dwise+adaptive)
27. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014. [find](https://scholar.google.com/scholar?q=Ian+Goodfellow%2C+Jean+Pouget%2DAbadie%2C+Mehdi+Mirza%2C+Bing+Xu%2C+David+Warde%2DFarley%2C+Sherjil+Ozair%2C+Aaron+Courville%2C+and+Yoshua+Bengio%2E+Generative+adversarial+nets%2E+In+Advances+in+neural+information+processi)
28. Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 708–719, New Orleans, Louisi… [find](https://scholar.google.com/scholar?q=Max+Grusky%2C+Mor+Naaman%2C+and+Yoav+Artzi%2E+Newsroom%3A+A+dataset+of+1%2E3+million+summaries+with+diverse+extractive+strategies%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
29. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587, 2016. [find](https://scholar.google.com/scholar?q=Kazuma+Hashimoto%2C+Caiming+Xiong%2C+Yoshimasa+Tsuruoka%2C+and+Richard+Socher%2E+A+joint+many%2Dtask+model%3A+Growing+a+neural+network+for+multiple+nlp+tasks%2E+arXiv+preprint+arXiv%3A1611%2E01587%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
30. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+Deep+residual+learning+for+image+recognition%2E+In+Proceedings+of+the+IEEE+conference+on+computer+vision+and+pattern+recognition%2C+pp%2E+770%E2%80%93778%2C+2016)
31. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693–1701, 2015. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+neural+information+proce) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
32. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+The+curious+case+of+neural+text+degeneration%2E+arXiv+preprint+arXiv%3A1904%2E09751%2C+2019%2E)
33. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+arXiv+preprint+arXiv%3A1801%2E06146%2C+2018%2E)
34. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462, 2016. [find](https://scholar.google.com/scholar?q=Hakan+Inan%2C+Khashayar+Khosravi%2C+and+Richard+Socher%2E+Tying+word+vectors+and+word+classifiers%3A+A+loss+framework+for+language+modeling%2E+arXiv+preprint+arXiv%3A1611%2E01462%2C+2016%2E)
35. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. Googles multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Melvin+Johnson%2C+Mike+Schuster%2C+Quoc+V+Le%2C+Maxim+Krikun%2C+Yonghui+Wu%2C+Zhifeng+Chen%2C+Nikhil+Thorat%2C+Fernanda+Vie%CC%81gas%2C+Martin+Wattenberg%2C+Greg+Corrado%2C+et+al%2E+Googles+multilingual+neural+machine+translati)
36. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+arXiv+preprint+arXiv%3A1705%2E03551%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
37. David Kaiser and Jonathan Moreno. Self-censorship is not enough. Nature, 492(7429):345–347, December 2012. doi: 10.1038/492345a. — [doi:10.1038/492345a](https://doi.org/10.1038/492345a) [find](https://scholar.google.com/scholar?q=David+Kaiser+and+Jonathan+Moreno%2E+Self%2Dcensorship+is+not+enough%2E+Nature%2C+492%287429%29%3A345%E2%80%93347%2C+December+2012%2E+doi%3A+10%2E1038%2F492345a%2E+%E2%80%94+%5Bdoi%3A10%2E1038%2F492345a%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1038%2F492345a%29)
38. Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017. — [arXiv:1706.05137](https://arxiv.org/abs/1706.05137) [find](https://scholar.google.com/scholar?q=Lukasz+Kaiser%2C+Aidan+N+Gomez%2C+Noam+Shazeer%2C+Ashish+Vaswani%2C+Niki+Parmar%2C+Llion+Jones%2C+and+Jakob+Uszkoreit%2E+One+model+to+learn+them+all%2E+arXiv+preprint+arXiv%3A1706%2E05137%2C+2017%2E+%E2%80%94+%5BarXiv%3A1706%2E05137%5D%28http)
39. Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. arXiv preprint arXiv:1803.03382, 2018. — [arXiv:1803.03382](https://arxiv.org/abs/1803.03382) [find](https://scholar.google.com/scholar?q=%C5%81ukasz+Kaiser%2C+Aurko+Roy%2C+Ashish+Vaswani%2C+Niki+Parmar%2C+Samy+Bengio%2C+Jakob+Uszkoreit%2C+and+Noam+Shazeer%2E+Fast+decoding+in+sequence+models+using+discrete+latent+variables%2E+arXiv+preprint+arXiv%3A1803%2E03382)
40. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286, 2019. — [arXiv:1904.09286](https://arxiv.org/abs/1904.09286) [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Unifying+question+answering+and+text+classification+via+span+extraction%2E+arXiv+preprint+arXiv%3A1904%2E09286%2C+2019%2E+%E2%80%94+%5BarXiv%3A1904%2E09)
41. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2C+2014%2E)
42. Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Max+Welling%2E+arXiv%3A1312%2E6114%2C+2013%2E) [our card](https://md.research.elamur.ai/free/en/papers/kingma-2013-auto-encoding-variational-bayes/)
43. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3294%E2%80%933)
44. Catherine Kobus, Josep Crego, and Jean Senellart. Domain control for neural machine translation. arXiv preprint arXiv:1612.06140, 2016. — [arXiv:1612.06140](https://arxiv.org/abs/1612.06140) [find](https://scholar.google.com/scholar?q=Catherine+Kobus%2C+Josep+Crego%2C+and+Jean+Senellart%2E+Domain+control+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A1612%2E06140%2C+2016%2E+%E2%80%94+%5BarXiv%3A1612%2E06140%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E06140%29)
45. Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. arXiv preprint arXiv:1908.08960, 2019. [find](https://scholar.google.com/scholar?q=Wojciech+Krys%CC%81cin%CC%81ski%2C+Nitish+Shirish+Keskar%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Neural+text+summarization%3A+A+critical+evaluation%2E+arXiv+preprint+arXiv%3A1908%2E08960%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/kryscinski-2019-neural-text-summarization-critical-evaluation/)
46. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:4… [find](https://scholar.google.com/scholar?q=Tom+Kwiatkowski%2C+Jennimaria+Palomaki%2C+Olivia+Redfield%2C+Michael+Collins%2C+Ankur+Parikh%2C+Chris+Alberti%2C+Danielle+Epstein%2C+Illia+Polosukhin%2C+Jacob+Devlin%2C+Kenton+Lee%2C+et+al%2E+Natural+questions%3A+a+benchmark) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/)
47. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2C+2019%2E)
48. Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. arXiv preprint arXiv:1907.05242, 2019. — [arXiv:1907.05242](https://arxiv.org/abs/1907.05242) [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Alexandre+Sablayrolles%2C+Marc%E2%80%99Aurelio+Ranzato%2C+Ludovic+Denoyer%2C+and+Herve%CC%81+Je%CC%81gou%2E+Large+memory+layers+with+product+keys%2E+arXiv+preprint+arXiv%3A1907%2E05242%2C+2019%2E+%E2%80%94+%5BarXiv%3A1907%2E05242%5D%28h)
49. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+The+winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+2012%2E)
50. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. Unsupervised question answering by cloze translation. arXiv preprint arXiv:1906.04980, 2019. [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Ludovic+Denoyer%2C+and+Sebastian+Riedel%2E+Unsupervised+question+answering+by+cloze+translation%2E+arXiv+preprint+arXiv%3A1906%2E04980%2C+2019%2E)
51. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015. [find](https://scholar.google.com/scholar?q=Minh%2DThang+Luong%2C+Quoc+V+Le%2C+Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Lukasz+Kaiser%2E+Multi%2Dtask+sequence+to+sequence+learning%2E+arXiv+preprint+arXiv%3A1511%2E06114%2C+2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-multi-task-sequence-sequence-learning/)
52. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 43–52. ACM, 2015. [find](https://scholar.google.com/scholar?q=Julian+McAuley%2C+Christopher+Targett%2C+Qinfeng+Shi%2C+and+Anton+Van+Den+Hengel%2E+Image%2Dbased+recommendations+on+styles+and+substitutes%2E+In+Proceedings+of+the+38th+International+ACM+SIGIR+Conference+on+Rese)
53. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294– 6305, 2017. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pp%2E+6294%E2%80%93+6305%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
54. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+Nitish+Shirish+Keskar%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+The+natural+language+decathlon%3A+Multitask+learning+as+question+answering%2E+arXiv+preprint+arXiv%3A1806%2E08730%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/)
55. Auto-encoding variational bayes. arXiv preprint Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing lstm language models. arXiv preprint arXiv:1708.02182, 2017. [find](https://scholar.google.com/scholar?q=Auto%2Dencoding+variational+bayes%2E+arXiv+preprint+Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+Regularizing+and+optimizing+lstm+language+models%2E+arXiv+preprint+arXiv%3A1708%2E02182%2C+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
56. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+neural+information+processing+system)
57. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19), January 2019. doi: 10.1145… [find](https://scholar.google.com/scholar?q=Margaret+Mitchell%2C+Simone+Wu%2C+Andrew+Zaldivar%2C+Parker+Barnes%2C+Lucy+Vasserman%2C+Ben+Hutchinson%2C+Elena+Spitzer%2C+Inioluwa+Deborah+Raji%2C+and+Timnit+Gebru%2E+Model+cards+for+model+reporting%2E+In+Proceedings+of)
58. Amit Moryossef, Roee Aharoni, and Yoav Goldberg. Filling gender & number gaps in neural machine translation with black-box context injection. arXiv preprint arXiv:1903.03467, 2019. [find](https://scholar.google.com/scholar?q=Amit+Moryossef%2C+Roee+Aharoni%2C+and+Yoav+Goldberg%2E+Filling+gender+%26+number+gaps+in+neural+machine+translation+with+black%2Dbox+context+injection%2E+arXiv+preprint+arXiv%3A1903%2E03467%2C+2019%2E)
59. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 807–814, 2010. [find](https://scholar.google.com/scholar?q=Vinod+Nair+and+Geoffrey+E+Hinton%2E+Rectified+linear+units+improve+restricted+boltzmann+machines%2E+In+Proceedings+of+the+27th+International+Conference+on+Machine+Learning+%28ICML%2D10%29%2C+pp%2E+807%E2%80%93814%2C+2010%2E)
60. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023, 2016. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Caglar+Gulcehre%2C+Bing+Xiang%2C+et+al%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+arXiv+preprint+arXiv%3A1602%2E06023%2C+2016%2E)
61. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [find](https://scholar.google.com/scholar?q=Matthew+E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+Deep+contextualized+word+representations%2E+arXiv+preprint+arXiv%3A1802%2E05365%2C+2018%2E)
62. Carol W Pfaff. Constraints on language mixing: intrasentential code-switching and borrowing in spanish/english. Language, pp. 291–318, 1979. [find](https://scholar.google.com/scholar?q=Carol+W+Pfaff%2E+Constraints+on+language+mixing%3A+intrasentential+code%2Dswitching+and+borrowing+in+spanish%2Fenglish%2E+Language%2C+pp%2E+291%E2%80%93318%2C+1979%2E)
63. Shana Poplack. Sometimes ill start a sentence in spanish y termino en espanol: toward a typology of code-switching1. Linguistics, 18(7-8):581–618, 1980. [find](https://scholar.google.com/scholar?q=Shana+Poplack%2E+Sometimes+ill+start+a+sentence+in+spanish+y+termino+en+espanol%3A+toward+a+typology+of+code%2Dswitching1%2E+Linguistics%2C+18%287%2D8%29%3A581%E2%80%93618%2C+1980%2E)
64. Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. [find](https://scholar.google.com/scholar?q=Ofir+Press+and+Lior+Wolf%2E+Using+the+output+embedding+to+improve+language+models%2E+arXiv+preprint+arXiv%3A1608%2E05859%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/press-2017-output-embedding-improve-language-models/)
65. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL [https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/langu](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/langu) ageunsupervised/language understand ing paper.pdf, 2018. [↗](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/langu)
66. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL [https://d4mucfpksywv.cloudfront.net/better-language-models/language](https://d4mucfpksywv.cloudfront.net/better-language-models/language) mo dels are unsupervised multitask learn ers.pdf, 2019. [↗](https://d4mucfpksywv.cloudfront.net/better-language-models/language)
67. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361, 2019. [find](https://scholar.google.com/scholar?q=Nazneen+Fatema+Rajani%2C+Bryan+McCann%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+Explain+yourself%21+leveraging+language+models+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1906%2E02361%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/)
68. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+arXiv+preprint+arXiv%3A1606%2E05250%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
69. Alexander M Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+arXiv+preprint+arXiv%3A1509%2E00685%2C+2015%2E)
70. Evan Sandhaus. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752, 2008. [find](https://scholar.google.com/scholar?q=Evan+Sandhaus%2E+The+new+york+times+annotated+corpus%2E+Linguistic+Data+Consortium%2C+Philadelphia%2C+6%2812%29%3Ae26752%2C+2008%2E)
71. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Answers unite! unsupervised metrics for reinforced summarization models. arXiv preprint arXiv:1909.01610, 2019. [find](https://scholar.google.com/scholar?q=Thomas+Scialom%2C+Sylvain+Lamprier%2C+Benjamin+Piwowarski%2C+and+Jacopo+Staiano%2E+Answers+unite%21+unsupervised+metrics+for+reinforced+summarization+models%2E+arXiv+preprint+arXiv%3A1909%2E01610%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/scialom-2019-answers-unite-unsupervised-metrics-reinforced/)
72. Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1073–1083, 2017. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J+Liu%2C+and+Christopher+D+Manning%2E+Get+to+the+point%3A+Summarization+with+pointer%2Dgenerator+networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Lin)
73. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+Neural+machine+translation+of+rare+words+with+subword+units%2E+arXiv+preprint+arXiv%3A1508%2E07909%2C+2015%2E)
74. Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+arXiv+preprint+arXiv%3A1804%2E04235%2C+2018%2E)
75. Jack Stilgoe, Richard Owen, and Phil Macnaghten. Developing a framework for responsible innovation. Research Policy, 42(9):1568–1580, November 2013. doi: 10.1016/j.respol.2013.05.008. — [doi:10.1016/j.respol.2013.05.008](https://doi.org/10.1016/j.respol.2013.05.008) [find](https://scholar.google.com/scholar?q=Jack+Stilgoe%2C+Richard+Owen%2C+and+Phil+Macnaghten%2E+Developing+a+framework+for+responsible+innovation%2E+Research+Policy%2C+42%289%29%3A1568%E2%80%931580%2C+November+2013%2E+doi%3A+10%2E1016%2Fj%2Erespol%2E2013%2E05%2E008%2E+%E2%80%94+%5Bdoi%3A10%2E1016%2Fj)
76. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+neural+information+processing+systems%2C+pp%2E+3104%E2%80%933112%2C+2014%2E)
77. Trieu H Trinh and Quoc V Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. [find](https://scholar.google.com/scholar?q=Trieu+H+Trinh+and+Quoc+V+Le%2E+A+simple+method+for+commonsense+reasoning%2E+arXiv+preprint+arXiv%3A1806%2E02847%2C+2018%2E)
78. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830, 2016. [find](https://scholar.google.com/scholar?q=Adam+Trischler%2C+Tong+Wang%2C+Xingdi+Yuan%2C+Justin+Harris%2C+Alessandro+Sordoni%2C+Philip+Bachman%2C+and+Kaheer+Suleman%2E+Newsqa%3A+A+machine+comprehension+dataset%2E+arXiv+preprint+arXiv%3A1611%2E09830%2C+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/)
79. Lav R. Varshney, Nitish Shirish Keskar, and Richard Socher. Pretrained AI models: Performativity, mobility, and change, September 2019. arXiv:1909.03290 [cs.CY]. — [arXiv:1909.03290](https://arxiv.org/abs/1909.03290) [find](https://scholar.google.com/scholar?q=Lav+R%2E+Varshney%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+Pretrained+AI+models%3A+Performativity%2C+mobility%2C+and+change%2C+September+2019%2E+arXiv%3A1909%2E03290+%5Bcs%2ECY%5D%2E+%E2%80%94+%5BarXiv%3A1909%2E03290%5D%28https%3A%2F%2Farxiv%2Eorg)
80. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sys… [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81+ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+Attention+is+all+you+need%2E+In+I%2E+Guyon%2C+U%2E+V%2E+Luxburg%2C+S%2E+Bengio%2C+H%2E+Walla)
81. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amapreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+R+Bowman%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+for+natural+language+understanding%2E+arXiv+preprint+arXiv%3A1804%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
82. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Ilia+Kulikov%2C+Stephen+Roller%2C+Emily+Dinan%2C+Kyunghyun+Cho%2C+and+Jason+Weston%2E+Neural+text+generation+with+unlikelihood+training%2E+arXiv+preprint+arXiv%3A1908%2E04319%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/)
83. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridging+th)
84. Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. Sumqe: a bert-based summary quality estimation model. arXiv preprint arXiv:1909.00578, 2019. — [arXiv:1909.00578](https://arxiv.org/abs/1909.00578) [find](https://scholar.google.com/scholar?q=Stratos+Xenouleas%2C+Prodromos+Malakasiotis%2C+Marianna+Apidianaki%2C+and+Ion+Androutsopoulos%2E+Sumqe%3A+a+bert%2Dbased+summary+quality+estimation+model%2E+arXiv+preprint+arXiv%3A1909%2E00578%2C+2019%2E+%E2%80%94+%5BarXiv%3A1909%2E0057)
85. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W+Cohen%2C+Ruslan+Salakhutdinov%2C+and+Christopher+D+Manning%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+arXiv+) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
86. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Hannah+Rashkin%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+Franziska+Roesner%2C+and+Yejin+Choi%2E+Defending+against+neural+fake+news%2E+arXiv+preprint+arXiv%3A1905%2E12616%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/)