---
title: "AlexaTM 20B — paper by Soltan et al. (2022)"
description: "Large language models that learn from a few examples in the prompt have mostly been decoder-only. 88 references, cited by 47 papers here."
authors: ["Saleh Soltan", "Shankar Ananthakrishnan", "Jack FitzGerald", "Rahul Gupta", "Wael Hamza", "Haidar Khan", "Charith Peris", "Stephen Rawls", "Andy Rosenbaum", "Anna Rumshisky", "Chandana Satya Prakash", "Mukund Sridhar", "Fabian Triefenbach", "Apurv Verma", "Gokhan Tur", "Prem Natarajan"]
year: 2022
venue: "arXiv:2208.01448"
source: "https://arxiv.org/abs/2208.01448"
doi: "10.48550/arxiv.2208.01448"
arxiv: "2208.01448"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

# AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model

**Authors:** Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, [Rahul Gupta](https://md.research.elamur.ai/free/en/authors/rahul-gupta/), [Wael Hamza](https://md.research.elamur.ai/free/en/authors/wael-hamza/), [Haidar Khan](https://md.research.elamur.ai/free/en/authors/haidar-khan/), [Charith Peris](https://md.research.elamur.ai/free/en/authors/charith-peris/), [Stephen Rawls](https://md.research.elamur.ai/free/en/authors/stephen-rawls/), [Andy Rosenbaum](https://md.research.elamur.ai/free/en/authors/andy-rosenbaum/), [Anna Rumshisky](https://md.research.elamur.ai/free/en/authors/anna-rumshisky/), Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, [Gokhan Tur](https://md.research.elamur.ai/free/en/authors/gokhan-tur/) and [Prem Natarajan](https://md.research.elamur.ai/free/en/authors/prem-natarajan/) · **Year:** 2022 · **Venue:** arXiv:2208.01448 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2208.01448) · [PDF](https://arxiv.org/pdf/2208.01448) · [DOI 10.48550/arxiv.2208.01448](https://doi.org/10.48550/arxiv.2208.01448) · [arXiv 2208.01448](https://arxiv.org/abs/2208.01448) · [OpenAlex](https://openalex.org/W4300506197)

**Citations:** 37 citations (OpenAlex, 2026-10-14) · [47](https://md.research.elamur.ai/free/en/papers/soltan-2022-alexatm-20b-few-shot-learning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models that learn from a few examples in the prompt have mostly been decoder-only. Their left-to-right attention can be a handicap when the input is long. This Amazon Alexa AI paper asks whether an encoder-decoder model can be an equally strong in-context learner.

The authors train AlexaTM 20B, a Pre-LN Transformer seq2seq model with about 20 billion parameters. It was trained on roughly one trillion tokens of Wikipedia and mC4 text in 12 languages. Pre-training combines two tasks: span denoising without mask tokens (80% of steps) and a Causal Language Modeling continuation task (20%). The encoder was initialized from an earlier 10B encoder.

Although much smaller than its rivals, the model matches or beats PaLM 540B on one-shot summarization. It also sets new one-shot translation results on Flores-101, especially for low-resource and non-English pairs, and surpasses GPT3 175B zero-shot on SuperGLUE and SQuADv2. The work suggests that mixed-objective seq2seq pre-training is a cheaper and competitive route to multilingual LLMs. Ukrainian is not among the covered languages.

## Key points
- **Data and setup**
  - Wikipedia supplies 119B tokens (9%) and mC4 supplies 1.2T tokens (91%).
  - Languages are rebalanced with α = 0.5 sampling, and Wikipedia is upsampled.
  - Text is mixed 7:3 in favor of a spoken-style format.
  - Tokenization uses a 150K unigram SentencePiece vocabulary.
- **Denoising objective:** 15% of tokens are dropped in Poisson-length spans (mean 3), with no [MASK] symbols. A [CLM] token tells the model to continue a text instead of reconstructing it.
- **Summarization and translation**
  - One-shot results are on par with or better than PaLM 540B on XSUM and MLSum (de, es).
  - On Flores-101, Arabic→Tamil reaches 21.8 spBLEU, compared with 0.9 for supervised M2M-124 615M.
- **Zero-shot multilingual tasks:** the model achieves the best reported results on XNLI, XCOPA, PAWS-X and XWinograd, beating XGLM 7.5B. It also leads on the Winogender bias benchmark.
- **Risks and cost**
  - Memorization drops slightly as context length grows.
  - Toxic prompts still raise the toxicity of continuations.
  - Training used 15,360 A100 GPU-days and about 98.2 tCO2e.
  - Weights are promised at github.com/amazon-research/alexa-teacher-models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🌍 Multilingual & cross-lingual NLP](https://md.research.elamur.ai/free/en/topics/multilingual-nlp/) · 882 works

## Cite

Cite as: Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur and Prem Natarajan. 2022. AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model. arXiv:2208.01448.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{soltan2022alexatm,
  title = {{AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model}},
  author = {Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur and Prem Natarajan},
  year = {2022},
  doi = {10.48550/arxiv.2208.01448},
  eprint = {2208.01448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.01448}
}
```

- [← previous Neuron-level Interpretation of Deep NLP Models: A Survey](https://md.research.elamur.ai/free/en/papers/sajjad-2022-neuron-level-interpretation-deep-nlp/)
- [next → Temporal Effects on Pre-trained Models for Language Processing Tasks](https://md.research.elamur.ai/free/en/papers/agarwal-2022-temporal-effects-pre-trained-models/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

88 references, 55 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bakshi, S., Batra, S., Heidari, P., Arun, A., Jain, S., and White, M. Structure-to-text generation with self-training, acceptability classifiers and context-conditioning for the gem shared task. In GEM, 2021. [find](https://scholar.google.com/scholar?q=Bakshi%2C+S%2E%2C+Batra%2C+S%2E%2C+Heidari%2C+P%2E%2C+Arun%2C+A%2E%2C+Jain%2C+S%2E%2C+and+White%2C+M%2E+Structure%2Dto%2Dtext+generation+with+self%2Dtraining%2C+acceptability+classifiers+and+context%2Dconditioning+for+the+gem+shared+task%2E+In+GE)
2. Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., and Hon, H.-W. [Unilmv2: Pseudo-masked language models for unified language model pre-training](https://md.research.elamur.ai/free/en/papers/bao-2020-unilmv2-pseudo-masked-language-models/). ArXiv, abs/2002.12804, 2020.
3. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454[–5476, Online, July 2020](https://md.research.elamur.ai/free/en/papers/blodgett-2020-language-technology-is-power-critical/). Association for Computational Linguistics. d…
4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+Agarwal%2C+S%2E%2C+Herbert%2DVoss%2C+A%2E%2C+Krueger%2C+G%2E%2C+Henighan%2C+T%2E%2C+Child%2C+R%2E%2C+Rames)
5. Curran Associates, Inc., 2020a. URL [https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf). [↗](https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf)
6. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T. J., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S.,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., and Zhang, C. Quantifying memorization across neural language models. ArXiv, abs/2202.07646, 2022. [find](https://scholar.google.com/scholar?q=Carlini%2C+N%2E%2C+Ippolito%2C+D%2E%2C+Jagielski%2C+M%2E%2C+Lee%2C+K%2E%2C+Trame%CC%80r%2C+F%2E%2C+and+Zhang%2C+C%2E+Quantifying+memorization+across+neural+language+models%2E+ArXiv%2C+abs%2F2202%2E07646%2C+2022%2E)
8. Chiu, J. and Zheng, S. Making deepspeed zero run efficiently on more-affordable hardware, Mar 2022. URL [https://www.amazon.science/blog/making-deepspeed-zero-run-efficiently-on-more-affordable-hardware](https://www.amazon.science/blog/making-deepspeed-zero-run-efficiently-on-more-affordable-hardware). [↗](https://www.amazon.science/blog/making-deepspeed-zero-run-efficiently-on-more-affordable-hardware)
9. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A. B., Barnes, P., Tay, Y., Shazeer, N. M., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B. C., Pope, R., Bradbury,… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
10. Conneau, A., Lample, G., Rinott, R., Williams, A., Bowman, S. R., Schwenk, H., and Stoyanov, V. Xnli: Evaluating cross-lingual sentence representations. In EMNLP, 2018. [find](https://scholar.google.com/scholar?q=Conneau%2C+A%2E%2C+Lample%2C+G%2E%2C+Rinott%2C+R%2E%2C+Williams%2C+A%2E%2C+Bowman%2C+S%2E+R%2E%2C+Schwenk%2C+H%2E%2C+and+Stoyanov%2C+V%2E+Xnli%3A+Evaluating+cross%2Dlingual+sentence+representations%2E+In+EMNLP%2C+2018%2E)
11. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. [Unsupervised cross-lingual representation learning at scale](https://md.research.elamur.ai/free/en/papers/conneau-2020-unsupervised-cross-lingual-representation-learning/). In ACL, 2020.
12. Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. [Plug and play language models: A simple approach to controlled text generation](https://md.research.elamur.ai/free/en/papers/dathathri-2019-plug-play-language-models-simple/), 2019.
13. Davy, B. Building an aws ec2 carbon emissions datase. [https://medium.com/teads-engineering/building-an-aws-ec2-carbon-emissions-dataset-3f0fd76c98ac](https://medium.com/teads-engineering/building-an-aws-ec2-carbon-emissions-dataset-3f0fd76c98ac), 2021. [↗](https://medium.com/teads-engineering/building-an-aws-ec2-carbon-emissions-dataset-3f0fd76c98ac)
14. Dev, S., Li, T., Phillips, J. M., and Srikumar, V. On Measuring and Mitigating Biased Inferences of Word Embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7659–7666, April 2020. doi: 10.1609/aaai.v34i05.6267. URL [https://ojs.aaai.org/index.php/AAAI/article/view/6267](https://ojs.aaai.org/index.php/AAAI/article/view/6267). — [doi:10.1609/aaai.v34i05.6267](https://doi.org/10.1609/aaai.v34i05.6267) [↗](https://ojs.aaai.org/index.php/AAAI/article/view/6267)
15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
16. Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., and Gupta, R. Bold: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021. [find](https://scholar.google.com/scholar?q=Dhamala%2C+J%2E%2C+Sun%2C+T%2E%2C+Kumar%2C+V%2E%2C+Krishna%2C+S%2E%2C+Pruksachatkun%2C+Y%2E%2C+Chang%2C+K%2E%2DW%2E%2C+and+Gupta%2C+R%2E+Bold%3A+Dataset+and+metrics+for+measuring+biases+in+open%2Dended+language+generation%2E+Proceedings+of+the+2021+A)
17. Dinan, E., Humeau, S., Chintagunta, B., and Weston, J. [Build it break it fix it for dialogue safety: Robustness from adversarial human attack](https://md.research.elamur.ai/free/en/papers/dinan-2019-build-it-break-it-fix/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proc…
18. Dinan, E., Fan, A., Williams, A., Urbanek, J., Kiela, D., and Weston, J. [Queens are powerful too: Mitigating gender bias in dialogue generation](https://md.research.elamur.ai/free/en/papers/dinan-2020-queens-are-powerful-too-mitigating/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8173–8188, Online, November 2020. Association for Co…
19. Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S. L., Hovy, D., Boureau, Y.-L., and Rieser, V. Anticipating safety issues in e2e conversational ai: Framework and tooling. ArXiv, abs/2107.03451, 2021. — [arXiv:2107.03451](https://arxiv.org/abs/2107.03451) [find](https://scholar.google.com/scholar?q=Dinan%2C+E%2E%2C+Abercrombie%2C+G%2E%2C+Bergman%2C+A%2E+S%2E%2C+Spruit%2C+S%2E+L%2E%2C+Hovy%2C+D%2E%2C+Boureau%2C+Y%2E%2DL%2E%2C+and+Rieser%2C+V%2E+Anticipating+safety+issues+in+e2e+conversational+ai%3A+Framework+and+tooling%2E+ArXiv%2C+abs%2F2107%2E03451%2C+2)
20. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. [Unified language model pre-training for natural language understanding and generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). ArXiv, abs/1905.03197, 2019.
21. Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., Zoph, B., Fedus, L., Bosma, M., Zhou, Z., Wang, T., Wang, Y. E., Webster, K., Pellat, M., Robinson, K., Meier-Hellstern, K. S., Duke, T., Dixon, L., Zhang, K., Le, Q. V., Wu, Y., Chen, Z., and… — [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://md.research.elamur.ai/free/en/papers/du-2021-glam-efficient-scaling-language-models/)
22. Dusek, O., Howcroft, D. M., and Rieser, V. [Semantic noise matters for neural natural language generation](https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/). ArXiv, abs/1911.03905, 2019.
23. Fan, A., Bhosale, S., Schwenk, H., Ma, Z., El-Kishky, A., Goyal, S., Baines, M., Çelebi, O., Wenzek, G., Chaudhary, V., Goyal, N., Birch, T., Liptchinsky, V., Edunov, S., Grave, E., Auli, M., and Joulin, A. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22:107:1–107:… [find](https://scholar.google.com/scholar?q=Fan%2C+A%2E%2C+Bhosale%2C+S%2E%2C+Schwenk%2C+H%2E%2C+Ma%2C+Z%2E%2C+El%2DKishky%2C+A%2E%2C+Goyal%2C+S%2E%2C+Baines%2C+M%2E%2C+C%CC%A7elebi%2C+O%2E%2C+Wenzek%2C+G%2E%2C+Chaudhary%2C+V%2E%2C+Goyal%2C+N%2E%2C+Birch%2C+T%2E%2C+Liptchinsky%2C+V%2E%2C+Edunov%2C+S%2E%2C+Grave%2C+E%2E%2C+Auli%2C+M%2E%2C+and+Jou)
24. Ferreira, T. C., Gardent, C., Ilinykh, N., van der Lee, C., Mille, S., Moussallem, D., and Shimorina, A. [The 2020 bilingual, bi-directional webnlg+ shared task: Overview and evaluation results (webnlg+ 2020)](https://md.research.elamur.ai/free/en/papers/ferreira-2020-2020-bilingual-bi-directional-webnlg/). In WEBNLG, 2020.
25. FitzGerald, J. G. M., Ananthakrishnan, S., Arkoudas, K., Bernardi, D., Bhagia, A., Bovi, C. D., Cao, J., Chada, R., Chauhan, A., Chen, L., Dwarakanath, A., Dwivedi, S., Gojayev, T., Gopalakrishnan, K., Gueudré, T., Hakkani-Tür, D. Z., Hamza, W., Hueser, J., Jose, K. M., Khan, H., Liu, B., Lu, J.,… — [arXiv:2206.07808](https://arxiv.org/abs/2206.07808) [find](https://scholar.google.com/scholar?q=FitzGerald%2C+J%2E+G%2E+M%2E%2C+Ananthakrishnan%2C+S%2E%2C+Arkoudas%2C+K%2E%2C+Bernardi%2C+D%2E%2C+Bhagia%2C+A%2E%2C+Bovi%2C+C%2E+D%2E%2C+Cao%2C+J%2E%2C+Chada%2C+R%2E%2C+Chauhan%2C+A%2E%2C+Chen%2C+L%2E%2C+Dwarakanath%2C+A%2E%2C+Dwivedi%2C+S%2E%2C+Gojayev%2C+T%2E%2C+Gopalakrishnan%2C+K%2E)
26. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. [Realtoxicityprompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). Findings of the Association for Computational Linguistics: EMNLP 2020, 2020. doi: 10.18653/v1/2020.findings-emnlp.301. URL http://dx.doi.org/10.18653/v1/2020.… [↗](http://dx.doi.org/10.18653/v1/2020.…)
27. Gehrmann, S., Adewumi, T. P., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., Du, W., Durmus, E., Dusek, O., Emezue, C. C., Gangal, V., Garbacea, C., Hashimoto, T. B., Hou, Y., Jernite, Y., Jhamtani, H., Ji, Y., Jolly, S., Kale, M.,… — [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://md.research.elamur.ai/free/en/papers/gehrmann-2021-gem-benchmark-natural-language-generation/)
28. Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., and Fan, A. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538, 2022. [find](https://scholar.google.com/scholar?q=Goyal%2C+N%2E%2C+Gao%2C+C%2E%2C+Chaudhary%2C+V%2E%2C+Chen%2C+P%2E%2DJ%2E%2C+Wenzek%2C+G%2E%2C+Ju%2C+D%2E%2C+Krishnan%2C+S%2E%2C+Ranzato%2C+M%2E%2C+Guzma%CC%81n%2C+F%2E%2C+and+Fan%2C+A%2E+The+flores%2D101+evaluation+benchmark+for+low%2Dresource+and+multilingual+machine+tr)
29. Gupta, U., Dhamala, J., Kumar, V., Verma, A., Pruksachatkun, Y., Krishna, S., Gupta, R., Chang, K.-W., Ver Steeg, G., and Galstyan, A. Mitigating gender bias in distilled language models via counterfactual role reversal. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 658… [find](https://scholar.google.com/scholar?q=Gupta%2C+U%2E%2C+Dhamala%2C+J%2E%2C+Kumar%2C+V%2E%2C+Verma%2C+A%2E%2C+Pruksachatkun%2C+Y%2E%2C+Krishna%2C+S%2E%2C+Gupta%2C+R%2E%2C+Chang%2C+K%2E%2DW%2E%2C+Ver+Steeg%2C+G%2E%2C+and+Galstyan%2C+A%2E+Mitigating+gender+bias+in+distilled+language+models+via+counterfa)
30. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
31. Honnibal, M. and Montani, I. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2017. [find](https://scholar.google.com/scholar?q=Honnibal%2C+M%2E+and+Montani%2C+I%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2C+2017%2E)
32. Izacard, G. and Grave, E. [Leveraging passage retrieval with generative models for open domain question answering](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/). In EACL, 2021.
33. Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private is private sgd? ArXiv, abs/2006.07709, 2020. — [arXiv:2006.07709](https://arxiv.org/abs/2006.07709) [find](https://scholar.google.com/scholar?q=Jagielski%2C+M%2E%2C+Ullman%2C+J%2E%2C+and+Oprea%2C+A%2E+Auditing+differentially+private+machine+learning%3A+How+private+is+private+sgd%3F+ArXiv%2C+abs%2F2006%2E07709%2C+2020%2E+%E2%80%94+%5BarXiv%3A2006%2E07709%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2006%2E0770)
34. Jayaraman, B. and Evans, D. Evaluating differentially private machine learning in practice. In 28th USENIX Security Symposium (USENIX Security 19), pp. 1895–1912, Santa Clara, CA, August 2019. USENIX Association. ISBN 978-1-939133-06-9. URL [https://www.usenix.org/conference/usenixsecurity19/presenta…](https://www.usenix.org/conference/usenixsecurity19/presenta…) [↗](https://www.usenix.org/conference/usenixsecurity19/presenta…)
35. Kandpal, N., Wallace, E., and Raffel, C. [Deduplicating training data mitigates privacy risks in language models](https://md.research.elamur.ai/free/en/papers/kandpal-2022-deduplicating-training-data-mitigates-privacy/). ArXiv, abs/2202.06539, 2022.
36. Khrushchev, M. Yandex publishes yalm 100b. it’s the largest gpt-like neural network in open source. [https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6](https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6), 2022. [↗](https://medium.com/yandex/yandex-publishes-yalm-100b-its-the-largest-gpt-like-neural-network-in-open-source-d1df53d0e9a6)
37. Kingma, D. P. and Ba, J. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). CoRR, abs/1412.6980, 2015.
38. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/), 2022. URL https://arxiv.org/abs/2205.11916. [↗](https://arxiv.org/abs/2205.11916)
39. Kudo, T. and Richardson, J. [Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In EMNLP, 2018.
40. Kurita, K., Vyas, N., Pareek, A., Black, A. W., and Tsvetkov, Y. Quantifying social biases in contextual word representations. 1st ACL Workshop on Gender Bias for Natural Language Processing. URL [https://par.nsf.gov/biblio/10098355](https://par.nsf.gov/biblio/10098355). Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Bu… [↗](https://par.nsf.gov/biblio/10098355)
41. Lester, B., Al-Rfou, R., and Constant, N. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). ArXiv, abs/2104.08691, 2021.
42. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. [Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In ACL, 2020.
43. Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., Davison, J., Šaško, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-Major, A., Schmid, P., Gugger, S., Delangue, C… — [Datasets: A Community Library for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/)
44. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190, 2021. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E+L%2E+and+Liang%2C+P%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+International+)
45. Liang, P. P., Wu, C., Morency, L.-P., and Salakhutdinov, R. [Towards understanding and mitigating social biases in language models](https://md.research.elamur.ai/free/en/papers/liang-2021-towards-understanding-mitigating-social-biases/). In ICML, 2021.
46. Lin, C.-Y. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In ACL 2004, 2004.
47. Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., Pasunuru, R., Shleifer, S., Koura, P. S., Chaudhary, V., O’Horo, B., Wang, J., Zettlemoyer, L., Kozareva, Z., Diab, M., Stoyanov, V., and Li, X. Few-shot learning with multilingual language… — [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/)
48. Liu, H., Dacon, J., Fan, W., Liu, H., Liu, Z., and Tang, J. [Does gender matter? towards fairness in dialogue systems](https://md.research.elamur.ai/free/en/papers/liu-2020-does-gender-matter-fairness-dialogue/). In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4403–4416, Barcelona, Spain (Online), December 2020a. International Committee on Computational L…
49. Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742, 2020b. [find](https://scholar.google.com/scholar?q=Liu%2C+Y%2E%2C+Gu%2C+J%2E%2C+Goyal%2C+N%2E%2C+Li%2C+X%2E%2C+Edunov%2C+S%2E%2C+Ghazvininejad%2C+M%2E%2C+Lewis%2C+M%2E%2C+and+Zettlemoyer%2C+L%2E+Multilingual+denoising+pre%2Dtraining+for+neural+machine+translation%2E+Transactions+of+the+Association+fo)
50. Majmudar, J., Dupuy, C., Peris, C. S., Smaili, S., Gupta, R., and Zemel, R. S. Differentially private decoding in large language models. ArXiv, abs/2205.13621, 2022. [find](https://scholar.google.com/scholar?q=Majmudar%2C+J%2E%2C+Dupuy%2C+C%2E%2C+Peris%2C+C%2E+S%2E%2C+Smaili%2C+S%2E%2C+Gupta%2C+R%2E%2C+and+Zemel%2C+R%2E+S%2E+Differentially+private+decoding+in+large+language+models%2E+ArXiv%2C+abs%2F2205%2E13621%2C+2022%2E)
51. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In EMNLP, 2018. — [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/)
52. Nasr, M., Song, S., Thakurta, A., Papernot, N., and Carlini, N. Adversary instantiation: Lower bounds for differentially private machine learning. 2021 IEEE Symposium on Security and Privacy (SP), pp. 866–882, 2021. [find](https://scholar.google.com/scholar?q=Nasr%2C+M%2E%2C+Song%2C+S%2E%2C+Thakurta%2C+A%2E%2C+Papernot%2C+N%2E%2C+and+Carlini%2C+N%2E+Adversary+instantiation%3A+Lower+bounds+for+differentially+private+machine+learning%2E+2021+IEEE+Symposium+on+Security+and+Privacy+%28SP%29%2C+pp%2E)
53. Novikova, J., Dusek, O., and Rieser, V. [The e2e dataset: New challenges for end-to-end generation](https://md.research.elamur.ai/free/en/papers/novikova-2017-e2e-dataset-new-challenges-end/). In SIGDIAL Conference, 2017.
54. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. J. Training language models to follow instruct… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
55. Patterson, D. A., Gonzalez, J., Le, Q. V., Liang, C., Munguia, L., Rothchild, D., So, D. R., Texier, M., and Dean, J. [Carbon emissions and large neural network training](https://md.research.elamur.ai/free/en/papers/patterson-2021-carbon-emissions-large-neural-network/). CoRR, abs/2104.10350, 2021. URL https://arxiv.org/abs/2104.10350. [↗](https://arxiv.org/abs/2104.10350)
56. Ponti, E., Glavavs, G., Majewska, O., Liu, Q., Vulic, I., and Korhonen, A. [Xcopa: A multilingual dataset for causal commonsense reasoning](https://md.research.elamur.ai/free/en/papers/ponti-2020-xcopa-multilingual-dataset-causal-commonsense/). In EMNLP, 2020.
57. Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E+and+Narasimhan%2C+K%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
58. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+and+Sutskever%2C+I%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
59. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang… [find](https://scholar.google.com/scholar?q=Rae%2C+J%2E+W%2E%2C+Borgeaud%2C+S%2E%2C+Cai%2C+T%2E%2C+Millican%2C+K%2E%2C+Hoffmann%2C+J%2E%2C+Song%2C+F%2E%2C+Aslanides%2C+J%2E%2C+Henderson%2C+S%2E%2C+Ring%2C+R%2E%2C+Young%2C+S%2E%2C+Rutherford%2C+E%2E%2C+Hennigan%2C+T%2E%2C+Menick%2C+J%2E%2C+Cassirer%2C+A%2E%2C+Powell%2C+R%2E%2C+van+den+)
60. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). ArXiv, abs/1910.10683, 2020.
61. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad. In ACL, 2018. — [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
62. Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pp. 3505–3506, New York, NY, USA, 2… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) [find](https://scholar.google.com/scholar?q=Rasley%2C+J%2E%2C+Rajbhandari%2C+S%2E%2C+Ruwase%2C+O%2E%2C+and+He%2C+Y%2E+Deepspeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+SIGKDD+Interna)
63. Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., and Weston, J. [Recipes for building an open-domain chatbot](https://md.research.elamur.ai/free/en/papers/roller-2021-recipes-building-open-domain-chatbot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p…
64. Roy, S. and Roth, D. [Solving general arithmetic word problems](https://md.research.elamur.ai/free/en/papers/roy-2015-solving-general-arithmetic-word-problems/), 2016. URL https://arxiv.org/abs/1608.01413. [↗](https://arxiv.org/abs/1608.01413)
65. Rudinger, R., Naradowsky, J., Leonard, B., and Van Durme, B. [Gender bias in coreference resolution](https://md.research.elamur.ai/free/en/papers/rudinger-2018-gender-bias-coreference-resolution/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 8–14, New Orleans, Louis…
66. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L. A., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., Dey, M., BARI, M. S., Xu, C., Thakker, U., Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N. V., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., She… — [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/)
67. Schick, T., Udupa, S., and Schütze, H. [Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/). Transactions of the Association for Computational Linguistics, 9:1408–1424, 2021. doi: 10.1162/tacl a 00434. URL https://aclanthology.org/2021.tacl-1.84. [↗](https://aclanthology.org/2021.tacl-1.84)
68. Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., and Staiano, J. [Mlsum: The multilingual summarization corpus](https://md.research.elamur.ai/free/en/papers/scialom-2020-mlsum-multilingual-summarization-corpus/). ArXiv, abs/2004.14900, 2020.
69. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. [The woman worked as a babysitter: On biases in language generation](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),…
70. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. [Societal biases in language generation: Progress and challenges](https://md.research.elamur.ai/free/en/papers/sheng-2021-societal-biases-language-generation-progress/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
71. Shoeybi, M., Patwary, M. A., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv, abs/1909.08053, 2019. [find](https://scholar.google.com/scholar?q=Shoeybi%2C+M%2E%2C+Patwary%2C+M%2E+A%2E%2C+Puri%2C+R%2E%2C+LeGresley%2C+P%2E%2C+Casper%2C+J%2E%2C+and+Catanzaro%2C+B%2E+Megatron%2Dlm%3A+Training+multi%2Dbillion+parameter+language+models+using+model+parallelism%2E+ArXiv%2C+abs%2F1909%2E08053%2C+2019%2E)
72. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. [Mass: Masked sequence to sequence pre-training for language generation](https://md.research.elamur.ai/free/en/papers/song-2019-mass-masked-sequence-sequence-pre/). In ICML 2019, June 2019. URL [https://www.microsoft.com/en-us/research/publication/mass-masked-sequence-to-sequence-pre-training-for-language-generation/](https://www.microsoft.com/en-us/research/publication/mass-masked-sequence-to-sequence-pre-training-for-language-generation/). [↗](https://www.microsoft.com/en-us/research/publication/mass-masked-sequence-to-sequence-pre-training-for-language-generation/)
73. Tay, Y., Dehghani, M., Tran, V. Q., Garcı́a, X., Bahri, D., Schuster, T., Zheng, H., Houlsby, N., and Metzler, D. Unifying language learning paradigms. ArXiv, abs/2205.[05131, 2022](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/).
74. Tikhonov, A. and Ryabinin, M. It’s all in the heads: Using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. ArXiv, abs/2106.[12066, 2021](https://md.research.elamur.ai/free/en/papers/tikhonov-2021-it-s-all-heads-attention/).
75. Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). ArXiv, abs/1706.03762, 2017.
76. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In NeurIPS, 2019.
77. Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. What language model architecture and pretraining objective work best for zero-shot generalization? In ICML, 2022. [find](https://scholar.google.com/scholar?q=Wang%2C+T%2E%2C+Roberts%2C+A%2E%2C+Hesslow%2C+D%2E%2C+Scao%2C+T%2E+L%2E%2C+Chung%2C+H%2E+W%2E%2C+Beltagy%2C+I%2E%2C+Launay%2C+J%2E%2C+and+Raffel%2C+C%2E+What+language+model+architecture+and+pretraining+objective+work+best+for+zero%2Dshot+generalization)
78. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models, 2022. [URL https://arxiv.org/abs/2201.11903](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). [↗](https://arxiv.org/abs/2201.11903)
79. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., Biles, C., Brown, S., Kenton, Z., Hawkins, W., Stepleton, T., Birhane, A., Hendricks, L. A., Rimell, L., Isaac, W., Haas, J., Legassick, S., Irving, G., and Gabriel, I. Taxon… — [doi:10.1145/3531146.3533088](https://doi.org/10.1145/3531146.3533088) [find](https://scholar.google.com/scholar?q=Weidinger%2C+L%2E%2C+Uesato%2C+J%2E%2C+Rauh%2C+M%2E%2C+Griffin%2C+C%2E%2C+Huang%2C+P%2E%2DS%2E%2C+Mellor%2C+J%2E%2C+Glaese%2C+A%2E%2C+Cheng%2C+M%2E%2C+Balle%2C+B%2E%2C+Kasirzadeh%2C+A%2E%2C+Biles%2C+C%2E%2C+Brown%2C+S%2E%2C+Kenton%2C+Z%2E%2C+Hawkins%2C+W%2E%2C+Stepleton%2C+T%2E%2C+Birhane%2C+A%2E%2C)
80. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. [Challenges in detoxifying language models](https://md.research.elamur.ai/free/en/papers/welbl-2021-challenges-detoxifying-language-models/). In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447–2469, Punta Cana, Dominican Republic, No…
81. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. Huggingface’s [transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). ArXiv, abs/1910.03771, 2019.
82. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
83. Xia, Y., Tan, X., Tian, F., Gao, F., He, D., Chen, W., Fan, Y., Gong, L., Leng, Y., Luo, R., Wang, Y., Wu, L., Zhu, J., Qin, T., and Liu, T.-Y. Microsoft research asia’s systems for wmt19. In WMT, 2019. [find](https://scholar.google.com/scholar?q=Xia%2C+Y%2E%2C+Tan%2C+X%2E%2C+Tian%2C+F%2E%2C+Gao%2C+F%2E%2C+He%2C+D%2E%2C+Chen%2C+W%2E%2C+Fan%2C+Y%2E%2C+Gong%2C+L%2E%2C+Leng%2C+Y%2E%2C+Luo%2C+R%2E%2C+Wang%2C+Y%2E%2C+Wu%2C+L%2E%2C+Zhu%2C+J%2E%2C+Qin%2C+T%2E%2C+and+Liu%2C+T%2E%2DY%2E+Microsoft+research+asia%E2%80%99s+systems+for+wmt19%2E+In+WMT%2C+201)
84. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y. On layer normalization in the transformer architecture. In ICML, 2020. [find](https://scholar.google.com/scholar?q=Xiong%2C+R%2E%2C+Yang%2C+Y%2E%2C+He%2C+D%2E%2C+Zheng%2C+K%2E%2C+Zheng%2C+S%2E%2C+Xing%2C+C%2E%2C+Zhang%2C+H%2E%2C+Lan%2C+Y%2E%2C+Wang%2C+L%2E%2C+and+Liu%2C+T%2E%2DY%2E+On+layer+normalization+in+the+transformer+architecture%2E+In+ICML%2C+2020%2E)
85. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. [mt5: A massively multilingual pre-trained text-to-text transformer](https://md.research.elamur.ai/free/en/papers/xue-2021-mt5-massively-multilingual-pre-trained/). In NAACL, 2021.
86. Yang, Y., Zhang, Y., Tar, C., and Baldridge, J. [Paws-x: A cross-lingual adversarial dataset for paraphrase identification](https://md.research.elamur.ai/free/en/papers/yang-2019-paws-x-cross-lingual-adversarial/). In EMNLP, 2019.
87. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. [Opt: Open pre-trained transformer language models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). ArXiv, abs/2205.01068,…
88. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N. M., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. 2022. [find](https://scholar.google.com/scholar?q=Zoph%2C+B%2E%2C+Bello%2C+I%2E%2C+Kumar%2C+S%2E%2C+Du%2C+N%2E%2C+Huang%2C+Y%2E%2C+Dean%2C+J%2E%2C+Shazeer%2C+N%2E+M%2E%2C+and+Fedus%2C+W%2E+St%2Dmoe%3A+Designing+stable+and+transferable+sparse+expert+models%2E+2022%2E)

*Updated 2 November 2026*