---
title: "Learning to Compress Prompts with Gist Tokens — Mu 2023"
description: "Long prompts, such as system instructions attached to every request, take up space in a model's context window."
authors: ["Jesse Mu", "Xiang Lisa Li", "Noah Goodman"]
year: 2023
venue: "arXiv:2304.08467"
source: "https://arxiv.org/abs/2304.08467"
doi: "10.48550/arxiv.2304.08467"
arxiv: "2304.08467"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/mu-2023-learning-compress-prompts-gist-tokens/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/) / Learning to Compress Prompts with Gist Tokens

# Learning to Compress Prompts with Gist Tokens

**Authors:** [Jesse Mu](https://md.research.elamur.ai/free/en/authors/jesse-mu/), [Xiang Lisa Li](https://md.research.elamur.ai/free/en/authors/xiang-lisa-li/), [Noah Goodman](https://md.research.elamur.ai/free/en/authors/noah-goodman/) · **Year:** 2023 · **Venue:** arXiv:2304.08467 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2304.08467) · [PDF](https://arxiv.org/pdf/2304.08467) · [DOI 10.48550/arxiv.2304.08467](https://doi.org/10.48550/arxiv.2304.08467) · [arXiv 2304.08467](https://arxiv.org/abs/2304.08467) · [OpenAlex](https://openalex.org/W7133221612)

**Citations:** 30 citations (openalex-snapshot · 2026-10-14) · [140](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Long prompts, such as system instructions attached to every request, take up space in a model's context window. They also have to be re-encoded or cached again and again, which costs compute and storage. Task-specific finetuning or distillation removes the prompt, but it requires a separately trained model for each task. The authors propose gisting: a language model learns to condense any instruction into the activations of a few special "gist" tokens, which can then be stored and reused. Training is ordinary instruction tuning with one change. The attention mask is altered so that tokens after the gist tokens cannot see the original prompt, which forces its content through the gist bottleneck. Because the model itself produces the compressed prefix, it generalizes zero-shot to unseen instructions. Experiments on LLaMA-7B and FLAN-T5-XXL, trained on the Alpaca+ dataset, show prompts shrunk by as much as 26x while output quality remains close to uncompressed baselines. This yields notable compute and storage savings, which matters for anyone serving many cached prompts at scale.

## Key points
- **Models and efficiency gains.** Gisting was tested on a decoder-only model (LLaMA-7B) and an encoder-decoder model (FLAN-T5-XXL).
  - Prompt compression reached up to 26x.
  - FLOPs dropped by up to 40%.
  - Wall-clock time improved by up to 4.2%.
- **Alpaca+ dataset.** It merges Self-Instruct and Stanford Alpaca data, which were generated by text-davinci-001 and text-davinci-003.
  - It contains 130,321 examples, with 104,664 unique tasks and 48,530 unique inputs.
  - About 59% of tasks have no input.
- **Evaluation splits.** Three held-out sets were used:
  - 1000 Seen prompts;
  - 1000 Unseen prompts;
  - 252 hand-written Human prompts from Self-Instruct, used as an out-of-distribution test.
- **Implementation.** The method needs only a single new vocabulary token plus a modified attention mask, roughly 10 lines of PyTorch. It adds no training cost beyond standard instruction finetuning.
- **Storage.** In LLaMA-7B, each cached token takes about 1.05 MB of KV storage, so gisting lets about 26x more prompts be cached in the same space. The authors caution that compression can drop nuances of the original instruction.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/)

## Cited by

Cited by

140 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 129 not yet.

- [Quiet-STaR: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/)
- [Adapting Language Models to Compress Contexts](https://md.research.elamur.ai/free/en/papers/chevalier-2023-adapting-language-models-compress-contexts/)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://md.research.elamur.ai/free/en/papers/ge-2023-model-tells-discard-adaptive-kv/)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://md.research.elamur.ai/free/en/papers/greshake-2023-ve-signed-up-compromising-real/)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://md.research.elamur.ai/free/en/papers/jiang-2023-llmlingua-compressing-prompts-accelerated-inference/)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://md.research.elamur.ai/free/en/papers/jiang-2023-longllmlingua-accelerating-enhancing-llms-long/)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://md.research.elamur.ai/free/en/papers/li-2023-compressing-context-enhance-inference-efficiency/)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://md.research.elamur.ai/free/en/papers/pan-2024-llmlingua-2-data-distillation-efficient/)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://md.research.elamur.ai/free/en/papers/xu-2023-recomp-improving-retrieval-augmented-lms/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{mu2023learning,
  title = {{Learning to Compress Prompts with Gist Tokens}},
  author = {Mu, Jesse and Li, Xiang Lisa and Goodman, Noah},
  year = {2023},
  doi = {10.48550/arxiv.2304.08467},
  eprint = {2304.08467},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08467}
}
```

- [← previous LeanDojo: Theorem Proving with Retrieval-Augmented Language Models](https://md.research.elamur.ai/free/en/papers/yang-2023-leandojo-theorem-proving-retrieval-augmented/)
- [next → LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/4/)

## References

43 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. [find](https://scholar.google.com/scholar?q=A%2E+Askell%2C+Y%2E+Bai%2C+A%2E+Chen%2C+D%2E+Drain%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+A%2E+Jones%2C+N%2E+Joseph%2C+B%2E+Mann%2C+N%2E+DasSarma%2C+et+al%2E+A+general+language+assistant+as+a+laboratory+for+alignment%2E+arXiv+preprint+arXiv%3A2112%2E0)
2. I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv:2004.05150, 2020. [find](https://scholar.google.com/scholar?q=I%2E+Beltagy%2C+M%2E+E%2E+Peters%2C+and+A%2E+Cohan%2E+Longformer%3A+The+long%2Ddocument+transformer%2E+arXiv%3A2004%2E05150%2C+2020%2E)
3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. [find](https://scholar.google.com/scholar?q=T%2E+Brown%2C+B%2E+Mann%2C+N%2E+Ryder%2C+M%2E+Subbiah%2C+J%2E+D%2E+Kaplan%2C+P%2E+Dhariwal%2C+A%2E+Neelakantan%2C+P%2E+Shyam%2C+G%2E+Sastry%2C+A%2E+Askell%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+Neural+Information+Process)
4. C. Chen. Transformer inference arithmetic. transformer-inference-arithmetic/, 2022. [find](https://scholar.google.com/scholar?q=C%2E+Chen%2E+Transformer+inference+arithmetic%2E+transformer%2Dinference%2Darithmetic%2F%2C+2022%2E)
5. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality. [https://vicuna.lmsys.org/](https://vicuna.lmsys.org/), 2023. [↗](https://vicuna.lmsys.org/)
6. R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. [find](https://scholar.google.com/scholar?q=R%2E+Child%2C+S%2E+Gray%2C+A%2E+Radford%2C+and+I%2E+Sutskever%2E+Generating+long+sequences+with+sparse+transformers%2E+arXiv+preprint+arXiv%3A1904%2E10509%2C+2019%2E)
7. E. Choi, Y. Jo, J. Jang, and M. Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint arXiv:2206.11349, 2022. — [arXiv:2206.11349](https://arxiv.org/abs/2206.11349) [find](https://scholar.google.com/scholar?q=E%2E+Choi%2C+Y%2E+Jo%2C+J%2E+Jang%2C+and+M%2E+Seo%2E+Prompt+injection%3A+Parameterization+of+fixed+inputs%2E+arXiv+preprint+arXiv%3A2206%2E11349%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E11349%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2206%2E11349%29)
8. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=H%2E+W%2E+Chung%2C+L%2E+Hou%2C+S%2E+Longpre%2C+B%2E+Zoph%2C+Y%2E+Tay%2C+W%2E+Fedus%2C+E%2E+Li%2C+X%2E+Wang%2C+M%2E+Dehghani%2C+S%2E+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+preprint+arXiv%3A2210%2E11416%2C+2022%2E)
9. Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978–2988, 2019. [find](https://scholar.google.com/scholar?q=Z%2E+Dai%2C+Z%2E+Yang%2C+Y%2E+Yang%2C+J%2E+G%2E+Carbonell%2C+Q%2E+Le%2C+and+R%2E+Salakhutdinov%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Associ)
10. W. J. Dixon and F. J. Massey Jr. Introduction to statistical analysis. McGraw-Hill, 1951. [find](https://scholar.google.com/scholar?q=W%2E+J%2E+Dixon+and+F%2E+J%2E+Massey+Jr%2E+Introduction+to+statistical+analysis%2E+McGraw%2DHill%2C+1951%2E)
11. X. Geng, A. Gudibande, H. Liu, E. Wallace, P. Abbeel, S. Levine, and D. Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/blog/2023/04/03/koala/](https://bair.berkeley.edu/blog/2023/04/03/koala/). [↗](https://bair.berkeley.edu/blog/2023/04/03/koala/)
12. F. Gilardi, M. Alizadeh, and M. Kubli. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Gilardi%2C+M%2E+Alizadeh%2C+and+M%2E+Kubli%2E+ChatGPT+outperforms+crowd%2Dworkers+for+text%2Dannotation+tasks%2E+arXiv+preprint+arXiv%3A2303%2E15056%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
13. D. Ha, A. Dai, and Q. V. Le. HyperNetworks. In International Conference on Learning Representations, 2017. [find](https://scholar.google.com/scholar?q=D%2E+Ha%2C+A%2E+Dai%2C+and+Q%2E+V%2E+Le%2E+HyperNetworks%2E+In+International+Conference+on+Learning+Representations%2C+2017%2E)
14. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Hoffmann%2C+S%2E+Borgeaud%2C+A%2E+Mensch%2C+E%2E+Buchatskaya%2C+T%2E+Cai%2C+E%2E+Rutherford%2C+D%2E+d%2E+L%2E+Casas%2C+L%2E+A%2E+Hendricks%2C+J%2E+Welbl%2C+A%2E+Clark%2C+et+al%2E+Training+compute%2Doptimal+large+language+models%2E+arXiv+preprint+a)
15. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019. [https://kipp.ly/blog/](https://kipp.ly/blog/) [↗](https://kipp.ly/blog/)
16. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=E%2E+J%2E+Hu%2C+Y%2E+Shen%2C+P%2E+Wallis%2C+Z%2E+Allen%2DZhu%2C+Y%2E+Li%2C+S%2E+Wang%2C+L%2E+Wang%2C+and+W%2E+Chen%2E+LoRA%3A+Low%2Drank+adaptation+of+large+language+models%2E+In+International+Conference+on+Learning+Representations%2C+2022%2E)
17. F. Huang, H. Kwak, and J. An. Is ChatGPT better than human annotators? Potential and limitations of ChatGPT in explaining implicit hate speech. arXiv preprint arXiv:2302.07736, 2023. [find](https://scholar.google.com/scholar?q=F%2E+Huang%2C+H%2E+Kwak%2C+and+J%2E+An%2E+Is+ChatGPT+better+than+human+annotators%3F+Potential+and+limitations+of+ChatGPT+in+explaining+implicit+hate+speech%2E+arXiv+preprint+arXiv%3A2302%2E07736%2C+2023%2E)
18. B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, 2021. [find](https://scholar.google.com/scholar?q=B%2E+Lester%2C+R%2E+Al%2DRfou%2C+and+N%2E+Constant%2E+The+power+of+scale+for+parameter%2Defficient+prompt+tuning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+3045%E2%80%93)
19. X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–4597, 2021. [find](https://scholar.google.com/scholar?q=X%2E+L%2E+Li+and+P%2E+Liang%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Internationa)
20. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004. [find](https://scholar.google.com/scholar?q=C%2E%2DY%2E+Lin%2E+ROUGE%3A+A+package+for+automatic+evaluation+of+summaries%2E+In+Text+summarization+branches+out%2C+pages+74%E2%80%9381%2C+2004%2E)
21. P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, and N. Shazeer. Generating Wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018. [find](https://scholar.google.com/scholar?q=P%2E+J%2E+Liu%2C+M%2E+Saleh%2C+E%2E+Pot%2C+B%2E+Goodrich%2C+R%2E+Sepassi%2C+L%2E+Kaiser%2C+and+N%2E+Shazeer%2E+Generating+Wikipedia+by+summarizing+long+sequences%2E+In+International+Conference+on+Learning+Representations%2C+2018%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2018-generating-wikipedia-summarizing-long-sequences/)
22. OpenAI. Introducing ChatGPT. [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt), 2022. [↗](https://openai.com/blog/chatgpt)
23. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, pages 27730–27744, 2022. [find](https://scholar.google.com/scholar?q=L%2E+Ouyang%2C+J%2E+Wu%2C+X%2E+Jiang%2C+D%2E+Almeida%2C+C%2E+Wainwright%2C+P%2E+Mishkin%2C+C%2E+Zhang%2C+S%2E+Agarwal%2C+K%2E+Slama%2C+A%2E+Ray%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+In+Advances+in+Ne)
24. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019. [find](https://scholar.google.com/scholar?q=A%2E+Paszke%2C+S%2E+Gross%2C+F%2E+Massa%2C+A%2E+Lerer%2C+J%2E+Bradbury%2C+G%2E+Chanan%2C+T%2E+Killeen%2C+Z%2E+Lin%2C+N%2E+Gimelshein%2C+L%2E+Antiga%2C+et+al%2E+PyTorch%3A+An+imperative+style%2C+high%2Dperformance+deep+learning+library%2E+Advances+in+) [our card](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
25. J. Phang, Y. Mao, P. He, and W. Chen. HyperTuning: Toward adapting large language models without back-propagation. arXiv preprint arXiv:2211.12485, 2022. — [arXiv:2211.12485](https://arxiv.org/abs/2211.12485) [find](https://scholar.google.com/scholar?q=J%2E+Phang%2C+Y%2E+Mao%2C+P%2E+He%2C+and+W%2E+Chen%2E+HyperTuning%3A+Toward+adapting+large+language+models+without+back%2Dpropagation%2E+arXiv+preprint+arXiv%3A2211%2E12485%2C+2022%2E+%E2%80%94+%5BarXiv%3A2211%2E12485%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F221)
26. R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. [find](https://scholar.google.com/scholar?q=R%2E+Pope%2C+S%2E+Douglas%2C+A%2E+Chowdhery%2C+J%2E+Devlin%2C+J%2E+Bradbury%2C+A%2E+Levskaya%2C+J%2E+Heek%2C+K%2E+Xiao%2C+S%2E+Agrawal%2C+and+J%2E+Dean%2E+Efficiently+scaling+transformer+inference%2E+arXiv+preprint+arXiv%3A2211%2E05102%2C+2022%2E)
27. J. W. Rae, A. Potapenko, S. M. Jayakumar, C. Hillier, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations, 2020. [find](https://scholar.google.com/scholar?q=J%2E+W%2E+Rae%2C+A%2E+Potapenko%2C+S%2E+M%2E+Jayakumar%2C+C%2E+Hillier%2C+and+T%2E+P%2E+Lillicrap%2E+Compressive+transformers+for+long%2Drange+sequence+modelling%2E+In+International+Conference+on+Learning+Representations%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/rae-2019-compressive-transformers-long-range-sequence/)
28. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified Text-to-Text Transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [find](https://scholar.google.com/scholar?q=C%2E+Raffel%2C+N%2E+Shazeer%2C+A%2E+Roberts%2C+K%2E+Lee%2C+S%2E+Narang%2C+M%2E+Matena%2C+Y%2E+Zhou%2C+W%2E+Li%2C+and+P%2E+J%2E+Liu%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+Text%2Dto%2DText+Transformer%2E+The+Journal+of+Machin)
29. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506, 2020. [find](https://scholar.google.com/scholar?q=J%2E+Rasley%2C+S%2E+Rajbhandari%2C+O%2E+Ruwase%2C+and+Y%2E+He%2E+DeepSpeed%3A+System+optimizations+enable+training+deep+learning+models+with+over+100+billion+parameters%2E+In+Proceedings+of+the+26th+ACM+SIGKDD+Internatio)
30. C. Snell, D. Klein, and R. Zhong. arXiv:2209.15189, 2022. [find](https://scholar.google.com/scholar?q=C%2E+Snell%2C+D%2E+Klein%2C+and+R%2E+Zhong%2E+arXiv%3A2209%2E15189%2C+2022%2E)
31. R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
32. Y. Tay, M. Dehghani, D. Bahri, and D. Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1–28, 2022. Learning by distilling context. arXiv preprint [find](https://scholar.google.com/scholar?q=Y%2E+Tay%2C+M%2E+Dehghani%2C+D%2E+Bahri%2C+and+D%2E+Metzler%2E+Efficient+transformers%3A+A+survey%2E+ACM+Computing+Surveys%2C+55%286%29%3A1%E2%80%9328%2C+2022%2E+Learning+by+distilling+context%2E+arXiv+preprint) [our card](https://md.research.elamur.ai/free/en/papers/tay-2020-efficient-transformers-survey/)
33. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+T%2E+Lavril%2C+G%2E+Izacard%2C+X%2E+Martinet%2C+M%2E%2DA%2E+Lachaux%2C+T%2E+Lacroix%2C+B%2E+Rozi%C3%A8re%2C+N%2E+Goyal%2C+E%2E+Hambro%2C+F%2E+Azhar%2C+et+al%2E+LLaMA%3A+Open+and+efficient+foundation+language+models%2E+arXiv+preprint+arXiv%3A)
34. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. [find](https://scholar.google.com/scholar?q=A%2E+Vaswani%2C+N%2E+Shazeer%2C+N%2E+Parmar%2C+J%2E+Uszkoreit%2C+L%2E+Jones%2C+A%2E+N%2E+Gomez%2C+%C5%81%2E+Kaiser%2C+and+I%2E+Polosukhin%2E+Attention+is+all+you+need%2E+Advances+in+Neural+Information+Processing+Systems%2C+30%2C+2017%2E)
35. J. Wang, Y. Liang, F. Meng, H. Shi, Z. Li, J. Xu, J. Qu, and J. Zhou. Is ChatGPT a good NLG evaluator? A preliminary study. arXiv preprint arXiv:2303.04048, 2023. [find](https://scholar.google.com/scholar?q=J%2E+Wang%2C+Y%2E+Liang%2C+F%2E+Meng%2C+H%2E+Shi%2C+Z%2E+Li%2C+J%2E+Xu%2C+J%2E+Qu%2C+and+J%2E+Zhou%2E+Is+ChatGPT+a+good+NLG+evaluator%3F+A+preliminary+study%2E+arXiv+preprint+arXiv%3A2303%2E04048%2C+2023%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/)
36. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+Y%2E+Kordi%2C+S%2E+Mishra%2C+A%2E+Liu%2C+N%2E+A%2E+Smith%2C+D%2E+Khashabi%2C+and+H%2E+Hajishirzi%2E+Self%2DInstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+arXiv%3A2212%2E10560%2C+2022%2E)
37. Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap, et al. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Languag… [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+S%2E+Mishra%2C+P%2E+Alipoormolabashi%2C+Y%2E+Kordi%2C+A%2E+Mirzaei%2C+A%2E+Naik%2C+A%2E+Ashok%2C+A%2E+S%2E+Dhanasekaran%2C+A%2E+Arunkumar%2C+D%2E+Stap%2C+et+al%2E+Super%2DNaturalInstructions%3A+Generalization+via+declarative+instructio)
38. J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+M%2E+Bosma%2C+V%2E+Zhao%2C+K%2E+Guu%2C+A%2E+W%2E+Yu%2C+B%2E+Lester%2C+N%2E+Du%2C+A%2E+M%2E+Dai%2C+and+Q%2E+V%2E+Le%2E+Finetuned+language+models+are+zero%2Dshot+learners%2E+In+International+Conference+on+Learning+Representations%2C+2022%2E)
39. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+X%2E+Wang%2C+D%2E+Schuurmans%2C+M%2E+Bosma%2C+F%2E+Xia%2C+E%2E+H%2E+Chi%2C+Q%2E+V%2E+Le%2C+D%2E+Zhou%2C+et+al%2E+Chain%2Dof%2Dthought+prompting+elicits+reasoning+in+large+language+models%2E+In+Advances+in+Neural+Information+Processi) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
40. D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5621–5634, Abu Dhabi, United Arab Emirates, Dec. 2022. Association f… [find](https://scholar.google.com/scholar?q=D%2E+Wingate%2C+M%2E+Shoeybi%2C+and+T%2E+Sorensen%2E+Prompt+compression+and+contrastive+conditioning+for+controllability+and+toxicity+reduction+in+language+models%2E+In+Findings+of+the+Association+for+Computational)
41. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In… [find](https://scholar.google.com/scholar?q=T%2E+Wolf%2C+L%2E+Debut%2C+V%2E+Sanh%2C+J%2E+Chaumond%2C+C%2E+Delangue%2C+A%2E+Moi%2C+P%2E+Cistac%2C+T%2E+Rault%2C+R%2E+Louf%2C+M%2E+Funtowicz%2C+J%2E+Davison%2C+S%2E+Shleifer%2C+P%2E+von+Platen%2C+C%2E+Ma%2C+Y%2E+Jernite%2C+J%2E+Plu%2C+C%2E+Xu%2C+T%2E+Le+Scao%2C+S%2E+Gugge) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
42. Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. [find](https://scholar.google.com/scholar?q=Y%2E+Wu%2C+M%2E+N%2E+Rabe%2C+D%2E+Hutchins%2C+and+C%2E+Szegedy%2E+Memorizing+transformers%2E+In+International+Conference+on+Learning+Representations%2C+2022%2E)
43. H. Zhang, Y. Gong, Y. Shen, W. Li, J. Lv, N. Duan, and W. Chen. Poolingformer: Long document modeling with pooling attention. In International Conference on Machine Learning, pages 12437–12446. PMLR, 2021. [find](https://scholar.google.com/scholar?q=H%2E+Zhang%2C+Y%2E+Gong%2C+Y%2E+Shen%2C+W%2E+Li%2C+J%2E+Lv%2C+N%2E+Duan%2C+and+W%2E+Chen%2E+Poolingformer%3A+Long+document+modeling+with+pooling+attention%2E+In+International+Conference+on+Machine+Learning%2C+pages+12437%E2%80%9312446%2E+PMLR%2C+)